2024-05-20T17:37:47.395 INFO:root:teuthology version: 0.0.1.dev287+gb9e3da8 2024-05-20T17:37:47.395 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --name pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} --archive /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661 --verbose -- /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661/orig.config.yaml 2024-05-20T17:37:47.448 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-20T17:37:47.532 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661 branch: wip-pdonnell-testing-20240517.203521-squid-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} email: pdonnell@redhat.com first_in_suite: false job_id: '7715661' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240517.203521-squid-debug ceph: cephfs: max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: d2b507c724ef6c8295ee0a35d185cdab167fd61d subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: d2b507c724ef6c8295ee0a35d185cdab167fd61d kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240517.203521-squid-debug sha1: d2b507c724ef6c8295ee0a35d185cdab167fd61d owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 9749 sha1: d2b507c724ef6c8295ee0a35d185cdab167fd61d sleep_before_teardown: 0 subset: 272/512 suite: fs suite_branch: wip-pdonnell-testing-20240517.203521-squid-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d2b507c724ef6c8295ee0a35d185cdab167fd61d/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d2b507c724ef6c8295ee0a35d185cdab167fd61d syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPB4duasGaOzlKwS5vH2l7B3dALopww4M6rSYsu+thQix/ga0U6Htb4DDMgKsrKg3RtG/tmMo435mVHaFWI/4AM= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGbrW7cO6QNQL8sO/FjyKFn9Ci2yCYaUPVO5nHLWVQkI5FOWSIRQ3f/0bOq0+IAI1Jk1YZyPx1AXJBun92VwE2I= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMwLI41redOVeKbRtGGLq3alK/cvveAMkaLalVldMHtRfdS+/r1jVa5/Rb7OkpWDdM/pXW4xfFiMpK/mQ4qqtvo= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a timestamp: 2024-05-20_15:44:59 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-20T17:37:47.533 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d2b507c724ef6c8295ee0a35d185cdab167fd61d/qa; will attempt to use it 2024-05-20T17:37:47.533 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d2b507c724ef6c8295ee0a35d185cdab167fd61d/qa/tasks 2024-05-20T17:37:47.534 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-20T17:37:47.535 INFO:teuthology.task.internal:Checking packages... 2024-05-20T17:37:47.554 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2b507c724ef6c8295ee0a35d185cdab167fd61d' 2024-05-20T17:37:47.554 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-20T17:37:47.554 INFO:teuthology.packaging:ref: None 2024-05-20T17:37:47.554 INFO:teuthology.packaging:tag: None 2024-05-20T17:37:47.554 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240517.203521-squid-debug 2024-05-20T17:37:47.554 INFO:teuthology.packaging:sha1: d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:37:47.555 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240517.203521-squid-debug 2024-05-20T17:37:47.690 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2744.gd2b507c7 2024-05-20T17:37:47.691 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-20T17:37:47.697 INFO:teuthology.task.internal:no buildpackages task found 2024-05-20T17:37:47.697 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-20T17:37:47.723 INFO:teuthology.task.internal:Saving configuration 2024-05-20T17:37:47.741 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-20T17:37:47.757 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-20T17:37:47.785 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-20 17:32:42.524131', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPB4duasGaOzlKwS5vH2l7B3dALopww4M6rSYsu+thQix/ga0U6Htb4DDMgKsrKg3RtG/tmMo435mVHaFWI/4AM='} 2024-05-20T17:37:47.806 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-20 17:32:42.522898', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGbrW7cO6QNQL8sO/FjyKFn9Ci2yCYaUPVO5nHLWVQkI5FOWSIRQ3f/0bOq0+IAI1Jk1YZyPx1AXJBun92VwE2I='} 2024-05-20T17:37:47.829 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-20 17:32:42.521350', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMwLI41redOVeKbRtGGLq3alK/cvveAMkaLalVldMHtRfdS+/r1jVa5/Rb7OkpWDdM/pXW4xfFiMpK/mQ4qqtvo='} 2024-05-20T17:37:47.829 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-20T17:37:47.835 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-20T17:37:47.835 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-20T17:37:47.835 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-20T17:37:47.835 INFO:teuthology.run_tasks:Running task console_log... 2024-05-20T17:37:47.945 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb22a267ac0>, signals=[15]) 2024-05-20T17:37:47.945 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-20T17:37:47.951 INFO:teuthology.task.internal:Opening connections... 2024-05-20T17:37:47.951 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-05-20T17:37:47.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-20T17:37:48.029 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-05-20T17:37:48.029 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-20T17:37:48.100 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-05-20T17:37:48.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-20T17:37:48.174 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-20T17:37:48.181 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-05-20T17:37:48.198 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-05-20T17:37:48.198 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-05-20T17:37:48.255 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-05-20T17:37:48.255 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-05-20T17:37:48.255 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-05-20T17:37:48.255 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-05-20T17:37:48.255 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-05-20T17:37:48.255 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-05-20T17:37:48.256 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-20T17:37:48.256 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-05-20T17:37:48.256 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-05-20T17:37:48.256 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-20T17:37:48.256 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-05-20T17:37:48.256 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-20T17:37:48.256 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-20T17:37:48.256 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-20T17:37:48.257 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-05-20T17:37:48.276 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-05-20T17:37:48.294 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-05-20T17:37:48.294 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-05-20T17:37:48.350 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-05-20T17:37:48.351 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2024-05-20T17:37:48.351 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-05-20T17:37:48.351 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-05-20T17:37:48.351 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2024-05-20T17:37:48.351 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2024-05-20T17:37:48.351 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-20T17:37:48.351 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-05-20T17:37:48.351 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2024-05-20T17:37:48.351 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-20T17:37:48.351 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-05-20T17:37:48.351 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-20T17:37:48.352 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-20T17:37:48.352 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-20T17:37:48.352 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-05-20T17:37:48.376 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-05-20T17:37:48.394 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-05-20T17:37:48.394 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-05-20T17:37:48.451 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-05-20T17:37:48.451 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-05-20T17:37:48.451 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-05-20T17:37:48.451 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-05-20T17:37:48.452 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-05-20T17:37:48.452 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-05-20T17:37:48.452 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-20T17:37:48.452 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-05-20T17:37:48.452 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-05-20T17:37:48.452 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-20T17:37:48.452 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-05-20T17:37:48.452 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-20T17:37:48.452 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-20T17:37:48.452 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-20T17:37:48.452 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-05-20T17:37:48.476 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-20T17:37:48.485 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-20T17:37:48.507 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-20T17:37:48.507 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-05-20T17:37:48.509 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-05-20T17:37:48.512 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-05-20T17:37:48.528 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-20T17:37:48.540 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-20T17:37:48.540 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-05-20T17:37:48.567 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-05-20T17:37:48.569 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-05-20T17:37:48.761 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-20T17:37:48.840 INFO:teuthology.run_tasks:Running task kernel... 2024-05-20T17:37:48.854 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-05-20T17:37:48.854 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-05-20T17:37:48.855 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-05-20T17:37:48.855 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-20T17:37:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T17:37:48.872 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-05-20T17:37:48.929 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-437.el9.x86_64 2024-05-20T17:37:48.930 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-437.el9.x86_64 2024-05-20T17:37:48.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2024-05-20T17:37:53.468 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.1 MB 00:02 2024-05-20T17:37:56.319 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-05-20T17:38:01.796 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.2 MB 00:01 2024-05-20T17:38:03.160 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-05-20T17:38:04.203 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-05-20T17:38:09.304 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-05-20T17:38:11.176 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-20T17:38:11.176 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-20T17:38:11.267 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-05-20T17:38:11.274 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-05-20T17:38:11.274 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-05-20T17:38:11.274 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-05-20T17:38:11.275 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-05-20T17:38:11.275 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-447.el9 baseos 576 k 2024-05-20T17:38:11.275 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-05-20T17:38:11.275 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-447.el9 baseos 16 M 2024-05-20T17:38:11.275 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-447.el9 baseos 34 M 2024-05-20T17:38:11.275 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-447.el9 baseos 28 M 2024-05-20T17:38:11.275 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:38:11.275 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-05-20T17:38:11.275 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-05-20T17:38:11.275 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-05-20T17:38:11.275 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:38:11.276 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 78 M 2024-05-20T17:38:11.276 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 123 M 2024-05-20T17:38:11.276 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-05-20T17:38:11.575 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-447.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:38:11.575 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-447.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:38:11.577 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-447.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:38:11.683 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-447.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:38:11.686 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-447.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:38:11.704 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-447.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:38:11.842 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-447.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:38:11.850 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-447.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:38:11.864 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-447.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:38:12.186 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-447.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:38:12.215 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-447.el9.x86_64.rpm (IP: 163.114.130.31) 2024-05-20T17:38:12.216 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-447.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:38:12.250 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-447.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:38:12.277 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-447.el9.x86_64.rpm (IP: 163.114.130.31) 2024-05-20T17:38:12.278 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-447.el9.x86_64.rpm (IP: 163.114.130.31) 2024-05-20T17:38:12.331 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-447.el9.x86_64.rpm (IP: 163.114.130.31) 2024-05-20T17:38:12.364 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-447.el9.x86_64.rpm (IP: 163.114.130.31) 2024-05-20T17:38:12.367 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-447.el9.x86_64.rpm (IP: 199.223.246.113) 2024-05-20T17:38:12.368 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-447.el9.x86_64.rpm (IP: 163.114.130.31) 2024-05-20T17:38:12.384 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-447.el9.x86_64.rpm (IP: 199.223.246.113) 2024-05-20T17:38:12.397 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-447.el9.x86_64.rpm (IP: 199.223.246.113) 2024-05-20T17:38:12.413 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-447.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:38:12.429 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-447.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:38:12.487 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-447.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:38:12.510 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-447.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:38:12.510 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-447.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:38:12.527 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-447.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:38:12.528 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-447.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:38:12.531 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-447.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:38:12.594 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-447.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:38:12.595 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-447.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:38:12.628 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-447.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:38:12.647 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-447.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:38:12.779 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-447.el9.x86_64.rpm 465 kB/s | 576 kB 00:01 2024-05-20T17:38:12.860 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-447.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:38:12.898 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-447.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:38:13.106 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-447.el9.x86_64.rpm 9.9 MB/s | 16 MB 00:01 2024-05-20T17:38:13.106 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-447.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:38:13.522 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-5.14.0-447.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-05-20T17:38:13.522 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-447.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:38:13.597 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-447.el9.x86_64.rpm (IP: 163.114.130.31) 2024-05-20T17:38:13.637 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-447.el9.x86_64.rpm (IP: 163.114.130.31) 2024-05-20T17:38:13.661 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-447.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:38:13.695 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-447.el9.x86_64.rpm (IP: 199.223.246.113) 2024-05-20T17:38:13.713 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-core-5.14.0-447.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-447.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:38:14.614 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-core-5.14.0-447.el9.x86_6 15 MB/s | 28 MB 00:01 2024-05-20T17:38:14.616 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-05-20T17:38:14.616 INFO:teuthology.orchestra.run.smithi129.stdout:Total 23 MB/s | 78 MB 00:03 2024-05-20T17:38:15.302 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-05-20T17:38:15.656 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-05-20T17:38:15.656 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-05-20T17:38:16.612 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-05-20T17:38:16.612 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-05-20T17:38:19.002 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-05-20T17:38:19.906 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-447.el9.x86_64 1/4 2024-05-20T17:38:20.022 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-20T17:38:22.642 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-20T17:38:22.773 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-20T17:38:27.517 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-20T17:38:27.567 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-447.el9.x86_64 4/4 2024-05-20T17:38:32.334 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-20T17:39:01.748 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 4/4 2024-05-20T17:39:01.786 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 4/4 2024-05-20T17:39:02.128 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-447.el9.x86_64 4/4 2024-05-20T17:39:02.128 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-447.el9.x86_64 1/4 2024-05-20T17:39:02.129 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-20T17:39:02.129 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-20T17:39:02.970 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-20T17:39:02.971 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:39:02.971 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-05-20T17:39:02.971 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-447.el9.x86_64 2024-05-20T17:39:02.971 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-447.el9.x86_64 2024-05-20T17:39:02.971 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-447.el9.x86_64 2024-05-20T17:39:02.971 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-447.el9.x86_64 2024-05-20T17:39:02.971 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:39:02.971 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-05-20T17:39:03.304 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-05-20T17:39:03.381 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-20T17:39:03.383 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-20T17:39:03.383 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-447.el9.x86_64 2024-05-20T17:39:03.383 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-20T17:39:03.383 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-20T17:39:03.383 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-05-20T17:39:03.383 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-05-20T17:39:03.384 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-05-20T17:39:03.433 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-20T17:39:03.433 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-20T17:39:03.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-05-20T17:39:03.463 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-05-20T17:39:03.463 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-05-20T17:39:03.463 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-05-20T17:39:03.463 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 68.el9 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8264254 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-05-20T17:39:03.464 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-05-20T17:39:03.465 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:39:03.465 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-20T17:39:03.465 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-20T17:39:03.465 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-20T17:39:03.465 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-05-20T17:39:03.465 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:39:03.465 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-05-20T17:39:03.466 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-20T17:39:03.466 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-447.el9.x86_64 2024-05-20T17:39:03.466 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-20T17:39:04.022 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-05-20T17:39:05.406 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-20T17:39:05.438 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-05-20T17:39:05.440 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-05-20T17:39:05.455 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.PSYEJc8Jco 2024-05-20T17:39:05.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PSYEJc8Jco 2024-05-20T17:39:05.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.PSYEJc8Jco 2024-05-20T17:39:05.680 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.PSYEJc8Jco is 6KB 2024-05-20T17:39:05.729 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.PSYEJc8Jco 2024-05-20T17:39:05.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-05-20T17:39:05.804 INFO:teuthology.orchestra.run.smithi129.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-20T17:39:05.804 INFO:teuthology.orchestra.run.smithi129.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-20T17:39:05.805 INFO:teuthology.orchestra.run.smithi129.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-20T17:39:05.805 INFO:teuthology.orchestra.run.smithi129.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64.conf 2024-05-20T17:39:05.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64 2024-05-20T17:39:05.953 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-05-20T17:39:35.956 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-20T17:39:35.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-05-20T17:39:35.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-20T17:39:54.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-05-20T17:40:03.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-05-20T17:40:03.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-20T17:40:06.594 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-05-20T17:40:18.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-05-20T17:40:18.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-20T17:40:50.622 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-05-20T17:40:50.976 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-05-20T17:40:50.976 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-447.el9.x86_64"... 2024-05-20T17:40:50.976 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-05-20T17:40:50.991 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-447.el9.x86_64 2024-05-20T17:40:50.991 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-447.el9.x86_64 vs 5.14.0-447.el9.x86_64 2024-05-20T17:40:50.991 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-20T17:40:50.991 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-20T17:40:51.992 DEBUG:teuthology.parallel:result is None 2024-05-20T17:40:51.992 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-20T17:40:52.003 INFO:teuthology.task.internal:Creating test directory... 2024-05-20T17:40:52.003 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-20T17:40:52.005 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-20T17:40:52.008 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-20T17:40:52.027 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-20T17:40:52.035 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-20T17:40:52.085 INFO:teuthology.task.internal:Creating archive directory... 2024-05-20T17:40:52.085 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-20T17:40:52.088 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-20T17:40:52.090 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-20T17:40:52.213 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-20T17:40:52.220 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-20T17:40:52.220 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-20T17:40:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T17:40:52.237 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-20T17:40:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T17:40:52.254 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-20T17:40:52.256 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-20T17:40:52.280 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-20T17:40:52.323 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-20T17:40:52.329 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-20T17:40:52.339 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-20T17:40:52.354 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-20T17:40:52.358 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-20T17:40:52.379 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-20T17:40:52.380 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-20T17:40:52.388 INFO:teuthology.task.internal:Configuring sudo... 2024-05-20T17:40:52.388 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-20T17:40:52.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-20T17:40:52.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-20T17:40:52.426 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-20T17:40:52.435 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-20T17:40:52.435 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-20T17:40:52.460 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-20T17:40:52.462 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-20T17:40:52.481 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-20T17:40:52.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-20T17:40:52.665 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-20T17:40:52.706 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-20T17:40:52.762 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:40:52.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-20T17:40:52.849 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-20T17:40:52.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-20T17:40:52.946 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-20T17:40:53.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-20T17:40:53.072 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-05-20T17:40:53.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-20T17:40:53.140 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-20T17:40:53.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-20T17:40:53.231 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-20T17:40:53.314 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-20T17:40:53.356 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-20T17:40:53.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-20T17:40:53.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-05-20T17:40:53.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-05-20T17:40:53.425 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-05-20T17:40:53.462 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-20T17:40:53.493 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-20T17:40:53.495 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-20T17:40:53.970 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-20T17:40:53.980 INFO:teuthology.task.internal:Starting timer... 2024-05-20T17:40:53.980 INFO:teuthology.run_tasks:Running task pcp... 2024-05-20T17:40:54.032 INFO:teuthology.run_tasks:Running task selinux... 2024-05-20T17:40:54.079 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-20T17:40:54.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-05-20T17:40:54.252 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-05-20T17:40:54.253 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2024-05-20T17:40:54.334 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2024-05-20T17:40:54.336 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-05-20T17:40:54.377 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-05-20T17:40:54.379 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-20T17:40:54.379 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-05-20T17:40:54.401 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-05-20T17:40:54.401 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2024-05-20T17:40:54.435 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2024-05-20T17:40:54.435 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-05-20T17:40:54.457 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-05-20T17:40:54.458 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi129.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-05-20T17:40:54.458 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-20T17:40:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T17:40:54.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-20T17:40:54.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T17:40:54.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-20T17:40:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T17:40:54.537 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-20T17:40:54.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-05-20T17:40:54.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2024-05-20T17:40:54.788 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-05-20T17:40:54.812 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-20T17:40:54.822 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-20T17:40:54.833 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-20T17:40:54.833 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi129.front.sepia.ceph.com,smithi154.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-20T17:45:41.743 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi129.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-05-20T17:45:41.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-05-20T17:45:41.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-20T17:45:41.824 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-05-20T17:45:41.860 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-05-20T17:45:41.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-05-20T17:45:41.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-20T17:45:41.933 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-05-20T17:45:42.013 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-05-20T17:45:42.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-05-20T17:45:42.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-20T17:45:42.089 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-05-20T17:45:42.171 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-05-20T17:45:42.171 INFO:teuthology.run_tasks:Running task clock... 2024-05-20T17:45:42.183 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-20T17:45:42.183 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-20T17:45:42.184 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-20T17:45:42.186 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-20T17:45:42.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-20T17:45:42.188 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-20T17:45:42.188 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-20T17:45:42.214 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-20T17:45:42.215 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-20T17:45:42.228 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-20T17:45:42.229 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-20T17:45:42.243 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-20T17:45:42.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2024-05-20T17:45:42.253 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-05-20T17:45:42.258 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-20T17:45:42.262 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2024-05-20T17:45:42.264 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-05-20T17:45:42.275 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-20T17:45:42.278 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-20T17:45:42.283 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-05-20T17:45:42.287 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-20T17:45:42.293 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-20T17:45:42.294 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-05-20T17:45:42.308 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-20T17:45:42.321 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-20T17:45:42.341 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-05-20T17:45:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-20T17:45:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-05-20T17:45:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.344 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.345 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.345 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.346 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-05-20T17:45:42.349 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-20T17:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-05-20T17:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.350 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.376 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-05-20T17:45:42.380 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-20T17:45:42.380 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-05-20T17:45:42.380 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.380 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.380 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.380 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T17:45:42.381 INFO:teuthology.run_tasks:Running task install... 2024-05-20T17:45:42.395 DEBUG:teuthology.task.install:project ceph 2024-05-20T17:45:42.395 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2b507c724ef6c8295ee0a35d185cdab167fd61d'}} 2024-05-20T17:45:42.395 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2b507c724ef6c8295ee0a35d185cdab167fd61d'} 2024-05-20T17:45:42.396 INFO:teuthology.task.install:Using flavor: default 2024-05-20T17:45:42.403 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-20T17:45:42.403 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-20T17:45:42.404 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2b507c724ef6c8295ee0a35d185cdab167fd61d', 'tag': None, 'wait_for_package': False} 2024-05-20T17:45:42.404 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:45:42.406 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2b507c724ef6c8295ee0a35d185cdab167fd61d', 'tag': None, 'wait_for_package': False} 2024-05-20T17:45:42.407 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:45:42.408 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2b507c724ef6c8295ee0a35d185cdab167fd61d', 'tag': None, 'wait_for_package': False} 2024-05-20T17:45:42.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:45:42.544 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/ 2024-05-20T17:45:42.544 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2744.gd2b507c7 2024-05-20T17:45:42.555 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/ 2024-05-20T17:45:42.555 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2744.gd2b507c7 2024-05-20T17:45:42.558 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/ 2024-05-20T17:45:42.558 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2744.gd2b507c7 2024-05-20T17:45:42.890 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-20T17:45:42.890 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-20T17:45:42.891 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-20T17:45:42.904 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-20T17:45:42.905 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-05-20T17:45:42.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-20T17:45:42.917 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-20T17:45:42.917 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2b507c724ef6c8295ee0a35d185cdab167fd61d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-20T17:45:42.931 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-20T17:45:42.931 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2b507c724ef6c8295ee0a35d185cdab167fd61d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-20T17:45:42.936 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-20T17:45:42.937 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:45:42.937 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-20T17:45:42.963 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-20T17:45:42.963 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2b507c724ef6c8295ee0a35d185cdab167fd61d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-20T17:45:42.982 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-20T17:45:42.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-20T17:45:43.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-20T17:45:43.057 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-20T17:45:43.069 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-20T17:45:43.104 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-20T17:45:43.120 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-05-20T17:45:43.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-05-20T17:45:43.133 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2024-05-20T17:45:43.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2024-05-20T17:45:43.168 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-05-20T17:45:43.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-05-20T17:45:43.346 INFO:teuthology.orchestra.run.smithi204.stdout:45 files removed 2024-05-20T17:45:43.359 INFO:teuthology.orchestra.run.smithi154.stdout:45 files removed 2024-05-20T17:45:43.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-20T17:45:43.387 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-20T17:45:43.402 INFO:teuthology.orchestra.run.smithi129.stdout:45 files removed 2024-05-20T17:45:43.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-20T17:45:44.696 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 84 kB/s | 83 kB 00:00 2024-05-20T17:45:44.734 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 76 kB/s | 83 kB 00:01 2024-05-20T17:45:44.838 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 69 kB/s | 83 kB 00:01 2024-05-20T17:45:45.463 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 16 kB/s | 11 kB 00:00 2024-05-20T17:45:45.601 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 14 kB/s | 11 kB 00:00 2024-05-20T17:45:45.663 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 15 kB/s | 11 kB 00:00 2024-05-20T17:45:46.263 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-05-20T17:45:46.418 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-05-20T17:45:46.456 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-05-20T17:45:46.931 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.1 MB 00:00 2024-05-20T17:45:47.512 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 7.6 MB/s | 8.1 MB 00:01 2024-05-20T17:45:48.335 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.1 MB 00:02 2024-05-20T17:45:49.418 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-05-20T17:45:49.997 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-05-20T17:45:52.988 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 19 MB 00:03 2024-05-20T17:45:54.036 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 6.0 MB/s | 6.2 MB 00:01 2024-05-20T17:45:54.221 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.2 MB 00:00 2024-05-20T17:45:55.693 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-05-20T17:45:55.896 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-05-20T17:45:57.089 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-05-20T17:45:57.328 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.2 MB 00:00 2024-05-20T17:45:58.846 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-05-20T17:45:58.876 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 21 MB 00:02 2024-05-20T17:46:01.572 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 21 MB 00:02 2024-05-20T17:46:02.303 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-05-20T17:46:04.090 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-05-20T17:46:04.422 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-20T17:46:04.422 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-20T17:46:04.432 INFO:teuthology.orchestra.run.smithi204.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-20T17:46:04.433 INFO:teuthology.orchestra.run.smithi204.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-20T17:46:04.434 INFO:teuthology.orchestra.run.smithi204.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-20T17:46:04.434 INFO:teuthology.orchestra.run.smithi204.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-20T17:46:04.436 INFO:teuthology.orchestra.run.smithi204.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-20T17:46:04.438 INFO:teuthology.orchestra.run.smithi204.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-20T17:46:04.439 INFO:teuthology.orchestra.run.smithi204.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-20T17:46:04.500 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-05-20T17:46:04.521 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-05-20T17:46:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-05-20T17:46:04.522 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-05-20T17:46:04.522 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-05-20T17:46:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-20T17:46:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 6.6 k 2024-05-20T17:46:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 5.4 M 2024-05-20T17:46:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 863 k 2024-05-20T17:46:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 147 k 2024-05-20T17:46:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 1.5 M 2024-05-20T17:46:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 142 k 2024-05-20T17:46:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 3.5 M 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 7.4 M 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 49 k 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 9.8 M 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 46 M 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 264 k 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 763 k 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-mirror x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 221 k 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-shell x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 27 k 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-top noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 18 k 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-20T17:46:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 32 k 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 756 k 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 127 k 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 161 k 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 320 k 2024-05-20T17:46:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 299 k 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 99 k 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 86 k 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 3.0 M 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 169 k 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-20T17:46:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 3.3 M 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 3.1 M 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 19 M 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 32 k 2024-05-20T17:46:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 2.3 M 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 249 k 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 4.9 M 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 18 M 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 16 k 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 25 k 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-20T17:46:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-20T17:46:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 158 k 2024-05-20T17:46:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-20T17:46:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-20T17:46:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-20T17:46:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-20T17:46:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-20T17:46:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 497 k 2024-05-20T17:46:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-20T17:46:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 5.0 M 2024-05-20T17:46:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-20T17:46:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 45 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 135 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-20T17:46:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-20T17:46:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-20T17:46:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-20T17:46:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-20T17:46:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-20T17:46:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-20T17:46:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-20T17:46:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-20T17:46:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-20T17:46:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-20T17:46:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-20T17:46:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-20T17:46:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-20T17:46:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-20T17:46:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-20T17:46:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-20T17:46:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-20T17:46:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-20T17:46:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-20T17:46:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-20T17:46:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-20T17:46:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-20T17:46:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-20T17:46:04.536 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-05-20T17:46:04.536 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-20T17:46:04.536 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:46:04.536 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-05-20T17:46:04.536 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-05-20T17:46:04.536 INFO:teuthology.orchestra.run.smithi204.stdout:Install 148 Packages 2024-05-20T17:46:04.536 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 11 Packages 2024-05-20T17:46:04.536 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:46:04.537 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 222 M 2024-05-20T17:46:04.537 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-05-20T17:46:05.496 INFO:teuthology.orchestra.run.smithi204.stdout:(1/159): ceph-19.0.0-2744.gd2b507c7.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-05-20T17:46:06.105 INFO:teuthology.orchestra.run.smithi204.stdout:(2/159): ceph-fuse-19.0.0-2744.gd2b507c7.el9.x8 1.4 MB/s | 863 kB 00:00 2024-05-20T17:46:06.191 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-20T17:46:06.192 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-20T17:46:06.202 INFO:teuthology.orchestra.run.smithi154.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-20T17:46:06.203 INFO:teuthology.orchestra.run.smithi154.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-20T17:46:06.204 INFO:teuthology.orchestra.run.smithi154.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-20T17:46:06.205 INFO:teuthology.orchestra.run.smithi154.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-20T17:46:06.206 INFO:teuthology.orchestra.run.smithi154.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-20T17:46:06.209 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-20T17:46:06.209 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-20T17:46:06.271 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-05-20T17:46:06.282 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================= 2024-05-20T17:46:06.282 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-05-20T17:46:06.282 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================= 2024-05-20T17:46:06.282 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-05-20T17:46:06.282 INFO:teuthology.orchestra.run.smithi154.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-20T17:46:06.282 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 6.6 k 2024-05-20T17:46:06.282 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 5.4 M 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 863 k 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 147 k 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 1.5 M 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 142 k 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 3.5 M 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 7.4 M 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 49 k 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 9.8 M 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 46 M 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 264 k 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 763 k 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 221 k 2024-05-20T17:46:06.283 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-shell x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 27 k 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 18 k 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 32 k 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 756 k 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 127 k 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-20T17:46:06.284 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 161 k 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 320 k 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 299 k 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 99 k 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 86 k 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 3.0 M 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 169 k 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-20T17:46:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-20T17:46:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-20T17:46:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-20T17:46:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 3.3 M 2024-05-20T17:46:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 3.1 M 2024-05-20T17:46:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-20T17:46:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-20T17:46:06.286 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-05-20T17:46:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-20T17:46:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 19 M 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 32 k 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 2.3 M 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 249 k 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 4.9 M 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 18 M 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 16 k 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 25 k 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-20T17:46:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 158 k 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 497 k 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 5.0 M 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-20T17:46:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-20T17:46:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-20T17:46:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-20T17:46:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-20T17:46:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 45 k 2024-05-20T17:46:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 135 k 2024-05-20T17:46:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-20T17:46:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-20T17:46:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-20T17:46:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-20T17:46:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-20T17:46:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-20T17:46:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-20T17:46:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-20T17:46:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-20T17:46:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-20T17:46:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================= 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout:Install 148 Packages 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 11 Packages 2024-05-20T17:46:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:46:06.296 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 222 M 2024-05-20T17:46:06.296 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-05-20T17:46:06.330 INFO:teuthology.orchestra.run.smithi204.stdout:(3/159): ceph-immutable-object-cache-19.0.0-274 653 kB/s | 147 kB 00:00 2024-05-20T17:46:06.722 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-05-20T17:46:07.082 INFO:teuthology.orchestra.run.smithi154.stdout:(1/159): ceph-19.0.0-2744.gd2b507c7.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-05-20T17:46:07.688 INFO:teuthology.orchestra.run.smithi204.stdout:(4/159): ceph-mds-19.0.0-2744.gd2b507c7.el9.x86 1.7 MB/s | 2.3 MB 00:01 2024-05-20T17:46:07.974 INFO:teuthology.orchestra.run.smithi154.stdout:(2/159): ceph-fuse-19.0.0-2744.gd2b507c7.el9.x8 968 kB/s | 863 kB 00:00 2024-05-20T17:46:08.150 INFO:teuthology.orchestra.run.smithi154.stdout:(3/159): ceph-immutable-object-cache-19.0.0-274 840 kB/s | 147 kB 00:00 2024-05-20T17:46:08.522 INFO:teuthology.orchestra.run.smithi204.stdout:(5/159): ceph-mgr-19.0.0-2744.gd2b507c7.el9.x86 1.8 MB/s | 1.5 MB 00:00 2024-05-20T17:46:08.589 INFO:teuthology.orchestra.run.smithi204.stdout:(6/159): ceph-base-19.0.0-2744.gd2b507c7.el9.x8 1.6 MB/s | 5.4 MB 00:03 2024-05-20T17:46:08.833 INFO:teuthology.orchestra.run.smithi154.stdout:(4/159): ceph-base-19.0.0-2744.gd2b507c7.el9.x8 2.6 MB/s | 5.4 MB 00:02 2024-05-20T17:46:08.882 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-20T17:46:08.883 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-20T17:46:08.894 INFO:teuthology.orchestra.run.smithi129.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-20T17:46:08.895 INFO:teuthology.orchestra.run.smithi129.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-20T17:46:08.895 INFO:teuthology.orchestra.run.smithi129.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-20T17:46:08.896 INFO:teuthology.orchestra.run.smithi129.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-20T17:46:08.898 INFO:teuthology.orchestra.run.smithi129.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-20T17:46:08.900 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-20T17:46:08.901 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-20T17:46:08.965 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-05-20T17:46:08.979 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-05-20T17:46:08.979 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 6.6 k 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 5.4 M 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 863 k 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 147 k 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 1.5 M 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 142 k 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 3.5 M 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 7.4 M 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 49 k 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 9.8 M 2024-05-20T17:46:08.980 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 46 M 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 264 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 763 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 221 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 27 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 18 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 32 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 756 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 127 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-20T17:46:08.981 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 161 k 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 320 k 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 299 k 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 99 k 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 86 k 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 3.0 M 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 169 k 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-05-20T17:46:08.982 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 3.3 M 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 3.1 M 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-20T17:46:08.983 INFO:teuthology.orchestra.run.smithi129.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 19 M 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 32 k 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 2.3 M 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 249 k 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 4.9 M 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 18 M 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2744.gd2b507c7.el9 ceph-noarch 16 k 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 25 k 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-20T17:46:08.984 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 158 k 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 497 k 2024-05-20T17:46:08.985 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 5.0 M 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-20T17:46:08.986 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 45 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.0.0-2744.gd2b507c7.el9 ceph 135 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-20T17:46:08.987 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-20T17:46:08.988 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-20T17:46:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-20T17:46:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-20T17:46:08.991 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-20T17:46:08.992 INFO:teuthology.orchestra.run.smithi129.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-20T17:46:08.992 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-20T17:46:08.992 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-20T17:46:08.992 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-05-20T17:46:08.992 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-20T17:46:08.992 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:46:08.992 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-05-20T17:46:08.992 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-05-20T17:46:08.992 INFO:teuthology.orchestra.run.smithi129.stdout:Install 148 Packages 2024-05-20T17:46:08.992 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 11 Packages 2024-05-20T17:46:08.992 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:46:08.993 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 222 M 2024-05-20T17:46:08.993 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-05-20T17:46:09.159 INFO:teuthology.orchestra.run.smithi154.stdout:(5/159): ceph-mds-19.0.0-2744.gd2b507c7.el9.x86 2.2 MB/s | 2.3 MB 00:01 2024-05-20T17:46:09.309 INFO:teuthology.orchestra.run.smithi154.stdout:(6/159): ceph-mgr-19.0.0-2744.gd2b507c7.el9.x86 3.1 MB/s | 1.5 MB 00:00 2024-05-20T17:46:09.932 INFO:teuthology.orchestra.run.smithi129.stdout:(1/159): ceph-19.0.0-2744.gd2b507c7.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-05-20T17:46:10.549 INFO:teuthology.orchestra.run.smithi129.stdout:(2/159): ceph-fuse-19.0.0-2744.gd2b507c7.el9.x8 1.4 MB/s | 863 kB 00:00 2024-05-20T17:46:10.618 INFO:teuthology.orchestra.run.smithi154.stdout:(7/159): ceph-mon-19.0.0-2744.gd2b507c7.el9.x86 3.3 MB/s | 4.9 MB 00:01 2024-05-20T17:46:10.633 INFO:teuthology.orchestra.run.smithi129.stdout:(3/159): ceph-immutable-object-cache-19.0.0-274 1.7 MB/s | 147 kB 00:00 2024-05-20T17:46:11.231 INFO:teuthology.orchestra.run.smithi204.stdout:(7/159): ceph-mon-19.0.0-2744.gd2b507c7.el9.x86 1.8 MB/s | 4.9 MB 00:02 2024-05-20T17:46:11.525 INFO:teuthology.orchestra.run.smithi129.stdout:(4/159): ceph-mds-19.0.0-2744.gd2b507c7.el9.x86 2.5 MB/s | 2.3 MB 00:00 2024-05-20T17:46:11.975 INFO:teuthology.orchestra.run.smithi129.stdout:(5/159): ceph-mgr-19.0.0-2744.gd2b507c7.el9.x86 3.3 MB/s | 1.5 MB 00:00 2024-05-20T17:46:12.684 INFO:teuthology.orchestra.run.smithi129.stdout:(6/159): ceph-base-19.0.0-2744.gd2b507c7.el9.x8 1.7 MB/s | 5.4 MB 00:03 2024-05-20T17:46:12.817 INFO:teuthology.orchestra.run.smithi129.stdout:(7/159): ceph-mon-19.0.0-2744.gd2b507c7.el9.x86 5.8 MB/s | 4.9 MB 00:00 2024-05-20T17:46:12.984 INFO:teuthology.orchestra.run.smithi154.stdout:(8/159): ceph-radosgw-19.0.0-2744.gd2b507c7.el9 4.2 MB/s | 9.8 MB 00:02 2024-05-20T17:46:13.084 INFO:teuthology.orchestra.run.smithi154.stdout:(9/159): ceph-selinux-19.0.0-2744.gd2b507c7.el9 252 kB/s | 25 kB 00:00 2024-05-20T17:46:14.259 INFO:teuthology.orchestra.run.smithi129.stdout:(8/159): ceph-radosgw-19.0.0-2744.gd2b507c7.el9 6.8 MB/s | 9.8 MB 00:01 2024-05-20T17:46:14.351 INFO:teuthology.orchestra.run.smithi129.stdout:(9/159): ceph-selinux-19.0.0-2744.gd2b507c7.el9 275 kB/s | 25 kB 00:00 2024-05-20T17:46:16.218 INFO:teuthology.orchestra.run.smithi154.stdout:(10/159): ceph-osd-19.0.0-2744.gd2b507c7.el9.x8 2.6 MB/s | 18 MB 00:06 2024-05-20T17:46:16.426 INFO:teuthology.orchestra.run.smithi154.stdout:(11/159): cephfs-mirror-19.0.0-2744.gd2b507c7.e 1.0 MB/s | 221 kB 00:00 2024-05-20T17:46:16.551 INFO:teuthology.orchestra.run.smithi154.stdout:(12/159): cephfs-shell-19.0.0-2744.gd2b507c7.el 217 kB/s | 27 kB 00:00 2024-05-20T17:46:16.660 INFO:teuthology.orchestra.run.smithi154.stdout:(13/159): libcephfs-devel-19.0.0-2744.gd2b507c7 292 kB/s | 32 kB 00:00 2024-05-20T17:46:17.060 INFO:teuthology.orchestra.run.smithi154.stdout:(14/159): libcephfs2-19.0.0-2744.gd2b507c7.el9. 1.8 MB/s | 756 kB 00:00 2024-05-20T17:46:17.114 INFO:teuthology.orchestra.run.smithi204.stdout:(8/159): ceph-radosgw-19.0.0-2744.gd2b507c7.el9 1.7 MB/s | 9.8 MB 00:05 2024-05-20T17:46:17.169 INFO:teuthology.orchestra.run.smithi154.stdout:(15/159): libcephsqlite-19.0.0-2744.gd2b507c7.e 1.4 MB/s | 158 kB 00:00 2024-05-20T17:46:17.214 INFO:teuthology.orchestra.run.smithi204.stdout:(9/159): ceph-selinux-19.0.0-2744.gd2b507c7.el9 252 kB/s | 25 kB 00:00 2024-05-20T17:46:17.286 INFO:teuthology.orchestra.run.smithi154.stdout:(16/159): librados-devel-19.0.0-2744.gd2b507c7. 1.1 MB/s | 127 kB 00:00 2024-05-20T17:46:17.628 INFO:teuthology.orchestra.run.smithi154.stdout:(17/159): libradosstriper1-19.0.0-2744.gd2b507c 1.4 MB/s | 497 kB 00:00 2024-05-20T17:46:18.798 INFO:teuthology.orchestra.run.smithi204.stdout:(10/159): ceph-common-19.0.0-2744.gd2b507c7.el9 1.4 MB/s | 19 MB 00:13 2024-05-20T17:46:19.064 INFO:teuthology.orchestra.run.smithi204.stdout:(11/159): cephfs-mirror-19.0.0-2744.gd2b507c7.e 829 kB/s | 221 kB 00:00 2024-05-20T17:46:19.186 INFO:teuthology.orchestra.run.smithi204.stdout:(12/159): cephfs-shell-19.0.0-2744.gd2b507c7.el 222 kB/s | 27 kB 00:00 2024-05-20T17:46:19.282 INFO:teuthology.orchestra.run.smithi204.stdout:(13/159): libcephfs-devel-19.0.0-2744.gd2b507c7 333 kB/s | 32 kB 00:00 2024-05-20T17:46:19.799 INFO:teuthology.orchestra.run.smithi204.stdout:(14/159): libcephfs2-19.0.0-2744.gd2b507c7.el9. 1.4 MB/s | 756 kB 00:00 2024-05-20T17:46:19.945 INFO:teuthology.orchestra.run.smithi154.stdout:(18/159): librgw2-19.0.0-2744.gd2b507c7.el9.x86 2.2 MB/s | 5.0 MB 00:02 2024-05-20T17:46:19.949 INFO:teuthology.orchestra.run.smithi204.stdout:(15/159): libcephsqlite-19.0.0-2744.gd2b507c7.e 1.0 MB/s | 158 kB 00:00 2024-05-20T17:46:20.061 INFO:teuthology.orchestra.run.smithi154.stdout:(19/159): python3-ceph-argparse-19.0.0-2744.gd2 388 kB/s | 45 kB 00:00 2024-05-20T17:46:20.108 INFO:teuthology.orchestra.run.smithi204.stdout:(16/159): librados-devel-19.0.0-2744.gd2b507c7. 801 kB/s | 127 kB 00:00 2024-05-20T17:46:20.178 INFO:teuthology.orchestra.run.smithi154.stdout:(20/159): python3-ceph-common-19.0.0-2744.gd2b5 1.1 MB/s | 135 kB 00:00 2024-05-20T17:46:20.295 INFO:teuthology.orchestra.run.smithi154.stdout:(21/159): python3-cephfs-19.0.0-2744.gd2b507c7. 1.4 MB/s | 161 kB 00:00 2024-05-20T17:46:20.504 INFO:teuthology.orchestra.run.smithi154.stdout:(22/159): python3-rados-19.0.0-2744.gd2b507c7.e 1.5 MB/s | 320 kB 00:00 2024-05-20T17:46:20.508 INFO:teuthology.orchestra.run.smithi204.stdout:(17/159): libradosstriper1-19.0.0-2744.gd2b507c 1.2 MB/s | 497 kB 00:00 2024-05-20T17:46:20.704 INFO:teuthology.orchestra.run.smithi154.stdout:(23/159): python3-rbd-19.0.0-2744.gd2b507c7.el9 1.5 MB/s | 299 kB 00:00 2024-05-20T17:46:20.813 INFO:teuthology.orchestra.run.smithi154.stdout:(24/159): python3-rgw-19.0.0-2744.gd2b507c7.el9 917 kB/s | 99 kB 00:00 2024-05-20T17:46:20.930 INFO:teuthology.orchestra.run.smithi154.stdout:(25/159): rbd-fuse-19.0.0-2744.gd2b507c7.el9.x8 735 kB/s | 86 kB 00:00 2024-05-20T17:46:21.092 INFO:teuthology.orchestra.run.smithi129.stdout:(10/159): ceph-osd-19.0.0-2744.gd2b507c7.el9.x8 2.1 MB/s | 18 MB 00:08 2024-05-20T17:46:21.334 INFO:teuthology.orchestra.run.smithi129.stdout:(11/159): ceph-common-19.0.0-2744.gd2b507c7.el9 1.6 MB/s | 19 MB 00:11 2024-05-20T17:46:21.359 INFO:teuthology.orchestra.run.smithi129.stdout:(12/159): cephfs-mirror-19.0.0-2744.gd2b507c7.e 828 kB/s | 221 kB 00:00 2024-05-20T17:46:21.443 INFO:teuthology.orchestra.run.smithi129.stdout:(13/159): cephfs-shell-19.0.0-2744.gd2b507c7.el 250 kB/s | 27 kB 00:00 2024-05-20T17:46:21.476 INFO:teuthology.orchestra.run.smithi129.stdout:(14/159): libcephfs-devel-19.0.0-2744.gd2b507c7 271 kB/s | 32 kB 00:00 2024-05-20T17:46:21.602 INFO:teuthology.orchestra.run.smithi129.stdout:(15/159): libcephsqlite-19.0.0-2744.gd2b507c7.e 1.2 MB/s | 158 kB 00:00 2024-05-20T17:46:21.718 INFO:teuthology.orchestra.run.smithi129.stdout:(16/159): librados-devel-19.0.0-2744.gd2b507c7. 1.1 MB/s | 127 kB 00:00 2024-05-20T17:46:22.011 INFO:teuthology.orchestra.run.smithi129.stdout:(17/159): libradosstriper1-19.0.0-2744.gd2b507c 1.7 MB/s | 497 kB 00:00 2024-05-20T17:46:22.044 INFO:teuthology.orchestra.run.smithi129.stdout:(18/159): libcephfs2-19.0.0-2744.gd2b507c7.el9. 1.2 MB/s | 756 kB 00:00 2024-05-20T17:46:22.161 INFO:teuthology.orchestra.run.smithi129.stdout:(19/159): python3-ceph-argparse-19.0.0-2744.gd2 387 kB/s | 45 kB 00:00 2024-05-20T17:46:22.278 INFO:teuthology.orchestra.run.smithi129.stdout:(20/159): python3-ceph-common-19.0.0-2744.gd2b5 1.1 MB/s | 135 kB 00:00 2024-05-20T17:46:22.313 INFO:teuthology.orchestra.run.smithi154.stdout:(26/159): rbd-mirror-19.0.0-2744.gd2b507c7.el9. 2.2 MB/s | 3.0 MB 00:01 2024-05-20T17:46:22.430 INFO:teuthology.orchestra.run.smithi154.stdout:(27/159): rbd-nbd-19.0.0-2744.gd2b507c7.el9.x86 1.4 MB/s | 169 kB 00:00 2024-05-20T17:46:22.470 INFO:teuthology.orchestra.run.smithi129.stdout:(21/159): python3-cephfs-19.0.0-2744.gd2b507c7. 843 kB/s | 161 kB 00:00 2024-05-20T17:46:22.525 INFO:teuthology.orchestra.run.smithi204.stdout:(18/159): ceph-osd-19.0.0-2744.gd2b507c7.el9.x8 1.3 MB/s | 18 MB 00:13 2024-05-20T17:46:22.581 INFO:teuthology.orchestra.run.smithi154.stdout:(28/159): ceph-grafana-dashboards-19.0.0-2744.g 212 kB/s | 32 kB 00:00 2024-05-20T17:46:22.633 INFO:teuthology.orchestra.run.smithi204.stdout:(19/159): python3-ceph-argparse-19.0.0-2744.gd2 417 kB/s | 45 kB 00:00 2024-05-20T17:46:22.706 INFO:teuthology.orchestra.run.smithi154.stdout:(29/159): ceph-mgr-cephadm-19.0.0-2744.gd2b507c 1.1 MB/s | 142 kB 00:00 2024-05-20T17:46:22.745 INFO:teuthology.orchestra.run.smithi129.stdout:(22/159): python3-rados-19.0.0-2744.gd2b507c7.e 1.1 MB/s | 320 kB 00:00 2024-05-20T17:46:22.834 INFO:teuthology.orchestra.run.smithi204.stdout:(20/159): python3-ceph-common-19.0.0-2744.gd2b5 676 kB/s | 135 kB 00:00 2024-05-20T17:46:23.020 INFO:teuthology.orchestra.run.smithi129.stdout:(23/159): python3-rbd-19.0.0-2744.gd2b507c7.el9 1.1 MB/s | 299 kB 00:00 2024-05-20T17:46:23.023 INFO:teuthology.orchestra.run.smithi154.stdout:(30/159): ceph-common-19.0.0-2744.gd2b507c7.el9 1.2 MB/s | 19 MB 00:16 2024-05-20T17:46:23.025 INFO:teuthology.orchestra.run.smithi204.stdout:(21/159): python3-cephfs-19.0.0-2744.gd2b507c7. 842 kB/s | 161 kB 00:00 2024-05-20T17:46:23.121 INFO:teuthology.orchestra.run.smithi129.stdout:(24/159): python3-rgw-19.0.0-2744.gd2b507c7.el9 994 kB/s | 99 kB 00:00 2024-05-20T17:46:23.238 INFO:teuthology.orchestra.run.smithi129.stdout:(25/159): rbd-fuse-19.0.0-2744.gd2b507c7.el9.x8 734 kB/s | 86 kB 00:00 2024-05-20T17:46:23.359 INFO:teuthology.orchestra.run.smithi204.stdout:(22/159): python3-rados-19.0.0-2744.gd2b507c7.e 960 kB/s | 320 kB 00:00 2024-05-20T17:46:23.485 INFO:teuthology.orchestra.run.smithi204.stdout:(23/159): librgw2-19.0.0-2744.gd2b507c7.el9.x86 1.7 MB/s | 5.0 MB 00:02 2024-05-20T17:46:23.593 INFO:teuthology.orchestra.run.smithi204.stdout:(24/159): python3-rgw-19.0.0-2744.gd2b507c7.el9 918 kB/s | 99 kB 00:00 2024-05-20T17:46:23.685 INFO:teuthology.orchestra.run.smithi204.stdout:(25/159): rbd-fuse-19.0.0-2744.gd2b507c7.el9.x8 937 kB/s | 86 kB 00:00 2024-05-20T17:46:23.711 INFO:teuthology.orchestra.run.smithi204.stdout:(26/159): python3-rbd-19.0.0-2744.gd2b507c7.el9 852 kB/s | 299 kB 00:00 2024-05-20T17:46:23.886 INFO:teuthology.orchestra.run.smithi204.stdout:(27/159): rbd-nbd-19.0.0-2744.gd2b507c7.el9.x86 969 kB/s | 169 kB 00:00 2024-05-20T17:46:24.036 INFO:teuthology.orchestra.run.smithi204.stdout:(28/159): ceph-grafana-dashboards-19.0.0-2744.g 212 kB/s | 32 kB 00:00 2024-05-20T17:46:24.195 INFO:teuthology.orchestra.run.smithi204.stdout:(29/159): ceph-mgr-cephadm-19.0.0-2744.gd2b507c 899 kB/s | 142 kB 00:00 2024-05-20T17:46:24.263 INFO:teuthology.orchestra.run.smithi129.stdout:(26/159): librgw2-19.0.0-2744.gd2b507c7.el9.x86 2.2 MB/s | 5.0 MB 00:02 2024-05-20T17:46:24.455 INFO:teuthology.orchestra.run.smithi129.stdout:(27/159): rbd-nbd-19.0.0-2744.gd2b507c7.el9.x86 885 kB/s | 169 kB 00:00 2024-05-20T17:46:24.564 INFO:teuthology.orchestra.run.smithi129.stdout:(28/159): ceph-grafana-dashboards-19.0.0-2744.g 293 kB/s | 32 kB 00:00 2024-05-20T17:46:24.582 INFO:teuthology.orchestra.run.smithi154.stdout:(31/159): ceph-mgr-dashboard-19.0.0-2744.gd2b50 1.9 MB/s | 3.5 MB 00:01 2024-05-20T17:46:24.672 INFO:teuthology.orchestra.run.smithi129.stdout:(29/159): ceph-mgr-cephadm-19.0.0-2744.gd2b507c 1.3 MB/s | 142 kB 00:00 2024-05-20T17:46:24.782 INFO:teuthology.orchestra.run.smithi154.stdout:(32/159): ceph-mgr-modules-core-19.0.0-2744.gd2 1.2 MB/s | 249 kB 00:00 2024-05-20T17:46:24.907 INFO:teuthology.orchestra.run.smithi154.stdout:(33/159): ceph-mgr-rook-19.0.0-2744.gd2b507c7.e 395 kB/s | 49 kB 00:00 2024-05-20T17:46:25.041 INFO:teuthology.orchestra.run.smithi154.stdout:(34/159): ceph-prometheus-alerts-19.0.0-2744.gd 123 kB/s | 16 kB 00:00 2024-05-20T17:46:25.392 INFO:teuthology.orchestra.run.smithi154.stdout:(35/159): ceph-volume-19.0.0-2744.gd2b507c7.el9 752 kB/s | 264 kB 00:00 2024-05-20T17:46:25.456 INFO:teuthology.orchestra.run.smithi129.stdout:(30/159): ceph-test-19.0.0-2744.gd2b507c7.el9.x 4.2 MB/s | 46 MB 00:11 2024-05-20T17:46:25.506 INFO:teuthology.orchestra.run.smithi129.stdout:(31/159): rbd-mirror-19.0.0-2744.gd2b507c7.el9. 1.3 MB/s | 3.0 MB 00:02 2024-05-20T17:46:25.662 INFO:teuthology.orchestra.run.smithi204.stdout:(30/159): rbd-mirror-19.0.0-2744.gd2b507c7.el9. 1.5 MB/s | 3.0 MB 00:01 2024-05-20T17:46:25.758 INFO:teuthology.orchestra.run.smithi154.stdout:(36/159): cephadm-19.0.0-2744.gd2b507c7.el9.noa 2.0 MB/s | 763 kB 00:00 2024-05-20T17:46:25.790 INFO:teuthology.orchestra.run.smithi129.stdout:(32/159): ceph-mgr-modules-core-19.0.0-2744.gd2 878 kB/s | 249 kB 00:00 2024-05-20T17:46:25.867 INFO:teuthology.orchestra.run.smithi154.stdout:(37/159): cephfs-top-19.0.0-2744.gd2b507c7.el9. 168 kB/s | 18 kB 00:00 2024-05-20T17:46:25.890 INFO:teuthology.orchestra.run.smithi129.stdout:(33/159): ceph-mgr-rook-19.0.0-2744.gd2b507c7.e 494 kB/s | 49 kB 00:00 2024-05-20T17:46:25.970 INFO:teuthology.orchestra.run.smithi204.stdout:(31/159): ceph-mgr-dashboard-19.0.0-2744.gd2b50 2.0 MB/s | 3.5 MB 00:01 2024-05-20T17:46:25.998 INFO:teuthology.orchestra.run.smithi129.stdout:(34/159): ceph-prometheus-alerts-19.0.0-2744.gd 152 kB/s | 16 kB 00:00 2024-05-20T17:46:26.008 INFO:teuthology.orchestra.run.smithi154.stdout:(38/159): fuse-2.9.9-15.el9.x86_64.rpm 568 kB/s | 80 kB 00:00 2024-05-20T17:46:26.134 INFO:teuthology.orchestra.run.smithi154.stdout:(39/159): libgfortran-11.4.1-3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2024-05-20T17:46:26.146 INFO:teuthology.orchestra.run.smithi204.stdout:(32/159): ceph-mgr-modules-core-19.0.0-2744.gd2 1.4 MB/s | 249 kB 00:00 2024-05-20T17:46:26.184 INFO:teuthology.orchestra.run.smithi154.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-05-20T17:46:26.207 INFO:teuthology.orchestra.run.smithi129.stdout:(35/159): ceph-volume-19.0.0-2744.gd2b507c7.el9 1.2 MB/s | 264 kB 00:00 2024-05-20T17:46:26.235 INFO:teuthology.orchestra.run.smithi154.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-05-20T17:46:26.246 INFO:teuthology.orchestra.run.smithi204.stdout:(33/159): ceph-mgr-rook-19.0.0-2744.gd2b507c7.e 494 kB/s | 49 kB 00:00 2024-05-20T17:46:26.318 INFO:teuthology.orchestra.run.smithi154.stdout:(42/159): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-05-20T17:46:26.360 INFO:teuthology.orchestra.run.smithi154.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-05-20T17:46:26.371 INFO:teuthology.orchestra.run.smithi204.stdout:(34/159): ceph-prometheus-alerts-19.0.0-2744.gd 131 kB/s | 16 kB 00:00 2024-05-20T17:46:26.410 INFO:teuthology.orchestra.run.smithi154.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 2.7 MB/s | 135 kB 00:00 2024-05-20T17:46:26.461 INFO:teuthology.orchestra.run.smithi154.stdout:(45/159): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-05-20T17:46:26.511 INFO:teuthology.orchestra.run.smithi154.stdout:(46/159): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-05-20T17:46:26.546 INFO:teuthology.orchestra.run.smithi204.stdout:(35/159): ceph-volume-19.0.0-2744.gd2b507c7.el9 1.5 MB/s | 264 kB 00:00 2024-05-20T17:46:26.608 INFO:teuthology.orchestra.run.smithi129.stdout:(36/159): ceph-mgr-dashboard-19.0.0-2744.gd2b50 1.8 MB/s | 3.5 MB 00:01 2024-05-20T17:46:26.716 INFO:teuthology.orchestra.run.smithi129.stdout:(37/159): cephfs-top-19.0.0-2744.gd2b507c7.el9. 168 kB/s | 18 kB 00:00 2024-05-20T17:46:26.736 INFO:teuthology.orchestra.run.smithi154.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 3.0 MB/s | 689 kB 00:00 2024-05-20T17:46:26.783 INFO:teuthology.orchestra.run.smithi129.stdout:(38/159): cephadm-19.0.0-2744.gd2b507c7.el9.noa 1.3 MB/s | 763 kB 00:00 2024-05-20T17:46:26.845 INFO:teuthology.orchestra.run.smithi154.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 8.5 MB/s | 943 kB 00:00 2024-05-20T17:46:26.887 INFO:teuthology.orchestra.run.smithi154.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-05-20T17:46:26.922 INFO:teuthology.orchestra.run.smithi204.stdout:(36/159): cephadm-19.0.0-2744.gd2b507c7.el9.noa 2.0 MB/s | 763 kB 00:00 2024-05-20T17:46:26.938 INFO:teuthology.orchestra.run.smithi154.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 434 kB/s | 22 kB 00:00 2024-05-20T17:46:26.996 INFO:teuthology.orchestra.run.smithi154.stdout:(51/159): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-05-20T17:46:27.038 INFO:teuthology.orchestra.run.smithi154.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 782 kB/s | 33 kB 00:00 2024-05-20T17:46:27.122 INFO:teuthology.orchestra.run.smithi204.stdout:(37/159): cephfs-top-19.0.0-2744.gd2b507c7.el9. 91 kB/s | 18 kB 00:00 2024-05-20T17:46:27.150 INFO:teuthology.orchestra.run.smithi129.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 185 kB/s | 80 kB 00:00 2024-05-20T17:46:27.197 INFO:teuthology.orchestra.run.smithi154.stdout:(53/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-05-20T17:46:27.230 INFO:teuthology.orchestra.run.smithi204.stdout:(38/159): fuse-2.9.9-15.el9.x86_64.rpm 743 kB/s | 80 kB 00:00 2024-05-20T17:46:27.239 INFO:teuthology.orchestra.run.smithi154.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 435 kB/s | 18 kB 00:00 2024-05-20T17:46:27.298 INFO:teuthology.orchestra.run.smithi154.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-05-20T17:46:27.348 INFO:teuthology.orchestra.run.smithi204.stdout:(39/159): libgfortran-11.4.1-3.el9.x86_64.rpm 6.7 MB/s | 803 kB 00:00 2024-05-20T17:46:27.364 INFO:teuthology.orchestra.run.smithi154.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-05-20T17:46:27.398 INFO:teuthology.orchestra.run.smithi204.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-05-20T17:46:27.432 INFO:teuthology.orchestra.run.smithi154.stdout:(57/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 678 kB/s | 45 kB 00:00 2024-05-20T17:46:27.448 INFO:teuthology.orchestra.run.smithi204.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-05-20T17:46:27.490 INFO:teuthology.orchestra.run.smithi154.stdout:(58/159): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-05-20T17:46:27.532 INFO:teuthology.orchestra.run.smithi204.stdout:(42/159): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-05-20T17:46:27.557 INFO:teuthology.orchestra.run.smithi154.stdout:(59/159): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-05-20T17:46:27.582 INFO:teuthology.orchestra.run.smithi204.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-20T17:46:27.599 INFO:teuthology.orchestra.run.smithi154.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 610 kB/s | 25 kB 00:00 2024-05-20T17:46:27.633 INFO:teuthology.orchestra.run.smithi204.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-05-20T17:46:27.649 INFO:teuthology.orchestra.run.smithi154.stdout:(61/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 952 kB/s | 47 kB 00:00 2024-05-20T17:46:27.683 INFO:teuthology.orchestra.run.smithi204.stdout:(45/159): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-05-20T17:46:27.708 INFO:teuthology.orchestra.run.smithi154.stdout:(62/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-05-20T17:46:27.734 INFO:teuthology.orchestra.run.smithi204.stdout:(46/159): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-05-20T17:46:27.758 INFO:teuthology.orchestra.run.smithi154.stdout:(63/159): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-05-20T17:46:27.792 INFO:teuthology.orchestra.run.smithi204.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 12 MB/s | 689 kB 00:00 2024-05-20T17:46:27.800 INFO:teuthology.orchestra.run.smithi154.stdout:(64/159): openblas-0.3.21-2.el9.x86_64.rpm 847 kB/s | 35 kB 00:00 2024-05-20T17:46:27.817 INFO:teuthology.orchestra.run.smithi129.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 293 kB/s | 195 kB 00:00 2024-05-20T17:46:27.859 INFO:teuthology.orchestra.run.smithi204.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-05-20T17:46:27.909 INFO:teuthology.orchestra.run.smithi129.stdout:(41/159): ceph-mgr-diskprediction-local-19.0.0- 3.0 MB/s | 7.4 MB 00:02 2024-05-20T17:46:27.909 INFO:teuthology.orchestra.run.smithi204.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-05-20T17:46:27.933 INFO:teuthology.orchestra.run.smithi204.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:27.967 INFO:teuthology.orchestra.run.smithi154.stdout:(65/159): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-05-20T17:46:28.034 INFO:teuthology.orchestra.run.smithi204.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:28.118 INFO:teuthology.orchestra.run.smithi204.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 104 kB/s | 22 kB 00:00 2024-05-20T17:46:28.252 INFO:teuthology.orchestra.run.smithi204.stdout:(51/159): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-05-20T17:46:28.302 INFO:teuthology.orchestra.run.smithi204.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 655 kB/s | 33 kB 00:00 2024-05-20T17:46:28.443 INFO:teuthology.orchestra.run.smithi154.stdout:(66/159): ceph-test-19.0.0-2744.gd2b507c7.el9.x 3.0 MB/s | 46 MB 00:15 2024-05-20T17:46:28.452 INFO:teuthology.orchestra.run.smithi204.stdout:(53/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-05-20T17:46:28.492 INFO:teuthology.orchestra.run.smithi129.stdout:(42/159): libgfortran-11.4.1-3.el9.x86_64.rpm 470 kB/s | 803 kB 00:01 2024-05-20T17:46:28.502 INFO:teuthology.orchestra.run.smithi204.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 366 kB/s | 18 kB 00:00 2024-05-20T17:46:28.543 INFO:teuthology.orchestra.run.smithi154.stdout:(67/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.7 MB/s | 4.4 MB 00:00 2024-05-20T17:46:28.553 INFO:teuthology.orchestra.run.smithi204.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-05-20T17:46:28.592 INFO:teuthology.orchestra.run.smithi129.stdout:(43/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 326 kB/s | 253 kB 00:00 2024-05-20T17:46:28.603 INFO:teuthology.orchestra.run.smithi204.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-05-20T17:46:28.635 INFO:teuthology.orchestra.run.smithi154.stdout:(68/159): postgresql-contrib-13.14-1.el9.x86_64 9.7 MB/s | 911 kB 00:00 2024-05-20T17:46:28.653 INFO:teuthology.orchestra.run.smithi204.stdout:(57/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-05-20T17:46:28.685 INFO:teuthology.orchestra.run.smithi154.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-05-20T17:46:28.712 INFO:teuthology.orchestra.run.smithi204.stdout:(58/159): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-05-20T17:46:28.727 INFO:teuthology.orchestra.run.smithi154.stdout:(70/159): postgresql-13.14-1.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-05-20T17:46:28.759 INFO:teuthology.orchestra.run.smithi129.stdout:(44/159): python3-cryptography-36.0.1-4.el9.x86 1.4 MB/s | 1.2 MB 00:00 2024-05-20T17:46:28.779 INFO:teuthology.orchestra.run.smithi204.stdout:(59/159): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 582 kB 00:00 2024-05-20T17:46:28.802 INFO:teuthology.orchestra.run.smithi154.stdout:(71/159): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-05-20T17:46:28.830 INFO:teuthology.orchestra.run.smithi204.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 508 kB/s | 25 kB 00:00 2024-05-20T17:46:28.844 INFO:teuthology.orchestra.run.smithi154.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-05-20T17:46:28.880 INFO:teuthology.orchestra.run.smithi204.stdout:(61/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 953 kB/s | 47 kB 00:00 2024-05-20T17:46:28.938 INFO:teuthology.orchestra.run.smithi204.stdout:(62/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-05-20T17:46:28.978 INFO:teuthology.orchestra.run.smithi154.stdout:(73/159): postgresql-server-13.14-1.el9.x86_64. 20 MB/s | 5.8 MB 00:00 2024-05-20T17:46:28.989 INFO:teuthology.orchestra.run.smithi204.stdout:(63/159): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-05-20T17:46:29.009 INFO:teuthology.orchestra.run.smithi129.stdout:(45/159): python3-ply-3.11-14.el9.noarch.rpm 206 kB/s | 106 kB 00:00 2024-05-20T17:46:29.039 INFO:teuthology.orchestra.run.smithi204.stdout:(64/159): openblas-0.3.21-2.el9.x86_64.rpm 705 kB/s | 35 kB 00:00 2024-05-20T17:46:29.070 INFO:teuthology.orchestra.run.smithi154.stdout:(74/159): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 6.0 MB 00:00 2024-05-20T17:46:29.095 INFO:teuthology.orchestra.run.smithi154.stdout:(75/159): python3-colorama-0.4.6-3.el9.noarch.r 524 kB/s | 61 kB 00:00 2024-05-20T17:46:29.102 INFO:teuthology.orchestra.run.smithi129.stdout:(46/159): python3-pycparser-2.20-6.el9.noarch.r 266 kB/s | 135 kB 00:00 2024-05-20T17:46:29.145 INFO:teuthology.orchestra.run.smithi154.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-05-20T17:46:29.171 INFO:teuthology.orchestra.run.smithi154.stdout:(77/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-05-20T17:46:29.188 INFO:teuthology.orchestra.run.smithi154.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 608 kB/s | 26 kB 00:00 2024-05-20T17:46:29.193 INFO:teuthology.orchestra.run.smithi129.stdout:(47/159): python3-requests-2.25.1-8.el9.noarch. 288 kB/s | 125 kB 00:00 2024-05-20T17:46:29.205 INFO:teuthology.orchestra.run.smithi154.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 574 kB/s | 19 kB 00:00 2024-05-20T17:46:29.216 INFO:teuthology.orchestra.run.smithi204.stdout:(65/159): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-05-20T17:46:29.239 INFO:teuthology.orchestra.run.smithi154.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-05-20T17:46:29.286 INFO:teuthology.orchestra.run.smithi129.stdout:(48/159): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 689 kB 00:00 2024-05-20T17:46:29.327 INFO:teuthology.orchestra.run.smithi129.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-05-20T17:46:29.331 INFO:teuthology.orchestra.run.smithi154.stdout:(81/159): ceph-mgr-diskprediction-local-19.0.0- 1.2 MB/s | 7.4 MB 00:06 2024-05-20T17:46:29.356 INFO:teuthology.orchestra.run.smithi154.stdout:(82/159): python3-markupsafe-1.1.1-12.el9.x86_6 230 kB/s | 35 kB 00:00 2024-05-20T17:46:29.361 INFO:teuthology.orchestra.run.smithi129.stdout:(50/159): bison-3.7.4-5.el9.x86_64.rpm 5.5 MB/s | 943 kB 00:00 2024-05-20T17:46:29.361 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 199.223.246.113) 2024-05-20T17:46:29.456 INFO:teuthology.orchestra.run.smithi154.stdout:(83/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-05-20T17:46:29.457 INFO:teuthology.orchestra.run.smithi204.stdout:(66/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-05-20T17:46:29.482 INFO:teuthology.orchestra.run.smithi154.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-05-20T17:46:29.487 INFO:teuthology.orchestra.run.smithi129.stdout:(51/159): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 313 kB 00:00 2024-05-20T17:46:29.507 INFO:teuthology.orchestra.run.smithi154.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-20T17:46:29.541 INFO:teuthology.orchestra.run.smithi154.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 722 kB/s | 42 kB 00:00 2024-05-20T17:46:29.548 INFO:teuthology.orchestra.run.smithi204.stdout:(67/159): ceph-mgr-diskprediction-local-19.0.0- 1.9 MB/s | 7.4 MB 00:03 2024-05-20T17:46:29.574 INFO:teuthology.orchestra.run.smithi154.stdout:(87/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 250 kB 00:00 2024-05-20T17:46:29.591 INFO:teuthology.orchestra.run.smithi204.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-20T17:46:29.599 INFO:teuthology.orchestra.run.smithi154.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-05-20T17:46:29.625 INFO:teuthology.orchestra.run.smithi154.stdout:(89/159): python3-protobuf-3.14.0-13.el9.noarch 2.2 MB/s | 269 kB 00:00 2024-05-20T17:46:29.641 INFO:teuthology.orchestra.run.smithi204.stdout:(69/159): postgresql-contrib-13.14-1.el9.x86_64 10 MB/s | 911 kB 00:00 2024-05-20T17:46:29.650 INFO:teuthology.orchestra.run.smithi154.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-20T17:46:29.653 INFO:teuthology.orchestra.run.smithi129.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 197 kB/s | 33 kB 00:00 2024-05-20T17:46:29.654 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:46:29.666 INFO:teuthology.orchestra.run.smithi204.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 1.8 MB/s | 133 kB 00:00 2024-05-20T17:46:29.675 INFO:teuthology.orchestra.run.smithi154.stdout:(91/159): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 279 kB 00:00 2024-05-20T17:46:29.696 INFO:teuthology.orchestra.run.smithi129.stdout:(53/159): python3-urllib3-1.26.5-5.el9.noarch.r 313 kB/s | 215 kB 00:00 2024-05-20T17:46:29.700 INFO:teuthology.orchestra.run.smithi154.stdout:(92/159): python3-requests-oauthlib-1.3.0-12.el 711 kB/s | 54 kB 00:00 2024-05-20T17:46:29.726 INFO:teuthology.orchestra.run.smithi154.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 827 kB/s | 42 kB 00:00 2024-05-20T17:46:29.753 INFO:teuthology.orchestra.run.smithi154.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 865 kB/s | 44 kB 00:00 2024-05-20T17:46:29.754 INFO:teuthology.orchestra.run.smithi129.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-05-20T17:46:29.813 INFO:teuthology.orchestra.run.smithi129.stdout:(55/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-05-20T17:46:29.838 INFO:teuthology.orchestra.run.smithi129.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 133 kB/s | 11 kB 00:00 2024-05-20T17:46:29.850 INFO:teuthology.orchestra.run.smithi204.stdout:(71/159): postgresql-server-13.14-1.el9.x86_64. 28 MB/s | 5.8 MB 00:00 2024-05-20T17:46:29.863 INFO:teuthology.orchestra.run.smithi129.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 160 kB 00:00 2024-05-20T17:46:29.864 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:46:29.868 INFO:teuthology.orchestra.run.smithi154.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-05-20T17:46:29.883 INFO:teuthology.orchestra.run.smithi204.stdout:(72/159): protobuf-3.14.0-13.el9.x86_64.rpm 4.7 MB/s | 1.0 MB 00:00 2024-05-20T17:46:29.889 INFO:teuthology.orchestra.run.smithi129.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 899 kB/s | 45 kB 00:00 2024-05-20T17:46:29.893 INFO:teuthology.orchestra.run.smithi154.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 259 kB/s | 36 kB 00:00 2024-05-20T17:46:29.917 INFO:teuthology.orchestra.run.smithi204.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-05-20T17:46:29.919 INFO:teuthology.orchestra.run.smithi154.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-05-20T17:46:29.922 INFO:teuthology.orchestra.run.smithi129.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-05-20T17:46:29.957 INFO:teuthology.orchestra.run.smithi129.stdout:(60/159): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-05-20T17:46:29.969 INFO:teuthology.orchestra.run.smithi154.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 305 kB 00:00 2024-05-20T17:46:29.981 INFO:teuthology.orchestra.run.smithi129.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 436 kB/s | 25 kB 00:00 2024-05-20T17:46:29.994 INFO:teuthology.orchestra.run.smithi154.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 768 kB/s | 58 kB 00:00 2024-05-20T17:46:30.006 INFO:teuthology.orchestra.run.smithi129.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-05-20T17:46:30.020 INFO:teuthology.orchestra.run.smithi154.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-05-20T17:46:30.032 INFO:teuthology.orchestra.run.smithi129.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-05-20T17:46:30.045 INFO:teuthology.orchestra.run.smithi154.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-20T17:46:30.057 INFO:teuthology.orchestra.run.smithi129.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-05-20T17:46:30.070 INFO:teuthology.orchestra.run.smithi154.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 893 kB/s | 45 kB 00:00 2024-05-20T17:46:30.082 INFO:teuthology.orchestra.run.smithi129.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-05-20T17:46:30.154 INFO:teuthology.orchestra.run.smithi154.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-05-20T17:46:30.218 INFO:teuthology.orchestra.run.smithi204.stdout:(74/159): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-05-20T17:46:30.218 INFO:teuthology.orchestra.run.smithi204.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 206.82.17.213) 2024-05-20T17:46:30.240 INFO:teuthology.orchestra.run.smithi204.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 206.82.17.213) 2024-05-20T17:46:30.249 INFO:teuthology.orchestra.run.smithi129.stdout:(66/159): openblas-openmp-0.3.21-2.el9.x86_64.r 25 MB/s | 4.7 MB 00:00 2024-05-20T17:46:30.268 INFO:teuthology.orchestra.run.smithi204.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-05-20T17:46:30.292 INFO:teuthology.orchestra.run.smithi204.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 206.82.17.213) 2024-05-20T17:46:30.316 INFO:teuthology.orchestra.run.smithi129.stdout:(67/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-05-20T17:46:30.326 INFO:teuthology.orchestra.run.smithi204.stdout:(76/159): python3-colorama-0.4.6-3.el9.noarch.r 150 kB/s | 61 kB 00:00 2024-05-20T17:46:30.358 INFO:teuthology.orchestra.run.smithi129.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-05-20T17:46:30.368 INFO:teuthology.orchestra.run.smithi204.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 616 kB/s | 26 kB 00:00 2024-05-20T17:46:30.388 INFO:teuthology.orchestra.run.smithi154.stdout:(104/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-05-20T17:46:30.391 INFO:teuthology.orchestra.run.smithi129.stdout:(69/159): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-05-20T17:46:30.410 INFO:teuthology.orchestra.run.smithi204.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 460 kB/s | 19 kB 00:00 2024-05-20T17:46:30.416 INFO:teuthology.orchestra.run.smithi129.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-05-20T17:46:30.430 INFO:teuthology.orchestra.run.smithi154.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-05-20T17:46:30.452 INFO:teuthology.orchestra.run.smithi204.stdout:(79/159): python3-jinja2-2.11.3-6.el9.noarch.rp 1.3 MB/s | 248 kB 00:00 2024-05-20T17:46:30.467 INFO:teuthology.orchestra.run.smithi129.stdout:(71/159): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-05-20T17:46:30.494 INFO:teuthology.orchestra.run.smithi204.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-05-20T17:46:30.497 INFO:teuthology.orchestra.run.smithi154.stdout:(106/159): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-05-20T17:46:30.520 INFO:teuthology.orchestra.run.smithi204.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 519 kB/s | 35 kB 00:00 2024-05-20T17:46:30.522 INFO:teuthology.orchestra.run.smithi154.stdout:(107/159): grpc-data-1.46.7-10.el9.noarch.rpm 211 kB/s | 19 kB 00:00 2024-05-20T17:46:30.534 INFO:teuthology.orchestra.run.smithi129.stdout:(72/159): postgresql-server-13.14-1.el9.x86_64. 41 MB/s | 5.8 MB 00:00 2024-05-20T17:46:30.559 INFO:teuthology.orchestra.run.smithi129.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 957 kB/s | 87 kB 00:00 2024-05-20T17:46:30.562 INFO:teuthology.orchestra.run.smithi204.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.9 MB/s | 250 kB 00:00 2024-05-20T17:46:30.572 INFO:teuthology.orchestra.run.smithi154.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 522 kB/s | 26 kB 00:00 2024-05-20T17:46:30.580 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 199.223.246.113) 2024-05-20T17:46:30.598 INFO:teuthology.orchestra.run.smithi154.stdout:(109/159): gperftools-libs-2.9.1-2.el9.x86_64.r 1.4 MB/s | 309 kB 00:00 2024-05-20T17:46:30.612 INFO:teuthology.orchestra.run.smithi204.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 4.4 MB/s | 222 kB 00:00 2024-05-20T17:46:30.640 INFO:teuthology.orchestra.run.smithi154.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 727 kB/s | 49 kB 00:00 2024-05-20T17:46:30.646 INFO:teuthology.orchestra.run.smithi204.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 2.2 MB/s | 77 kB 00:00 2024-05-20T17:46:30.665 INFO:teuthology.orchestra.run.smithi154.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-05-20T17:46:30.668 INFO:teuthology.orchestra.run.smithi129.stdout:(74/159): python3-babel-2.9.1-2.el9.noarch.rpm 45 MB/s | 6.0 MB 00:00 2024-05-20T17:46:30.668 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 216.158.133.254) 2024-05-20T17:46:30.712 INFO:teuthology.orchestra.run.smithi204.stdout:(85/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-05-20T17:46:30.718 INFO:teuthology.orchestra.run.smithi129.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-05-20T17:46:30.719 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 216.158.133.254) 2024-05-20T17:46:30.724 INFO:teuthology.orchestra.run.smithi154.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-05-20T17:46:30.734 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 199.223.246.113) 2024-05-20T17:46:30.738 INFO:teuthology.orchestra.run.smithi204.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 459 kB/s | 42 kB 00:00 2024-05-20T17:46:30.761 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 216.158.133.254) 2024-05-20T17:46:30.771 INFO:teuthology.orchestra.run.smithi204.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-05-20T17:46:30.775 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 206.82.17.213) 2024-05-20T17:46:30.784 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 216.158.133.254) 2024-05-20T17:46:30.797 INFO:teuthology.orchestra.run.smithi204.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-05-20T17:46:30.822 INFO:teuthology.orchestra.run.smithi204.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-05-20T17:46:30.824 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 206.82.17.213) 2024-05-20T17:46:30.848 INFO:teuthology.orchestra.run.smithi204.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-20T17:46:30.849 INFO:teuthology.orchestra.run.smithi154.stdout:(113/159): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-05-20T17:46:30.873 INFO:teuthology.orchestra.run.smithi204.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-20T17:46:30.891 INFO:teuthology.orchestra.run.smithi154.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.6 MB/s | 839 kB 00:00 2024-05-20T17:46:30.908 INFO:teuthology.orchestra.run.smithi154.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 744 kB/s | 43 kB 00:00 2024-05-20T17:46:30.914 INFO:teuthology.orchestra.run.smithi204.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-05-20T17:46:30.916 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 206.82.17.213) 2024-05-20T17:46:30.940 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 206.82.17.213) 2024-05-20T17:46:30.941 INFO:teuthology.orchestra.run.smithi154.stdout:(116/159): python3-asyncssh-2.13.2-5.el9.noarch 2.5 MB/s | 548 kB 00:00 2024-05-20T17:46:30.958 INFO:teuthology.orchestra.run.smithi154.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-05-20T17:46:30.959 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:30.965 INFO:teuthology.orchestra.run.smithi204.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 883 kB/s | 44 kB 00:00 2024-05-20T17:46:30.975 INFO:teuthology.orchestra.run.smithi154.stdout:(118/159): python3-cachetools-4.2.4-1.el9.noarc 383 kB/s | 32 kB 00:00 2024-05-20T17:46:30.982 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:31.001 INFO:teuthology.orchestra.run.smithi154.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-05-20T17:46:31.015 INFO:teuthology.orchestra.run.smithi204.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-20T17:46:31.052 INFO:teuthology.orchestra.run.smithi154.stdout:(120/159): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-05-20T17:46:31.074 INFO:teuthology.orchestra.run.smithi204.stdout:(95/159): qatlib-service-24.02.0-1.el9.x86_64.r 615 kB/s | 36 kB 00:00 2024-05-20T17:46:31.076 INFO:teuthology.orchestra.run.smithi154.stdout:(121/159): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-05-20T17:46:31.102 INFO:teuthology.orchestra.run.smithi154.stdout:(122/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-05-20T17:46:31.125 INFO:teuthology.orchestra.run.smithi204.stdout:(96/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-05-20T17:46:31.143 INFO:teuthology.orchestra.run.smithi154.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-05-20T17:46:31.160 INFO:teuthology.orchestra.run.smithi154.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-05-20T17:46:31.186 INFO:teuthology.orchestra.run.smithi154.stdout:(125/159): python3-jaraco-classes-3.2.1-5.el9.n 422 kB/s | 18 kB 00:00 2024-05-20T17:46:31.236 INFO:teuthology.orchestra.run.smithi154.stdout:(126/159): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-05-20T17:46:31.253 INFO:teuthology.orchestra.run.smithi154.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-05-20T17:46:31.270 INFO:teuthology.orchestra.run.smithi154.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-05-20T17:46:31.287 INFO:teuthology.orchestra.run.smithi154.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 585 kB/s | 20 kB 00:00 2024-05-20T17:46:31.291 INFO:teuthology.orchestra.run.smithi204.stdout:(97/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-05-20T17:46:31.312 INFO:teuthology.orchestra.run.smithi154.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-05-20T17:46:31.317 INFO:teuthology.orchestra.run.smithi204.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 305 kB 00:00 2024-05-20T17:46:31.337 INFO:teuthology.orchestra.run.smithi154.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-05-20T17:46:31.342 INFO:teuthology.orchestra.run.smithi204.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-20T17:46:31.354 INFO:teuthology.orchestra.run.smithi154.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-05-20T17:46:31.362 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 72.4.120.219) 2024-05-20T17:46:31.363 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 72.4.120.219) 2024-05-20T17:46:31.363 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:31.388 INFO:teuthology.orchestra.run.smithi154.stdout:(133/159): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-05-20T17:46:31.410 INFO:teuthology.orchestra.run.smithi204.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.0 MB/s | 94 kB 00:00 2024-05-20T17:46:31.413 INFO:teuthology.orchestra.run.smithi154.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-05-20T17:46:31.434 INFO:teuthology.orchestra.run.smithi204.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 692 kB/s | 64 kB 00:00 2024-05-20T17:46:31.439 INFO:teuthology.orchestra.run.smithi154.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-05-20T17:46:31.455 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:31.458 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 72.4.120.219) 2024-05-20T17:46:31.460 INFO:teuthology.orchestra.run.smithi204.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-05-20T17:46:31.464 INFO:teuthology.orchestra.run.smithi154.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 558 kB/s | 28 kB 00:00 2024-05-20T17:46:31.468 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 72.4.120.219) 2024-05-20T17:46:31.489 INFO:teuthology.orchestra.run.smithi154.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-05-20T17:46:31.523 INFO:teuthology.orchestra.run.smithi154.stdout:(138/159): python3-kubernetes-26.1.0-2.el9.noar 3.8 MB/s | 1.0 MB 00:00 2024-05-20T17:46:31.548 INFO:teuthology.orchestra.run.smithi154.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-05-20T17:46:31.574 INFO:teuthology.orchestra.run.smithi154.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 704 kB/s | 59 kB 00:00 2024-05-20T17:46:31.599 INFO:teuthology.orchestra.run.smithi154.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-05-20T17:46:31.619 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:46:31.623 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 140.211.166.134) 2024-05-20T17:46:31.624 INFO:teuthology.orchestra.run.smithi154.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-05-20T17:46:31.626 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 140.211.166.134) 2024-05-20T17:46:31.650 INFO:teuthology.orchestra.run.smithi154.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-05-20T17:46:31.675 INFO:teuthology.orchestra.run.smithi154.stdout:(144/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-05-20T17:46:31.700 INFO:teuthology.orchestra.run.smithi154.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-05-20T17:46:31.718 INFO:teuthology.orchestra.run.smithi204.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 79 kB/s | 22 kB 00:00 2024-05-20T17:46:31.725 INFO:teuthology.orchestra.run.smithi154.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-05-20T17:46:31.751 INFO:teuthology.orchestra.run.smithi154.stdout:(147/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-05-20T17:46:31.877 INFO:teuthology.orchestra.run.smithi204.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 3.4 MB/s | 548 kB 00:00 2024-05-20T17:46:31.919 INFO:teuthology.orchestra.run.smithi204.stdout:(105/159): gperftools-libs-2.9.1-2.el9.x86_64.r 7.3 MB/s | 309 kB 00:00 2024-05-20T17:46:31.959 INFO:teuthology.orchestra.run.smithi154.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-05-20T17:46:31.961 INFO:teuthology.orchestra.run.smithi204.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-05-20T17:46:31.970 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:46:31.994 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 140.211.166.134) 2024-05-20T17:46:32.016 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 140.211.166.134) 2024-05-20T17:46:32.131 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 163.114.132.32) 2024-05-20T17:46:32.156 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 163.114.132.32) 2024-05-20T17:46:32.177 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 163.114.132.32) 2024-05-20T17:46:32.178 INFO:teuthology.orchestra.run.smithi204.stdout:(107/159): libarrow-9.0.0-11.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-05-20T17:46:32.220 INFO:teuthology.orchestra.run.smithi204.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-05-20T17:46:32.262 INFO:teuthology.orchestra.run.smithi204.stdout:(109/159): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-05-20T17:46:32.287 INFO:teuthology.orchestra.run.smithi204.stdout:(110/159): protobuf-compiler-3.14.0-13.el9.x86_ 1.0 MB/s | 863 kB 00:00 2024-05-20T17:46:32.304 INFO:teuthology.orchestra.run.smithi204.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-05-20T17:46:32.351 INFO:teuthology.orchestra.run.smithi154.stdout:(149/159): NetworkManager-1.47.90-1.el9.x86_64. 5.9 MB/s | 2.3 MB 00:00 2024-05-20T17:46:32.379 INFO:teuthology.orchestra.run.smithi204.stdout:(112/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-05-20T17:46:32.430 INFO:teuthology.orchestra.run.smithi204.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-05-20T17:46:32.455 INFO:teuthology.orchestra.run.smithi204.stdout:(114/159): luarocks-3.9.2-5.el9.noarch.rpm 904 kB/s | 151 kB 00:00 2024-05-20T17:46:32.480 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-devel-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.191-4.el9.x86_64.rpm (IP: 163.114.132.32) 2024-05-20T17:46:32.489 INFO:teuthology.orchestra.run.smithi204.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 743 kB/s | 43 kB 00:00 2024-05-20T17:46:32.506 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-colorama-0.4.6-3.el9.noarch.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-colorama-0.4.6-3.el9.noarch.rpm (IP: 163.114.132.32) 2024-05-20T17:46:32.514 INFO:teuthology.orchestra.run.smithi204.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-05-20T17:46:32.530 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 163.114.132.32) 2024-05-20T17:46:32.556 INFO:teuthology.orchestra.run.smithi204.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 223 kB/s | 15 kB 00:00 2024-05-20T17:46:32.560 INFO:teuthology.orchestra.run.smithi154.stdout:(150/159): NetworkManager-libnm-1.47.90-1.el9.x 8.7 MB/s | 1.8 MB 00:00 2024-05-20T17:46:32.568 INFO:teuthology.orchestra.run.smithi129.stdout:(76/159): elfutils-libelf-devel-0.191-4.el9.x86 6.7 kB/s | 22 kB 00:03 2024-05-20T17:46:32.581 INFO:teuthology.orchestra.run.smithi204.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-05-20T17:46:32.602 INFO:teuthology.orchestra.run.smithi154.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 1.0 MB/s | 41 kB 00:00 2024-05-20T17:46:32.610 INFO:teuthology.orchestra.run.smithi129.stdout:(77/159): python3-colorama-0.4.6-3.el9.noarch.r 30 kB/s | 61 kB 00:02 2024-05-20T17:46:32.623 INFO:teuthology.orchestra.run.smithi204.stdout:(119/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.1 MB/s | 303 kB 00:00 2024-05-20T17:46:32.644 INFO:teuthology.orchestra.run.smithi129.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 340 kB/s | 26 kB 00:00 2024-05-20T17:46:32.648 INFO:teuthology.orchestra.run.smithi204.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-05-20T17:46:32.652 INFO:teuthology.orchestra.run.smithi154.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 4.8 MB/s | 248 kB 00:00 2024-05-20T17:46:32.674 INFO:teuthology.orchestra.run.smithi204.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-05-20T17:46:32.675 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:32.711 INFO:teuthology.orchestra.run.smithi129.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 191 kB/s | 19 kB 00:00 2024-05-20T17:46:32.716 INFO:teuthology.orchestra.run.smithi204.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-05-20T17:46:32.736 INFO:teuthology.orchestra.run.smithi154.stdout:(153/159): librados2-19.0.0-2744.gd2b507c7.el9. 3.3 MB/s | 3.3 MB 00:01 2024-05-20T17:46:32.737 INFO:teuthology.orchestra.run.smithi129.stdout:(80/159): python3-jinja2-2.11.3-6.el9.noarch.rp 123 kB/s | 248 kB 00:02 2024-05-20T17:46:32.758 INFO:teuthology.orchestra.run.smithi204.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-05-20T17:46:32.761 INFO:teuthology.orchestra.run.smithi129.stdout:(81/159): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-05-20T17:46:32.787 INFO:teuthology.orchestra.run.smithi129.stdout:(82/159): python3-markupsafe-1.1.1-12.el9.x86_6 458 kB/s | 35 kB 00:00 2024-05-20T17:46:32.800 INFO:teuthology.orchestra.run.smithi204.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 424 kB/s | 18 kB 00:00 2024-05-20T17:46:32.821 INFO:teuthology.orchestra.run.smithi129.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-05-20T17:46:32.834 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:32.842 INFO:teuthology.orchestra.run.smithi204.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 557 kB/s | 23 kB 00:00 2024-05-20T17:46:32.853 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 199.223.246.113) 2024-05-20T17:46:32.860 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:32.862 INFO:teuthology.orchestra.run.smithi129.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 1.8 MB/s | 77 kB 00:00 2024-05-20T17:46:32.882 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:32.884 INFO:teuthology.orchestra.run.smithi204.stdout:(126/159): python3-jaraco-functools-3.5.0-2.el9 468 kB/s | 19 kB 00:00 2024-05-20T17:46:32.888 INFO:teuthology.orchestra.run.smithi129.stdout:(85/159): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-05-20T17:46:32.892 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:46:32.899 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 199.223.246.113) 2024-05-20T17:46:32.913 INFO:teuthology.orchestra.run.smithi129.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 839 kB/s | 42 kB 00:00 2024-05-20T17:46:32.926 INFO:teuthology.orchestra.run.smithi204.stdout:(127/159): python3-jaraco-text-3.2.0-6.el9.noar 472 kB/s | 20 kB 00:00 2024-05-20T17:46:32.936 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:46:32.947 INFO:teuthology.orchestra.run.smithi129.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-05-20T17:46:32.972 INFO:teuthology.orchestra.run.smithi129.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-05-20T17:46:32.997 INFO:teuthology.orchestra.run.smithi129.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-05-20T17:46:32.999 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:46:33.009 INFO:teuthology.orchestra.run.smithi204.stdout:(128/159): python3-grpcio-1.46.7-10.el9.x86_64. 5.6 MB/s | 2.0 MB 00:00 2024-05-20T17:46:33.023 INFO:teuthology.orchestra.run.smithi129.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-20T17:46:33.043 INFO:teuthology.orchestra.run.smithi204.stdout:(129/159): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-05-20T17:46:33.048 INFO:teuthology.orchestra.run.smithi129.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-20T17:46:33.053 INFO:teuthology.orchestra.run.smithi154.stdout:(154/159): librbd1-19.0.0-2744.gd2b507c7.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-05-20T17:46:33.053 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:46:33.063 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:33.068 INFO:teuthology.orchestra.run.smithi204.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 791 kB/s | 46 kB 00:00 2024-05-20T17:46:33.070 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:33.094 INFO:teuthology.orchestra.run.smithi204.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-05-20T17:46:33.119 INFO:teuthology.orchestra.run.smithi204.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-05-20T17:46:33.123 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:33.132 INFO:teuthology.orchestra.run.smithi129.stdout:(92/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-05-20T17:46:33.133 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:33.140 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 199.223.246.113) 2024-05-20T17:46:33.142 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:33.144 INFO:teuthology.orchestra.run.smithi204.stdout:(133/159): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-05-20T17:46:33.157 INFO:teuthology.orchestra.run.smithi129.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 384 kB/s | 42 kB 00:00 2024-05-20T17:46:33.162 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:33.170 INFO:teuthology.orchestra.run.smithi204.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-05-20T17:46:33.181 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:46:33.182 INFO:teuthology.orchestra.run.smithi129.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 884 kB/s | 44 kB 00:00 2024-05-20T17:46:33.195 INFO:teuthology.orchestra.run.smithi204.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-05-20T17:46:33.200 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 216.158.133.254) 2024-05-20T17:46:33.207 INFO:teuthology.orchestra.run.smithi129.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-20T17:46:33.209 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 104.167.241.199) 2024-05-20T17:46:33.218 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 104.167.241.199) 2024-05-20T17:46:33.218 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:33.220 INFO:teuthology.orchestra.run.smithi204.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-05-20T17:46:33.232 INFO:teuthology.orchestra.run.smithi129.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 714 kB/s | 36 kB 00:00 2024-05-20T17:46:33.246 INFO:teuthology.orchestra.run.smithi204.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-05-20T17:46:33.258 INFO:teuthology.orchestra.run.smithi129.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-05-20T17:46:33.271 INFO:teuthology.orchestra.run.smithi204.stdout:(138/159): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-05-20T17:46:33.283 INFO:teuthology.orchestra.run.smithi129.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-05-20T17:46:33.296 INFO:teuthology.orchestra.run.smithi204.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-05-20T17:46:33.308 INFO:teuthology.orchestra.run.smithi129.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-20T17:46:33.309 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:33.322 INFO:teuthology.orchestra.run.smithi204.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-05-20T17:46:33.334 INFO:teuthology.orchestra.run.smithi129.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-05-20T17:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 104.167.241.199) 2024-05-20T17:46:33.347 INFO:teuthology.orchestra.run.smithi204.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-05-20T17:46:33.351 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 104.167.241.199) 2024-05-20T17:46:33.359 INFO:teuthology.orchestra.run.smithi129.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-20T17:46:33.376 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 104.167.241.199) 2024-05-20T17:46:33.384 INFO:teuthology.orchestra.run.smithi129.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 892 kB/s | 45 kB 00:00 2024-05-20T17:46:33.385 INFO:teuthology.orchestra.run.smithi154.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 104.167.241.199) 2024-05-20T17:46:33.389 INFO:teuthology.orchestra.run.smithi204.stdout:(142/159): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-05-20T17:46:33.414 INFO:teuthology.orchestra.run.smithi204.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-05-20T17:46:33.436 INFO:teuthology.orchestra.run.smithi154.stdout:(155/159): elfutils-debuginfod-client-0.191-4.e 47 kB/s | 37 kB 00:00 2024-05-20T17:46:33.456 INFO:teuthology.orchestra.run.smithi204.stdout:(144/159): python3-zc-lockfile-2.0-10.el9.noarc 482 kB/s | 20 kB 00:00 2024-05-20T17:46:33.481 INFO:teuthology.orchestra.run.smithi204.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-05-20T17:46:33.507 INFO:teuthology.orchestra.run.smithi204.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-05-20T17:46:33.520 INFO:teuthology.orchestra.run.smithi154.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 264 kB/s | 207 kB 00:00 2024-05-20T17:46:33.545 INFO:teuthology.orchestra.run.smithi154.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 528 kB/s | 260 kB 00:00 2024-05-20T17:46:33.626 INFO:teuthology.orchestra.run.smithi129.stdout:(103/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-05-20T17:46:33.629 INFO:teuthology.orchestra.run.smithi154.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 7.0 MB/s | 1.3 MB 00:00 2024-05-20T17:46:33.640 INFO:teuthology.orchestra.run.smithi204.stdout:(147/159): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-05-20T17:46:33.660 INFO:teuthology.orchestra.run.smithi129.stdout:(104/159): protobuf-compiler-3.14.0-13.el9.x86_ 3.1 MB/s | 863 kB 00:00 2024-05-20T17:46:33.718 INFO:teuthology.orchestra.run.smithi129.stdout:(105/159): lua-devel-5.4.4-4.el9.x86_64.rpm 62 kB/s | 22 kB 00:00 2024-05-20T17:46:33.779 INFO:teuthology.orchestra.run.smithi154.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 9.3 MB/s | 2.4 MB 00:00 2024-05-20T17:46:33.783 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-05-20T17:46:33.783 INFO:teuthology.orchestra.run.smithi154.stdout:Total 8.1 MB/s | 222 MB 00:27 2024-05-20T17:46:33.877 INFO:teuthology.orchestra.run.smithi129.stdout:(106/159): abseil-cpp-20211102.0-3.el9.x86_64.r 2.1 MB/s | 548 kB 00:00 2024-05-20T17:46:33.902 INFO:teuthology.orchestra.run.smithi129.stdout:(107/159): gperftools-libs-2.9.1-2.el9.x86_64.r 1.4 MB/s | 309 kB 00:00 2024-05-20T17:46:33.927 INFO:teuthology.orchestra.run.smithi129.stdout:(108/159): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-05-20T17:46:34.061 INFO:teuthology.orchestra.run.smithi129.stdout:(109/159): libarrow-9.0.0-11.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-05-20T17:46:34.086 INFO:teuthology.orchestra.run.smithi129.stdout:(110/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 165 kB/s | 26 kB 00:00 2024-05-20T17:46:34.112 INFO:teuthology.orchestra.run.smithi129.stdout:(111/159): liboath-2.6.7-2.el9.x86_64.rpm 266 kB/s | 49 kB 00:00 2024-05-20T17:46:34.137 INFO:teuthology.orchestra.run.smithi129.stdout:(112/159): libunwind-1.6.2-1.el9.x86_64.rpm 895 kB/s | 67 kB 00:00 2024-05-20T17:46:34.162 INFO:teuthology.orchestra.run.smithi129.stdout:(113/159): luarocks-3.9.2-5.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-05-20T17:46:34.196 INFO:teuthology.orchestra.run.smithi129.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-05-20T17:46:34.221 INFO:teuthology.orchestra.run.smithi129.stdout:(115/159): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-05-20T17:46:34.238 INFO:teuthology.orchestra.run.smithi129.stdout:(116/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 587 kB/s | 43 kB 00:00 2024-05-20T17:46:34.255 INFO:teuthology.orchestra.run.smithi129.stdout:(117/159): python3-cachetools-4.2.4-1.el9.noarc 552 kB/s | 32 kB 00:00 2024-05-20T17:46:34.280 INFO:teuthology.orchestra.run.smithi129.stdout:(118/159): python3-certifi-2021.10.8-2.el9.noar 360 kB/s | 15 kB 00:00 2024-05-20T17:46:34.305 INFO:teuthology.orchestra.run.smithi129.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-05-20T17:46:34.331 INFO:teuthology.orchestra.run.smithi129.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-05-20T17:46:34.356 INFO:teuthology.orchestra.run.smithi129.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-05-20T17:46:34.373 INFO:teuthology.orchestra.run.smithi129.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 4.9 MB/s | 206 kB 00:00 2024-05-20T17:46:34.415 INFO:teuthology.orchestra.run.smithi129.stdout:(123/159): python3-grpcio-1.46.7-10.el9.x86_64. 25 MB/s | 2.0 MB 00:00 2024-05-20T17:46:34.440 INFO:teuthology.orchestra.run.smithi129.stdout:(124/159): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-05-20T17:46:34.465 INFO:teuthology.orchestra.run.smithi129.stdout:(125/159): python3-jaraco-8.2.1-3.el9.noarch.rp 119 kB/s | 11 kB 00:00 2024-05-20T17:46:34.490 INFO:teuthology.orchestra.run.smithi129.stdout:(126/159): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-05-20T17:46:34.515 INFO:teuthology.orchestra.run.smithi129.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-05-20T17:46:34.541 INFO:teuthology.orchestra.run.smithi129.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-05-20T17:46:34.566 INFO:teuthology.orchestra.run.smithi129.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-05-20T17:46:34.608 INFO:teuthology.orchestra.run.smithi129.stdout:(130/159): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-05-20T17:46:34.633 INFO:teuthology.orchestra.run.smithi129.stdout:(131/159): python3-logutils-0.3.5-21.el9.noarch 503 kB/s | 46 kB 00:00 2024-05-20T17:46:34.658 INFO:teuthology.orchestra.run.smithi129.stdout:(132/159): python3-more-itertools-8.12.0-2.el9. 860 kB/s | 79 kB 00:00 2024-05-20T17:46:34.684 INFO:teuthology.orchestra.run.smithi129.stdout:(133/159): python3-natsort-7.1.1-5.el9.noarch.r 769 kB/s | 58 kB 00:00 2024-05-20T17:46:34.709 INFO:teuthology.orchestra.run.smithi129.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-05-20T17:46:34.734 INFO:teuthology.orchestra.run.smithi129.stdout:(135/159): python3-portend-3.1.0-2.el9.noarch.r 219 kB/s | 16 kB 00:00 2024-05-20T17:46:34.759 INFO:teuthology.orchestra.run.smithi129.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-05-20T17:46:34.785 INFO:teuthology.orchestra.run.smithi129.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-05-20T17:46:34.810 INFO:teuthology.orchestra.run.smithi129.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-05-20T17:46:34.835 INFO:teuthology.orchestra.run.smithi129.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-05-20T17:46:34.861 INFO:teuthology.orchestra.run.smithi129.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-05-20T17:46:34.885 INFO:teuthology.orchestra.run.smithi129.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 476 kB/s | 36 kB 00:00 2024-05-20T17:46:34.911 INFO:teuthology.orchestra.run.smithi129.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-05-20T17:46:34.936 INFO:teuthology.orchestra.run.smithi129.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-05-20T17:46:34.953 INFO:teuthology.orchestra.run.smithi129.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-05-20T17:46:34.978 INFO:teuthology.orchestra.run.smithi129.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 6.3 MB/s | 427 kB 00:00 2024-05-20T17:46:34.990 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-05-20T17:46:35.003 INFO:teuthology.orchestra.run.smithi129.stdout:(146/159): python3-zc-lockfile-2.0-10.el9.noarc 404 kB/s | 20 kB 00:00 2024-05-20T17:46:35.029 INFO:teuthology.orchestra.run.smithi129.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-05-20T17:46:35.108 INFO:teuthology.orchestra.run.smithi204.stdout:(148/159): librbd1-19.0.0-2744.gd2b507c7.el9.x8 2.1 MB/s | 3.1 MB 00:01 2024-05-20T17:46:35.149 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-05-20T17:46:35.150 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-05-20T17:46:35.204 INFO:teuthology.orchestra.run.smithi129.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-05-20T17:46:35.216 INFO:teuthology.orchestra.run.smithi204.stdout:(149/159): librados2-19.0.0-2744.gd2b507c7.el9. 1.9 MB/s | 3.3 MB 00:01 2024-05-20T17:46:35.383 INFO:teuthology.orchestra.run.smithi204.stdout:(150/159): NetworkManager-1.47.90-1.el9.x86_64. 8.4 MB/s | 2.3 MB 00:00 2024-05-20T17:46:35.433 INFO:teuthology.orchestra.run.smithi204.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 835 kB/s | 41 kB 00:00 2024-05-20T17:46:35.484 INFO:teuthology.orchestra.run.smithi204.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 4.9 MB/s | 248 kB 00:00 2024-05-20T17:46:35.526 INFO:teuthology.orchestra.run.smithi204.stdout:(153/159): NetworkManager-libnm-1.47.90-1.el9.x 5.8 MB/s | 1.8 MB 00:00 2024-05-20T17:46:35.526 INFO:teuthology.orchestra.run.smithi204.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:35.968 INFO:teuthology.orchestra.run.smithi204.stdout:(154/159): ceph-test-19.0.0-2744.gd2b507c7.el9. 2.5 MB/s | 46 MB 00:18 2024-05-20T17:46:35.996 INFO:teuthology.orchestra.run.smithi204.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:36.093 INFO:teuthology.orchestra.run.smithi204.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:36.094 INFO:teuthology.orchestra.run.smithi204.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:36.116 INFO:teuthology.orchestra.run.smithi204.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 206.82.17.213) 2024-05-20T17:46:36.151 INFO:teuthology.orchestra.run.smithi204.stdout:(155/159): elfutils-debuginfod-client-0.191-4.e 55 kB/s | 37 kB 00:00 2024-05-20T17:46:36.235 INFO:teuthology.orchestra.run.smithi204.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 1.1 MB/s | 260 kB 00:00 2024-05-20T17:46:36.277 INFO:teuthology.orchestra.run.smithi204.stdout:(157/159): elfutils-libelf-0.191-4.el9.x86_64.r 275 kB/s | 207 kB 00:00 2024-05-20T17:46:36.335 INFO:teuthology.orchestra.run.smithi204.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 7.4 MB/s | 1.3 MB 00:00 2024-05-20T17:46:36.388 INFO:teuthology.orchestra.run.smithi129.stdout:(149/159): NetworkManager-1.47.90-1.el9.x86_64. 2.0 MB/s | 2.3 MB 00:01 2024-05-20T17:46:36.394 INFO:teuthology.orchestra.run.smithi204.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-05-20T17:46:36.399 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-05-20T17:46:36.399 INFO:teuthology.orchestra.run.smithi204.stdout:Total 7.0 MB/s | 222 MB 00:31 2024-05-20T17:46:36.871 INFO:teuthology.orchestra.run.smithi129.stdout:(150/159): librbd1-19.0.0-2744.gd2b507c7.el9.x8 1.7 MB/s | 3.1 MB 00:01 2024-05-20T17:46:36.923 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-05-20T17:46:36.924 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-05-20T17:46:36.955 INFO:teuthology.orchestra.run.smithi129.stdout:(151/159): librados2-19.0.0-2744.gd2b507c7.el9. 1.7 MB/s | 3.3 MB 00:01 2024-05-20T17:46:37.138 INFO:teuthology.orchestra.run.smithi129.stdout:(152/159): NetworkManager-team-1.47.90-1.el9.x8 156 kB/s | 41 kB 00:00 2024-05-20T17:46:37.296 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 163.114.132.32) 2024-05-20T17:46:37.565 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-05-20T17:46:37.651 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 163.114.132.32) 2024-05-20T17:46:37.689 INFO:teuthology.orchestra.run.smithi129.stdout:(153/159): NetworkManager-tui-1.47.90-1.el9.x86 338 kB/s | 248 kB 00:00 2024-05-20T17:46:37.689 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:37.724 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-05-20T17:46:37.724 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-05-20T17:46:37.747 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:37.850 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 163.114.132.32) 2024-05-20T17:46:37.907 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:46:38.072 INFO:teuthology.orchestra.run.smithi129.stdout:(154/159): NetworkManager-libnm-1.47.90-1.el9.x 1.1 MB/s | 1.8 MB 00:01 2024-05-20T17:46:38.203 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 163.114.132.32) 2024-05-20T17:46:38.222 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:38.230 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 163.114.132.32) 2024-05-20T17:46:38.258 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-debuginfod-client-0.191-4.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:46:38.277 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:38.322 INFO:teuthology.orchestra.run.smithi129.stdout:(155/159): elfutils-debuginfod-client-0.191-4.e 31 kB/s | 37 kB 00:01 2024-05-20T17:46:38.357 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:46:38.455 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libelf-0.191-4.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libelf-0.191-4.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:46:38.539 INFO:teuthology.orchestra.run.smithi129.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 243 kB/s | 207 kB 00:00 2024-05-20T17:46:38.598 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 163.114.132.32) 2024-05-20T17:46:38.616 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:38.691 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 72.4.120.219) 2024-05-20T17:46:38.771 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:46:38.851 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] elfutils-libs-0.191-4.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/elfutils-libs-0.191-4.el9.x86_64.rpm (IP: 140.211.166.134) 2024-05-20T17:46:38.915 INFO:teuthology.orchestra.run.smithi129.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 308 kB/s | 260 kB 00:00 2024-05-20T17:46:39.495 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-05-20T17:46:39.496 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-05-20T17:46:40.865 INFO:teuthology.orchestra.run.smithi129.stdout:(158/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 1.0 MB/s | 2.4 MB 00:02 2024-05-20T17:46:40.898 INFO:teuthology.orchestra.run.smithi129.stdout:(159/159): openssl-3.2.1-1.el9.x86_64.rpm 542 kB/s | 1.3 MB 00:02 2024-05-20T17:46:40.905 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-05-20T17:46:40.905 INFO:teuthology.orchestra.run.smithi129.stdout:Total 6.9 MB/s | 222 MB 00:31 2024-05-20T17:46:42.116 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-05-20T17:46:42.275 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-05-20T17:46:42.275 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-05-20T17:46:43.248 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-05-20T17:46:43.320 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-20T17:46:43.358 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-20T17:46:43.420 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-20T17:46:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-20T17:46:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-20T17:46:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-20T17:46:43.876 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-20T17:46:43.980 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 8/170 2024-05-20T17:46:44.105 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 8/170 2024-05-20T17:46:44.140 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 9/170 2024-05-20T17:46:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 9/170 2024-05-20T17:46:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:19.0.0-2744.gd2b507c7.el9.x86_64 10/170 2024-05-20T17:46:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-20T17:46:44.478 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-20T17:46:44.485 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-05-20T17:46:44.485 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-05-20T17:46:44.526 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-20T17:46:44.554 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 14/170 2024-05-20T17:46:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 14/170 2024-05-20T17:46:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86 15/170 2024-05-20T17:46:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86 15/170 2024-05-20T17:46:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-20T17:46:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-20T17:46:44.922 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-20T17:46:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-20T17:46:45.237 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-20T17:46:45.297 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-20T17:46:45.401 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-20T17:46:45.482 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-20T17:46:45.681 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-20T17:46:45.701 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-05-20T17:46:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-20T17:46:45.788 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-20T17:46:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-20T17:46:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-20T17:46:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-20T17:46:45.929 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-05-20T17:46:45.932 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-20T17:46:45.964 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-05-20T17:46:45.967 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-20T17:46:46.009 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-20T17:46:46.030 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-05-20T17:46:46.063 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-05-20T17:46:46.109 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-05-20T17:46:46.141 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-05-20T17:46:46.169 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-05-20T17:46:46.239 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-05-20T17:46:46.292 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-05-20T17:46:46.330 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-05-20T17:46:46.334 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-20T17:46:46.366 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-05-20T17:46:46.442 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-05-20T17:46:46.459 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 8/170 2024-05-20T17:46:46.476 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:19.0.0-2744.gd2b507c7.el9. 34/170 2024-05-20T17:46:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:19.0.0-2744.gd2b507c7.el 35/170 2024-05-20T17:46:46.548 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:19.0.0-2744.gd2b507c7.el9.x86_6 36/170 2024-05-20T17:46:46.568 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 8/170 2024-05-20T17:46:46.594 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 9/170 2024-05-20T17:46:46.618 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-05-20T17:46:46.663 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-05-20T17:46:46.669 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 9/170 2024-05-20T17:46:46.698 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:19.0.0-2744.gd2b507c7.el9.x86_64 10/170 2024-05-20T17:46:46.731 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-20T17:46:46.738 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-05-20T17:46:46.880 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-05-20T17:46:46.913 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-20T17:46:46.933 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-05-20T17:46:46.963 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-20T17:46:46.992 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 14/170 2024-05-20T17:46:47.096 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 14/170 2024-05-20T17:46:47.126 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86 15/170 2024-05-20T17:46:47.172 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86 15/170 2024-05-20T17:46:47.194 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-20T17:46:47.217 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-20T17:46:47.411 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-20T17:46:47.461 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-05-20T17:46:47.537 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-20T17:46:47.545 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-05-20T17:46:47.582 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-05-20T17:46:47.618 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-05-20T17:46:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-05-20T17:46:47.685 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-20T17:46:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:19.0.0-2744.gd2b507c7.el9.x86_6 47/170 2024-05-20T17:46:47.728 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-20T17:46:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-05-20T17:46:47.838 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-20T17:46:47.900 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-05-20T17:46:47.913 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-20T17:46:47.951 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-05-20T17:46:48.005 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-05-20T17:46:48.087 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-20T17:46:48.102 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-20T17:46:48.118 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-20T17:46:48.154 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-20T17:46:48.201 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-20T17:46:48.210 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-20T17:46:48.237 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-20T17:46:48.280 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-20T17:46:48.305 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-20T17:46:48.322 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-20T17:46:48.344 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-05-20T17:46:48.378 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-05-20T17:46:48.443 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-05-20T17:46:48.460 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-20T17:46:48.478 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-05-20T17:46:48.508 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-20T17:46:48.521 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-05-20T17:46:48.549 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-20T17:46:48.553 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-05-20T17:46:48.582 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-05-20T17:46:48.644 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-20T17:46:48.650 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-05-20T17:46:48.682 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-20T17:46:48.704 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-05-20T17:46:48.722 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-20T17:46:48.735 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-05-20T17:46:48.748 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-20T17:46:48.770 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-05-20T17:46:48.772 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-20T17:46:48.837 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-05-20T17:46:48.848 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-20T17:46:48.873 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:19.0.0-2744.gd2b507c7.el9. 34/170 2024-05-20T17:46:48.916 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:19.0.0-2744.gd2b507c7.el 35/170 2024-05-20T17:46:48.945 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:19.0.0-2744.gd2b507c7.el9.x86_6 36/170 2024-05-20T17:46:48.976 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-20T17:46:49.015 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-05-20T17:46:49.022 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-20T17:46:49.050 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-20T17:46:49.060 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-05-20T17:46:49.084 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-20T17:46:49.132 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-20T17:46:49.134 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-05-20T17:46:49.163 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-20T17:46:49.193 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-20T17:46:49.227 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-20T17:46:49.261 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-20T17:46:49.266 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-05-20T17:46:49.303 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-20T17:46:49.321 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-05-20T17:46:49.337 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-20T17:46:49.432 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-20T17:46:49.484 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-20T17:46:49.515 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-20T17:46:49.562 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-20T17:46:49.662 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-20T17:46:49.708 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-20T17:46:49.844 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-05-20T17:46:49.933 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-05-20T17:46:49.971 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-05-20T17:46:49.999 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-05-20T17:46:50.048 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-05-20T17:46:50.087 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:19.0.0-2744.gd2b507c7.el9.x86_6 47/170 2024-05-20T17:46:50.128 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-05-20T17:46:50.279 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-05-20T17:46:50.331 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-05-20T17:46:50.386 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-05-20T17:46:50.435 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-20T17:46:50.451 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-20T17:46:50.481 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-20T17:46:50.481 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-20T17:46:50.514 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-20T17:46:50.517 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-20T17:46:50.564 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-20T17:46:50.601 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-20T17:46:50.613 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-20T17:46:50.644 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-20T17:46:50.661 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-20T17:46:50.686 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-20T17:46:50.706 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-20T17:46:50.718 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-05-20T17:46:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-20T17:46:50.795 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-20T17:46:50.822 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-20T17:46:50.833 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-20T17:46:50.842 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-20T17:46:50.863 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-20T17:46:50.896 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-20T17:46:50.908 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-20T17:46:50.945 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-20T17:46:50.979 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-20T17:46:50.991 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-20T17:46:51.022 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-20T17:46:51.029 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-20T17:46:51.030 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-20T17:46:51.056 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-20T17:46:51.070 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-20T17:46:51.095 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-20T17:46:51.120 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-20T17:46:51.134 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-20T17:46:51.195 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-20T17:46:51.322 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-20T17:46:51.361 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-20T17:46:51.365 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-20T17:46:51.389 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-20T17:46:51.423 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-20T17:46:51.462 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-20T17:46:51.494 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-20T17:46:51.509 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 8/170 2024-05-20T17:46:51.524 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-20T17:46:51.558 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-20T17:46:51.600 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-20T17:46:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 8/170 2024-05-20T17:46:51.642 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-20T17:46:51.664 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 9/170 2024-05-20T17:46:51.676 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-20T17:46:51.739 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 9/170 2024-05-20T17:46:51.763 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-20T17:46:51.775 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.0.0-2744.gd2b507c7.el9.x86_64 10/170 2024-05-20T17:46:51.781 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-20T17:46:51.815 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-20T17:46:51.831 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-20T17:46:51.846 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-20T17:46:51.893 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-20T17:46:51.896 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-20T17:46:51.993 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-20T17:46:52.025 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-20T17:46:52.047 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-20T17:46:52.075 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-20T17:46:52.103 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 14/170 2024-05-20T17:46:52.207 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 14/170 2024-05-20T17:46:52.229 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86 15/170 2024-05-20T17:46:52.292 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86 15/170 2024-05-20T17:46:52.313 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-20T17:46:52.337 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-20T17:46:52.540 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-20T17:46:52.718 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-20T17:46:52.762 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-20T17:46:52.812 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-20T17:46:52.845 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-20T17:46:52.878 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-20T17:46:52.938 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-20T17:46:52.951 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-20T17:46:53.000 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-20T17:46:53.030 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-20T17:46:53.045 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-20T17:46:53.106 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-20T17:46:53.124 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-20T17:46:53.147 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-20T17:46:53.165 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-20T17:46:53.236 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-20T17:46:53.264 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-20T17:46:53.297 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-20T17:46:53.317 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-20T17:46:53.349 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-20T17:46:53.379 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-20T17:46:53.427 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-20T17:46:53.457 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-20T17:46:53.522 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-20T17:46:53.553 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-05-20T17:46:53.595 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-05-20T17:46:53.617 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-20T17:46:53.652 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 103/170 2024-05-20T17:46:53.659 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-05-20T17:46:53.695 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-05-20T17:46:53.738 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-05-20T17:46:53.743 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 103/170 2024-05-20T17:46:53.771 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-05-20T17:46:53.773 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 104/170 2024-05-20T17:46:53.799 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-05-20T17:46:53.809 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-20T17:46:53.867 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-05-20T17:46:53.920 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-05-20T17:46:53.993 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-05-20T17:46:54.028 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-05-20T17:46:54.100 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-20T17:46:54.103 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-05-20T17:46:54.147 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.0.0-2744.gd2b507c7.el9. 34/170 2024-05-20T17:46:54.190 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.0.0-2744.gd2b507c7.el 35/170 2024-05-20T17:46:54.211 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-20T17:46:54.219 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.0.0-2744.gd2b507c7.el9.x86_6 36/170 2024-05-20T17:46:54.287 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-20T17:46:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-05-20T17:46:54.321 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 107/170 2024-05-20T17:46:54.334 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-05-20T17:46:54.392 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 107/170 2024-05-20T17:46:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-05-20T17:46:54.415 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 108/170 2024-05-20T17:46:54.541 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-05-20T17:46:54.595 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-05-20T17:46:55.137 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-05-20T17:46:55.248 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-05-20T17:46:55.278 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-05-20T17:46:55.307 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-05-20T17:46:55.346 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-05-20T17:46:55.384 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.0.0-2744.gd2b507c7.el9.x86_6 47/170 2024-05-20T17:46:55.427 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-05-20T17:46:55.433 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-20T17:46:55.510 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-20T17:46:55.537 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-20T17:46:55.571 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-20T17:46:55.581 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-05-20T17:46:55.637 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-05-20T17:46:55.692 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-05-20T17:46:55.764 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-20T17:46:55.804 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-20T17:46:55.841 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-20T17:46:55.887 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-20T17:46:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-20T17:46:55.924 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-20T17:46:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 103/170 2024-05-20T17:46:55.966 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-20T17:46:56.008 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-20T17:46:56.016 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 103/170 2024-05-20T17:46:56.046 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 104/170 2024-05-20T17:46:56.091 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-20T17:46:56.143 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-20T17:46:56.186 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-20T17:46:56.226 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-20T17:46:56.320 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-20T17:46:56.352 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-20T17:46:56.394 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-20T17:46:56.426 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-20T17:46:56.450 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-20T17:46:56.555 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-20T17:46:56.557 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-20T17:46:56.587 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 107/170 2024-05-20T17:46:56.592 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 109/170 2024-05-20T17:46:56.628 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 109/170 2024-05-20T17:46:56.657 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 107/170 2024-05-20T17:46:56.676 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-20T17:46:56.680 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 108/170 2024-05-20T17:46:56.724 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-20T17:46:56.760 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-20T17:46:56.795 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-20T17:46:56.834 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-20T17:46:56.865 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-20T17:46:56.895 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-20T17:46:56.930 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-20T17:46:56.963 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-20T17:46:57.005 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-20T17:46:57.039 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-20T17:46:57.124 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-20T17:46:57.169 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-20T17:46:57.184 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 109/170 2024-05-20T17:46:57.200 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-20T17:46:57.223 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 110/170 2024-05-20T17:46:57.247 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-20T17:46:57.306 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 110/170 2024-05-20T17:46:57.306 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-20T17:46:57.306 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-20T17:46:57.306 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:46:57.333 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 111/170 2024-05-20T17:46:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-20T17:46:57.393 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-20T17:46:58.107 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-20T17:46:58.158 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-20T17:46:58.191 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-20T17:46:58.281 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-20T17:46:58.338 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-20T17:46:58.382 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-20T17:46:58.470 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-20T17:46:58.502 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-20T17:46:58.680 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-20T17:46:58.708 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-20T17:46:58.779 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-20T17:46:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 109/170 2024-05-20T17:46:58.944 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 109/170 2024-05-20T17:46:59.422 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-20T17:46:59.499 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 109/170 2024-05-20T17:46:59.533 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-20T17:46:59.539 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 110/170 2024-05-20T17:46:59.631 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 110/170 2024-05-20T17:46:59.631 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-20T17:46:59.631 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-20T17:46:59.631 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:46:59.665 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 111/170 2024-05-20T17:47:00.787 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-20T17:47:00.872 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-20T17:47:00.899 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-20T17:47:00.949 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-20T17:47:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-20T17:47:01.320 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 103/170 2024-05-20T17:47:01.410 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 103/170 2024-05-20T17:47:01.442 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 104/170 2024-05-20T17:47:01.484 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-20T17:47:01.962 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-20T17:47:01.999 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 107/170 2024-05-20T17:47:02.062 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 107/170 2024-05-20T17:47:02.092 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 108/170 2024-05-20T17:47:04.326 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 109/170 2024-05-20T17:47:04.364 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 109/170 2024-05-20T17:47:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 109/170 2024-05-20T17:47:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 110/170 2024-05-20T17:47:05.069 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 110/170 2024-05-20T17:47:05.069 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-20T17:47:05.070 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-20T17:47:05.070 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:05.094 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 111/170 2024-05-20T17:47:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 111/170 2024-05-20T17:47:09.127 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2024-05-20T17:47:09.127 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2024-05-20T17:47:09.127 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2024-05-20T17:47:09.127 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2024-05-20T17:47:09.127 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2024-05-20T17:47:09.127 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /root 2024-05-20T17:47:09.127 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2024-05-20T17:47:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:09.288 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 112/170 2024-05-20T17:47:09.303 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 112/170 2024-05-20T17:47:09.303 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:09.303 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-20T17:47:09.304 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 113/170 2024-05-20T17:47:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 113/170 2024-05-20T17:47:10.927 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:10.927 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-20T17:47:10.927 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-20T17:47:10.927 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-20T17:47:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:11.152 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 114/170 2024-05-20T17:47:11.174 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 114/170 2024-05-20T17:47:11.174 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:11.174 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-20T17:47:11.174 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-20T17:47:11.174 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-20T17:47:11.174 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:11.597 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 111/170 2024-05-20T17:47:11.597 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-05-20T17:47:11.597 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-05-20T17:47:11.597 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-05-20T17:47:11.598 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-05-20T17:47:11.598 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-05-20T17:47:11.598 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-05-20T17:47:11.598 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-05-20T17:47:11.598 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:11.644 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 115/170 2024-05-20T17:47:11.666 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 115/170 2024-05-20T17:47:11.666 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:11.667 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-20T17:47:11.667 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-20T17:47:11.667 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-20T17:47:11.667 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:11.774 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-20T17:47:11.786 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 112/170 2024-05-20T17:47:11.802 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 112/170 2024-05-20T17:47:11.802 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:11.802 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-20T17:47:11.802 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:11.832 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-20T17:47:11.887 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-20T17:47:12.084 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-20T17:47:12.137 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-20T17:47:12.215 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-20T17:47:12.304 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-20T17:47:12.453 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-20T17:47:12.495 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-20T17:47:12.526 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-20T17:47:12.554 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-20T17:47:13.006 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 127/170 2024-05-20T17:47:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 127/170 2024-05-20T17:47:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2744.gd2b507c7.e 128/170 2024-05-20T17:47:13.130 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2744.gd2b507c7. 129/170 2024-05-20T17:47:13.167 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noa 130/170 2024-05-20T17:47:13.381 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 113/170 2024-05-20T17:47:13.406 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 113/170 2024-05-20T17:47:13.406 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:13.406 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-20T17:47:13.406 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-20T17:47:13.406 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-20T17:47:13.406 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:13.631 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 114/170 2024-05-20T17:47:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 114/170 2024-05-20T17:47:13.653 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:13.653 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-20T17:47:13.653 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-20T17:47:13.653 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-20T17:47:13.653 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:14.115 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 115/170 2024-05-20T17:47:14.138 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 115/170 2024-05-20T17:47:14.138 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:14.138 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-20T17:47:14.138 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-20T17:47:14.138 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-20T17:47:14.138 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noa 130/170 2024-05-20T17:47:14.245 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-20T17:47:14.269 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.n 131/170 2024-05-20T17:47:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-20T17:47:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-20T17:47:14.536 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-20T17:47:14.591 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-20T17:47:14.669 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-20T17:47:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-20T17:47:14.897 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-20T17:47:14.941 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-20T17:47:14.973 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-20T17:47:15.001 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-20T17:47:15.238 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.n 131/170 2024-05-20T17:47:15.269 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b 132/170 2024-05-20T17:47:15.371 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b 132/170 2024-05-20T17:47:15.478 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 127/170 2024-05-20T17:47:15.508 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 127/170 2024-05-20T17:47:15.540 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2744.gd2b507c7.e 128/170 2024-05-20T17:47:15.542 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2744.gd2b507c7.el 133/170 2024-05-20T17:47:15.571 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 134/170 2024-05-20T17:47:15.594 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 134/170 2024-05-20T17:47:15.594 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:15.594 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-20T17:47:15.594 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-20T17:47:15.594 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-20T17:47:15.594 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:15.601 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2744.gd2b507c7. 129/170 2024-05-20T17:47:15.631 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noa 130/170 2024-05-20T17:47:15.638 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 135/170 2024-05-20T17:47:15.649 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 135/170 2024-05-20T17:47:15.747 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:19.0.0-2744.gd2b507c7.el9.x86_64 136/170 2024-05-20T17:47:15.803 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 137/170 2024-05-20T17:47:16.498 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 111/170 2024-05-20T17:47:16.498 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-05-20T17:47:16.498 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-05-20T17:47:16.498 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-05-20T17:47:16.498 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-05-20T17:47:16.498 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-05-20T17:47:16.498 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-05-20T17:47:16.498 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-05-20T17:47:16.498 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:16.664 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noa 130/170 2024-05-20T17:47:16.665 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 112/170 2024-05-20T17:47:16.681 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 112/170 2024-05-20T17:47:16.681 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:16.681 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-20T17:47:16.681 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:16.700 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.n 131/170 2024-05-20T17:47:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-20T17:47:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 139/170 2024-05-20T17:47:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 139/170 2024-05-20T17:47:16.772 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:16.772 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-20T17:47:16.772 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-20T17:47:16.772 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-20T17:47:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:16.816 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2744.gd2b50 140/170 2024-05-20T17:47:16.838 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2744.gd2b50 140/170 2024-05-20T17:47:16.838 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:16.839 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-20T17:47:16.839 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:16.883 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 141/170 2024-05-20T17:47:16.906 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 141/170 2024-05-20T17:47:16.906 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:16.906 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-20T17:47:16.906 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-20T17:47:16.906 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-20T17:47:16.906 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:17.218 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 142/170 2024-05-20T17:47:17.240 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 142/170 2024-05-20T17:47:17.240 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:17.240 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-20T17:47:17.240 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-20T17:47:17.240 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-20T17:47:17.240 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:17.665 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.n 131/170 2024-05-20T17:47:17.708 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b 132/170 2024-05-20T17:47:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b 132/170 2024-05-20T17:47:17.975 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2744.gd2b507c7.el 133/170 2024-05-20T17:47:18.002 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 134/170 2024-05-20T17:47:18.025 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 134/170 2024-05-20T17:47:18.025 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:18.025 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-20T17:47:18.025 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-20T17:47:18.025 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-20T17:47:18.025 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:18.070 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 135/170 2024-05-20T17:47:18.081 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 135/170 2024-05-20T17:47:18.179 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:19.0.0-2744.gd2b507c7.el9.x86_64 136/170 2024-05-20T17:47:18.226 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 137/170 2024-05-20T17:47:18.277 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 113/170 2024-05-20T17:47:18.304 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 113/170 2024-05-20T17:47:18.304 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:18.304 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-20T17:47:18.304 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-20T17:47:18.304 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-20T17:47:18.304 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:18.527 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 114/170 2024-05-20T17:47:18.550 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 114/170 2024-05-20T17:47:18.550 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:18.550 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-20T17:47:18.550 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-20T17:47:18.550 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-20T17:47:18.550 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:19.011 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 115/170 2024-05-20T17:47:19.034 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 115/170 2024-05-20T17:47:19.034 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:19.034 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-20T17:47:19.035 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-20T17:47:19.035 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-20T17:47:19.035 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:19.139 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-20T17:47:19.141 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-20T17:47:19.172 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 139/170 2024-05-20T17:47:19.195 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 139/170 2024-05-20T17:47:19.195 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:19.195 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-20T17:47:19.195 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-20T17:47:19.195 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-20T17:47:19.195 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:19.198 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-20T17:47:19.248 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2744.gd2b50 140/170 2024-05-20T17:47:19.262 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-20T17:47:19.269 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2744.gd2b50 140/170 2024-05-20T17:47:19.269 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:19.269 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-20T17:47:19.269 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:19.349 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 141/170 2024-05-20T17:47:19.371 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 141/170 2024-05-20T17:47:19.371 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:19.371 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-20T17:47:19.371 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-20T17:47:19.372 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-20T17:47:19.372 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:19.449 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-20T17:47:19.495 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-20T17:47:19.573 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-20T17:47:19.670 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-20T17:47:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 142/170 2024-05-20T17:47:19.705 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 142/170 2024-05-20T17:47:19.705 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:19.705 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-20T17:47:19.705 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-20T17:47:19.705 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-20T17:47:19.705 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:19.808 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-20T17:47:19.845 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-20T17:47:19.881 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-20T17:47:19.905 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-20T17:47:20.359 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 127/170 2024-05-20T17:47:20.387 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 127/170 2024-05-20T17:47:20.418 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2744.gd2b507c7.e 128/170 2024-05-20T17:47:20.480 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2744.gd2b507c7. 129/170 2024-05-20T17:47:20.510 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noa 130/170 2024-05-20T17:47:21.539 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noa 130/170 2024-05-20T17:47:21.578 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.n 131/170 2024-05-20T17:47:21.813 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:19.0.0-2744.gd2b507c7.el9.x86_64 143/170 2024-05-20T17:47:21.862 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 144/170 2024-05-20T17:47:21.904 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 145/170 2024-05-20T17:47:21.935 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-20T17:47:21.962 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-20T17:47:22.150 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-shell-2:19.0.0-2744.gd2b507c7.el9.x86_64 148/170 2024-05-20T17:47:22.192 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-20T17:47:22.224 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:19.0.0-2744.gd2b507c7.el9.x86_ 150/170 2024-05-20T17:47:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-20T17:47:22.294 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-20T17:47:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-20T17:47:22.540 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.n 131/170 2024-05-20T17:47:22.570 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b 132/170 2024-05-20T17:47:22.672 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b 132/170 2024-05-20T17:47:22.835 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2744.gd2b507c7.el 133/170 2024-05-20T17:47:22.881 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 134/170 2024-05-20T17:47:22.904 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 134/170 2024-05-20T17:47:22.904 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:22.904 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-20T17:47:22.904 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-20T17:47:22.904 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-20T17:47:22.904 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:22.948 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 135/170 2024-05-20T17:47:22.959 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 135/170 2024-05-20T17:47:22.964 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-20T17:47:23.005 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-20T17:47:23.024 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-20T17:47:23.057 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.0.0-2744.gd2b507c7.el9.x86_64 136/170 2024-05-20T17:47:23.114 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 137/170 2024-05-20T17:47:23.877 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-top-2:19.0.0-2744.gd2b507c7.el9.noarch 155/170 2024-05-20T17:47:23.967 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-20T17:47:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-20T17:47:24.022 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-20T17:47:24.050 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 139/170 2024-05-20T17:47:24.074 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 139/170 2024-05-20T17:47:24.074 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:24.074 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-20T17:47:24.074 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-20T17:47:24.074 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-20T17:47:24.074 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-20T17:47:24.118 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2744.gd2b50 140/170 2024-05-20T17:47:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-20T17:47:24.140 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2744.gd2b50 140/170 2024-05-20T17:47:24.140 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:24.140 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-20T17:47:24.140 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-20T17:47:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-20T17:47:24.185 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 141/170 2024-05-20T17:47:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-20T17:47:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-20T17:47:24.208 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 141/170 2024-05-20T17:47:24.208 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:24.208 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-20T17:47:24.208 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-20T17:47:24.208 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-20T17:47:24.208 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:19.0.0-2744.gd2b507c7.el9.x86_64 143/170 2024-05-20T17:47:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-20T17:47:24.287 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 144/170 2024-05-20T17:47:24.298 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-20T17:47:24.298 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-20T17:47:24.327 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 145/170 2024-05-20T17:47:24.359 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-20T17:47:24.386 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-20T17:47:24.391 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-20T17:47:24.416 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-20T17:47:24.421 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-20T17:47:24.421 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-20T17:47:24.519 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 142/170 2024-05-20T17:47:24.542 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 142/170 2024-05-20T17:47:24.542 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-20T17:47:24.542 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-20T17:47:24.542 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-20T17:47:24.542 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-20T17:47:24.542 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:24.570 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-shell-2:19.0.0-2744.gd2b507c7.el9.x86_64 148/170 2024-05-20T17:47:24.610 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-20T17:47:24.610 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-20T17:47:24.616 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-20T17:47:24.657 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:19.0.0-2744.gd2b507c7.el9.x86_ 150/170 2024-05-20T17:47:24.677 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-20T17:47:24.697 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-20T17:47:24.702 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-20T17:47:24.727 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-20T17:47:24.765 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-20T17:47:24.773 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-20T17:47:25.398 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-20T17:47:25.438 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-20T17:47:25.457 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-20T17:47:26.299 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-top-2:19.0.0-2744.gd2b507c7.el9.noarch 155/170 2024-05-20T17:47:26.377 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-20T17:47:26.418 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-20T17:47:26.516 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-20T17:47:26.555 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-20T17:47:26.580 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-20T17:47:26.606 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-20T17:47:26.632 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-20T17:47:26.632 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-20T17:47:26.717 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-20T17:47:26.749 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-20T17:47:26.750 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-20T17:47:26.826 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-20T17:47:26.851 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-20T17:47:26.856 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-20T17:47:26.856 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-20T17:47:27.036 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-20T17:47:27.036 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-20T17:47:27.120 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-20T17:47:27.154 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-20T17:47:27.218 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-20T17:47:29.099 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.0.0-2744.gd2b507c7.el9.x86_64 143/170 2024-05-20T17:47:29.165 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 144/170 2024-05-20T17:47:29.206 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 145/170 2024-05-20T17:47:29.239 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-20T17:47:29.257 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-20T17:47:29.445 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-shell-2:19.0.0-2744.gd2b507c7.el9.x86_64 148/170 2024-05-20T17:47:29.488 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-20T17:47:29.530 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.0.0-2744.gd2b507c7.el9.x86_ 150/170 2024-05-20T17:47:29.569 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-20T17:47:29.598 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-20T17:47:29.644 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-20T17:47:30.250 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-20T17:47:30.285 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-20T17:47:30.304 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-20T17:47:30.576 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:19.0.0-2744.gd2b507c7.el9.x86_64 1/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 2/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 3/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 4/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2744.gd2b50 5/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 6/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 7/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 8/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 9/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 10/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 11/170 2024-05-20T17:47:30.577 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:19.0.0-2744.gd2b507c7.el9.x86_64 12/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 13/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-shell-2:19.0.0-2744.gd2b507c7.el9.x86_64 14/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:19.0.0-2744.gd2b507c7.el9.x86_ 15/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 16/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 17/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:19.0.0-2744.gd2b507c7.el9.x86_6 18/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86 19/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 20/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2744.gd2b507c7.el 21/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:19.0.0-2744.gd2b507c7.el9. 22/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:19.0.0-2744.gd2b507c7.el9.x86_6 23/170 2024-05-20T17:47:30.578 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:19.0.0-2744.gd2b507c7.el9.x86_64 24/170 2024-05-20T17:47:30.579 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 25/170 2024-05-20T17:47:30.579 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 26/170 2024-05-20T17:47:30.579 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 27/170 2024-05-20T17:47:30.591 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 28/170 2024-05-20T17:47:30.592 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 29/170 2024-05-20T17:47:30.592 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2744.gd2b507c7. 30/170 2024-05-20T17:47:30.592 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noa 31/170 2024-05-20T17:47:30.592 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.n 32/170 2024-05-20T17:47:30.592 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b 33/170 2024-05-20T17:47:30.592 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2744.gd2b507c7.el 34/170 2024-05-20T17:47:30.592 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 35/170 2024-05-20T17:47:30.592 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2744.gd2b507c7.e 36/170 2024-05-20T17:47:30.592 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 37/170 2024-05-20T17:47:30.592 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 38/170 2024-05-20T17:47:30.592 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-top-2:19.0.0-2744.gd2b507c7.el9.noarch 39/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-20T17:47:30.593 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-20T17:47:30.594 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-20T17:47:30.595 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-20T17:47:30.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-20T17:47:30.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-20T17:47:30.598 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-20T17:47:30.599 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-20T17:47:30.600 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 149/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 151/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-20T17:47:30.601 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-20T17:47:30.602 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-20T17:47:30.602 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-20T17:47:30.602 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-20T17:47:30.602 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-20T17:47:30.602 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-20T17:47:30.602 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-20T17:47:30.602 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-20T17:47:30.602 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-20T17:47:30.602 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-20T17:47:30.602 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-20T17:47:30.602 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-20T17:47:30.603 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-20T17:47:30.603 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-20T17:47:30.603 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-20T17:47:31.138 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-top-2:19.0.0-2744.gd2b507c7.el9.noarch 155/170 2024-05-20T17:47:31.232 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-20T17:47:31.274 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-20T17:47:31.357 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-20T17:47:31.385 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-20T17:47:31.410 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-20T17:47:31.436 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-20T17:47:31.462 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-20T17:47:31.462 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-20T17:47:31.555 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-20T17:47:31.588 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-20T17:47:31.588 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-20T17:47:31.673 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-20T17:47:31.698 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-20T17:47:31.703 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-20T17:47:31.703 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-20T17:47:31.892 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-20T17:47:31.892 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-20T17:47:31.976 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-20T17:47:32.009 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-20T17:47:32.075 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-20T17:47:33.087 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-20T17:47:33.087 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:19.0.0-2744.gd2b507c7.el9.x86_64 1/170 2024-05-20T17:47:33.087 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 2/170 2024-05-20T17:47:33.087 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 3/170 2024-05-20T17:47:33.087 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 4/170 2024-05-20T17:47:33.087 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2744.gd2b50 5/170 2024-05-20T17:47:33.087 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 6/170 2024-05-20T17:47:33.087 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 7/170 2024-05-20T17:47:33.087 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 8/170 2024-05-20T17:47:33.087 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 9/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 10/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 11/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:19.0.0-2744.gd2b507c7.el9.x86_64 12/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 13/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-shell-2:19.0.0-2744.gd2b507c7.el9.x86_64 14/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:19.0.0-2744.gd2b507c7.el9.x86_ 15/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 16/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 17/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:19.0.0-2744.gd2b507c7.el9.x86_6 18/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86 19/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 20/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2744.gd2b507c7.el 21/170 2024-05-20T17:47:33.088 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:19.0.0-2744.gd2b507c7.el9. 22/170 2024-05-20T17:47:33.089 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:19.0.0-2744.gd2b507c7.el9.x86_6 23/170 2024-05-20T17:47:33.089 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:19.0.0-2744.gd2b507c7.el9.x86_64 24/170 2024-05-20T17:47:33.089 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 25/170 2024-05-20T17:47:33.089 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 26/170 2024-05-20T17:47:33.089 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 27/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 28/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 29/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2744.gd2b507c7. 30/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noa 31/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.n 32/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b 33/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2744.gd2b507c7.el 34/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 35/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2744.gd2b507c7.e 36/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 37/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 38/170 2024-05-20T17:47:33.101 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-top-2:19.0.0-2744.gd2b507c7.el9.noarch 39/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-20T17:47:33.102 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-20T17:47:33.103 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-20T17:47:33.104 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-20T17:47:33.105 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-20T17:47:33.106 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-20T17:47:33.107 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-20T17:47:33.108 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-20T17:47:33.109 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 149/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 151/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-20T17:47:33.110 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-20T17:47:33.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-20T17:47:33.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-20T17:47:33.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-20T17:47:33.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-20T17:47:33.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-20T17:47:33.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-20T17:47:33.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-20T17:47:33.111 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-20T17:47:33.389 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-20T17:47:33.389 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:33.389 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2024-05-20T17:47:33.389 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:33.389 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:33.389 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:33.389 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:33.389 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: automake-1.16.2-8.el9.noarch 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: bison-3.7.4-5.el9.x86_64 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.390 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:33.391 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-shell-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: flex-2.6.4-9.el9.x86_64 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-20T17:47:33.392 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-20T17:47:33.393 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-20T17:47:33.393 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-20T17:47:33.393 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-20T17:47:33.393 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-20T17:47:33.393 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.393 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.393 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.393 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-20T17:47:33.393 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-20T17:47:33.393 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-20T17:47:33.393 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: lua-5.4.4-4.el9.x86_64 2024-05-20T17:47:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-13.14-1.el9.x86_64 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-20T17:47:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-20T17:47:33.396 INFO:teuthology.orchestra.run.smithi154.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-20T17:47:33.397 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-20T17:47:33.398 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply-3.11-14.el9.noarch 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-20T17:47:33.399 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-20T17:47:33.400 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: re2-1:20211101-3.el9.x86_64 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-20T17:47:33.401 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-20T17:47:33.402 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-20T17:47:33.402 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-05-20T17:47:33.402 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-05-20T17:47:33.885 DEBUG:teuthology.parallel:result is None 2024-05-20T17:47:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-20T17:47:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:35.919 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-05-20T17:47:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-20T17:47:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: automake-1.16.2-8.el9.noarch 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: bison-3.7.4-5.el9.x86_64 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-shell-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-top-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: flex-2.6.4-9.el9.x86_64 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-20T17:47:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-20T17:47:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-20T17:47:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-20T17:47:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-20T17:47:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-20T17:47:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: lua-5.4.4-4.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-20T17:47:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-20T17:47:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-13.14-1.el9.x86_64 2024-05-20T17:47:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-20T17:47:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-20T17:47:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-20T17:47:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-20T17:47:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-20T17:47:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-20T17:47:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-20T17:47:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-20T17:47:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-20T17:47:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-20T17:47:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-20T17:47:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-20T17:47:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-20T17:47:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-20T17:47:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-20T17:47:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-20T17:47:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-20T17:47:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-20T17:47:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-20T17:47:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-20T17:47:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-20T17:47:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-20T17:47:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-20T17:47:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-20T17:47:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-20T17:47:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-20T17:47:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-20T17:47:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: re2-1:20211101-3.el9.x86_64 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-20T17:47:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-20T17:47:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-20T17:47:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-20T17:47:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:47:35.935 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-05-20T17:47:36.360 DEBUG:teuthology.parallel:result is None 2024-05-20T17:47:37.901 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-20T17:47:37.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.0.0-2744.gd2b507c7.el9.x86_64 1/170 2024-05-20T17:47:37.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 2/170 2024-05-20T17:47:37.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 3/170 2024-05-20T17:47:37.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 4/170 2024-05-20T17:47:37.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2744.gd2b50 5/170 2024-05-20T17:47:37.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 6/170 2024-05-20T17:47:37.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 7/170 2024-05-20T17:47:37.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 8/170 2024-05-20T17:47:37.902 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 9/170 2024-05-20T17:47:37.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 10/170 2024-05-20T17:47:37.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 11/170 2024-05-20T17:47:37.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.0.0-2744.gd2b507c7.el9.x86_64 12/170 2024-05-20T17:47:37.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 13/170 2024-05-20T17:47:37.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-shell-2:19.0.0-2744.gd2b507c7.el9.x86_64 14/170 2024-05-20T17:47:37.903 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.0.0-2744.gd2b507c7.el9.x86_ 15/170 2024-05-20T17:47:37.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 16/170 2024-05-20T17:47:37.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 17/170 2024-05-20T17:47:37.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.0.0-2744.gd2b507c7.el9.x86_6 18/170 2024-05-20T17:47:37.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86 19/170 2024-05-20T17:47:37.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 20/170 2024-05-20T17:47:37.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2744.gd2b507c7.el 21/170 2024-05-20T17:47:37.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.0.0-2744.gd2b507c7.el9. 22/170 2024-05-20T17:47:37.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.0.0-2744.gd2b507c7.el9.x86_6 23/170 2024-05-20T17:47:37.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.0.0-2744.gd2b507c7.el9.x86_64 24/170 2024-05-20T17:47:37.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 25/170 2024-05-20T17:47:37.904 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 26/170 2024-05-20T17:47:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 27/170 2024-05-20T17:47:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 28/170 2024-05-20T17:47:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 29/170 2024-05-20T17:47:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2744.gd2b507c7. 30/170 2024-05-20T17:47:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noa 31/170 2024-05-20T17:47:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.n 32/170 2024-05-20T17:47:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b 33/170 2024-05-20T17:47:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2744.gd2b507c7.el 34/170 2024-05-20T17:47:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 35/170 2024-05-20T17:47:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2744.gd2b507c7.e 36/170 2024-05-20T17:47:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 37/170 2024-05-20T17:47:37.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 38/170 2024-05-20T17:47:37.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-top-2:19.0.0-2744.gd2b507c7.el9.noarch 39/170 2024-05-20T17:47:37.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-20T17:47:37.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-20T17:47:37.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-20T17:47:37.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-20T17:47:37.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-20T17:47:37.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-20T17:47:37.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-20T17:47:37.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-20T17:47:37.906 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-20T17:47:37.907 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-20T17:47:37.908 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-20T17:47:37.908 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-20T17:47:37.908 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-20T17:47:37.908 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-20T17:47:37.908 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-20T17:47:37.908 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-20T17:47:37.908 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-20T17:47:37.908 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-20T17:47:37.908 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-20T17:47:37.908 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-20T17:47:37.908 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-20T17:47:37.909 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-20T17:47:37.909 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-20T17:47:37.909 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-20T17:47:37.909 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-20T17:47:37.909 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-20T17:47:37.909 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-20T17:47:37.909 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-20T17:47:37.909 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-20T17:47:37.909 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-20T17:47:37.909 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-20T17:47:37.909 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-20T17:47:37.910 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-20T17:47:37.910 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-20T17:47:37.910 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-20T17:47:37.910 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-20T17:47:37.910 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-20T17:47:37.910 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-20T17:47:37.910 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-20T17:47:37.910 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-20T17:47:37.910 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-20T17:47:37.910 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-20T17:47:37.910 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-20T17:47:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-20T17:47:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-20T17:47:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-20T17:47:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-20T17:47:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-20T17:47:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-20T17:47:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-20T17:47:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-20T17:47:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-20T17:47:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-20T17:47:37.911 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-20T17:47:37.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-20T17:47:37.913 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-20T17:47:37.914 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-20T17:47:37.914 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-20T17:47:37.914 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-20T17:47:37.914 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-20T17:47:37.914 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-20T17:47:37.914 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-20T17:47:37.914 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-20T17:47:37.914 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-20T17:47:37.914 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-20T17:47:37.914 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-20T17:47:37.914 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-20T17:47:37.915 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-20T17:47:37.915 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-20T17:47:37.915 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-20T17:47:37.915 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-20T17:47:37.915 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-20T17:47:37.915 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-20T17:47:37.915 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-20T17:47:37.915 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-20T17:47:37.915 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-20T17:47:37.915 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 149/170 2024-05-20T17:47:37.915 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 151/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-20T17:47:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-20T17:47:37.917 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-20T17:47:37.917 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-20T17:47:37.917 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-20T17:47:37.917 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-20T17:47:37.917 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-20T17:47:37.917 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-20T17:47:37.917 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-20T17:47:40.743 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-20T17:47:40.743 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:40.743 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-20T17:47:40.744 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: automake-1.16.2-8.el9.noarch 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: bison-3.7.4-5.el9.x86_64 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.745 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.746 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top-2:19.0.0-2744.gd2b507c7.el9.noarch 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: flex-2.6.4-9.el9.x86_64 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-20T17:47:40.747 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.748 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-20T17:47:40.749 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-13.14-1.el9.x86_64 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-20T17:47:40.750 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-20T17:47:40.751 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-20T17:47:40.752 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-20T17:47:40.753 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-20T17:47:40.754 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.755 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.756 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.0.0-2744.gd2b507c7.el9.x86_64 2024-05-20T17:47:40.756 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-3.el9.x86_64 2024-05-20T17:47:40.756 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-20T17:47:40.756 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-20T17:47:40.756 INFO:teuthology.orchestra.run.smithi129.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-20T17:47:40.756 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-20T17:47:40.756 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-20T17:47:40.756 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-20T17:47:40.756 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:47:40.757 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-05-20T17:47:41.210 DEBUG:teuthology.parallel:result is None 2024-05-20T17:47:41.211 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:47:41.434 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-20T17:47:41.459 INFO:teuthology.orchestra.run.smithi129.stdout:19.0.0-2744.gd2b507c7.el9 2024-05-20T17:47:41.459 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2744.gd2b507c7.el9 2024-05-20T17:47:41.459 INFO:teuthology.task.install:The correct ceph version 19.0.0-2744.gd2b507c7 is installed. 2024-05-20T17:47:41.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:47:41.727 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-20T17:47:41.753 INFO:teuthology.orchestra.run.smithi154.stdout:19.0.0-2744.gd2b507c7.el9 2024-05-20T17:47:41.753 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2744.gd2b507c7.el9 2024-05-20T17:47:41.753 INFO:teuthology.task.install:The correct ceph version 19.0.0-2744.gd2b507c7 is installed. 2024-05-20T17:47:41.755 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:47:41.892 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-20T17:47:41.917 INFO:teuthology.orchestra.run.smithi204.stdout:19.0.0-2744.gd2b507c7.el9 2024-05-20T17:47:41.918 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2744.gd2b507c7.el9 2024-05-20T17:47:41.918 INFO:teuthology.task.install:The correct ceph version 19.0.0-2744.gd2b507c7 is installed. 2024-05-20T17:47:41.920 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-20T17:47:41.920 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:47:41.920 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-20T17:47:41.953 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-05-20T17:47:41.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-20T17:47:41.984 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-20T17:47:41.984 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-20T17:47:42.015 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-20T17:47:42.015 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:47:42.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-05-20T17:47:42.044 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-20T17:47:42.112 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-05-20T17:47:42.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-05-20T17:47:42.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-20T17:47:42.205 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-20T17:47:42.205 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-05-20T17:47:42.230 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-20T17:47:42.294 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-20T17:47:42.294 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:47:42.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-20T17:47:42.323 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-20T17:47:42.392 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-05-20T17:47:42.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-20T17:47:42.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-20T17:47:42.481 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-20T17:47:42.481 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-20T17:47:42.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-20T17:47:42.570 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-20T17:47:42.570 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:47:42.570 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-05-20T17:47:42.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-20T17:47:42.670 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-05-20T17:47:42.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-05-20T17:47:42.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-20T17:47:42.756 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-20T17:47:42.756 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-05-20T17:47:42.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-20T17:47:42.843 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-20T17:47:42.943 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'd2b507c724ef6c8295ee0a35d185cdab167fd61d', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-20T17:47:42.943 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:47:42.944 INFO:tasks.cephadm:Cluster fsid is 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:47:42.944 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-20T17:47:42.944 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.129', 'mon.b': '172.21.15.154', 'mon.c': '172.21.15.204'} 2024-05-20T17:47:42.945 INFO:tasks.cephadm:First mon is mon.a on smithi129 2024-05-20T17:47:42.945 INFO:tasks.cephadm:First mgr is x 2024-05-20T17:47:42.945 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-20T17:47:42.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo hostname $(hostname -s) 2024-05-20T17:47:42.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname $(hostname -s) 2024-05-20T17:47:42.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname $(hostname -s) 2024-05-20T17:47:43.022 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-20T17:47:43.022 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:47:43.162 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd2b507c724ef6c8295ee0a35d185cdab167fd61d', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79503/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2744-gd2b507c7', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2744.gd2b507c7'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-17 22:05:47.367530', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240517.203521-squid-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-20T17:47:43.320 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-pdonnell-testing-20240517.203521-squid-debug, sha1 d2b507c724ef6c8295ee0a35d185cdab167fd61d from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:47:43.321 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/x86_64/flavors/default/cephadm 2024-05-20T17:47:43.322 DEBUG:teuthology.orchestra.run.smithi129:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-20T17:47:44.777 INFO:teuthology.orchestra.run.smithi129.stdout:-rw-r--r--. 1 ubuntu ubuntu 782070 May 20 17:47 /home/ubuntu/cephtest/cephadm 2024-05-20T17:47:44.777 DEBUG:teuthology.orchestra.run.smithi154:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-20T17:47:46.087 INFO:teuthology.orchestra.run.smithi154.stdout:-rw-r--r--. 1 ubuntu ubuntu 782070 May 20 17:47 /home/ubuntu/cephtest/cephadm 2024-05-20T17:47:46.087 DEBUG:teuthology.orchestra.run.smithi204:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240517.203521-squid-debug/d2b507c724ef6c8295ee0a35d185cdab167fd61d/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-20T17:47:47.343 INFO:teuthology.orchestra.run.smithi204.stdout:-rw-r--r--. 1 ubuntu ubuntu 782070 May 20 17:47 /home/ubuntu/cephtest/cephadm 2024-05-20T17:47:47.344 DEBUG:teuthology.orchestra.run.smithi129:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-20T17:47:47.364 DEBUG:teuthology.orchestra.run.smithi154:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-20T17:47:47.383 DEBUG:teuthology.orchestra.run.smithi204:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-20T17:47:47.417 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d on all hosts... 2024-05-20T17:47:47.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d pull 2024-05-20T17:47:47.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d pull 2024-05-20T17:47:47.425 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d pull 2024-05-20T17:47:48.268 INFO:teuthology.orchestra.run.smithi154.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d... 2024-05-20T17:47:48.306 INFO:teuthology.orchestra.run.smithi129.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d... 2024-05-20T17:47:48.389 INFO:teuthology.orchestra.run.smithi204.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d... 2024-05-20T17:48:21.235 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-05-20T17:48:21.235 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_version": "ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev)", 2024-05-20T17:48:21.235 INFO:teuthology.orchestra.run.smithi154.stdout: "image_id": "cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3", 2024-05-20T17:48:21.235 INFO:teuthology.orchestra.run.smithi154.stdout: "repo_digests": [ 2024-05-20T17:48:21.235 INFO:teuthology.orchestra.run.smithi154.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708" 2024-05-20T17:48:21.235 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-05-20T17:48:21.236 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-05-20T17:48:35.396 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-05-20T17:48:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_version": "ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev)", 2024-05-20T17:48:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: "image_id": "cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3", 2024-05-20T17:48:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: "repo_digests": [ 2024-05-20T17:48:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708" 2024-05-20T17:48:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-05-20T17:48:35.397 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-05-20T17:48:40.540 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T17:48:40.540 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev)", 2024-05-20T17:48:40.540 INFO:teuthology.orchestra.run.smithi129.stdout: "image_id": "cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3", 2024-05-20T17:48:40.540 INFO:teuthology.orchestra.run.smithi129.stdout: "repo_digests": [ 2024-05-20T17:48:40.540 INFO:teuthology.orchestra.run.smithi129.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708" 2024-05-20T17:48:40.540 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T17:48:40.540 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T17:48:40.564 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph 2024-05-20T17:48:40.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph 2024-05-20T17:48:40.639 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph 2024-05-20T17:48:40.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /etc/ceph 2024-05-20T17:48:40.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /etc/ceph 2024-05-20T17:48:40.724 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /etc/ceph 2024-05-20T17:48:40.752 INFO:tasks.cephadm:Writing seed config... 2024-05-20T17:48:40.753 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-20T17:48:40.753 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-20T17:48:40.753 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-20T17:48:40.753 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-20T17:48:40.753 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-20T17:48:40.754 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-20T17:48:40.755 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-20T17:48:40.756 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-20T17:48:40.757 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-20T17:48:40.757 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-20T17:48:40.757 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-05-20T17:48:40.757 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:48:40.758 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-20T17:48:40.773 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-20T17:48:40.773 DEBUG:teuthology.orchestra.run.smithi129:mon.a> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.a.service 2024-05-20T17:48:40.815 DEBUG:teuthology.orchestra.run.smithi129:mgr.x> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.x.service 2024-05-20T17:48:40.857 INFO:tasks.cephadm:Bootstrapping... 2024-05-20T17:48:40.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d -v bootstrap --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.129 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-20T17:48:41.047 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-05-20T17:48:41.047 INFO:teuthology.orchestra.run.smithi129.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d', '-v', 'bootstrap', '--fsid', '0f1f0dd2-16d1-11ef-bc9a-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.129', '--skip-admin-label'] 2024-05-20T17:48:41.047 INFO:teuthology.orchestra.run.smithi129.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-20T17:48:41.048 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying podman|docker is present... 2024-05-20T17:48:41.072 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stdout 5.0.2 2024-05-20T17:48:41.072 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying lvm2 is present... 2024-05-20T17:48:41.073 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying time synchronization is in place... 2024-05-20T17:48:41.080 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-20T17:48:41.080 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-20T17:48:41.087 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-20T17:48:41.087 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-05-20T17:48:41.094 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout enabled 2024-05-20T17:48:41.100 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout active 2024-05-20T17:48:41.100 INFO:teuthology.orchestra.run.smithi129.stdout:Unit chronyd.service is enabled and running 2024-05-20T17:48:41.100 INFO:teuthology.orchestra.run.smithi129.stdout:Repeating the final host check... 2024-05-20T17:48:41.123 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stdout 5.0.2 2024-05-20T17:48:41.123 INFO:teuthology.orchestra.run.smithi129.stdout:podman (/bin/podman) version 5.0.2 is present 2024-05-20T17:48:41.123 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl is present 2024-05-20T17:48:41.124 INFO:teuthology.orchestra.run.smithi129.stdout:lvcreate is present 2024-05-20T17:48:41.130 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-20T17:48:41.130 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-20T17:48:41.136 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-20T17:48:41.137 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-05-20T17:48:41.143 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout enabled 2024-05-20T17:48:41.150 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout active 2024-05-20T17:48:41.150 INFO:teuthology.orchestra.run.smithi129.stdout:Unit chronyd.service is enabled and running 2024-05-20T17:48:41.150 INFO:teuthology.orchestra.run.smithi129.stdout:Host looks OK 2024-05-20T17:48:41.150 INFO:teuthology.orchestra.run.smithi129.stdout:Cluster fsid: 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:48:41.150 INFO:teuthology.orchestra.run.smithi129.stdout:Acquiring lock 140335124837760 on /run/cephadm/0f1f0dd2-16d1-11ef-bc9a-c7b262605968.lock 2024-05-20T17:48:41.150 INFO:teuthology.orchestra.run.smithi129.stdout:Lock 140335124837760 acquired on /run/cephadm/0f1f0dd2-16d1-11ef-bc9a-c7b262605968.lock 2024-05-20T17:48:41.151 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying IP 172.21.15.129 port 3300 ... 2024-05-20T17:48:41.151 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying IP 172.21.15.129 port 6789 ... 2024-05-20T17:48:41.151 INFO:teuthology.orchestra.run.smithi129.stdout:Base mon IP(s) is [172.21.15.129:3300, 172.21.15.129:6789], mon addrv is [v2:172.21.15.129:3300,v1:172.21.15.129:6789] 2024-05-20T17:48:41.154 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.129 metric 100 2024-05-20T17:48:41.154 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.129 metric 100 2024-05-20T17:48:41.157 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-20T17:48:41.157 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-05-20T17:48:41.157 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium 2024-05-20T17:48:41.161 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-20T17:48:41.162 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-20T17:48:41.162 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-20T17:48:41.162 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-20T17:48:41.162 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7261/64 scope link 2024-05-20T17:48:41.162 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-20T17:48:41.162 INFO:teuthology.orchestra.run.smithi129.stdout:Mon IP `172.21.15.129` is in CIDR network `172.21.0.0/20` 2024-05-20T17:48:41.162 INFO:teuthology.orchestra.run.smithi129.stdout:Mon IP `172.21.15.129` is in CIDR network `172.21.0.0/20` 2024-05-20T17:48:41.162 INFO:teuthology.orchestra.run.smithi129.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-20T17:48:41.162 INFO:teuthology.orchestra.run.smithi129.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-20T17:48:41.162 INFO:teuthology.orchestra.run.smithi129.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d... 2024-05-20T17:48:41.693 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stdout cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 2024-05-20T17:48:41.694 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d... 2024-05-20T17:48:41.694 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Getting image source signatures 2024-05-20T17:48:41.694 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Copying blob sha256:506d33f63dccdaac2c87f708e3d3ef98b8f2fc00666d50b72896243431ca9bae 2024-05-20T17:48:41.694 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Copying blob sha256:d0e808cd455827ca3108db1b0125dcee9ffc148daa1d7dde963f550bc831d501 2024-05-20T17:48:41.694 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Copying config sha256:cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 2024-05-20T17:48:41.694 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-20T17:48:42.623 INFO:teuthology.orchestra.run.smithi129.stdout:ceph: stdout ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T17:48:42.623 INFO:teuthology.orchestra.run.smithi129.stdout:Ceph version: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T17:48:42.623 INFO:teuthology.orchestra.run.smithi129.stdout:Extracting ceph user uid/gid from container image... 2024-05-20T17:48:43.479 INFO:teuthology.orchestra.run.smithi129.stdout:stat: stdout 167 167 2024-05-20T17:48:43.479 INFO:teuthology.orchestra.run.smithi129.stdout:Creating initial keys... 2024-05-20T17:48:44.369 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph-authtool: stdout AQD7jEtmRKGMLxAAjVHDl3fdTZTdznXaTtxN9w== 2024-05-20T17:48:45.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph-authtool: stdout AQD8jEtmmPycLBAAtk4XLRloRarTGq0/hwSLMw== 2024-05-20T17:48:46.274 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph-authtool: stdout AQD9jEtm1QiuJxAAK6QtHtEJDumDyGE6Vkl4SQ== 2024-05-20T17:48:46.275 INFO:teuthology.orchestra.run.smithi129.stdout:Creating initial monmap... 2024-05-20T17:48:47.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-20T17:48:47.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-20T17:48:47.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:48:47.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-20T17:48:47.106 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool for a [v2:172.21.15.129:3300,v1:172.21.15.129:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-20T17:48:47.106 INFO:teuthology.orchestra.run.smithi129.stdout:setting min_mon_release = quincy 2024-05-20T17:48:47.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: set fsid to 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:48:47.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-20T17:48:47.106 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:48:47.106 INFO:teuthology.orchestra.run.smithi129.stdout:Creating mon... 2024-05-20T17:48:48.062 INFO:teuthology.orchestra.run.smithi129.stdout:create mon.a on 2024-05-20T17:48:48.265 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-20T17:48:48.423 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-20T17:48:48.613 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968.target → /etc/systemd/system/ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968.target. 2024-05-20T17:48:48.613 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968.target → /etc/systemd/system/ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968.target. 2024-05-20T17:48:48.854 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.a 2024-05-20T17:48:48.854 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.a.service: Unit ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.a.service not loaded. 2024-05-20T17:48:49.030 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968.target.wants/ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.a.service → /etc/systemd/system/ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@.service. 2024-05-20T17:48:49.088 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:49 smithi129 systemd[1]: Starting Ceph mon.a for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:48:49.353 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:49 smithi129 podman[31641]: 2024-05-20 17:48:49.253517527 +0000 UTC m=+0.086519738 container create 2965d9d38e4815bcd94cd9f40e17b2e6c010a6df2b0e9d6cccf67bae0ce916f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T17:48:49.353 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:49 smithi129 podman[31641]: 2024-05-20 17:48:49.184458596 +0000 UTC m=+0.017460803 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:48:49.545 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-20T17:48:49.545 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout disabled 2024-05-20T17:48:49.553 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-20T17:48:49.553 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-05-20T17:48:49.553 INFO:teuthology.orchestra.run.smithi129.stdout:firewalld.service is not enabled 2024-05-20T17:48:49.553 INFO:teuthology.orchestra.run.smithi129.stdout:Not possible to enable service . firewalld.service is not available 2024-05-20T17:48:49.554 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mon to start... 2024-05-20T17:48:49.554 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mon... 2024-05-20T17:48:49.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:49 smithi129 podman[31641]: 2024-05-20 17:48:49.447726607 +0000 UTC m=+0.280728805 container init 2965d9d38e4815bcd94cd9f40e17b2e6c010a6df2b0e9d6cccf67bae0ce916f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-20T17:48:49.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:49 smithi129 podman[31641]: 2024-05-20 17:48:49.45096427 +0000 UTC m=+0.283966479 container start 2965d9d38e4815bcd94cd9f40e17b2e6c010a6df2b0e9d6cccf67bae0ce916f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-05-20T17:48:49.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:49 smithi129 ceph-mon[31660]: mkfs 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:48:49.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:49 smithi129 ceph-mon[31660]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-20T17:48:49.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:49 smithi129 bash[31641]: 2965d9d38e4815bcd94cd9f40e17b2e6c010a6df2b0e9d6cccf67bae0ce916f2 2024-05-20T17:48:49.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:49 smithi129 systemd[1]: Started Ceph mon.a for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:48:50.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-20T17:48:50.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: monmap epoch 1 2024-05-20T17:48:50.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:48:50.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: last_changed 2024-05-20T17:48:46.628733+0000 2024-05-20T17:48:50.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: created 2024-05-20T17:48:46.628733+0000 2024-05-20T17:48:50.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: min_mon_release 19 (squid) 2024-05-20T17:48:50.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: election_strategy: 1 2024-05-20T17:48:50.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-05-20T17:48:50.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: fsmap 2024-05-20T17:48:50.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: osdmap e1: 0 total, 0 up, 0 in 2024-05-20T17:48:50.666 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: mgrmap e1: no daemons active 2024-05-20T17:48:50.666 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:50 smithi129 ceph-mon[31660]: from='client.? 172.21.15.129:0/3575595364' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-20T17:48:50.992 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout cluster: 2024-05-20T17:48:50.998 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout id: 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:48:50.998 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-20T17:48:50.998 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:48:50.998 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout services: 2024-05-20T17:48:50.998 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.876699s) 2024-05-20T17:48:50.998 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-20T17:48:50.998 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-20T17:48:50.998 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:48:50.998 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout data: 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout pgs: 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.303+0000 7f4c4649d700 1 Processor -- start 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.303+0000 7f4c4649d700 1 -- start start 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.303+0000 7f4c4649d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c401094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.303+0000 7f4c4649d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4c40109a80 con 0x7f4c401090a0 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.303+0000 7f4c3ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c401094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.304+0000 7f4c3ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c401094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36288/0 (socket says 172.21.15.129:36288) 2024-05-20T17:48:50.999 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.304+0000 7f4c3ffff700 1 -- 172.21.15.129:0/18835883 learned_addr learned my addr 172.21.15.129:0/18835883 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.304+0000 7f4c3ffff700 1 -- 172.21.15.129:0/18835883 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c40109c00 con 0x7f4c401090a0 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.304+0000 7f4c3ffff700 1 --2- 172.21.15.129:0/18835883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c401094b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4c300098a0 tx=0x7f4c30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45252b7602015b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.304+0000 7f4c3f7fe700 1 -- 172.21.15.129:0/18835883 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c30004160 con 0x7f4c401090a0 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.304+0000 7f4c3f7fe700 1 -- 172.21.15.129:0/18835883 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c30004300 con 0x7f4c401090a0 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.304+0000 7f4c3f7fe700 1 -- 172.21.15.129:0/18835883 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c300036a0 con 0x7f4c401090a0 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.305+0000 7f4c4649d700 1 -- 172.21.15.129:0/18835883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 msgr2=0x7f4c401094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.305+0000 7f4c4649d700 1 --2- 172.21.15.129:0/18835883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c401094b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4c300098a0 tx=0x7f4c30009bb0 comp rx=0 tx=0).stop 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.305+0000 7f4c4649d700 1 -- 172.21.15.129:0/18835883 shutdown_connections 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.305+0000 7f4c4649d700 1 --2- 172.21.15.129:0/18835883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c401094b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.305+0000 7f4c4649d700 1 -- 172.21.15.129:0/18835883 >> 172.21.15.129:0/18835883 conn(0x7f4c40102a00 msgr2=0x7f4c40104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.305+0000 7f4c4649d700 1 -- 172.21.15.129:0/18835883 shutdown_connections 2024-05-20T17:48:51.000 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.305+0000 7f4c4649d700 1 -- 172.21.15.129:0/18835883 wait complete. 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.306+0000 7f4c4649d700 1 Processor -- start 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.306+0000 7f4c4649d700 1 -- start start 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.306+0000 7f4c4649d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c40107630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.306+0000 7f4c4649d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4c40108120 con 0x7f4c401090a0 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.306+0000 7f4c3ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c40107630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.306+0000 7f4c3ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c40107630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36292/0 (socket says 172.21.15.129:36292) 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.306+0000 7f4c3ffff700 1 -- 172.21.15.129:0/3575595364 learned_addr learned my addr 172.21.15.129:0/3575595364 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.306+0000 7f4c3ffff700 1 -- 172.21.15.129:0/3575595364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c40107b70 con 0x7f4c401090a0 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.307+0000 7f4c3ffff700 1 --2- 172.21.15.129:0/3575595364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c40107630 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4c30000c00 tx=0x7f4c30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.307+0000 7f4c3dffb700 1 -- 172.21.15.129:0/3575595364 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c30003db0 con 0x7f4c401090a0 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.307+0000 7f4c3dffb700 1 -- 172.21.15.129:0/3575595364 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c30011630 con 0x7f4c401090a0 2024-05-20T17:48:51.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.307+0000 7f4c3dffb700 1 -- 172.21.15.129:0/3575595364 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c30011950 con 0x7f4c401090a0 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.307+0000 7f4c4649d700 1 -- 172.21.15.129:0/3575595364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4019f430 con 0x7f4c401090a0 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.307+0000 7f4c4649d700 1 -- 172.21.15.129:0/3575595364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c4019f8b0 con 0x7f4c401090a0 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.307+0000 7f4c3dffb700 1 -- 172.21.15.129:0/3575595364 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4c30011af0 con 0x7f4c401090a0 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.307+0000 7f4c3dffb700 1 -- 172.21.15.129:0/3575595364 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4c3001c450 con 0x7f4c401090a0 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.308+0000 7f4c4649d700 1 -- 172.21.15.129:0/3575595364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c40063ca0 con 0x7f4c401090a0 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.310+0000 7f4c3dffb700 1 -- 172.21.15.129:0/3575595364 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4c30032020 con 0x7f4c401090a0 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.379+0000 7f4c4649d700 1 -- 172.21.15.129:0/3575595364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f4c4002e060 con 0x7f4c401090a0 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.380+0000 7f4c3dffb700 1 -- 172.21.15.129:0/3575595364 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f4c40109c00 con 0x7f4c401090a0 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.380+0000 7f4c4649d700 1 -- 172.21.15.129:0/3575595364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 msgr2=0x7f4c40107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.380+0000 7f4c4649d700 1 --2- 172.21.15.129:0/3575595364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c40107630 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4c30000c00 tx=0x7f4c30004750 comp rx=0 tx=0).stop 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.380+0000 7f4c4649d700 1 -- 172.21.15.129:0/3575595364 shutdown_connections 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.380+0000 7f4c4649d700 1 --2- 172.21.15.129:0/3575595364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c401090a0 0x7f4c40107630 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:48:51.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.380+0000 7f4c4649d700 1 -- 172.21.15.129:0/3575595364 >> 172.21.15.129:0/3575595364 conn(0x7f4c40102a00 msgr2=0x7f4c4018f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:48:51.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.380+0000 7f4c4649d700 1 -- 172.21.15.129:0/3575595364 shutdown_connections 2024-05-20T17:48:51.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:50.380+0000 7f4c4649d700 1 -- 172.21.15.129:0/3575595364 wait complete. 2024-05-20T17:48:51.003 INFO:teuthology.orchestra.run.smithi129.stdout:mon is available 2024-05-20T17:48:51.003 INFO:teuthology.orchestra.run.smithi129.stdout:Assimilating anything we can from ceph.conf... 2024-05-20T17:48:51.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:51 smithi129 ceph-mon[31660]: from='client.? 172.21.15.129:0/6032936' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-20T17:48:51.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:51 smithi129 ceph-mon[31660]: from='client.? 172.21.15.129:0/6032936' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-20T17:48:52.089 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:48:52.089 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [global] 2024-05-20T17:48:52.089 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout fsid = 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.129:3300,v1:172.21.15.129:6789] 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [osd] 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-20T17:48:52.090 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.446+0000 7ff5317e1700 1 Processor -- start 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.447+0000 7ff5317e1700 1 -- start start 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.447+0000 7ff5317e1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c07cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.447+0000 7ff5317e1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c07d000 con 0x7ff52c07e5a0 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.447+0000 7ff52affd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c07cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.447+0000 7ff52affd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c07cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36302/0 (socket says 172.21.15.129:36302) 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.447+0000 7ff52affd700 1 -- 172.21.15.129:0/4103178023 learned_addr learned my addr 172.21.15.129:0/4103178023 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.447+0000 7ff52affd700 1 -- 172.21.15.129:0/4103178023 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52c07d180 con 0x7ff52c07e5a0 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.447+0000 7ff52affd700 1 --2- 172.21.15.129:0/4103178023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c07cac0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0098a0 tx=0x7ff51c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33105895a00ed364 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:48:52.091 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.448+0000 7ff529ffb700 1 -- 172.21.15.129:0/4103178023 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff51c004160 con 0x7ff52c07e5a0 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.448+0000 7ff529ffb700 1 -- 172.21.15.129:0/4103178023 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff51c004300 con 0x7ff52c07e5a0 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.448+0000 7ff529ffb700 1 -- 172.21.15.129:0/4103178023 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff51c0036a0 con 0x7ff52c07e5a0 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.448+0000 7ff5317e1700 1 -- 172.21.15.129:0/4103178023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 msgr2=0x7ff52c07cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.448+0000 7ff5317e1700 1 --2- 172.21.15.129:0/4103178023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c07cac0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff51c0098a0 tx=0x7ff51c009bb0 comp rx=0 tx=0).stop 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.448+0000 7ff5317e1700 1 -- 172.21.15.129:0/4103178023 shutdown_connections 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.448+0000 7ff5317e1700 1 --2- 172.21.15.129:0/4103178023 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c07cac0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.448+0000 7ff5317e1700 1 -- 172.21.15.129:0/4103178023 >> 172.21.15.129:0/4103178023 conn(0x7ff52c104310 msgr2=0x7ff52c106780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.448+0000 7ff5317e1700 1 -- 172.21.15.129:0/4103178023 shutdown_connections 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.448+0000 7ff5317e1700 1 -- 172.21.15.129:0/4103178023 wait complete. 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.449+0000 7ff5317e1700 1 Processor -- start 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.449+0000 7ff5317e1700 1 -- start start 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.449+0000 7ff5317e1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:48:52.092 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.449+0000 7ff5317e1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff52c07d6b0 con 0x7ff52c07e5a0 2024-05-20T17:48:52.093 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.449+0000 7ff52affd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:48:52.093 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.450+0000 7ff52affd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c19e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36312/0 (socket says 172.21.15.129:36312) 2024-05-20T17:48:52.093 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.450+0000 7ff52affd700 1 -- 172.21.15.129:0/6032936 learned_addr learned my addr 172.21.15.129:0/6032936 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:48:52.093 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.450+0000 7ff52affd700 1 -- 172.21.15.129:0/6032936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52c19eda0 con 0x7ff52c07e5a0 2024-05-20T17:48:52.093 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.450+0000 7ff52affd700 1 --2- 172.21.15.129:0/6032936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c19e860 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff51c000c00 tx=0x7ff51c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:48:52.093 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.450+0000 7ff513fff700 1 -- 172.21.15.129:0/6032936 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff51c003db0 con 0x7ff52c07e5a0 2024-05-20T17:48:52.093 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.450+0000 7ff513fff700 1 -- 172.21.15.129:0/6032936 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff51c011630 con 0x7ff52c07e5a0 2024-05-20T17:48:52.093 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.450+0000 7ff5317e1700 1 -- 172.21.15.129:0/6032936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52c19f030 con 0x7ff52c07e5a0 2024-05-20T17:48:52.093 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.450+0000 7ff513fff700 1 -- 172.21.15.129:0/6032936 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff51c011950 con 0x7ff52c07e5a0 2024-05-20T17:48:52.093 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.450+0000 7ff5317e1700 1 -- 172.21.15.129:0/6032936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52c1a1c40 con 0x7ff52c07e5a0 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.450+0000 7ff513fff700 1 -- 172.21.15.129:0/6032936 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff51c011af0 con 0x7ff52c07e5a0 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.450+0000 7ff513fff700 1 -- 172.21.15.129:0/6032936 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff51c01c450 con 0x7ff52c07e5a0 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.451+0000 7ff5317e1700 1 -- 172.21.15.129:0/6032936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff52c063ca0 con 0x7ff52c07e5a0 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.453+0000 7ff513fff700 1 -- 172.21.15.129:0/6032936 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff51c032020 con 0x7ff52c07e5a0 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.520+0000 7ff5317e1700 1 -- 172.21.15.129:0/6032936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ff52c02df60 con 0x7ff52c07e5a0 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.524+0000 7ff513fff700 1 -- 172.21.15.129:0/6032936 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7ff51c004260 con 0x7ff52c07e5a0 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.525+0000 7ff513fff700 1 -- 172.21.15.129:0/6032936 <== mon.0 v2:172.21.15.129:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff51c01c6a0 con 0x7ff52c07e5a0 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.525+0000 7ff5317e1700 1 -- 172.21.15.129:0/6032936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 msgr2=0x7ff52c19e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.525+0000 7ff5317e1700 1 --2- 172.21.15.129:0/6032936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c19e860 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff51c000c00 tx=0x7ff51c004750 comp rx=0 tx=0).stop 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.525+0000 7ff5317e1700 1 -- 172.21.15.129:0/6032936 shutdown_connections 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.525+0000 7ff5317e1700 1 --2- 172.21.15.129:0/6032936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff52c07e5a0 0x7ff52c19e860 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:48:52.094 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.525+0000 7ff5317e1700 1 -- 172.21.15.129:0/6032936 >> 172.21.15.129:0/6032936 conn(0x7ff52c104310 msgr2=0x7ff52c104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:48:52.095 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.525+0000 7ff5317e1700 1 -- 172.21.15.129:0/6032936 shutdown_connections 2024-05-20T17:48:52.095 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:51.525+0000 7ff5317e1700 1 -- 172.21.15.129:0/6032936 wait complete. 2024-05-20T17:48:52.095 INFO:teuthology.orchestra.run.smithi129.stdout:Generating new minimal ceph.conf... 2024-05-20T17:48:52.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:52 smithi129 ceph-mon[31660]: from='client.? 172.21.15.129:0/4148365710' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:48:53.069 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.465+0000 7fb98448b700 1 Processor -- start 2024-05-20T17:48:53.069 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.465+0000 7fb98448b700 1 -- start start 2024-05-20T17:48:53.069 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.465+0000 7fb98448b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c1090a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:48:53.069 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.465+0000 7fb98448b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c109690 con 0x7fb97c108c90 2024-05-20T17:48:53.069 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.465+0000 7fb982227700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c1090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:48:53.069 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.465+0000 7fb982227700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c1090a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36318/0 (socket says 172.21.15.129:36318) 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.465+0000 7fb982227700 1 -- 172.21.15.129:0/4236165277 learned_addr learned my addr 172.21.15.129:0/4236165277 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.466+0000 7fb982227700 1 -- 172.21.15.129:0/4236165277 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97c109d70 con 0x7fb97c108c90 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.466+0000 7fb982227700 1 --2- 172.21.15.129:0/4236165277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c1090a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb9780098a0 tx=0x7fb978009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ecfd0ed5ffda710 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.466+0000 7fb981225700 1 -- 172.21.15.129:0/4236165277 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb978004160 con 0x7fb97c108c90 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.466+0000 7fb981225700 1 -- 172.21.15.129:0/4236165277 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb978004300 con 0x7fb97c108c90 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.466+0000 7fb981225700 1 -- 172.21.15.129:0/4236165277 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb978011b50 con 0x7fb97c108c90 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.467+0000 7fb98448b700 1 -- 172.21.15.129:0/4236165277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 msgr2=0x7fb97c1090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.467+0000 7fb98448b700 1 --2- 172.21.15.129:0/4236165277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c1090a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb9780098a0 tx=0x7fb978009bb0 comp rx=0 tx=0).stop 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.467+0000 7fb98448b700 1 -- 172.21.15.129:0/4236165277 shutdown_connections 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.467+0000 7fb98448b700 1 --2- 172.21.15.129:0/4236165277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c1090a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.467+0000 7fb98448b700 1 -- 172.21.15.129:0/4236165277 >> 172.21.15.129:0/4236165277 conn(0x7fb97c104310 msgr2=0x7fb97c106780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:48:53.070 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.467+0000 7fb98448b700 1 -- 172.21.15.129:0/4236165277 shutdown_connections 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.467+0000 7fb98448b700 1 -- 172.21.15.129:0/4236165277 wait complete. 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.467+0000 7fb98448b700 1 Processor -- start 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.467+0000 7fb98448b700 1 -- start start 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb98448b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c07e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb98448b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c10d120 con 0x7fb97c108c90 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb982227700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c07e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb982227700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c07e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36334/0 (socket says 172.21.15.129:36334) 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb982227700 1 -- 172.21.15.129:0/4148365710 learned_addr learned my addr 172.21.15.129:0/4148365710 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb982227700 1 -- 172.21.15.129:0/4148365710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97c07e7f0 con 0x7fb97c108c90 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb982227700 1 --2- 172.21.15.129:0/4148365710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c07e2b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb9780046e0 tx=0x7fb978011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb9737fe700 1 -- 172.21.15.129:0/4148365710 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9780039e0 con 0x7fb97c108c90 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb9737fe700 1 -- 172.21.15.129:0/4148365710 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb978003b80 con 0x7fb97c108c90 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb98448b700 1 -- 172.21.15.129:0/4148365710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb97c07ca40 con 0x7fb97c108c90 2024-05-20T17:48:53.071 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb9737fe700 1 -- 172.21.15.129:0/4148365710 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb978024690 con 0x7fb97c108c90 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.468+0000 7fb98448b700 1 -- 172.21.15.129:0/4148365710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb97c07cf20 con 0x7fb97c108c90 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.469+0000 7fb9737fe700 1 -- 172.21.15.129:0/4148365710 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb978022020 con 0x7fb97c108c90 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.469+0000 7fb9737fe700 1 -- 172.21.15.129:0/4148365710 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb978011e00 con 0x7fb97c108c90 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.469+0000 7fb98448b700 1 -- 172.21.15.129:0/4148365710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb97c063ca0 con 0x7fb97c108c90 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.472+0000 7fb9737fe700 1 -- 172.21.15.129:0/4148365710 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb978043dd0 con 0x7fb97c108c90 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.539+0000 7fb98448b700 1 -- 172.21.15.129:0/4148365710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fb97c07d1a0 con 0x7fb97c108c90 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.539+0000 7fb9737fe700 1 -- 172.21.15.129:0/4148365710 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7fb978032020 con 0x7fb97c108c90 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.540+0000 7fb98448b700 1 -- 172.21.15.129:0/4148365710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 msgr2=0x7fb97c07e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.540+0000 7fb98448b700 1 --2- 172.21.15.129:0/4148365710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c07e2b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb9780046e0 tx=0x7fb978011c10 comp rx=0 tx=0).stop 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.540+0000 7fb98448b700 1 -- 172.21.15.129:0/4148365710 shutdown_connections 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.540+0000 7fb98448b700 1 --2- 172.21.15.129:0/4148365710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c108c90 0x7fb97c07e2b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.540+0000 7fb98448b700 1 -- 172.21.15.129:0/4148365710 >> 172.21.15.129:0/4148365710 conn(0x7fb97c104310 msgr2=0x7fb97c104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:48:53.072 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.540+0000 7fb98448b700 1 -- 172.21.15.129:0/4148365710 shutdown_connections 2024-05-20T17:48:53.073 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:52.540+0000 7fb98448b700 1 -- 172.21.15.129:0/4148365710 wait complete. 2024-05-20T17:48:53.073 INFO:teuthology.orchestra.run.smithi129.stdout:Restarting the monitor... 2024-05-20T17:48:53.340 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:53 smithi129 systemd[1]: Stopping Ceph mon.a for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:48:53.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:53 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a[31656]: 2024-05-20T17:48:53.338+0000 7face6c66700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T17:48:53.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:53 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a[31656]: 2024-05-20T17:48:53.338+0000 7face6c66700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-20T17:48:53.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:53 smithi129 podman[31905]: 2024-05-20 17:48:53.564540377 +0000 UTC m=+0.331510971 container died 2965d9d38e4815bcd94cd9f40e17b2e6c010a6df2b0e9d6cccf67bae0ce916f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T17:48:54.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:53 smithi129 podman[31905]: 2024-05-20 17:48:53.90589898 +0000 UTC m=+0.672869571 container cleanup 2965d9d38e4815bcd94cd9f40e17b2e6c010a6df2b0e9d6cccf67bae0ce916f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , ceph=True) 2024-05-20T17:48:54.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:53 smithi129 bash[31905]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a 2024-05-20T17:48:54.516 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 podman[31918]: 2024-05-20 17:48:54.190252849 +0000 UTC m=+0.629785623 container remove 2965d9d38e4815bcd94cd9f40e17b2e6c010a6df2b0e9d6cccf67bae0ce916f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-20T17:48:54.516 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-20T17:48:54.516 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 systemd[1]: Stopped Ceph mon.a for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:48:54.516 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 systemd[1]: Starting Ceph mon.a for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:48:54.767 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 podman[32010]: 2024-05-20 17:48:54.516278441 +0000 UTC m=+0.097582283 container create 532f464357bd499cee617b2abc843bf027ef1bdd71505be37b3d46216e1aa8d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 podman[32010]: 2024-05-20 17:48:54.435599603 +0000 UTC m=+0.016903457 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 podman[32010]: 2024-05-20 17:48:54.718954404 +0000 UTC m=+0.300258267 container init 532f464357bd499cee617b2abc843bf027ef1bdd71505be37b3d46216e1aa8d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 podman[32010]: 2024-05-20 17:48:54.722105569 +0000 UTC m=+0.303409418 container start 532f464357bd499cee617b2abc843bf027ef1bdd71505be37b3d46216e1aa8d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c) 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: set uid:gid to 167:167 (ceph:ceph) 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev), process ceph-mon, pid 2 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: pidfile_write: ignore empty --pid-file 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: load: jerasure load: lrc 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: RocksDB version: 7.9.2 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Git sha 0 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Compile date 2024-05-17 21:21:31 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: DB SUMMARY 2024-05-20T17:48:54.768 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: DB Session ID: 7BFSLJ2VP5WEG2F78A7S 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: CURRENT file: CURRENT 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: IDENTITY file: IDENTITY 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104730 ; 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.error_if_exists: 0 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.create_if_missing: 0 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.paranoid_checks: 1 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.env: 0x55a34a5d24a0 2024-05-20T17:48:54.769 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.fs: PosixFileSystem 2024-05-20T17:48:54.770 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.info_log: 0x55a34c71bd40 2024-05-20T17:48:54.770 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_file_opening_threads: 16 2024-05-20T17:48:54.770 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.statistics: (nil) 2024-05-20T17:48:54.770 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.use_fsync: 0 2024-05-20T17:48:54.770 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_log_file_size: 0 2024-05-20T17:48:54.770 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-20T17:48:54.770 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-20T17:48:54.770 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.keep_log_file_num: 1000 2024-05-20T17:48:54.770 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.recycle_log_file_num: 0 2024-05-20T17:48:54.770 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.allow_fallocate: 1 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.allow_mmap_reads: 0 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.allow_mmap_writes: 0 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.use_direct_reads: 0 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.create_missing_column_families: 0 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.db_log_dir: 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.wal_dir: 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-20T17:48:54.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.advise_random_on_open: 1 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.db_write_buffer_size: 0 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.write_buffer_manager: 0x55a34c6ef2c0 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.rate_limiter: (nil) 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.wal_recovery_mode: 2 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.enable_thread_tracking: 0 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.enable_pipelined_write: 0 2024-05-20T17:48:54.772 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.unordered_write: 0 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.row_cache: None 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.wal_filter: None 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.allow_ingest_behind: 0 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.two_write_queues: 0 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.manual_wal_flush: 0 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.wal_compression: 0 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.atomic_flush: 0 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-20T17:48:54.773 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.log_readahead_size: 0 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.best_efforts_recovery: 0 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.allow_data_in_errors: 0 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.db_host_id: __hostname__ 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_background_jobs: 2 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_background_compactions: -1 2024-05-20T17:48:54.774 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_subcompactions: 1 2024-05-20T17:48:54.775 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-20T17:48:54.775 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-20T17:48:54.775 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-20T17:48:54.775 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_total_wal_size: 0 2024-05-20T17:48:54.775 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-20T17:48:54.775 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-20T17:48:54.775 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-20T17:48:54.775 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-20T17:48:54.775 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_open_files: -1 2024-05-20T17:48:54.776 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bytes_per_sync: 0 2024-05-20T17:48:54.776 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-20T17:48:54.776 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_readahead_size: 0 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_background_flushes: -1 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Compression algorithms supported: 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: kZSTD supported: 0 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: kXpressCompression supported: 0 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: kLZ4HCCompression supported: 1 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: kZlibCompression supported: 1 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: kSnappyCompression supported: 1 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: kLZ4Compression supported: 1 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: kBZip2Compression supported: 0 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-20T17:48:54.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-20T17:48:54.778 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-20T17:48:54.778 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-20T17:48:54.778 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-20T17:48:54.778 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.merge_operator: 2024-05-20T17:48:54.778 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_filter: None 2024-05-20T17:48:54.778 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_filter_factory: None 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.sst_partitioner_factory: None 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a34c71be40) 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout: cache_index_and_filter_blocks: 1 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout: pin_top_level_index_and_filter: 1 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout: index_type: 0 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout: data_block_index_type: 0 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout: index_shortening: 1 2024-05-20T17:48:54.779 INFO:journalctl@ceph.mon.a.smithi129.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: checksum: 4 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: no_block_cache: 0 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_cache: 0x55a34c71f090 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_cache_name: BinnedLRUCache 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_cache_options: 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: capacity : 536870912 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: num_shard_bits : 4 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: strict_capacity_limit : 0 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: high_pri_pool_ratio: 0.000 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_cache_compressed: (nil) 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: persistent_cache: (nil) 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_size: 4096 2024-05-20T17:48:54.780 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_size_deviation: 10 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_restart_interval: 16 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: index_block_restart_interval: 1 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: metadata_block_size: 4096 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: partition_filters: 0 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: use_delta_encoding: 1 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: filter_policy: bloomfilter 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: whole_key_filtering: 1 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: verify_compression: 0 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: read_amp_bytes_per_bit: 0 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: format_version: 5 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: enable_index_compression: 1 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_align: 0 2024-05-20T17:48:54.781 INFO:journalctl@ceph.mon.a.smithi129.stdout: max_auto_readahead_size: 262144 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout: prepopulate_block_cache: 0 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout: initial_auto_readahead_size: 8192 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout: num_file_reads_for_auto_readahead: 2 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.write_buffer_size: 33554432 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_write_buffer_number: 2 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compression: NoCompression 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bottommost_compression: Disabled 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.prefix_extractor: nullptr 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.num_levels: 7 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-20T17:48:54.782 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compression_opts.level: 32767 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compression_opts.strategy: 0 2024-05-20T17:48:54.783 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compression_opts.enabled: false 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.target_file_size_base: 67108864 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-20T17:48:54.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.arena_block_size: 1048576 2024-05-20T17:48:54.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.disable_auto_compactions: 0 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-20T17:48:54.786 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.table_properties_collectors: 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.inplace_update_support: 0 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.bloom_locality: 0 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.max_successive_merges: 0 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.paranoid_file_checks: 0 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.force_consistency_checks: 1 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.report_bg_io_stats: 0 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.ttl: 2592000 2024-05-20T17:48:54.787 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.enable_blob_files: false 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.min_blob_size: 0 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.blob_file_size: 268435456 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.blob_file_starting_level: 0 2024-05-20T17:48:54.788 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1fbcad2e-ca76-4768-85ee-0ab1b7202bbd 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716227334762597, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716227334763783, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99819, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 97731, "index_size": 306, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14165, "raw_average_key_size": 49, "raw_value_size": 90070, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716227334, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1fbcad2e-ca76-4768-85ee-0ab1b7202bbd", "db_session_id": "7BFSLJ2VP5WEG2F78A7S", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716227334763865, "job": 1, "event": "recovery_finished"} 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a34c705c00 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: DB pointer 0x55a3514f2000 2024-05-20T17:48:54.789 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout: ** DB Stats ** 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout: ** Compaction Stats [default] ** 2024-05-20T17:48:54.790 INFO:journalctl@ceph.mon.a.smithi129.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: L0 2/0 99.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: Sum 2/0 99.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: ** Compaction Stats [default] ** 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-05-20T17:48:54.791 INFO:journalctl@ceph.mon.a.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: Cumulative compaction: 0.00 GB write, 29.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: Interval compaction: 0.00 GB write, 29.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: Block cache BinnedLRUCache@0x55a34c71f090#2 capacity: 512.00 MB usage: 1.53 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.03 KB,0.000196695%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-20T17:48:54.792 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: starting mon.a rank 0 at public addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] at bind addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: mon.a@-1(???) e1 preinit fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: mon.a@-1(???).mds e1 new map 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: mon.a@-1(???).mds e1 print_map 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout: e1 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout: legacy client fscid: -1 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout: No filesystems configured 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-20T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-20T17:48:54.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-20T17:48:54.798 INFO:teuthology.orchestra.run.smithi129.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-20T17:48:55.023 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 bash[32010]: 532f464357bd499cee617b2abc843bf027ef1bdd71505be37b3d46216e1aa8d0 2024-05-20T17:48:55.023 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 systemd[1]: Started Ceph mon.a for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:48:55.023 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-20T17:48:55.024 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: monmap epoch 1 2024-05-20T17:48:55.024 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:48:55.024 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: last_changed 2024-05-20T17:48:46.628733+0000 2024-05-20T17:48:55.024 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: created 2024-05-20T17:48:46.628733+0000 2024-05-20T17:48:55.024 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: min_mon_release 19 (squid) 2024-05-20T17:48:55.024 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: election_strategy: 1 2024-05-20T17:48:55.024 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-05-20T17:48:55.024 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: fsmap 2024-05-20T17:48:55.024 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: osdmap e1: 0 total, 0 up, 0 in 2024-05-20T17:48:55.024 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:54 smithi129 ceph-mon[32029]: mgrmap e1: no daemons active 2024-05-20T17:48:55.867 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.235+0000 7f888aef6700 1 Processor -- start 2024-05-20T17:48:55.868 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.235+0000 7f888aef6700 1 -- start start 2024-05-20T17:48:55.868 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.235+0000 7f888aef6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f8884107f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:48:55.868 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.235+0000 7f888aef6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8884108540 con 0x7f8884107b40 2024-05-20T17:48:55.868 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.236+0000 7f8888c92700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f8884107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:48:55.868 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.236+0000 7f8888c92700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f8884107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36346/0 (socket says 172.21.15.129:36346) 2024-05-20T17:48:55.868 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.236+0000 7f8888c92700 1 -- 172.21.15.129:0/977587373 learned_addr learned my addr 172.21.15.129:0/977587373 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:48:55.868 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.236+0000 7f8888c92700 1 -- 172.21.15.129:0/977587373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8884108c70 con 0x7f8884107b40 2024-05-20T17:48:55.868 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.236+0000 7f8888c92700 1 --2- 172.21.15.129:0/977587373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f8884107f50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f88780098a0 tx=0x7f8878009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ec1ade7200ff93 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:48:55.869 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.236+0000 7f88837fe700 1 -- 172.21.15.129:0/977587373 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8878004160 con 0x7f8884107b40 2024-05-20T17:48:55.869 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.236+0000 7f88837fe700 1 -- 172.21.15.129:0/977587373 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8878004300 con 0x7f8884107b40 2024-05-20T17:48:55.869 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.236+0000 7f88837fe700 1 -- 172.21.15.129:0/977587373 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8878011b50 con 0x7f8884107b40 2024-05-20T17:48:55.869 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.237+0000 7f888aef6700 1 -- 172.21.15.129:0/977587373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 msgr2=0x7f8884107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:48:55.869 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.237+0000 7f888aef6700 1 --2- 172.21.15.129:0/977587373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f8884107f50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f88780098a0 tx=0x7f8878009bb0 comp rx=0 tx=0).stop 2024-05-20T17:48:55.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.237+0000 7f888aef6700 1 -- 172.21.15.129:0/977587373 shutdown_connections 2024-05-20T17:48:55.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.237+0000 7f888aef6700 1 --2- 172.21.15.129:0/977587373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f8884107f50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:48:55.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.237+0000 7f888aef6700 1 -- 172.21.15.129:0/977587373 >> 172.21.15.129:0/977587373 conn(0x7f8884103240 msgr2=0x7f8884105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:48:55.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.237+0000 7f888aef6700 1 -- 172.21.15.129:0/977587373 shutdown_connections 2024-05-20T17:48:55.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.237+0000 7f888aef6700 1 -- 172.21.15.129:0/977587373 wait complete. 2024-05-20T17:48:55.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.238+0000 7f888aef6700 1 Processor -- start 2024-05-20T17:48:55.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.238+0000 7f888aef6700 1 -- start start 2024-05-20T17:48:55.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.238+0000 7f888aef6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f88841063a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:48:55.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.238+0000 7f888aef6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f888410c020 con 0x7f8884107b40 2024-05-20T17:48:55.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.238+0000 7f8888c92700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f88841063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:48:55.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.238+0000 7f8888c92700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f88841063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36358/0 (socket says 172.21.15.129:36358) 2024-05-20T17:48:55.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.238+0000 7f8888c92700 1 -- 172.21.15.129:0/2992320696 learned_addr learned my addr 172.21.15.129:0/2992320696 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:48:55.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.238+0000 7f8888c92700 1 -- 172.21.15.129:0/2992320696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88841068e0 con 0x7f8884107b40 2024-05-20T17:48:55.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.239+0000 7f8888c92700 1 --2- 172.21.15.129:0/2992320696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f88841063a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f88780046e0 tx=0x7f8878011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:48:55.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.239+0000 7f8881ffb700 1 -- 172.21.15.129:0/2992320696 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88780039e0 con 0x7f8884107b40 2024-05-20T17:48:55.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.239+0000 7f8881ffb700 1 -- 172.21.15.129:0/2992320696 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8878003b80 con 0x7f8884107b40 2024-05-20T17:48:55.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.239+0000 7f8881ffb700 1 -- 172.21.15.129:0/2992320696 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8878024690 con 0x7f8884107b40 2024-05-20T17:48:55.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.239+0000 7f888aef6700 1 -- 172.21.15.129:0/2992320696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8884104b30 con 0x7f8884107b40 2024-05-20T17:48:55.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.239+0000 7f888aef6700 1 -- 172.21.15.129:0/2992320696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8884105010 con 0x7f8884107b40 2024-05-20T17:48:55.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.239+0000 7f8881ffb700 1 -- 172.21.15.129:0/2992320696 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8878022020 con 0x7f8884107b40 2024-05-20T17:48:55.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.239+0000 7f8881ffb700 1 -- 172.21.15.129:0/2992320696 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8878011e00 con 0x7f8884107b40 2024-05-20T17:48:55.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.240+0000 7f888aef6700 1 -- 172.21.15.129:0/2992320696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8884063ca0 con 0x7f8884107b40 2024-05-20T17:48:55.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.242+0000 7f8881ffb700 1 -- 172.21.15.129:0/2992320696 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8878003d30 con 0x7f8884107b40 2024-05-20T17:48:55.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.308+0000 7f888aef6700 1 -- 172.21.15.129:0/2992320696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f8884196000 con 0x7f8884107b40 2024-05-20T17:48:55.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.313+0000 7f8881ffb700 1 -- 172.21.15.129:0/2992320696 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f887801a9f0 con 0x7f8884107b40 2024-05-20T17:48:55.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.313+0000 7f888aef6700 1 -- 172.21.15.129:0/2992320696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 msgr2=0x7f88841063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:48:55.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.313+0000 7f888aef6700 1 --2- 172.21.15.129:0/2992320696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f88841063a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f88780046e0 tx=0x7f8878011c10 comp rx=0 tx=0).stop 2024-05-20T17:48:55.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.313+0000 7f888aef6700 1 -- 172.21.15.129:0/2992320696 shutdown_connections 2024-05-20T17:48:55.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.313+0000 7f888aef6700 1 --2- 172.21.15.129:0/2992320696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8884107b40 0x7f88841063a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:48:55.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.313+0000 7f888aef6700 1 -- 172.21.15.129:0/2992320696 >> 172.21.15.129:0/2992320696 conn(0x7f8884103240 msgr2=0x7f88841934b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:48:55.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.314+0000 7f888aef6700 1 -- 172.21.15.129:0/2992320696 shutdown_connections 2024-05-20T17:48:55.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:55.314+0000 7f888aef6700 1 -- 172.21.15.129:0/2992320696 wait complete. 2024-05-20T17:48:55.875 INFO:teuthology.orchestra.run.smithi129.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-20T17:48:55.892 INFO:teuthology.orchestra.run.smithi129.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-20T17:48:55.893 INFO:teuthology.orchestra.run.smithi129.stdout:Creating mgr... 2024-05-20T17:48:55.893 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-20T17:48:55.893 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-20T17:48:56.133 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.x 2024-05-20T17:48:56.133 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.x.service: Unit ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.x.service not loaded. 2024-05-20T17:48:56.281 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968.target.wants/ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@.service. 2024-05-20T17:48:56.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:56 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2992320696' entity='client.admin' 2024-05-20T17:48:56.781 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:56 smithi129 podman[32239]: 2024-05-20 17:48:56.716793593 +0000 UTC m=+0.314699734 container start 6908fde9dc21654a293de409c9376bca8ba2c350f4f114c0f06a584285cfb6c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x, org.label-schema.build-date=20240514, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c) 2024-05-20T17:48:56.781 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:56 smithi129 bash[32239]: 6908fde9dc21654a293de409c9376bca8ba2c350f4f114c0f06a584285cfb6c9 2024-05-20T17:48:56.795 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-20T17:48:56.795 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout disabled 2024-05-20T17:48:56.802 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-20T17:48:56.802 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-05-20T17:48:56.802 INFO:teuthology.orchestra.run.smithi129.stdout:firewalld.service is not enabled 2024-05-20T17:48:56.802 INFO:teuthology.orchestra.run.smithi129.stdout:Not possible to enable service . firewalld.service is not available 2024-05-20T17:48:56.811 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-20T17:48:56.811 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout disabled 2024-05-20T17:48:56.818 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-20T17:48:56.818 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-05-20T17:48:56.819 INFO:teuthology.orchestra.run.smithi129.stdout:firewalld.service is not enabled 2024-05-20T17:48:56.819 INFO:teuthology.orchestra.run.smithi129.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-20T17:48:56.819 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mgr to start... 2024-05-20T17:48:56.819 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mgr... 2024-05-20T17:48:57.061 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:56 smithi129 systemd[1]: Started Ceph mgr.x for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:48:57.061 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:56 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:56.885+0000 7fa1a5ee4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-20T17:48:57.376 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:57 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:57.075+0000 7fa1a5ee4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:48:57.376 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:57 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:57.179+0000 7fa1a5ee4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-20T17:48:57.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:48:57 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/313667798' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-20T17:48:57.665 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:57 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:57.507+0000 7fa1a5ee4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-20T17:48:57.665 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:57 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:57.593+0000 7fa1a5ee4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-20T17:48:57.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:48:57.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-05-20T17:48:57.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsid": "0f1f0dd2-16d1-11ef-bc9a-c7b262605968", 2024-05-20T17:48:57.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "health": { 2024-05-20T17:48:57.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-20T17:48:57.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-20T17:48:57.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-20T17:48:57.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:48:57.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-20T17:48:57.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-20T17:48:57.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 0 2024-05-20T17:48:57.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-05-20T17:48:57.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-20T17:48:57.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "a" 2024-05-20T17:48:57.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-05-20T17:48:57.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-20T17:48:57.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-20T17:48:57.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:48:57.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-20T17:48:57.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-20T17:48:57.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:48:57.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-20T17:48:57.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:48:57.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-20T17:48:57.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-20T17:48:57.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-20T17:48:57.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-20T17:48:57.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "restful" 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modified": "2024-05-20T17:48:49.505690+0000", 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-20T17:48:57.981 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-05-20T17:48:57.982 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.295+0000 7f94c1bb8700 1 Processor -- start 2024-05-20T17:48:57.982 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.295+0000 7f94c1bb8700 1 -- start start 2024-05-20T17:48:57.982 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.295+0000 7f94c1bb8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc107f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:48:57.982 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.295+0000 7f94c1bb8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc108510 con 0x7f94bc107b10 2024-05-20T17:48:57.982 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.296+0000 7f94bb7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:48:57.982 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.296+0000 7f94bb7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc107f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36384/0 (socket says 172.21.15.129:36384) 2024-05-20T17:48:57.982 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.296+0000 7f94bb7fe700 1 -- 172.21.15.129:0/2857983066 learned_addr learned my addr 172.21.15.129:0/2857983066 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:48:57.982 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.296+0000 7f94bb7fe700 1 -- 172.21.15.129:0/2857983066 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc108c40 con 0x7f94bc107b10 2024-05-20T17:48:57.982 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.296+0000 7f94bb7fe700 1 --2- 172.21.15.129:0/2857983066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc107f20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0098a0 tx=0x7f94ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dda27a4e3599707 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:48:57.982 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.296+0000 7f94ba7fc700 1 -- 172.21.15.129:0/2857983066 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94ac004160 con 0x7f94bc107b10 2024-05-20T17:48:57.982 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.296+0000 7f94ba7fc700 1 -- 172.21.15.129:0/2857983066 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f94ac004300 con 0x7f94bc107b10 2024-05-20T17:48:57.983 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.297+0000 7f94ba7fc700 1 -- 172.21.15.129:0/2857983066 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94ac011b50 con 0x7f94bc107b10 2024-05-20T17:48:57.983 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.297+0000 7f94c1bb8700 1 -- 172.21.15.129:0/2857983066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 msgr2=0x7f94bc107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:48:57.983 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.297+0000 7f94c1bb8700 1 --2- 172.21.15.129:0/2857983066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc107f20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0098a0 tx=0x7f94ac009bb0 comp rx=0 tx=0).stop 2024-05-20T17:48:57.983 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.297+0000 7f94c1bb8700 1 -- 172.21.15.129:0/2857983066 shutdown_connections 2024-05-20T17:48:57.983 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.297+0000 7f94c1bb8700 1 --2- 172.21.15.129:0/2857983066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc107f20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:48:57.983 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.297+0000 7f94c1bb8700 1 -- 172.21.15.129:0/2857983066 >> 172.21.15.129:0/2857983066 conn(0x7f94bc1031d0 msgr2=0x7f94bc105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:48:57.983 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.297+0000 7f94c1bb8700 1 -- 172.21.15.129:0/2857983066 shutdown_connections 2024-05-20T17:48:57.983 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.298+0000 7f94c1bb8700 1 -- 172.21.15.129:0/2857983066 wait complete. 2024-05-20T17:48:57.983 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.298+0000 7f94c1bb8700 1 Processor -- start 2024-05-20T17:48:57.983 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.298+0000 7f94c1bb8700 1 -- start start 2024-05-20T17:48:57.983 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94c1bb8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:48:57.984 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94c1bb8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc10bff0 con 0x7f94bc107b10 2024-05-20T17:48:57.984 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94bb7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:48:57.984 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94bb7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36392/0 (socket says 172.21.15.129:36392) 2024-05-20T17:48:57.984 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94bb7fe700 1 -- 172.21.15.129:0/313667798 learned_addr learned my addr 172.21.15.129:0/313667798 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:48:57.984 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94bb7fe700 1 -- 172.21.15.129:0/313667798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc19d4e0 con 0x7f94bc107b10 2024-05-20T17:48:57.984 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94bb7fe700 1 --2- 172.21.15.129:0/313667798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc19cfa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0046e0 tx=0x7f94ac011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:48:57.984 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94b8ff9700 1 -- 172.21.15.129:0/313667798 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94ac0039e0 con 0x7f94bc107b10 2024-05-20T17:48:57.984 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94c1bb8700 1 -- 172.21.15.129:0/313667798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94bc19d770 con 0x7f94bc107b10 2024-05-20T17:48:57.984 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94b8ff9700 1 -- 172.21.15.129:0/313667798 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f94ac003b80 con 0x7f94bc107b10 2024-05-20T17:48:57.984 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94b8ff9700 1 -- 172.21.15.129:0/313667798 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94ac024690 con 0x7f94bc107b10 2024-05-20T17:48:57.984 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.299+0000 7f94c1bb8700 1 -- 172.21.15.129:0/313667798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc1a0380 con 0x7f94bc107b10 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.300+0000 7f94b8ff9700 1 -- 172.21.15.129:0/313667798 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f94ac022020 con 0x7f94bc107b10 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.300+0000 7f94b8ff9700 1 -- 172.21.15.129:0/313667798 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f94ac011e00 con 0x7f94bc107b10 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.300+0000 7f94c1bb8700 1 -- 172.21.15.129:0/313667798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94bc063ca0 con 0x7f94bc107b10 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.303+0000 7f94b8ff9700 1 -- 172.21.15.129:0/313667798 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f94ac003d30 con 0x7f94bc107b10 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.375+0000 7f94c1bb8700 1 -- 172.21.15.129:0/313667798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f94bc02df60 con 0x7f94bc107b10 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.375+0000 7f94b8ff9700 1 -- 172.21.15.129:0/313667798 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f94ac01a9f0 con 0x7f94bc107b10 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.376+0000 7f94c1bb8700 1 -- 172.21.15.129:0/313667798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 msgr2=0x7f94bc19cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.376+0000 7f94c1bb8700 1 --2- 172.21.15.129:0/313667798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc19cfa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0046e0 tx=0x7f94ac011c10 comp rx=0 tx=0).stop 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.376+0000 7f94c1bb8700 1 -- 172.21.15.129:0/313667798 shutdown_connections 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.376+0000 7f94c1bb8700 1 --2- 172.21.15.129:0/313667798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94bc107b10 0x7f94bc19cfa0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.376+0000 7f94c1bb8700 1 -- 172.21.15.129:0/313667798 >> 172.21.15.129:0/313667798 conn(0x7f94bc1031d0 msgr2=0x7f94bc193530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:48:57.985 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.377+0000 7f94c1bb8700 1 -- 172.21.15.129:0/313667798 shutdown_connections 2024-05-20T17:48:57.986 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:48:57.377+0000 7f94c1bb8700 1 -- 172.21.15.129:0/313667798 wait complete. 2024-05-20T17:48:57.986 INFO:teuthology.orchestra.run.smithi129.stdout:mgr not available, waiting (1/15)... 2024-05-20T17:48:59.113 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:58 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:58.788+0000 7fa1a5ee4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-20T17:48:59.113 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:58 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:58.956+0000 7fa1a5ee4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-20T17:48:59.414 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:59 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:59.131+0000 7fa1a5ee4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-20T17:48:59.718 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:59 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:59.418+0000 7fa1a5ee4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-20T17:48:59.718 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:59 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:59.509+0000 7fa1a5ee4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-20T17:49:00.095 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:59 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:59.717+0000 7fa1a5ee4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-20T17:49:00.095 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:48:59 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:48:59.912+0000 7fa1a5ee4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:49:00.914 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:00 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:00.538+0000 7fa1a5ee4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-20T17:49:01.269 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:00 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:00.965+0000 7fa1a5ee4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-20T17:49:01.269 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:01 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:01.063+0000 7fa1a5ee4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-20T17:49:01.269 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:01 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:01.268+0000 7fa1a5ee4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-20T17:49:02.136 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:01 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:01.798+0000 7fa1a5ee4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-20T17:49:02.137 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:01 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:01.891+0000 7fa1a5ee4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-20T17:49:02.414 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:02 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:02.135+0000 7fa1a5ee4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-20T17:49:02.415 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:02 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:02.271+0000 7fa1a5ee4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-20T17:49:02.415 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:02 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:02.366+0000 7fa1a5ee4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-20T17:49:02.799 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:02 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:02.486+0000 7fa1a5ee4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-20T17:49:02.799 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:02 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:02.798+0000 7fa1a5ee4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-20T17:49:03.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:03 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3794882659' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-20T17:49:03.876 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:03 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:03.438+0000 7fa1a5ee4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-20T17:49:03.876 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:03 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:03.665+0000 7fa1a5ee4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-20T17:49:04.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: Activating manager daemon x 2024-05-20T17:49:04.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: mgrmap e2: x(active, starting, since 0.00346198s) 2024-05-20T17:49:04.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: from='mgr.14100 172.21.15.129:0/490069625' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-20T17:49:04.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: from='mgr.14100 172.21.15.129:0/490069625' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-20T17:49:04.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: from='mgr.14100 172.21.15.129:0/490069625' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-20T17:49:04.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: from='mgr.14100 172.21.15.129:0/490069625' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:49:04.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: from='mgr.14100 172.21.15.129:0/490069625' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-20T17:49:04.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: Manager daemon x is now available 2024-05-20T17:49:04.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: from='mgr.14100 172.21.15.129:0/490069625' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-20T17:49:04.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: from='mgr.14100 172.21.15.129:0/490069625' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-20T17:49:04.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: from='mgr.14100 172.21.15.129:0/490069625' entity='mgr.x' 2024-05-20T17:49:04.666 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: from='mgr.14100 172.21.15.129:0/490069625' entity='mgr.x' 2024-05-20T17:49:04.666 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:04 smithi129 ceph-mon[32029]: from='mgr.14100 172.21.15.129:0/490069625' entity='mgr.x' 2024-05-20T17:49:06.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:05 smithi129 ceph-mon[32029]: mgrmap e3: x(active, since 1.00773s) 2024-05-20T17:49:07.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:06 smithi129 ceph-mon[32029]: mgrmap e4: x(active, since 2s) 2024-05-20T17:49:09.515 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:49:09.515 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-05-20T17:49:09.515 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsid": "0f1f0dd2-16d1-11ef-bc9a-c7b262605968", 2024-05-20T17:49:09.515 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "health": { 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 0 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "a" 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-05-20T17:49:09.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-20T17:49:09.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:49:09.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-20T17:49:09.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-20T17:49:09.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:09.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-20T17:49:09.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:49:09.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-20T17:49:09.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-20T17:49:09.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-20T17:49:09.525 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-20T17:49:09.525 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-20T17:49:09.525 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-20T17:49:09.525 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:09.525 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-20T17:49:09.525 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-20T17:49:09.525 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-20T17:49:09.525 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-20T17:49:09.525 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-20T17:49:09.525 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-20T17:49:09.526 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "restful" 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modified": "2024-05-20T17:48:49.505690+0000", 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-20T17:49:09.527 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.106+0000 7f12c5fdb700 1 Processor -- start 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.106+0000 7f12c5fdb700 1 -- start start 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.107+0000 7f12c5fdb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c010b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.107+0000 7f12c5fdb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f12c010bd50 con 0x7f12c007bc80 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.107+0000 7f12bf7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c010b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.107+0000 7f12bf7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c010b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54152/0 (socket says 172.21.15.129:54152) 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.107+0000 7f12bf7fe700 1 -- 172.21.15.129:0/1340270577 learned_addr learned my addr 172.21.15.129:0/1340270577 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.107+0000 7f12bf7fe700 1 -- 172.21.15.129:0/1340270577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c010bed0 con 0x7f12c007bc80 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.107+0000 7f12bf7fe700 1 --2- 172.21.15.129:0/1340270577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c010b740 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f12b00098a0 tx=0x7f12b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fba51eda3c8eb68 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.107+0000 7f12be7fc700 1 -- 172.21.15.129:0/1340270577 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b0004160 con 0x7f12c007bc80 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.108+0000 7f12be7fc700 1 -- 172.21.15.129:0/1340270577 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f12b0004300 con 0x7f12c007bc80 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.108+0000 7f12be7fc700 1 -- 172.21.15.129:0/1340270577 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b0011b50 con 0x7f12c007bc80 2024-05-20T17:49:09.528 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.108+0000 7f12c5fdb700 1 -- 172.21.15.129:0/1340270577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 msgr2=0x7f12c010b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.108+0000 7f12c5fdb700 1 --2- 172.21.15.129:0/1340270577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c010b740 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f12b00098a0 tx=0x7f12b0009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.108+0000 7f12c5fdb700 1 -- 172.21.15.129:0/1340270577 shutdown_connections 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.108+0000 7f12c5fdb700 1 --2- 172.21.15.129:0/1340270577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c010b740 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.108+0000 7f12c5fdb700 1 -- 172.21.15.129:0/1340270577 >> 172.21.15.129:0/1340270577 conn(0x7f12c01031d0 msgr2=0x7f12c0105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.108+0000 7f12c5fdb700 1 -- 172.21.15.129:0/1340270577 shutdown_connections 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.108+0000 7f12c5fdb700 1 -- 172.21.15.129:0/1340270577 wait complete. 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.109+0000 7f12c5fdb700 1 Processor -- start 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.110+0000 7f12c5fdb700 1 -- start start 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.110+0000 7f12c5fdb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.110+0000 7f12c5fdb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f12c010cfb0 con 0x7f12c007bc80 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.110+0000 7f12bf7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.110+0000 7f12bf7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c019cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54154/0 (socket says 172.21.15.129:54154) 2024-05-20T17:49:09.529 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.110+0000 7f12bf7fe700 1 -- 172.21.15.129:0/3794882659 learned_addr learned my addr 172.21.15.129:0/3794882659 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.111+0000 7f12bf7fe700 1 -- 172.21.15.129:0/3794882659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c019d430 con 0x7f12c007bc80 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.111+0000 7f12bf7fe700 1 --2- 172.21.15.129:0/3794882659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c019cef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009fb0 tx=0x7f12b0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.111+0000 7f12bcff9700 1 -- 172.21.15.129:0/3794882659 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b00038b0 con 0x7f12c007bc80 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.111+0000 7f12bcff9700 1 -- 172.21.15.129:0/3794882659 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f12b0011e50 con 0x7f12c007bc80 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.111+0000 7f12c5fdb700 1 -- 172.21.15.129:0/3794882659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c019d6c0 con 0x7f12c007bc80 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.111+0000 7f12bcff9700 1 -- 172.21.15.129:0/3794882659 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b0019550 con 0x7f12c007bc80 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.111+0000 7f12c5fdb700 1 -- 172.21.15.129:0/3794882659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c01a02d0 con 0x7f12c007bc80 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.111+0000 7f12bcff9700 1 -- 172.21.15.129:0/3794882659 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f12b0019b90 con 0x7f12c007bc80 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.111+0000 7f12bcff9700 1 -- 172.21.15.129:0/3794882659 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f12b0024730 con 0x7f12c007bc80 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.112+0000 7f12c5fdb700 1 -- 172.21.15.129:0/3794882659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c0063ca0 con 0x7f12c007bc80 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.116+0000 7f12bcff9700 1 -- 172.21.15.129:0/3794882659 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f12b0003a50 con 0x7f12c007bc80 2024-05-20T17:49:09.530 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.190+0000 7f12c5fdb700 1 -- 172.21.15.129:0/3794882659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f12c01a04a0 con 0x7f12c007bc80 2024-05-20T17:49:09.531 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.191+0000 7f12bcff9700 1 -- 172.21.15.129:0/3794882659 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f12b00249d0 con 0x7f12c007bc80 2024-05-20T17:49:09.531 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.192+0000 7f12c5fdb700 1 -- 172.21.15.129:0/3794882659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 msgr2=0x7f12c019cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:09.531 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.192+0000 7f12c5fdb700 1 --2- 172.21.15.129:0/3794882659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c019cef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009fb0 tx=0x7f12b0011bf0 comp rx=0 tx=0).stop 2024-05-20T17:49:09.531 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.192+0000 7f12c5fdb700 1 -- 172.21.15.129:0/3794882659 shutdown_connections 2024-05-20T17:49:09.531 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.192+0000 7f12c5fdb700 1 --2- 172.21.15.129:0/3794882659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c007bc80 0x7f12c019cef0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:09.531 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.192+0000 7f12c5fdb700 1 -- 172.21.15.129:0/3794882659 >> 172.21.15.129:0/3794882659 conn(0x7f12c01031d0 msgr2=0x7f12c0104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:09.531 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.192+0000 7f12c5fdb700 1 -- 172.21.15.129:0/3794882659 shutdown_connections 2024-05-20T17:49:09.531 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:03.192+0000 7f12c5fdb700 1 -- 172.21.15.129:0/3794882659 wait complete. 2024-05-20T17:49:09.531 INFO:teuthology.orchestra.run.smithi129.stdout:mgr not available, waiting (2/15)... 2024-05-20T17:49:12.631 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:12 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/565233101' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-20T17:49:13.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:49:13.048 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-05-20T17:49:13.048 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsid": "0f1f0dd2-16d1-11ef-bc9a-c7b262605968", 2024-05-20T17:49:13.048 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "health": { 2024-05-20T17:49:13.048 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 0 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "a" 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:49:13.049 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-20T17:49:13.050 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-20T17:49:13.050 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:13.050 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-20T17:49:13.050 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:49:13.050 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-20T17:49:13.050 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-20T17:49:13.050 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-20T17:49:13.058 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-20T17:49:13.058 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-20T17:49:13.058 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-20T17:49:13.058 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:13.058 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-20T17:49:13.058 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-20T17:49:13.058 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-20T17:49:13.058 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-20T17:49:13.059 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "restful" 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modified": "2024-05-20T17:48:49.505690+0000", 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-05-20T17:49:13.060 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.949+0000 7f024dfb9700 1 Processor -- start 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.949+0000 7f024dfb9700 1 -- start start 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.949+0000 7f024dfb9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f02481090a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.949+0000 7f024dfb9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0248109690 con 0x7f0248108c90 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.950+0000 7f02477fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f02481090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.950+0000 7f02477fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f02481090a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60962/0 (socket says 172.21.15.129:60962) 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.950+0000 7f02477fe700 1 -- 172.21.15.129:0/3836186900 learned_addr learned my addr 172.21.15.129:0/3836186900 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.950+0000 7f02477fe700 1 -- 172.21.15.129:0/3836186900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0248109d70 con 0x7f0248108c90 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.950+0000 7f02477fe700 1 --2- 172.21.15.129:0/3836186900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f02481090a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02380098a0 tx=0x7f0238009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43e7ade72f31c13 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.950+0000 7f02467fc700 1 -- 172.21.15.129:0/3836186900 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0238004160 con 0x7f0248108c90 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.950+0000 7f02467fc700 1 -- 172.21.15.129:0/3836186900 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0238004300 con 0x7f0248108c90 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.950+0000 7f02467fc700 1 -- 172.21.15.129:0/3836186900 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0238011b50 con 0x7f0248108c90 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.951+0000 7f024dfb9700 1 -- 172.21.15.129:0/3836186900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 msgr2=0x7f02481090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.951+0000 7f024dfb9700 1 --2- 172.21.15.129:0/3836186900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f02481090a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02380098a0 tx=0x7f0238009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.951+0000 7f024dfb9700 1 -- 172.21.15.129:0/3836186900 shutdown_connections 2024-05-20T17:49:13.061 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.951+0000 7f024dfb9700 1 --2- 172.21.15.129:0/3836186900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f02481090a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.951+0000 7f024dfb9700 1 -- 172.21.15.129:0/3836186900 >> 172.21.15.129:0/3836186900 conn(0x7f0248104310 msgr2=0x7f0248106780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.951+0000 7f024dfb9700 1 -- 172.21.15.129:0/3836186900 shutdown_connections 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.951+0000 7f024dfb9700 1 -- 172.21.15.129:0/3836186900 wait complete. 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.952+0000 7f024dfb9700 1 Processor -- start 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.952+0000 7f024dfb9700 1 -- start start 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.952+0000 7f024dfb9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f024807e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.952+0000 7f024dfb9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f024810d120 con 0x7f0248108c90 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.952+0000 7f02477fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f024807e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.952+0000 7f02477fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f024807e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60964/0 (socket says 172.21.15.129:60964) 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.952+0000 7f02477fe700 1 -- 172.21.15.129:0/565233101 learned_addr learned my addr 172.21.15.129:0/565233101 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.953+0000 7f02477fe700 1 -- 172.21.15.129:0/565233101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024807e7f0 con 0x7f0248108c90 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.953+0000 7f02477fe700 1 --2- 172.21.15.129:0/565233101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f024807e2b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0238009fb0 tx=0x7f0238011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.953+0000 7f0244ff9700 1 -- 172.21.15.129:0/565233101 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02380038b0 con 0x7f0248108c90 2024-05-20T17:49:13.062 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.953+0000 7f024dfb9700 1 -- 172.21.15.129:0/565233101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f024807ca40 con 0x7f0248108c90 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.953+0000 7f0244ff9700 1 -- 172.21.15.129:0/565233101 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0238011e50 con 0x7f0248108c90 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.953+0000 7f0244ff9700 1 -- 172.21.15.129:0/565233101 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f023801a430 con 0x7f0248108c90 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.953+0000 7f024dfb9700 1 -- 172.21.15.129:0/565233101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f024807cf20 con 0x7f0248108c90 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.954+0000 7f0244ff9700 1 -- 172.21.15.129:0/565233101 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f023801a730 con 0x7f0248108c90 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.954+0000 7f024dfb9700 1 -- 172.21.15.129:0/565233101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0248063ca0 con 0x7f0248108c90 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.954+0000 7f0244ff9700 1 --2- 172.21.15.129:0/565233101 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f0230039070 0x7f023003b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.954+0000 7f0244ff9700 1 -- 172.21.15.129:0/565233101 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f023804d110 con 0x7f0248108c90 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.954+0000 7f0246ffd700 1 --2- 172.21.15.129:0/565233101 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f0230039070 0x7f023003b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.955+0000 7f0246ffd700 1 --2- 172.21.15.129:0/565233101 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f0230039070 0x7f023003b580 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f023c006fd0 tx=0x7f023c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:11.960+0000 7f0244ff9700 1 -- 172.21.15.129:0/565233101 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0238024440 con 0x7f0248108c90 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.234+0000 7f024dfb9700 1 -- 172.21.15.129:0/565233101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f024807d660 con 0x7f0248108c90 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.235+0000 7f0244ff9700 1 -- 172.21.15.129:0/565233101 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f023802d770 con 0x7f0248108c90 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.237+0000 7f024dfb9700 1 -- 172.21.15.129:0/565233101 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f0230039070 msgr2=0x7f023003b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:13.063 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.237+0000 7f024dfb9700 1 --2- 172.21.15.129:0/565233101 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f0230039070 0x7f023003b580 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f023c006fd0 tx=0x7f023c006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:13.064 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.237+0000 7f024dfb9700 1 -- 172.21.15.129:0/565233101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 msgr2=0x7f024807e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:13.064 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.237+0000 7f024dfb9700 1 --2- 172.21.15.129:0/565233101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f024807e2b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0238009fb0 tx=0x7f0238011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:13.064 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.237+0000 7f024dfb9700 1 -- 172.21.15.129:0/565233101 shutdown_connections 2024-05-20T17:49:13.064 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.237+0000 7f024dfb9700 1 --2- 172.21.15.129:0/565233101 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f0230039070 0x7f023003b580 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:13.064 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.237+0000 7f024dfb9700 1 --2- 172.21.15.129:0/565233101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0248108c90 0x7f024807e2b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:13.064 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.237+0000 7f024dfb9700 1 -- 172.21.15.129:0/565233101 >> 172.21.15.129:0/565233101 conn(0x7f0248104310 msgr2=0x7f0248104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:13.064 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.238+0000 7f024dfb9700 1 -- 172.21.15.129:0/565233101 shutdown_connections 2024-05-20T17:49:13.064 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:12.238+0000 7f024dfb9700 1 -- 172.21.15.129:0/565233101 wait complete. 2024-05-20T17:49:13.064 INFO:teuthology.orchestra.run.smithi129.stdout:mgr is available 2024-05-20T17:49:13.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:13 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/364565014' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-20T17:49:14.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:49:14.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [global] 2024-05-20T17:49:14.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout fsid = 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:49:14.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [osd] 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-20T17:49:14.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.447+0000 7f8807dfe700 1 Processor -- start 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.447+0000 7f8807dfe700 1 -- start start 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.447+0000 7f8807dfe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880010bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.447+0000 7f8807dfe700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f880007c620 con 0x7f8800109720 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.447+0000 7f8805b9a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880010bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.447+0000 7f8805b9a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880010bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60970/0 (socket says 172.21.15.129:60970) 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.447+0000 7f8805b9a700 1 -- 172.21.15.129:0/1007386427 learned_addr learned my addr 172.21.15.129:0/1007386427 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.448+0000 7f8805b9a700 1 -- 172.21.15.129:0/1007386427 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880010c080 con 0x7f8800109720 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.448+0000 7f8805b9a700 1 --2- 172.21.15.129:0/1007386427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880010bb40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0098a0 tx=0x7f87fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9163aba107a56037 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.448+0000 7f8804b98700 1 -- 172.21.15.129:0/1007386427 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87fc004160 con 0x7f8800109720 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.448+0000 7f8804b98700 1 -- 172.21.15.129:0/1007386427 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f87fc004300 con 0x7f8800109720 2024-05-20T17:49:14.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.448+0000 7f8804b98700 1 -- 172.21.15.129:0/1007386427 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87fc011b50 con 0x7f8800109720 2024-05-20T17:49:14.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.449+0000 7f8807dfe700 1 -- 172.21.15.129:0/1007386427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 msgr2=0x7f880010bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:14.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.449+0000 7f8807dfe700 1 --2- 172.21.15.129:0/1007386427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880010bb40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0098a0 tx=0x7f87fc009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:14.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.449+0000 7f8807dfe700 1 -- 172.21.15.129:0/1007386427 shutdown_connections 2024-05-20T17:49:14.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.449+0000 7f8807dfe700 1 --2- 172.21.15.129:0/1007386427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880010bb40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:14.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.449+0000 7f8807dfe700 1 -- 172.21.15.129:0/1007386427 >> 172.21.15.129:0/1007386427 conn(0x7f8800103200 msgr2=0x7f8800105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:14.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.449+0000 7f8807dfe700 1 -- 172.21.15.129:0/1007386427 shutdown_connections 2024-05-20T17:49:14.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.449+0000 7f8807dfe700 1 -- 172.21.15.129:0/1007386427 wait complete. 2024-05-20T17:49:14.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.449+0000 7f8807dfe700 1 Processor -- start 2024-05-20T17:49:14.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.450+0000 7f8807dfe700 1 -- start start 2024-05-20T17:49:14.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.450+0000 7f8807dfe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880019cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:14.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.450+0000 7f8807dfe700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f880010cfe0 con 0x7f8800109720 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.450+0000 7f8805b9a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880019cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.450+0000 7f8805b9a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880019cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60974/0 (socket says 172.21.15.129:60974) 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.450+0000 7f8805b9a700 1 -- 172.21.15.129:0/364565014 learned_addr learned my addr 172.21.15.129:0/364565014 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.450+0000 7f8805b9a700 1 -- 172.21.15.129:0/364565014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880019d450 con 0x7f8800109720 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.450+0000 7f8805b9a700 1 --2- 172.21.15.129:0/364565014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880019cf10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f87fc009fb0 tx=0x7f87fc011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.450+0000 7f87f6ffd700 1 -- 172.21.15.129:0/364565014 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87fc0038b0 con 0x7f8800109720 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.450+0000 7f8807dfe700 1 -- 172.21.15.129:0/364565014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f880019d6e0 con 0x7f8800109720 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.450+0000 7f87f6ffd700 1 -- 172.21.15.129:0/364565014 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f87fc011e50 con 0x7f8800109720 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.451+0000 7f87f6ffd700 1 -- 172.21.15.129:0/364565014 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87fc019550 con 0x7f8800109720 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.451+0000 7f8807dfe700 1 -- 172.21.15.129:0/364565014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88001a02f0 con 0x7f8800109720 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.451+0000 7f87f6ffd700 1 -- 172.21.15.129:0/364565014 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f87fc019850 con 0x7f8800109720 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.451+0000 7f8807dfe700 1 -- 172.21.15.129:0/364565014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8800063ca0 con 0x7f8800109720 2024-05-20T17:49:14.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.452+0000 7f87f6ffd700 1 --2- 172.21.15.129:0/364565014 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f87ec039070 0x7f87ec03b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.452+0000 7f87f6ffd700 1 -- 172.21.15.129:0/364565014 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f87fc04d290 con 0x7f8800109720 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.452+0000 7f8805399700 1 --2- 172.21.15.129:0/364565014 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f87ec039070 0x7f87ec03b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.452+0000 7f8805399700 1 --2- 172.21.15.129:0/364565014 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f87ec039070 0x7f87ec03b580 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f87f0006fd0 tx=0x7f87f0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.457+0000 7f87f6ffd700 1 -- 172.21.15.129:0/364565014 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f87fc04ccf0 con 0x7f8800109720 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.649+0000 7f8807dfe700 1 -- 172.21.15.129:0/364565014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f88001a1af0 con 0x7f8800109720 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.651+0000 7f87f6ffd700 1 -- 172.21.15.129:0/364565014 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f87fc00fda0 con 0x7f8800109720 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.653+0000 7f8807dfe700 1 -- 172.21.15.129:0/364565014 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f87ec039070 msgr2=0x7f87ec03b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.653+0000 7f8807dfe700 1 --2- 172.21.15.129:0/364565014 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f87ec039070 0x7f87ec03b580 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f87f0006fd0 tx=0x7f87f0006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.653+0000 7f8807dfe700 1 -- 172.21.15.129:0/364565014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 msgr2=0x7f880019cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.653+0000 7f8807dfe700 1 --2- 172.21.15.129:0/364565014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880019cf10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f87fc009fb0 tx=0x7f87fc011bf0 comp rx=0 tx=0).stop 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.653+0000 7f8807dfe700 1 -- 172.21.15.129:0/364565014 shutdown_connections 2024-05-20T17:49:14.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.653+0000 7f8807dfe700 1 --2- 172.21.15.129:0/364565014 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f87ec039070 0x7f87ec03b580 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:14.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.653+0000 7f8807dfe700 1 --2- 172.21.15.129:0/364565014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8800109720 0x7f880019cf10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:14.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.653+0000 7f8807dfe700 1 -- 172.21.15.129:0/364565014 >> 172.21.15.129:0/364565014 conn(0x7f8800103200 msgr2=0x7f8800104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:14.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.653+0000 7f8807dfe700 1 -- 172.21.15.129:0/364565014 shutdown_connections 2024-05-20T17:49:14.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:13.653+0000 7f8807dfe700 1 -- 172.21.15.129:0/364565014 wait complete. 2024-05-20T17:49:14.190 INFO:teuthology.orchestra.run.smithi129.stdout:Enabling cephadm module... 2024-05-20T17:49:15.155 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:14 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1626161564' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-20T17:49:15.156 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:14 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: ignoring --setuser ceph since I am not root 2024-05-20T17:49:15.156 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:14 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: ignoring --setgroup ceph since I am not root 2024-05-20T17:49:15.156 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:15 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:15.154+0000 7f5b7db97200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-20T17:49:15.418 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:15 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:15.342+0000 7f5b7db97200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:49:15.761 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:15 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:15.448+0000 7f5b7db97200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-20T17:49:15.761 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:15 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:15.760+0000 7f5b7db97200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-20T17:49:15.974 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.598+0000 7f3cc5755700 1 Processor -- start 2024-05-20T17:49:15.974 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.598+0000 7f3cc5755700 1 -- start start 2024-05-20T17:49:15.974 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.598+0000 7f3cc5755700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc0107410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:15.974 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.598+0000 7f3cc5755700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc0107a00 con 0x7f3cc0107000 2024-05-20T17:49:15.974 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.598+0000 7f3cbeffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc0107410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:15.974 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.598+0000 7f3cbeffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc0107410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60976/0 (socket says 172.21.15.129:60976) 2024-05-20T17:49:15.974 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.598+0000 7f3cbeffd700 1 -- 172.21.15.129:0/3931531557 learned_addr learned my addr 172.21.15.129:0/3931531557 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:15.975 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.599+0000 7f3cbeffd700 1 -- 172.21.15.129:0/3931531557 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc0107b80 con 0x7f3cc0107000 2024-05-20T17:49:15.975 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.599+0000 7f3cbeffd700 1 --2- 172.21.15.129:0/3931531557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc0107410 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00098a0 tx=0x7f3cb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b93aee28b38919 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:15.975 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.599+0000 7f3cbdffb700 1 -- 172.21.15.129:0/3931531557 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cb0004160 con 0x7f3cc0107000 2024-05-20T17:49:15.975 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.599+0000 7f3cbdffb700 1 -- 172.21.15.129:0/3931531557 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3cb0004300 con 0x7f3cc0107000 2024-05-20T17:49:15.975 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.599+0000 7f3cbdffb700 1 -- 172.21.15.129:0/3931531557 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cb0011b50 con 0x7f3cc0107000 2024-05-20T17:49:15.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.600+0000 7f3cc5755700 1 -- 172.21.15.129:0/3931531557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 msgr2=0x7f3cc0107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:15.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.600+0000 7f3cc5755700 1 --2- 172.21.15.129:0/3931531557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc0107410 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00098a0 tx=0x7f3cb0009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:15.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.600+0000 7f3cc5755700 1 -- 172.21.15.129:0/3931531557 shutdown_connections 2024-05-20T17:49:15.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.600+0000 7f3cc5755700 1 --2- 172.21.15.129:0/3931531557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc0107410 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:15.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.600+0000 7f3cc5755700 1 -- 172.21.15.129:0/3931531557 >> 172.21.15.129:0/3931531557 conn(0x7f3cc0102a00 msgr2=0x7f3cc0104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:15.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.600+0000 7f3cc5755700 1 -- 172.21.15.129:0/3931531557 shutdown_connections 2024-05-20T17:49:15.976 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.600+0000 7f3cc5755700 1 -- 172.21.15.129:0/3931531557 wait complete. 2024-05-20T17:49:15.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.600+0000 7f3cc5755700 1 Processor -- start 2024-05-20T17:49:15.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.601+0000 7f3cc5755700 1 -- start start 2024-05-20T17:49:15.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.601+0000 7f3cc5755700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:15.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.601+0000 7f3cc5755700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc010d040 con 0x7f3cc0107000 2024-05-20T17:49:15.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.601+0000 7f3cbeffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:15.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.601+0000 7f3cbeffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc019cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60990/0 (socket says 172.21.15.129:60990) 2024-05-20T17:49:15.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.601+0000 7f3cbeffd700 1 -- 172.21.15.129:0/1626161564 learned_addr learned my addr 172.21.15.129:0/1626161564 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:15.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.601+0000 7f3cbeffd700 1 -- 172.21.15.129:0/1626161564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc019d480 con 0x7f3cc0107000 2024-05-20T17:49:15.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.601+0000 7f3cbeffd700 1 --2- 172.21.15.129:0/1626161564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc019cf40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00046e0 tx=0x7f3cb0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:15.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.601+0000 7f3ca7fff700 1 -- 172.21.15.129:0/1626161564 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cb00039e0 con 0x7f3cc0107000 2024-05-20T17:49:15.977 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.601+0000 7f3cc5755700 1 -- 172.21.15.129:0/1626161564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc019d710 con 0x7f3cc0107000 2024-05-20T17:49:15.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.601+0000 7f3ca7fff700 1 -- 172.21.15.129:0/1626161564 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3cb0003b80 con 0x7f3cc0107000 2024-05-20T17:49:15.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.602+0000 7f3ca7fff700 1 -- 172.21.15.129:0/1626161564 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cb00247c0 con 0x7f3cc0107000 2024-05-20T17:49:15.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.602+0000 7f3cc5755700 1 -- 172.21.15.129:0/1626161564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cc01a0320 con 0x7f3cc0107000 2024-05-20T17:49:15.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.602+0000 7f3ca7fff700 1 -- 172.21.15.129:0/1626161564 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f3cb0022020 con 0x7f3cc0107000 2024-05-20T17:49:15.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.602+0000 7f3cc5755700 1 -- 172.21.15.129:0/1626161564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cc0063ca0 con 0x7f3cc0107000 2024-05-20T17:49:15.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.603+0000 7f3ca7fff700 1 --2- 172.21.15.129:0/1626161564 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f3ca80390c0 0x7f3ca803b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:15.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.603+0000 7f3ca7fff700 1 -- 172.21.15.129:0/1626161564 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3cb0011e00 con 0x7f3cc0107000 2024-05-20T17:49:15.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.603+0000 7f3cbe7fc700 1 --2- 172.21.15.129:0/1626161564 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f3ca80390c0 0x7f3ca803b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:15.978 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.603+0000 7f3cbe7fc700 1 --2- 172.21.15.129:0/1626161564 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f3ca80390c0 0x7f3ca803b5d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4006fd0 tx=0x7f3cb4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:15.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.608+0000 7f3ca7fff700 1 -- 172.21.15.129:0/1626161564 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3cb004ccc0 con 0x7f3cc0107000 2024-05-20T17:49:15.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.868+0000 7f3cc5755700 1 -- 172.21.15.129:0/1626161564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f3cc01a09a0 con 0x7f3cc0107000 2024-05-20T17:49:15.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.920+0000 7f3ca7fff700 1 -- 172.21.15.129:0/1626161564 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f3cb000fdd0 con 0x7f3cc0107000 2024-05-20T17:49:15.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.922+0000 7f3ca7fff700 1 -- 172.21.15.129:0/1626161564 <== mon.0 v2:172.21.15.129:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f3cb0081850 con 0x7f3cc0107000 2024-05-20T17:49:15.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.923+0000 7f3cc5755700 1 -- 172.21.15.129:0/1626161564 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f3ca80390c0 msgr2=0x7f3ca803b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:15.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.923+0000 7f3cc5755700 1 --2- 172.21.15.129:0/1626161564 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f3ca80390c0 0x7f3ca803b5d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4006fd0 tx=0x7f3cb4006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:15.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.923+0000 7f3cc5755700 1 -- 172.21.15.129:0/1626161564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 msgr2=0x7f3cc019cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:15.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.923+0000 7f3cc5755700 1 --2- 172.21.15.129:0/1626161564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc019cf40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00046e0 tx=0x7f3cb0011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:15.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.923+0000 7f3cc5755700 1 -- 172.21.15.129:0/1626161564 shutdown_connections 2024-05-20T17:49:15.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.923+0000 7f3cc5755700 1 --2- 172.21.15.129:0/1626161564 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f3ca80390c0 0x7f3ca803b5d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:15.979 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.923+0000 7f3cc5755700 1 --2- 172.21.15.129:0/1626161564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cc0107000 0x7f3cc019cf40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:15.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.923+0000 7f3cc5755700 1 -- 172.21.15.129:0/1626161564 >> 172.21.15.129:0/1626161564 conn(0x7f3cc0102a00 msgr2=0x7f3cc0104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:15.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.923+0000 7f3cc5755700 1 -- 172.21.15.129:0/1626161564 shutdown_connections 2024-05-20T17:49:15.980 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:14.923+0000 7f3cc5755700 1 -- 172.21.15.129:0/1626161564 wait complete. 2024-05-20T17:49:16.094 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:15 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1626161564' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-20T17:49:16.094 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:15 smithi129 ceph-mon[32029]: mgrmap e5: x(active, since 11s) 2024-05-20T17:49:16.094 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:15 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:15.847+0000 7f5b7db97200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-20T17:49:17.086 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:16 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/98843364' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-20T17:49:17.356 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:17 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:17.085+0000 7f5b7db97200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-20T17:49:17.356 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:17 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:17.261+0000 7f5b7db97200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-20T17:49:17.364 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-05-20T17:49:17.364 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-20T17:49:17.364 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-20T17:49:17.364 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-20T17:49:17.364 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-20T17:49:17.364 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-05-20T17:49:17.364 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.480+0000 7f340ed38700 1 Processor -- start 2024-05-20T17:49:17.364 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.480+0000 7f340ed38700 1 -- start start 2024-05-20T17:49:17.364 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.480+0000 7f340ed38700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f3408107490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.480+0000 7f340ed38700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3408107aa0 con 0x7f3408107020 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.480+0000 7f340cad4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f3408107490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.480+0000 7f340cad4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f3408107490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32804/0 (socket says 172.21.15.129:32804) 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.480+0000 7f340cad4700 1 -- 172.21.15.129:0/746833229 learned_addr learned my addr 172.21.15.129:0/746833229 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.481+0000 7f340cad4700 1 -- 172.21.15.129:0/746833229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340810c0c0 con 0x7f3408107020 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.481+0000 7f340cad4700 1 --2- 172.21.15.129:0/746833229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f3408107490 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f33fc0098a0 tx=0x7f33fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb4641c6f60495b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.481+0000 7f34077fe700 1 -- 172.21.15.129:0/746833229 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33fc004160 con 0x7f3408107020 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.481+0000 7f34077fe700 1 -- 172.21.15.129:0/746833229 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f33fc004300 con 0x7f3408107020 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.481+0000 7f34077fe700 1 -- 172.21.15.129:0/746833229 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33fc011b50 con 0x7f3408107020 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.482+0000 7f340ed38700 1 -- 172.21.15.129:0/746833229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 msgr2=0x7f3408107490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.482+0000 7f340ed38700 1 --2- 172.21.15.129:0/746833229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f3408107490 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f33fc0098a0 tx=0x7f33fc009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.482+0000 7f340ed38700 1 -- 172.21.15.129:0/746833229 shutdown_connections 2024-05-20T17:49:17.365 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.482+0000 7f340ed38700 1 --2- 172.21.15.129:0/746833229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f3408107490 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.482+0000 7f340ed38700 1 -- 172.21.15.129:0/746833229 >> 172.21.15.129:0/746833229 conn(0x7f34081028f0 msgr2=0x7f3408104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.482+0000 7f340ed38700 1 -- 172.21.15.129:0/746833229 shutdown_connections 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.483+0000 7f340ed38700 1 -- 172.21.15.129:0/746833229 wait complete. 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.483+0000 7f340ed38700 1 Processor -- start 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.483+0000 7f340ed38700 1 -- start start 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.483+0000 7f340ed38700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f340819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.483+0000 7f340ed38700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f340810d080 con 0x7f3408107020 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.483+0000 7f340cad4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f340819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.483+0000 7f340cad4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f340819cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32806/0 (socket says 172.21.15.129:32806) 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.483+0000 7f340cad4700 1 -- 172.21.15.129:0/98843364 learned_addr learned my addr 172.21.15.129:0/98843364 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.484+0000 7f340cad4700 1 -- 172.21.15.129:0/98843364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340819d530 con 0x7f3408107020 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.484+0000 7f340cad4700 1 --2- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f340819cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009fb0 tx=0x7f33fc011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:17.366 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.484+0000 7f3405ffb700 1 -- 172.21.15.129:0/98843364 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33fc0038b0 con 0x7f3408107020 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.484+0000 7f340ed38700 1 -- 172.21.15.129:0/98843364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f340819d7c0 con 0x7f3408107020 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.484+0000 7f3405ffb700 1 -- 172.21.15.129:0/98843364 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f33fc011e50 con 0x7f3408107020 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.484+0000 7f3405ffb700 1 -- 172.21.15.129:0/98843364 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33fc019550 con 0x7f3408107020 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.484+0000 7f340ed38700 1 -- 172.21.15.129:0/98843364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34081a03d0 con 0x7f3408107020 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.485+0000 7f3405ffb700 1 -- 172.21.15.129:0/98843364 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f33fc019850 con 0x7f3408107020 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.486+0000 7f340ed38700 1 -- 172.21.15.129:0/98843364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3408063ca0 con 0x7f3408107020 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.486+0000 7f3405ffb700 1 --2- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f33f80390c0 0x7f33f803b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.486+0000 7f3405ffb700 1 -- 172.21.15.129:0/98843364 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f33fc04d250 con 0x7f3408107020 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.486+0000 7f3407fff700 1 -- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f33f80390c0 msgr2=0x7f33f803b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/3614898033 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.486+0000 7f3407fff700 1 --2- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f33f80390c0 0x7f33f803b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.491+0000 7f3405ffb700 1 -- 172.21.15.129:0/98843364 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f33fc00fda0 con 0x7f3408107020 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.687+0000 7f3407fff700 1 -- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f33f80390c0 msgr2=0x7f33f803b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/3614898033 2024-05-20T17:49:17.367 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.687+0000 7f3407fff700 1 --2- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f33f80390c0 0x7f33f803b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.753+0000 7f340ed38700 1 -- 172.21.15.129:0/98843364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f34081a0910 con 0x7f3408107020 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.754+0000 7f3405ffb700 1 -- 172.21.15.129:0/98843364 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f33fc02d960 con 0x7f3408107020 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.756+0000 7f340ed38700 1 -- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f33f80390c0 msgr2=0x7f33f803b5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.756+0000 7f340ed38700 1 --2- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f33f80390c0 0x7f33f803b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.756+0000 7f340ed38700 1 -- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 msgr2=0x7f340819cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.756+0000 7f340ed38700 1 --2- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f340819cff0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009fb0 tx=0x7f33fc011bf0 comp rx=0 tx=0).stop 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.756+0000 7f340ed38700 1 -- 172.21.15.129:0/98843364 shutdown_connections 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.756+0000 7f340ed38700 1 --2- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f33f80390c0 0x7f33f803b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.756+0000 7f340ed38700 1 --2- 172.21.15.129:0/98843364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3408107020 0x7f340819cff0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.756+0000 7f340ed38700 1 -- 172.21.15.129:0/98843364 >> 172.21.15.129:0/98843364 conn(0x7f34081028f0 msgr2=0x7f3408103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.756+0000 7f340ed38700 1 -- 172.21.15.129:0/98843364 shutdown_connections 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:16.757+0000 7f340ed38700 1 -- 172.21.15.129:0/98843364 wait complete. 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for the mgr to restart... 2024-05-20T17:49:17.368 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mgr epoch 5... 2024-05-20T17:49:17.633 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:17 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:17.440+0000 7f5b7db97200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-20T17:49:17.914 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:17 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:17.750+0000 7f5b7db97200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-20T17:49:17.915 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:17 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:17.844+0000 7f5b7db97200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-20T17:49:18.245 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:18 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:18.049+0000 7f5b7db97200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-20T17:49:18.664 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:18 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:18.244+0000 7f5b7db97200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:49:19.153 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:18 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:18.846+0000 7f5b7db97200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-20T17:49:19.414 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:19 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:19.272+0000 7f5b7db97200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-20T17:49:19.414 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:19 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:19.370+0000 7f5b7db97200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-20T17:49:19.914 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:19 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:19.574+0000 7f5b7db97200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-20T17:49:20.414 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:20.097+0000 7f5b7db97200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-20T17:49:20.414 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:20.190+0000 7f5b7db97200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-20T17:49:20.754 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:20.434+0000 7f5b7db97200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-20T17:49:20.754 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:20.572+0000 7f5b7db97200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-20T17:49:20.754 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:20.672+0000 7f5b7db97200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-20T17:49:21.097 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:20.791+0000 7f5b7db97200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-20T17:49:21.097 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:21 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:21.096+0000 7f5b7db97200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-20T17:49:21.978 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:21 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:21.703+0000 7f5b7db97200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-20T17:49:21.978 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:21 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:21.924+0000 7f5b7db97200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-20T17:49:22.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: Active manager daemon x restarted 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: Activating manager daemon x 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: osdmap e2: 0 total, 0 up, 0 in 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: mgrmap e6: x(active, starting, since 0.00384804s) 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: Manager daemon x is now available 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:49:22.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:21 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:49:23.244 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:22 smithi129 ceph-mon[32029]: Found migration_current of "None". Setting to last migration. 2024-05-20T17:49:23.245 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:22 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-20T17:49:23.245 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:22 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-20T17:49:23.245 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:22 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:23.245 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:22 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:23.245 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:22 smithi129 ceph-mon[32029]: mgrmap e7: x(active, since 1.00612s) 2024-05-20T17:49:23.537 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-05-20T17:49:23.538 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-20T17:49:23.538 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-20T17:49:23.538 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-05-20T17:49:23.538 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.744+0000 7f40e2cfe700 1 Processor -- start 2024-05-20T17:49:23.538 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.744+0000 7f40e2cfe700 1 -- start start 2024-05-20T17:49:23.538 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.744+0000 7f40e2cfe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc10b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:23.538 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.745+0000 7f40e2cfe700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc10bcc0 con 0x7f40dc07bc60 2024-05-20T17:49:23.538 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.745+0000 7f40e0a9a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc10b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:23.538 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.745+0000 7f40e0a9a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc10b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32816/0 (socket says 172.21.15.129:32816) 2024-05-20T17:49:23.538 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.745+0000 7f40e0a9a700 1 -- 172.21.15.129:0/221977215 learned_addr learned my addr 172.21.15.129:0/221977215 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:23.538 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.745+0000 7f40e0a9a700 1 -- 172.21.15.129:0/221977215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40dc10be40 con 0x7f40dc07bc60 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.745+0000 7f40e0a9a700 1 --2- 172.21.15.129:0/221977215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc10b6d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f40d00098a0 tx=0x7f40d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a4b5eb2cb5324c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.745+0000 7f40db7fe700 1 -- 172.21.15.129:0/221977215 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40d0004160 con 0x7f40dc07bc60 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.745+0000 7f40db7fe700 1 -- 172.21.15.129:0/221977215 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f40d0004300 con 0x7f40dc07bc60 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.745+0000 7f40db7fe700 1 -- 172.21.15.129:0/221977215 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40d0011b50 con 0x7f40dc07bc60 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.746+0000 7f40e2cfe700 1 -- 172.21.15.129:0/221977215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 msgr2=0x7f40dc10b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.746+0000 7f40e2cfe700 1 --2- 172.21.15.129:0/221977215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc10b6d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f40d00098a0 tx=0x7f40d0009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.746+0000 7f40e2cfe700 1 -- 172.21.15.129:0/221977215 shutdown_connections 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.746+0000 7f40e2cfe700 1 --2- 172.21.15.129:0/221977215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc10b6d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.746+0000 7f40e2cfe700 1 -- 172.21.15.129:0/221977215 >> 172.21.15.129:0/221977215 conn(0x7f40dc103100 msgr2=0x7f40dc105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.746+0000 7f40e2cfe700 1 -- 172.21.15.129:0/221977215 shutdown_connections 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.746+0000 7f40e2cfe700 1 -- 172.21.15.129:0/221977215 wait complete. 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.747+0000 7f40e2cfe700 1 Processor -- start 2024-05-20T17:49:23.539 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.747+0000 7f40e2cfe700 1 -- start start 2024-05-20T17:49:23.540 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.747+0000 7f40e2cfe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:23.540 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.747+0000 7f40e2cfe700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc10cf20 con 0x7f40dc07bc60 2024-05-20T17:49:23.540 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.747+0000 7f40e0a9a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:23.540 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.747+0000 7f40e0a9a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc19cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32830/0 (socket says 172.21.15.129:32830) 2024-05-20T17:49:23.540 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.747+0000 7f40e0a9a700 1 -- 172.21.15.129:0/3277917399 learned_addr learned my addr 172.21.15.129:0/3277917399 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:23.540 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.747+0000 7f40e0a9a700 1 -- 172.21.15.129:0/3277917399 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40dc19d420 con 0x7f40dc07bc60 2024-05-20T17:49:23.540 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.748+0000 7f40e0a9a700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc19cee0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f40d0011850 tx=0x7f40d0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:23.540 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.748+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40d00039c0 con 0x7f40dc07bc60 2024-05-20T17:49:23.540 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.748+0000 7f40e2cfe700 1 -- 172.21.15.129:0/3277917399 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40dc19d6b0 con 0x7f40dc07bc60 2024-05-20T17:49:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.748+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f40d0003b60 con 0x7f40dc07bc60 2024-05-20T17:49:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.748+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40d00196c0 con 0x7f40dc07bc60 2024-05-20T17:49:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.748+0000 7f40e2cfe700 1 -- 172.21.15.129:0/3277917399 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40dc1a02c0 con 0x7f40dc07bc60 2024-05-20T17:49:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.749+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f40d0019980 con 0x7f40dc07bc60 2024-05-20T17:49:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.749+0000 7f40d9ffb700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 0x7f40cc03b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.749+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 --> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f40cc03bd30 con 0x7f40cc039110 2024-05-20T17:49:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.749+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f40d0011e00 con 0x7f40dc07bc60 2024-05-20T17:49:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.749+0000 7f40dbfff700 1 -- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 msgr2=0x7f40cc03b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/3614898033 2024-05-20T17:49:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.749+0000 7f40dbfff700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 0x7f40cc03b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:49:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.950+0000 7f40dbfff700 1 -- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 msgr2=0x7f40cc03b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/3614898033 2024-05-20T17:49:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:17.950+0000 7f40dbfff700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 0x7f40cc03b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-20T17:49:23.542 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:18.351+0000 7f40dbfff700 1 -- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 msgr2=0x7f40cc03b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/3614898033 2024-05-20T17:49:23.542 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:18.351+0000 7f40dbfff700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 0x7f40cc03b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-20T17:49:23.542 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:19.152+0000 7f40dbfff700 1 -- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 msgr2=0x7f40cc03b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/3614898033 2024-05-20T17:49:23.542 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:19.152+0000 7f40dbfff700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 0x7f40cc03b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-20T17:49:23.542 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:20.753+0000 7f40dbfff700 1 -- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 msgr2=0x7f40cc03b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/3614898033 2024-05-20T17:49:23.542 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:20.753+0000 7f40dbfff700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 0x7f40cc03b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-20T17:49:23.542 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:21.928+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f40d002d960 con 0x7f40dc07bc60 2024-05-20T17:49:23.542 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:21.928+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 msgr2=0x7f40cc03b620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:49:23.542 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:21.928+0000 7f40d9ffb700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 0x7f40cc03b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:23.542 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.929+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f40d004f300 con 0x7f40dc07bc60 2024-05-20T17:49:23.542 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.930+0000 7f40d9ffb700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f40cc03ca90 0x7f40cc03eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.930+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 --> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f40cc03bd30 con 0x7f40cc03ca90 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.930+0000 7f40dbfff700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f40cc03ca90 0x7f40cc03eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.931+0000 7f40dbfff700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f40cc03ca90 0x7f40cc03eeb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f40c8003a10 tx=0x7f40c8009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.931+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 <== mgr.14118 v2:172.21.15.129:6800/1916241944 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f40cc03bd30 con 0x7f40cc03ca90 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.938+0000 7f40e2cfe700 1 -- 172.21.15.129:0/3277917399 --> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f40dc063ca0 con 0x7f40cc03ca90 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.938+0000 7f40d9ffb700 1 -- 172.21.15.129:0/3277917399 <== mgr.14118 v2:172.21.15.129:6800/1916241944 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f40dc063ca0 con 0x7f40cc03ca90 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.939+0000 7f40e2cfe700 1 -- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f40cc03ca90 msgr2=0x7f40cc03eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.939+0000 7f40e2cfe700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f40cc03ca90 0x7f40cc03eeb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f40c8003a10 tx=0x7f40c8009290 comp rx=0 tx=0).stop 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.939+0000 7f40e2cfe700 1 -- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 msgr2=0x7f40dc19cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.939+0000 7f40e2cfe700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc19cee0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f40d0011850 tx=0x7f40d0011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.939+0000 7f40e2cfe700 1 -- 172.21.15.129:0/3277917399 shutdown_connections 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.939+0000 7f40e2cfe700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40dc07bc60 0x7f40dc19cee0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:23.543 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.939+0000 7f40e2cfe700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f40cc03ca90 0x7f40cc03eeb0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:23.544 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.939+0000 7f40e2cfe700 1 --2- 172.21.15.129:0/3277917399 >> [v2:172.21.15.129:6800/3614898033,v1:172.21.15.129:6801/3614898033] conn(0x7f40cc039110 0x7f40cc03b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:23.544 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.939+0000 7f40e2cfe700 1 -- 172.21.15.129:0/3277917399 >> 172.21.15.129:0/3277917399 conn(0x7f40dc103100 msgr2=0x7f40dc103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:23.544 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.939+0000 7f40e2cfe700 1 -- 172.21.15.129:0/3277917399 shutdown_connections 2024-05-20T17:49:23.544 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:22.939+0000 7f40e2cfe700 1 -- 172.21.15.129:0/3277917399 wait complete. 2024-05-20T17:49:23.544 INFO:teuthology.orchestra.run.smithi129.stdout:mgr epoch 5 is available 2024-05-20T17:49:23.544 INFO:teuthology.orchestra.run.smithi129.stdout:Setting orchestrator backend to cephadm... 2024-05-20T17:49:24.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:23 smithi129 ceph-mon[32029]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-20T17:49:24.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:23 smithi129 ceph-mon[32029]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-20T17:49:24.767 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.951+0000 7fc2ecd0c700 1 Processor -- start 2024-05-20T17:49:24.767 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.951+0000 7fc2ecd0c700 1 -- start start 2024-05-20T17:49:24.767 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.951+0000 7fc2ecd0c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e810b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:24.767 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.951+0000 7fc2ecd0c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e810bd50 con 0x7fc2e807bc60 2024-05-20T17:49:24.767 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.951+0000 7fc2e659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e810b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:24.767 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.952+0000 7fc2e659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e810b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58572/0 (socket says 172.21.15.129:58572) 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.952+0000 7fc2e659c700 1 -- 172.21.15.129:0/3299049535 learned_addr learned my addr 172.21.15.129:0/3299049535 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.952+0000 7fc2e659c700 1 -- 172.21.15.129:0/3299049535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e810bed0 con 0x7fc2e807bc60 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.952+0000 7fc2e659c700 1 --2- 172.21.15.129:0/3299049535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e810b760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80098a0 tx=0x7fc2d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5f3d7c7001ee06 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.952+0000 7fc2e559a700 1 -- 172.21.15.129:0/3299049535 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2d8004160 con 0x7fc2e807bc60 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.952+0000 7fc2e559a700 1 -- 172.21.15.129:0/3299049535 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc2d8004300 con 0x7fc2e807bc60 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.952+0000 7fc2e559a700 1 -- 172.21.15.129:0/3299049535 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2d8011b50 con 0x7fc2e807bc60 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.953+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/3299049535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 msgr2=0x7fc2e810b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.953+0000 7fc2ecd0c700 1 --2- 172.21.15.129:0/3299049535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e810b760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80098a0 tx=0x7fc2d8009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.953+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/3299049535 shutdown_connections 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.953+0000 7fc2ecd0c700 1 --2- 172.21.15.129:0/3299049535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e810b760 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.953+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/3299049535 >> 172.21.15.129:0/3299049535 conn(0x7fc2e81031d0 msgr2=0x7fc2e8105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.953+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/3299049535 shutdown_connections 2024-05-20T17:49:24.768 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.953+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/3299049535 wait complete. 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.954+0000 7fc2ecd0c700 1 Processor -- start 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.954+0000 7fc2ecd0c700 1 -- start start 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.954+0000 7fc2ecd0c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.954+0000 7fc2ecd0c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e810cfb0 con 0x7fc2e807bc60 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.954+0000 7fc2e659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.954+0000 7fc2e659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e819cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58584/0 (socket says 172.21.15.129:58584) 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.954+0000 7fc2e659c700 1 -- 172.21.15.129:0/1163533050 learned_addr learned my addr 172.21.15.129:0/1163533050 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.955+0000 7fc2e659c700 1 -- 172.21.15.129:0/1163533050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e819d460 con 0x7fc2e807bc60 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.955+0000 7fc2e659c700 1 --2- 172.21.15.129:0/1163533050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e819cf20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8009fb0 tx=0x7fc2d8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.955+0000 7fc2d77fe700 1 -- 172.21.15.129:0/1163533050 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2d80038b0 con 0x7fc2e807bc60 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.955+0000 7fc2d77fe700 1 -- 172.21.15.129:0/1163533050 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc2d8011e50 con 0x7fc2e807bc60 2024-05-20T17:49:24.769 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.955+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/1163533050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2e819d6f0 con 0x7fc2e807bc60 2024-05-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.955+0000 7fc2d77fe700 1 -- 172.21.15.129:0/1163533050 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2d8019550 con 0x7fc2e807bc60 2024-05-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.955+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/1163533050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2e81a0300 con 0x7fc2e807bc60 2024-05-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.956+0000 7fc2d77fe700 1 -- 172.21.15.129:0/1163533050 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fc2d8019850 con 0x7fc2e807bc60 2024-05-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.956+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/1163533050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2e8063ca0 con 0x7fc2e807bc60 2024-05-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.956+0000 7fc2d77fe700 1 --2- 172.21.15.129:0/1163533050 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc2d0038fb0 0x7fc2d003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.956+0000 7fc2d77fe700 1 -- 172.21.15.129:0/1163533050 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc2d804d340 con 0x7fc2e807bc60 2024-05-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.956+0000 7fc2e5d9b700 1 --2- 172.21.15.129:0/1163533050 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc2d0038fb0 0x7fc2d003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.957+0000 7fc2e5d9b700 1 --2- 172.21.15.129:0/1163533050 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc2d0038fb0 0x7fc2d003b4c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc006fd0 tx=0x7fc2dc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.962+0000 7fc2d77fe700 1 -- 172.21.15.129:0/1163533050 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc2d8023c90 con 0x7fc2e807bc60 2024-05-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:23.984+0000 7fc2d77fe700 1 -- 172.21.15.129:0/1163533050 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc2d8003a50 con 0x7fc2e807bc60 2024-05-20T17:49:24.770 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.180+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/1163533050 --> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fc2e8105160 con 0x7fc2d0038fb0 2024-05-20T17:49:24.771 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.185+0000 7fc2d77fe700 1 -- 172.21.15.129:0/1163533050 <== mgr.14118 v2:172.21.15.129:6800/1916241944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc2e8105160 con 0x7fc2d0038fb0 2024-05-20T17:49:24.771 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.188+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/1163533050 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc2d0038fb0 msgr2=0x7fc2d003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:24.771 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.188+0000 7fc2ecd0c700 1 --2- 172.21.15.129:0/1163533050 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc2d0038fb0 0x7fc2d003b4c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc006fd0 tx=0x7fc2dc006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:24.771 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.188+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/1163533050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 msgr2=0x7fc2e819cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:24.771 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.188+0000 7fc2ecd0c700 1 --2- 172.21.15.129:0/1163533050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e819cf20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8009fb0 tx=0x7fc2d8011bf0 comp rx=0 tx=0).stop 2024-05-20T17:49:24.771 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.188+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/1163533050 shutdown_connections 2024-05-20T17:49:24.771 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.188+0000 7fc2ecd0c700 1 --2- 172.21.15.129:0/1163533050 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc2d0038fb0 0x7fc2d003b4c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:24.771 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.188+0000 7fc2ecd0c700 1 --2- 172.21.15.129:0/1163533050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2e807bc60 0x7fc2e819cf20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:24.771 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.188+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/1163533050 >> 172.21.15.129:0/1163533050 conn(0x7fc2e81031d0 msgr2=0x7fc2e8104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:24.771 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.188+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/1163533050 shutdown_connections 2024-05-20T17:49:24.771 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:24.188+0000 7fc2ecd0c700 1 -- 172.21.15.129:0/1163533050 wait complete. 2024-05-20T17:49:25.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:24 smithi129 ceph-mon[32029]: [20/May/2024:17:49:23] ENGINE Bus STARTING 2024-05-20T17:49:25.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:24 smithi129 ceph-mon[32029]: mgrmap e8: x(active, since 2s) 2024-05-20T17:49:25.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:24 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:25.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:24 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:49:25.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:24 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:49:26.142 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:25 smithi129 ceph-mon[32029]: [20/May/2024:17:49:24] ENGINE Serving on https://172.21.15.129:7150 2024-05-20T17:49:26.143 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:25 smithi129 ceph-mon[32029]: [20/May/2024:17:49:24] ENGINE Serving on http://172.21.15.129:8765 2024-05-20T17:49:26.143 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:25 smithi129 ceph-mon[32029]: [20/May/2024:17:49:24] ENGINE Bus STARTED 2024-05-20T17:49:26.143 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:25 smithi129 ceph-mon[32029]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:49:26.149 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-20T17:49:26.149 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.361+0000 7fc3a7286700 1 Processor -- start 2024-05-20T17:49:26.150 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.361+0000 7fc3a7286700 1 -- start start 2024-05-20T17:49:26.150 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.362+0000 7fc3a7286700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a01084b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:26.150 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.362+0000 7fc3a7286700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a0108a80 con 0x7fc3a01080a0 2024-05-20T17:49:26.150 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.362+0000 7fc3a5022700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a01084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:26.150 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.362+0000 7fc3a5022700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a01084b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58588/0 (socket says 172.21.15.129:58588) 2024-05-20T17:49:26.150 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.362+0000 7fc3a5022700 1 -- 172.21.15.129:0/3349791045 learned_addr learned my addr 172.21.15.129:0/3349791045 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:26.150 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.362+0000 7fc3a5022700 1 -- 172.21.15.129:0/3349791045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a0108c00 con 0x7fc3a01080a0 2024-05-20T17:49:26.150 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.362+0000 7fc3a5022700 1 --2- 172.21.15.129:0/3349791045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a01084b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc3900098a0 tx=0x7fc390009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdefb97c93396527 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:26.150 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.362+0000 7fc397fff700 1 -- 172.21.15.129:0/3349791045 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc390004160 con 0x7fc3a01080a0 2024-05-20T17:49:26.150 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.363+0000 7fc397fff700 1 -- 172.21.15.129:0/3349791045 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc390004300 con 0x7fc3a01080a0 2024-05-20T17:49:26.150 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.363+0000 7fc397fff700 1 -- 172.21.15.129:0/3349791045 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc390011b50 con 0x7fc3a01080a0 2024-05-20T17:49:26.151 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.363+0000 7fc3a7286700 1 -- 172.21.15.129:0/3349791045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 msgr2=0x7fc3a01084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:26.151 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.363+0000 7fc3a7286700 1 --2- 172.21.15.129:0/3349791045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a01084b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc3900098a0 tx=0x7fc390009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:26.151 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.363+0000 7fc3a7286700 1 -- 172.21.15.129:0/3349791045 shutdown_connections 2024-05-20T17:49:26.151 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.363+0000 7fc3a7286700 1 --2- 172.21.15.129:0/3349791045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a01084b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:26.151 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.363+0000 7fc3a7286700 1 -- 172.21.15.129:0/3349791045 >> 172.21.15.129:0/3349791045 conn(0x7fc3a0102970 msgr2=0x7fc3a0104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:26.151 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.363+0000 7fc3a7286700 1 -- 172.21.15.129:0/3349791045 shutdown_connections 2024-05-20T17:49:26.151 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.364+0000 7fc3a7286700 1 -- 172.21.15.129:0/3349791045 wait complete. 2024-05-20T17:49:26.151 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.364+0000 7fc3a7286700 1 Processor -- start 2024-05-20T17:49:26.151 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.364+0000 7fc3a7286700 1 -- start start 2024-05-20T17:49:26.151 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.364+0000 7fc3a7286700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a0107680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.364+0000 7fc3a7286700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a0109c10 con 0x7fc3a01080a0 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.364+0000 7fc3a5022700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a0107680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.365+0000 7fc3a5022700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a0107680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58598/0 (socket says 172.21.15.129:58598) 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.365+0000 7fc3a5022700 1 -- 172.21.15.129:0/1618294057 learned_addr learned my addr 172.21.15.129:0/1618294057 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.365+0000 7fc3a5022700 1 -- 172.21.15.129:0/1618294057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a0107bc0 con 0x7fc3a01080a0 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.365+0000 7fc3a5022700 1 --2- 172.21.15.129:0/1618294057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a0107680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc390009fb0 tx=0x7fc390011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.365+0000 7fc3967fc700 1 -- 172.21.15.129:0/1618294057 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3900038b0 con 0x7fc3a01080a0 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.365+0000 7fc3967fc700 1 -- 172.21.15.129:0/1618294057 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc390011e50 con 0x7fc3a01080a0 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.365+0000 7fc3a7286700 1 -- 172.21.15.129:0/1618294057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a019fc50 con 0x7fc3a01080a0 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.365+0000 7fc3967fc700 1 -- 172.21.15.129:0/1618294057 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc39001a5c0 con 0x7fc3a01080a0 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.365+0000 7fc3a7286700 1 -- 172.21.15.129:0/1618294057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a01a00d0 con 0x7fc3a01080a0 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.366+0000 7fc3967fc700 1 -- 172.21.15.129:0/1618294057 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc39001a8c0 con 0x7fc3a01080a0 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.366+0000 7fc3a7286700 1 -- 172.21.15.129:0/1618294057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3a0063ca0 con 0x7fc3a01080a0 2024-05-20T17:49:26.152 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.366+0000 7fc3967fc700 1 --2- 172.21.15.129:0/1618294057 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc38c0390c0 0x7fc38c03b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.367+0000 7fc3967fc700 1 -- 172.21.15.129:0/1618294057 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc39004d4d0 con 0x7fc3a01080a0 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.367+0000 7fc3a4821700 1 --2- 172.21.15.129:0/1618294057 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc38c0390c0 0x7fc38c03b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.367+0000 7fc3a4821700 1 --2- 172.21.15.129:0/1618294057 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc38c0390c0 0x7fc38c03b5d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc39c006fd0 tx=0x7fc39c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.372+0000 7fc3967fc700 1 -- 172.21.15.129:0/1618294057 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc39002d910 con 0x7fc3a01080a0 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.567+0000 7fc3a7286700 1 -- 172.21.15.129:0/1618294057 --> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fc3a0104900 con 0x7fc38c0390c0 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.568+0000 7fc3967fc700 1 -- 172.21.15.129:0/1618294057 <== mgr.14118 v2:172.21.15.129:6800/1916241944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fc3a0104900 con 0x7fc38c0390c0 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.570+0000 7fc3a7286700 1 -- 172.21.15.129:0/1618294057 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc38c0390c0 msgr2=0x7fc38c03b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.571+0000 7fc3a7286700 1 --2- 172.21.15.129:0/1618294057 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc38c0390c0 0x7fc38c03b5d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc39c006fd0 tx=0x7fc39c006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.571+0000 7fc3a7286700 1 -- 172.21.15.129:0/1618294057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 msgr2=0x7fc3a0107680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.571+0000 7fc3a7286700 1 --2- 172.21.15.129:0/1618294057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a0107680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc390009fb0 tx=0x7fc390011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.571+0000 7fc3a7286700 1 -- 172.21.15.129:0/1618294057 shutdown_connections 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.571+0000 7fc3a7286700 1 --2- 172.21.15.129:0/1618294057 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fc38c0390c0 0x7fc38c03b5d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:26.153 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.571+0000 7fc3a7286700 1 --2- 172.21.15.129:0/1618294057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3a01080a0 0x7fc3a0107680 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:26.154 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.571+0000 7fc3a7286700 1 -- 172.21.15.129:0/1618294057 >> 172.21.15.129:0/1618294057 conn(0x7fc3a0102970 msgr2=0x7fc3a01041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:26.154 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.571+0000 7fc3a7286700 1 -- 172.21.15.129:0/1618294057 shutdown_connections 2024-05-20T17:49:26.154 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:25.571+0000 7fc3a7286700 1 -- 172.21.15.129:0/1618294057 wait complete. 2024-05-20T17:49:26.154 INFO:teuthology.orchestra.run.smithi129.stdout:Generating ssh key... 2024-05-20T17:49:27.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:26 smithi129 ceph-mon[32029]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:49:27.464 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: Generating public/private rsa key pair. 2024-05-20T17:49:27.464 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: Your identification has been saved in /tmp/tmp7dd6b6bh/key. 2024-05-20T17:49:27.464 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: Your public key has been saved in /tmp/tmp7dd6b6bh/key.pub. 2024-05-20T17:49:27.464 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: The key fingerprint is: 2024-05-20T17:49:27.464 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: SHA256:T8CtOQjGOYw/kK9WvYFRoh2AJe9Mx16AtZIREO/VLA8 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:49:27.464 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: The key's randomart image is: 2024-05-20T17:49:27.465 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: +---[RSA 3072]----+ 2024-05-20T17:49:27.465 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: |+==== . | 2024-05-20T17:49:27.465 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: |.+.@ X . . | 2024-05-20T17:49:27.465 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: | @ E + o . | 2024-05-20T17:49:27.465 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: | = X @ . + | 2024-05-20T17:49:27.465 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: | + B = S . | 2024-05-20T17:49:27.465 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: | o . o + | 2024-05-20T17:49:27.465 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: | o . . | 2024-05-20T17:49:27.465 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: | . | 2024-05-20T17:49:27.465 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: | | 2024-05-20T17:49:27.465 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: +----[SHA256]-----+ 2024-05-20T17:49:27.738 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.570+0000 7fb094683700 1 Processor -- start 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.570+0000 7fb094683700 1 -- start start 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.570+0000 7fb094683700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c10b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.570+0000 7fb094683700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c10bdf0 con 0x7fb08c07c510 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.570+0000 7fb09241f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c10b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.570+0000 7fb09241f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c10b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58610/0 (socket says 172.21.15.129:58610) 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.570+0000 7fb09241f700 1 -- 172.21.15.129:0/41593058 learned_addr learned my addr 172.21.15.129:0/41593058 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.571+0000 7fb09241f700 1 -- 172.21.15.129:0/41593058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb08c10bf70 con 0x7fb08c07c510 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.571+0000 7fb09241f700 1 --2- 172.21.15.129:0/41593058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c10b7e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0880098a0 tx=0x7fb088009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445157e9df8132ab server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.571+0000 7fb09141d700 1 -- 172.21.15.129:0/41593058 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb088004160 con 0x7fb08c07c510 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.571+0000 7fb09141d700 1 -- 172.21.15.129:0/41593058 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb088004300 con 0x7fb08c07c510 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.571+0000 7fb09141d700 1 -- 172.21.15.129:0/41593058 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb088011b50 con 0x7fb08c07c510 2024-05-20T17:49:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.572+0000 7fb094683700 1 -- 172.21.15.129:0/41593058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 msgr2=0x7fb08c10b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.572+0000 7fb094683700 1 --2- 172.21.15.129:0/41593058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c10b7e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0880098a0 tx=0x7fb088009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.572+0000 7fb094683700 1 -- 172.21.15.129:0/41593058 shutdown_connections 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.572+0000 7fb094683700 1 --2- 172.21.15.129:0/41593058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c10b7e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.572+0000 7fb094683700 1 -- 172.21.15.129:0/41593058 >> 172.21.15.129:0/41593058 conn(0x7fb08c103290 msgr2=0x7fb08c1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.572+0000 7fb094683700 1 -- 172.21.15.129:0/41593058 shutdown_connections 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.572+0000 7fb094683700 1 -- 172.21.15.129:0/41593058 wait complete. 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.573+0000 7fb094683700 1 Processor -- start 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.573+0000 7fb094683700 1 -- start start 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.573+0000 7fb094683700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c108f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.573+0000 7fb094683700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb08c10cfe0 con 0x7fb08c07c510 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.573+0000 7fb09241f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.573+0000 7fb09241f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c108f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58622/0 (socket says 172.21.15.129:58622) 2024-05-20T17:49:27.740 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.573+0000 7fb09241f700 1 -- 172.21.15.129:0/2023676788 learned_addr learned my addr 172.21.15.129:0/2023676788 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.573+0000 7fb09241f700 1 -- 172.21.15.129:0/2023676788 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb08c109490 con 0x7fb08c07c510 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.573+0000 7fb09241f700 1 --2- 172.21.15.129:0/2023676788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c108f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb088011850 tx=0x7fb088011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.574+0000 7fb0837fe700 1 -- 172.21.15.129:0/2023676788 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb088003af0 con 0x7fb08c07c510 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.574+0000 7fb0837fe700 1 -- 172.21.15.129:0/2023676788 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb088020070 con 0x7fb08c07c510 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.574+0000 7fb0837fe700 1 -- 172.21.15.129:0/2023676788 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0880195a0 con 0x7fb08c07c510 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.574+0000 7fb094683700 1 -- 172.21.15.129:0/2023676788 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb08c1076e0 con 0x7fb08c07c510 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.574+0000 7fb094683700 1 -- 172.21.15.129:0/2023676788 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb08c107bc0 con 0x7fb08c07c510 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.575+0000 7fb0837fe700 1 -- 172.21.15.129:0/2023676788 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb088019850 con 0x7fb08c07c510 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.575+0000 7fb094683700 1 -- 172.21.15.129:0/2023676788 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb08c063ca0 con 0x7fb08c07c510 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.575+0000 7fb0837fe700 1 --2- 172.21.15.129:0/2023676788 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fb078039160 0x7fb07803b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.575+0000 7fb0837fe700 1 -- 172.21.15.129:0/2023676788 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb088011e00 con 0x7fb08c07c510 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.575+0000 7fb091c1e700 1 --2- 172.21.15.129:0/2023676788 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fb078039160 0x7fb07803b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:27.741 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.575+0000 7fb091c1e700 1 --2- 172.21.15.129:0/2023676788 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fb078039160 0x7fb07803b670 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb07c006fd0 tx=0x7fb07c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.581+0000 7fb0837fe700 1 -- 172.21.15.129:0/2023676788 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb088003c90 con 0x7fb08c07c510 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:26.777+0000 7fb094683700 1 -- 172.21.15.129:0/2023676788 --> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fb08c108370 con 0x7fb078039160 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:27.180+0000 7fb0837fe700 1 -- 172.21.15.129:0/2023676788 <== mgr.14118 v2:172.21.15.129:6800/1916241944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb08c108370 con 0x7fb078039160 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:27.183+0000 7fb094683700 1 -- 172.21.15.129:0/2023676788 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fb078039160 msgr2=0x7fb07803b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:27.183+0000 7fb094683700 1 --2- 172.21.15.129:0/2023676788 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fb078039160 0x7fb07803b670 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb07c006fd0 tx=0x7fb07c006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:27.183+0000 7fb094683700 1 -- 172.21.15.129:0/2023676788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 msgr2=0x7fb08c108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:27.183+0000 7fb094683700 1 --2- 172.21.15.129:0/2023676788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c108f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb088011850 tx=0x7fb088011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:27.183+0000 7fb094683700 1 -- 172.21.15.129:0/2023676788 shutdown_connections 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:27.183+0000 7fb094683700 1 --2- 172.21.15.129:0/2023676788 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fb078039160 0x7fb07803b670 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:27.183+0000 7fb094683700 1 --2- 172.21.15.129:0/2023676788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb08c07c510 0x7fb08c108f50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:27.183+0000 7fb094683700 1 -- 172.21.15.129:0/2023676788 >> 172.21.15.129:0/2023676788 conn(0x7fb08c103290 msgr2=0x7fb08c104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:27.742 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:27.183+0000 7fb094683700 1 -- 172.21.15.129:0/2023676788 shutdown_connections 2024-05-20T17:49:27.743 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:27.183+0000 7fb094683700 1 -- 172.21.15.129:0/2023676788 wait complete. 2024-05-20T17:49:28.374 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:27 smithi129 ceph-mon[32029]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:49:28.374 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:27 smithi129 ceph-mon[32029]: Generating ssh key... 2024-05-20T17:49:28.374 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:27 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:28.374 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:27 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:28.903 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHGwKD8K6rp9Hhw2NEccATSZ12j5Oga2cFfFtVTSjlWwnilwx/4DuXALXS3PeQ7HMMSxqqJ+QR7IQK2gRNZ6rkOUjkFs8LsImwI4K4tD5+r4fTqqDLBNH1P577LFEc68U/pnfrVdb0xd95dEALFplq5uNUZizD6HA8QviW5OyELNeaFVKafw2dEMK+33I4djdKpLY59Tuw59HMDG8c6NSjpcL4h02JYGd6r/LBSOaFhObjIC5Hdr3l0/0Ajlz73/Is9Jc3tZGnqj2sh1LGfkzm/KMA7VR8HfCYETDyU63zEGTDusNGT8VkQQneS60jbUfu3+3vlVSIlocM+s6P4c/bElrBln0LPzA76uy9bzZ3TAZvgpESBPw8u1pIhlItkVtGwDls2yluXnWsj8BqNOiTbfg38cSABedeodTMJqdlK21AQ/p7fzJPx4gG9MZTRJ2QnBSUdPNxXTquVL4as+Ubvpe4mEGO5HUlGVIVW+SKYkWR0pMGZ4FbCybtIA9BiH8= ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:49:28.903 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.168+0000 7f672fbe4700 1 Processor -- start 2024-05-20T17:49:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.168+0000 7f672fbe4700 1 -- start start 2024-05-20T17:49:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.168+0000 7f672fbe4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672810b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.168+0000 7f672fbe4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f672810bd50 con 0x7f672807bc60 2024-05-20T17:49:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.169+0000 7f672d980700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672810b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.169+0000 7f672d980700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672810b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58638/0 (socket says 172.21.15.129:58638) 2024-05-20T17:49:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.169+0000 7f672d980700 1 -- 172.21.15.129:0/1310107850 learned_addr learned my addr 172.21.15.129:0/1310107850 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.169+0000 7f672d980700 1 -- 172.21.15.129:0/1310107850 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672810bed0 con 0x7f672807bc60 2024-05-20T17:49:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.169+0000 7f672d980700 1 --2- 172.21.15.129:0/1310107850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672810b760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f67240098a0 tx=0x7f6724009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf77b6f4fb8e90 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.169+0000 7f672c97e700 1 -- 172.21.15.129:0/1310107850 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6724004160 con 0x7f672807bc60 2024-05-20T17:49:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.169+0000 7f672c97e700 1 -- 172.21.15.129:0/1310107850 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6724004300 con 0x7f672807bc60 2024-05-20T17:49:28.904 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.169+0000 7f672c97e700 1 -- 172.21.15.129:0/1310107850 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6724011b50 con 0x7f672807bc60 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.170+0000 7f672fbe4700 1 -- 172.21.15.129:0/1310107850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 msgr2=0x7f672810b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.170+0000 7f672fbe4700 1 --2- 172.21.15.129:0/1310107850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672810b760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f67240098a0 tx=0x7f6724009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.170+0000 7f672fbe4700 1 -- 172.21.15.129:0/1310107850 shutdown_connections 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.170+0000 7f672fbe4700 1 --2- 172.21.15.129:0/1310107850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672810b760 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.170+0000 7f672fbe4700 1 -- 172.21.15.129:0/1310107850 >> 172.21.15.129:0/1310107850 conn(0x7f67281031d0 msgr2=0x7f6728105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.170+0000 7f672fbe4700 1 -- 172.21.15.129:0/1310107850 shutdown_connections 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.170+0000 7f672fbe4700 1 -- 172.21.15.129:0/1310107850 wait complete. 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.171+0000 7f672fbe4700 1 Processor -- start 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.171+0000 7f672fbe4700 1 -- start start 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.171+0000 7f672fbe4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.171+0000 7f672fbe4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f672810cfb0 con 0x7f672807bc60 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.171+0000 7f672d980700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:28.905 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.171+0000 7f672d980700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672819cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58642/0 (socket says 172.21.15.129:58642) 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.171+0000 7f672d980700 1 -- 172.21.15.129:0/2763002012 learned_addr learned my addr 172.21.15.129:0/2763002012 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.171+0000 7f672d980700 1 -- 172.21.15.129:0/2763002012 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672819d460 con 0x7f672807bc60 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.172+0000 7f672d980700 1 --2- 172.21.15.129:0/2763002012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672819cf20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6724011850 tx=0x7f6724011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.172+0000 7f671effd700 1 -- 172.21.15.129:0/2763002012 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6724003af0 con 0x7f672807bc60 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.172+0000 7f671effd700 1 -- 172.21.15.129:0/2763002012 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6724020070 con 0x7f672807bc60 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.172+0000 7f672fbe4700 1 -- 172.21.15.129:0/2763002012 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672819d6f0 con 0x7f672807bc60 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.172+0000 7f671effd700 1 -- 172.21.15.129:0/2763002012 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67240195a0 con 0x7f672807bc60 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.172+0000 7f672fbe4700 1 -- 172.21.15.129:0/2763002012 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67281a0300 con 0x7f672807bc60 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.173+0000 7f671effd700 1 -- 172.21.15.129:0/2763002012 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f6724019850 con 0x7f672807bc60 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.173+0000 7f672fbe4700 1 -- 172.21.15.129:0/2763002012 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6728063ca0 con 0x7f672807bc60 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.173+0000 7f671effd700 1 --2- 172.21.15.129:0/2763002012 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f6714039160 0x7f671403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:28.906 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.173+0000 7f671effd700 1 -- 172.21.15.129:0/2763002012 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6724011e00 con 0x7f672807bc60 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.173+0000 7f672d17f700 1 --2- 172.21.15.129:0/2763002012 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f6714039160 0x7f671403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.174+0000 7f672d17f700 1 --2- 172.21.15.129:0/2763002012 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f6714039160 0x7f671403b670 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6718006fd0 tx=0x7f6718006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.178+0000 7f671effd700 1 -- 172.21.15.129:0/2763002012 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6724003c90 con 0x7f672807bc60 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.374+0000 7f672fbe4700 1 -- 172.21.15.129:0/2763002012 --> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f6728105160 con 0x7f6714039160 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.375+0000 7f671effd700 1 -- 172.21.15.129:0/2763002012 <== mgr.14118 v2:172.21.15.129:6800/1916241944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f6728105160 con 0x7f6714039160 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.377+0000 7f672fbe4700 1 -- 172.21.15.129:0/2763002012 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f6714039160 msgr2=0x7f671403b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.377+0000 7f672fbe4700 1 --2- 172.21.15.129:0/2763002012 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f6714039160 0x7f671403b670 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6718006fd0 tx=0x7f6718006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.377+0000 7f672fbe4700 1 -- 172.21.15.129:0/2763002012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 msgr2=0x7f672819cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.377+0000 7f672fbe4700 1 --2- 172.21.15.129:0/2763002012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672819cf20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6724011850 tx=0x7f6724011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.377+0000 7f672fbe4700 1 -- 172.21.15.129:0/2763002012 shutdown_connections 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.377+0000 7f672fbe4700 1 --2- 172.21.15.129:0/2763002012 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f6714039160 0x7f671403b670 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.377+0000 7f672fbe4700 1 --2- 172.21.15.129:0/2763002012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f672807bc60 0x7f672819cf20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:28.907 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.377+0000 7f672fbe4700 1 -- 172.21.15.129:0/2763002012 >> 172.21.15.129:0/2763002012 conn(0x7f67281031d0 msgr2=0x7f6728104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:28.908 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.377+0000 7f672fbe4700 1 -- 172.21.15.129:0/2763002012 shutdown_connections 2024-05-20T17:49:28.908 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:28.377+0000 7f672fbe4700 1 -- 172.21.15.129:0/2763002012 wait complete. 2024-05-20T17:49:28.908 INFO:teuthology.orchestra.run.smithi129.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-20T17:49:28.908 INFO:teuthology.orchestra.run.smithi129.stdout:Adding key to root@localhost authorized_keys... 2024-05-20T17:49:28.908 INFO:teuthology.orchestra.run.smithi129.stdout:Adding host smithi129... 2024-05-20T17:49:29.290 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:29 smithi129 ceph-mon[32029]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:49:30.134 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:30 smithi129 ceph-mon[32029]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi129", "addr": "172.21.15.129", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:49:31.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:31 smithi129 ceph-mon[32029]: Deploying cephadm binary to smithi129 2024-05-20T17:49:34.011 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout Added host 'smithi129' with addr '172.21.15.129' 2024-05-20T17:49:34.011 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.336+0000 7fe33598f700 1 Processor -- start 2024-05-20T17:49:34.011 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.336+0000 7fe33598f700 1 -- start start 2024-05-20T17:49:34.011 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.336+0000 7fe33598f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe330107f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:34.011 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.336+0000 7fe33598f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe330108540 con 0x7fe330107b40 2024-05-20T17:49:34.011 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.337+0000 7fe32effd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe330107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:34.011 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.337+0000 7fe32effd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe330107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58652/0 (socket says 172.21.15.129:58652) 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.337+0000 7fe32effd700 1 -- 172.21.15.129:0/3226867017 learned_addr learned my addr 172.21.15.129:0/3226867017 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.337+0000 7fe32effd700 1 -- 172.21.15.129:0/3226867017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe330108c70 con 0x7fe330107b40 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.337+0000 7fe32effd700 1 --2- 172.21.15.129:0/3226867017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe330107f50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe3200098a0 tx=0x7fe320009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885ccefee8c2aa21 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.337+0000 7fe32e7fc700 1 -- 172.21.15.129:0/3226867017 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe320004160 con 0x7fe330107b40 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.337+0000 7fe32e7fc700 1 -- 172.21.15.129:0/3226867017 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe320004300 con 0x7fe330107b40 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.337+0000 7fe32e7fc700 1 -- 172.21.15.129:0/3226867017 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe320011b50 con 0x7fe330107b40 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.338+0000 7fe33598f700 1 -- 172.21.15.129:0/3226867017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 msgr2=0x7fe330107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.338+0000 7fe33598f700 1 --2- 172.21.15.129:0/3226867017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe330107f50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe3200098a0 tx=0x7fe320009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.338+0000 7fe33598f700 1 -- 172.21.15.129:0/3226867017 shutdown_connections 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.338+0000 7fe33598f700 1 --2- 172.21.15.129:0/3226867017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe330107f50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.338+0000 7fe33598f700 1 -- 172.21.15.129:0/3226867017 >> 172.21.15.129:0/3226867017 conn(0x7fe330103240 msgr2=0x7fe330105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:34.012 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.338+0000 7fe33598f700 1 -- 172.21.15.129:0/3226867017 shutdown_connections 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.338+0000 7fe33598f700 1 -- 172.21.15.129:0/3226867017 wait complete. 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.339+0000 7fe33598f700 1 Processor -- start 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.339+0000 7fe33598f700 1 -- start start 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.339+0000 7fe33598f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe33019be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.339+0000 7fe33598f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe33010c020 con 0x7fe330107b40 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.339+0000 7fe32effd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe33019be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.339+0000 7fe32effd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe33019be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58658/0 (socket says 172.21.15.129:58658) 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.339+0000 7fe32effd700 1 -- 172.21.15.129:0/3036899725 learned_addr learned my addr 172.21.15.129:0/3036899725 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.339+0000 7fe32effd700 1 -- 172.21.15.129:0/3036899725 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33019c3a0 con 0x7fe330107b40 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.340+0000 7fe32effd700 1 --2- 172.21.15.129:0/3036899725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe33019be60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe320011b60 tx=0x7fe320011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.340+0000 7fe33498d700 1 -- 172.21.15.129:0/3036899725 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe320003bf0 con 0x7fe330107b40 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.340+0000 7fe33498d700 1 -- 172.21.15.129:0/3036899725 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe320020070 con 0x7fe330107b40 2024-05-20T17:49:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.340+0000 7fe33598f700 1 -- 172.21.15.129:0/3036899725 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe33019c630 con 0x7fe330107b40 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.340+0000 7fe33498d700 1 -- 172.21.15.129:0/3036899725 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe32001b700 con 0x7fe330107b40 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.340+0000 7fe33598f700 1 -- 172.21.15.129:0/3036899725 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe33007b810 con 0x7fe330107b40 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.341+0000 7fe33498d700 1 -- 172.21.15.129:0/3036899725 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fe32001ba00 con 0x7fe330107b40 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.341+0000 7fe33598f700 1 -- 172.21.15.129:0/3036899725 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe330063ca0 con 0x7fe330107b40 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.341+0000 7fe33498d700 1 --2- 172.21.15.129:0/3036899725 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe31c039110 0x7fe31c03b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.341+0000 7fe33498d700 1 -- 172.21.15.129:0/3036899725 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe320011e00 con 0x7fe330107b40 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.341+0000 7fe327fff700 1 --2- 172.21.15.129:0/3036899725 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe31c039110 0x7fe31c03b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.341+0000 7fe327fff700 1 --2- 172.21.15.129:0/3036899725 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe31c039110 0x7fe31c03b620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe318006fd0 tx=0x7fe318006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.346+0000 7fe33498d700 1 -- 172.21.15.129:0/3036899725 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe320003d90 con 0x7fe330107b40 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:29.547+0000 7fe33598f700 1 -- 172.21.15.129:0/3036899725 --> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi129", "addr": "172.21.15.129", "target": ["mon-mgr", ""]}) -- 0x7fe330105250 con 0x7fe31c039110 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:33.572+0000 7fe33498d700 1 -- 172.21.15.129:0/3036899725 <== mgr.14118 v2:172.21.15.129:6800/1916241944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fe330105250 con 0x7fe31c039110 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:33.575+0000 7fe33598f700 1 -- 172.21.15.129:0/3036899725 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe31c039110 msgr2=0x7fe31c03b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:33.575+0000 7fe33598f700 1 --2- 172.21.15.129:0/3036899725 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe31c039110 0x7fe31c03b620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe318006fd0 tx=0x7fe318006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:34.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:33.575+0000 7fe33598f700 1 -- 172.21.15.129:0/3036899725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 msgr2=0x7fe33019be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:34.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:33.575+0000 7fe33598f700 1 --2- 172.21.15.129:0/3036899725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe33019be60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe320011b60 tx=0x7fe320011c40 comp rx=0 tx=0).stop 2024-05-20T17:49:34.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:33.575+0000 7fe33598f700 1 -- 172.21.15.129:0/3036899725 shutdown_connections 2024-05-20T17:49:34.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:33.575+0000 7fe33598f700 1 --2- 172.21.15.129:0/3036899725 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe31c039110 0x7fe31c03b620 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:34.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:33.575+0000 7fe33598f700 1 --2- 172.21.15.129:0/3036899725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe330107b40 0x7fe33019be60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:34.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:33.575+0000 7fe33598f700 1 -- 172.21.15.129:0/3036899725 >> 172.21.15.129:0/3036899725 conn(0x7fe330103240 msgr2=0x7fe330104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:34.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:33.575+0000 7fe33598f700 1 -- 172.21.15.129:0/3036899725 shutdown_connections 2024-05-20T17:49:34.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:33.575+0000 7fe33598f700 1 -- 172.21.15.129:0/3036899725 wait complete. 2024-05-20T17:49:34.015 INFO:teuthology.orchestra.run.smithi129.stdout:Deploying unmanaged mon service... 2024-05-20T17:49:34.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:34 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:34.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:34 smithi129 ceph-mon[32029]: Added host smithi129 2024-05-20T17:49:34.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:34 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:49:35.411 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-20T17:49:35.411 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.418+0000 7f58d6773700 1 Processor -- start 2024-05-20T17:49:35.411 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.418+0000 7f58d6773700 1 -- start start 2024-05-20T17:49:35.411 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.418+0000 7f58d6773700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d0107f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:35.411 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.418+0000 7f58d6773700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f58d0108540 con 0x7f58d0107b40 2024-05-20T17:49:35.411 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.418+0000 7f58cffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d0107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:35.411 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.419+0000 7f58cffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d0107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51888/0 (socket says 172.21.15.129:51888) 2024-05-20T17:49:35.411 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.419+0000 7f58cffff700 1 -- 172.21.15.129:0/1098881809 learned_addr learned my addr 172.21.15.129:0/1098881809 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:35.411 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.419+0000 7f58cffff700 1 -- 172.21.15.129:0/1098881809 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d0108c70 con 0x7f58d0107b40 2024-05-20T17:49:35.411 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.419+0000 7f58cffff700 1 --2- 172.21.15.129:0/1098881809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d0107f50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58c00098a0 tx=0x7f58c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619c23161d1812ba server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.419+0000 7f58cf7fe700 1 -- 172.21.15.129:0/1098881809 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c0004160 con 0x7f58d0107b40 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.419+0000 7f58cf7fe700 1 -- 172.21.15.129:0/1098881809 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f58c0004300 con 0x7f58d0107b40 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.419+0000 7f58cf7fe700 1 -- 172.21.15.129:0/1098881809 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c0011b50 con 0x7f58d0107b40 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.420+0000 7f58d6773700 1 -- 172.21.15.129:0/1098881809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 msgr2=0x7f58d0107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.420+0000 7f58d6773700 1 --2- 172.21.15.129:0/1098881809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d0107f50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58c00098a0 tx=0x7f58c0009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.420+0000 7f58d6773700 1 -- 172.21.15.129:0/1098881809 shutdown_connections 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.420+0000 7f58d6773700 1 --2- 172.21.15.129:0/1098881809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d0107f50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.420+0000 7f58d6773700 1 -- 172.21.15.129:0/1098881809 >> 172.21.15.129:0/1098881809 conn(0x7f58d0103240 msgr2=0x7f58d0105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.420+0000 7f58d6773700 1 -- 172.21.15.129:0/1098881809 shutdown_connections 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.420+0000 7f58d6773700 1 -- 172.21.15.129:0/1098881809 wait complete. 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.421+0000 7f58d6773700 1 Processor -- start 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.421+0000 7f58d6773700 1 -- start start 2024-05-20T17:49:35.412 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.421+0000 7f58d6773700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d007c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.421+0000 7f58d6773700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f58d010c020 con 0x7f58d0107b40 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.421+0000 7f58cffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d007c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.421+0000 7f58cffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d007c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51904/0 (socket says 172.21.15.129:51904) 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.421+0000 7f58cffff700 1 -- 172.21.15.129:0/116970673 learned_addr learned my addr 172.21.15.129:0/116970673 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.421+0000 7f58cffff700 1 -- 172.21.15.129:0/116970673 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d007c6d0 con 0x7f58d0107b40 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.421+0000 7f58cffff700 1 --2- 172.21.15.129:0/116970673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d007c190 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009fb0 tx=0x7f58c0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.422+0000 7f58cdffb700 1 -- 172.21.15.129:0/116970673 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c00038b0 con 0x7f58d0107b40 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.422+0000 7f58cdffb700 1 -- 172.21.15.129:0/116970673 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f58c0011e50 con 0x7f58d0107b40 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.422+0000 7f58cdffb700 1 -- 172.21.15.129:0/116970673 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c001a430 con 0x7f58d0107b40 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.422+0000 7f58d6773700 1 -- 172.21.15.129:0/116970673 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d019fd10 con 0x7f58d0107b40 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.422+0000 7f58d6773700 1 -- 172.21.15.129:0/116970673 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d01a0190 con 0x7f58d0107b40 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.423+0000 7f58d6773700 1 -- 172.21.15.129:0/116970673 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d0063ca0 con 0x7f58d0107b40 2024-05-20T17:49:35.413 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.423+0000 7f58cdffb700 1 -- 172.21.15.129:0/116970673 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f58c001a730 con 0x7f58d0107b40 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.423+0000 7f58cdffb700 1 --2- 172.21.15.129:0/116970673 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f58bc0390c0 0x7f58bc03b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.423+0000 7f58cdffb700 1 -- 172.21.15.129:0/116970673 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f58c004d350 con 0x7f58d0107b40 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.423+0000 7f58c7fff700 1 --2- 172.21.15.129:0/116970673 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f58bc0390c0 0x7f58bc03b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.423+0000 7f58c7fff700 1 --2- 172.21.15.129:0/116970673 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f58bc0390c0 0x7f58bc03b5d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f58b8006fd0 tx=0x7f58b8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.428+0000 7f58cdffb700 1 -- 172.21.15.129:0/116970673 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f58c0024440 con 0x7f58d0107b40 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.650+0000 7f58d6773700 1 -- 172.21.15.129:0/116970673 --> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f58d0105250 con 0x7f58bc0390c0 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.658+0000 7f58cdffb700 1 -- 172.21.15.129:0/116970673 <== mgr.14118 v2:172.21.15.129:6800/1916241944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f58d0105250 con 0x7f58bc0390c0 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.661+0000 7f58d6773700 1 -- 172.21.15.129:0/116970673 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f58bc0390c0 msgr2=0x7f58bc03b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.661+0000 7f58d6773700 1 --2- 172.21.15.129:0/116970673 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f58bc0390c0 0x7f58bc03b5d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f58b8006fd0 tx=0x7f58b8006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.661+0000 7f58d6773700 1 -- 172.21.15.129:0/116970673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 msgr2=0x7f58d007c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.661+0000 7f58d6773700 1 --2- 172.21.15.129:0/116970673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d007c190 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009fb0 tx=0x7f58c0011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.661+0000 7f58d6773700 1 -- 172.21.15.129:0/116970673 shutdown_connections 2024-05-20T17:49:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.661+0000 7f58d6773700 1 --2- 172.21.15.129:0/116970673 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f58bc0390c0 0x7f58bc03b5d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:35.415 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.661+0000 7f58d6773700 1 --2- 172.21.15.129:0/116970673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58d0107b40 0x7f58d007c190 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:35.415 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.661+0000 7f58d6773700 1 -- 172.21.15.129:0/116970673 >> 172.21.15.129:0/116970673 conn(0x7f58d0103240 msgr2=0x7f58d0104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:35.415 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.661+0000 7f58d6773700 1 -- 172.21.15.129:0/116970673 shutdown_connections 2024-05-20T17:49:35.415 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:34.661+0000 7f58d6773700 1 -- 172.21.15.129:0/116970673 wait complete. 2024-05-20T17:49:35.415 INFO:teuthology.orchestra.run.smithi129.stdout:Deploying unmanaged mgr service... 2024-05-20T17:49:35.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:35 smithi129 ceph-mon[32029]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:49:35.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:35 smithi129 ceph-mon[32029]: Saving service mon spec with placement count:5 2024-05-20T17:49:35.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:35 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:36.703 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-20T17:49:36.703 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.807+0000 7fcc9942a700 1 Processor -- start 2024-05-20T17:49:36.703 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.807+0000 7fcc9942a700 1 -- start start 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.808+0000 7fcc9942a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc941076a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.808+0000 7fcc9942a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcc94107c90 con 0x7fcc94107290 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.808+0000 7fcc92ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc941076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.808+0000 7fcc92ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc941076a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51910/0 (socket says 172.21.15.129:51910) 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.808+0000 7fcc92ffd700 1 -- 172.21.15.129:0/1018306336 learned_addr learned my addr 172.21.15.129:0/1018306336 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.808+0000 7fcc92ffd700 1 -- 172.21.15.129:0/1018306336 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc940750e0 con 0x7fcc94107290 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.808+0000 7fcc92ffd700 1 --2- 172.21.15.129:0/1018306336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc941076a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc840098a0 tx=0x7fcc84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6d423b567afc14 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.808+0000 7fcc91ffb700 1 -- 172.21.15.129:0/1018306336 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc84004160 con 0x7fcc94107290 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.809+0000 7fcc91ffb700 1 -- 172.21.15.129:0/1018306336 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcc84004300 con 0x7fcc94107290 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.809+0000 7fcc91ffb700 1 -- 172.21.15.129:0/1018306336 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc84011b50 con 0x7fcc94107290 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.809+0000 7fcc9942a700 1 -- 172.21.15.129:0/1018306336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 msgr2=0x7fcc941076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:36.704 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.809+0000 7fcc9942a700 1 --2- 172.21.15.129:0/1018306336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc941076a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcc840098a0 tx=0x7fcc84009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.809+0000 7fcc9942a700 1 -- 172.21.15.129:0/1018306336 shutdown_connections 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.809+0000 7fcc9942a700 1 --2- 172.21.15.129:0/1018306336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc941076a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.809+0000 7fcc9942a700 1 -- 172.21.15.129:0/1018306336 >> 172.21.15.129:0/1018306336 conn(0x7fcc941028f0 msgr2=0x7fcc94104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.809+0000 7fcc9942a700 1 -- 172.21.15.129:0/1018306336 shutdown_connections 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.809+0000 7fcc9942a700 1 -- 172.21.15.129:0/1018306336 wait complete. 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.810+0000 7fcc9942a700 1 Processor -- start 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.810+0000 7fcc9942a700 1 -- start start 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.810+0000 7fcc9942a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc9419be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.810+0000 7fcc9942a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcc94108bb0 con 0x7fcc94107290 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.810+0000 7fcc92ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc9419be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.811+0000 7fcc92ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc9419be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51924/0 (socket says 172.21.15.129:51924) 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.811+0000 7fcc92ffd700 1 -- 172.21.15.129:0/2804776569 learned_addr learned my addr 172.21.15.129:0/2804776569 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:36.705 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.811+0000 7fcc92ffd700 1 -- 172.21.15.129:0/2804776569 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9419c370 con 0x7fcc94107290 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.811+0000 7fcc92ffd700 1 --2- 172.21.15.129:0/2804776569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc9419be30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcc84009fb0 tx=0x7fcc84011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.811+0000 7fcc7bfff700 1 -- 172.21.15.129:0/2804776569 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc840038b0 con 0x7fcc94107290 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.811+0000 7fcc9942a700 1 -- 172.21.15.129:0/2804776569 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc9419c600 con 0x7fcc94107290 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.811+0000 7fcc7bfff700 1 -- 172.21.15.129:0/2804776569 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcc84011e50 con 0x7fcc94107290 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.811+0000 7fcc7bfff700 1 -- 172.21.15.129:0/2804776569 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc8401a5c0 con 0x7fcc94107290 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.811+0000 7fcc9942a700 1 -- 172.21.15.129:0/2804776569 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc9419f210 con 0x7fcc94107290 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.812+0000 7fcc7bfff700 1 -- 172.21.15.129:0/2804776569 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fcc8401a8c0 con 0x7fcc94107290 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.812+0000 7fcc7bfff700 1 --2- 172.21.15.129:0/2804776569 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcc7c039110 0x7fcc7c03b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.812+0000 7fcc7bfff700 1 -- 172.21.15.129:0/2804776569 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcc8404e690 con 0x7fcc94107290 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.812+0000 7fcc9942a700 1 -- 172.21.15.129:0/2804776569 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc94063ca0 con 0x7fcc94107290 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.812+0000 7fcc927fc700 1 --2- 172.21.15.129:0/2804776569 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcc7c039110 0x7fcc7c03b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:36.706 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.813+0000 7fcc927fc700 1 --2- 172.21.15.129:0/2804776569 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcc7c039110 0x7fcc7c03b620 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcc88006fd0 tx=0x7fcc88006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:35.818+0000 7fcc7bfff700 1 -- 172.21.15.129:0/2804776569 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcc8404df60 con 0x7fcc94107290 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.029+0000 7fcc9942a700 1 -- 172.21.15.129:0/2804776569 --> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fcc94103660 con 0x7fcc7c039110 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.038+0000 7fcc7bfff700 1 -- 172.21.15.129:0/2804776569 <== mgr.14118 v2:172.21.15.129:6800/1916241944 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fcc94103660 con 0x7fcc7c039110 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.040+0000 7fcc9942a700 1 -- 172.21.15.129:0/2804776569 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcc7c039110 msgr2=0x7fcc7c03b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.040+0000 7fcc9942a700 1 --2- 172.21.15.129:0/2804776569 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcc7c039110 0x7fcc7c03b620 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcc88006fd0 tx=0x7fcc88006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.040+0000 7fcc9942a700 1 -- 172.21.15.129:0/2804776569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 msgr2=0x7fcc9419be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.040+0000 7fcc9942a700 1 --2- 172.21.15.129:0/2804776569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc9419be30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcc84009fb0 tx=0x7fcc84011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.040+0000 7fcc9942a700 1 -- 172.21.15.129:0/2804776569 shutdown_connections 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.040+0000 7fcc9942a700 1 --2- 172.21.15.129:0/2804776569 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcc7c039110 0x7fcc7c03b620 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.040+0000 7fcc9942a700 1 --2- 172.21.15.129:0/2804776569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc94107290 0x7fcc9419be30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.040+0000 7fcc9942a700 1 -- 172.21.15.129:0/2804776569 >> 172.21.15.129:0/2804776569 conn(0x7fcc941028f0 msgr2=0x7fcc94102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.040+0000 7fcc9942a700 1 -- 172.21.15.129:0/2804776569 shutdown_connections 2024-05-20T17:49:36.707 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:36.040+0000 7fcc9942a700 1 -- 172.21.15.129:0/2804776569 wait complete. 2024-05-20T17:49:37.336 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:37 smithi129 ceph-mon[32029]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:49:37.336 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:37 smithi129 ceph-mon[32029]: Saving service mgr spec with placement count:2 2024-05-20T17:49:37.336 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:37 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:37.923 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.129+0000 7fccfb5be700 1 Processor -- start 2024-05-20T17:49:37.924 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.129+0000 7fccfb5be700 1 -- start start 2024-05-20T17:49:37.924 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.129+0000 7fccfb5be700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf4107f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:37.924 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.129+0000 7fccfb5be700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fccf4108540 con 0x7fccf4107b40 2024-05-20T17:49:37.924 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.129+0000 7fccf935a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf4107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:37.924 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.129+0000 7fccf935a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf4107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51940/0 (socket says 172.21.15.129:51940) 2024-05-20T17:49:37.924 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.129+0000 7fccf935a700 1 -- 172.21.15.129:0/1082813615 learned_addr learned my addr 172.21.15.129:0/1082813615 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:37.924 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.130+0000 7fccf935a700 1 -- 172.21.15.129:0/1082813615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf4108c70 con 0x7fccf4107b40 2024-05-20T17:49:37.924 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.130+0000 7fccf935a700 1 --2- 172.21.15.129:0/1082813615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf4107f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fccf00098a0 tx=0x7fccf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f7e8f38de05228 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:37.924 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.130+0000 7fccebfff700 1 -- 172.21.15.129:0/1082813615 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccf0004160 con 0x7fccf4107b40 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.130+0000 7fccebfff700 1 -- 172.21.15.129:0/1082813615 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fccf0004300 con 0x7fccf4107b40 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.130+0000 7fccebfff700 1 -- 172.21.15.129:0/1082813615 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccf0011b50 con 0x7fccf4107b40 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.131+0000 7fccfb5be700 1 -- 172.21.15.129:0/1082813615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 msgr2=0x7fccf4107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.131+0000 7fccfb5be700 1 --2- 172.21.15.129:0/1082813615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf4107f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fccf00098a0 tx=0x7fccf0009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.131+0000 7fccfb5be700 1 -- 172.21.15.129:0/1082813615 shutdown_connections 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.131+0000 7fccfb5be700 1 --2- 172.21.15.129:0/1082813615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf4107f50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.131+0000 7fccfb5be700 1 -- 172.21.15.129:0/1082813615 >> 172.21.15.129:0/1082813615 conn(0x7fccf4103180 msgr2=0x7fccf4105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.131+0000 7fccfb5be700 1 -- 172.21.15.129:0/1082813615 shutdown_connections 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.131+0000 7fccfb5be700 1 -- 172.21.15.129:0/1082813615 wait complete. 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.131+0000 7fccfb5be700 1 Processor -- start 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.131+0000 7fccfb5be700 1 -- start start 2024-05-20T17:49:37.925 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccfb5be700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf419bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccfb5be700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fccf410c020 con 0x7fccf4107b40 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccf935a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf419bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccf935a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf419bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51948/0 (socket says 172.21.15.129:51948) 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccf935a700 1 -- 172.21.15.129:0/370285670 learned_addr learned my addr 172.21.15.129:0/370285670 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccf935a700 1 -- 172.21.15.129:0/370285670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf419c420 con 0x7fccf4107b40 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccf935a700 1 --2- 172.21.15.129:0/370285670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf419bee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fccf00046e0 tx=0x7fccf0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccea7fc700 1 -- 172.21.15.129:0/370285670 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccf00039e0 con 0x7fccf4107b40 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccea7fc700 1 -- 172.21.15.129:0/370285670 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fccf0003b80 con 0x7fccf4107b40 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccea7fc700 1 -- 172.21.15.129:0/370285670 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccf0024690 con 0x7fccf4107b40 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccfb5be700 1 -- 172.21.15.129:0/370285670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf419c6b0 con 0x7fccf4107b40 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.132+0000 7fccfb5be700 1 -- 172.21.15.129:0/370285670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf419cad0 con 0x7fccf4107b40 2024-05-20T17:49:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.133+0000 7fccea7fc700 1 -- 172.21.15.129:0/370285670 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fccf0022020 con 0x7fccf4107b40 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.133+0000 7fccfb5be700 1 -- 172.21.15.129:0/370285670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf4063ca0 con 0x7fccf4107b40 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.134+0000 7fccea7fc700 1 --2- 172.21.15.129:0/370285670 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcce0039160 0x7fcce003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.134+0000 7fccea7fc700 1 -- 172.21.15.129:0/370285670 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fccf0011e00 con 0x7fccf4107b40 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.134+0000 7fccf8b59700 1 --2- 172.21.15.129:0/370285670 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcce0039160 0x7fcce003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.134+0000 7fccf8b59700 1 --2- 172.21.15.129:0/370285670 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcce0039160 0x7fcce003b670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcce4006fd0 tx=0x7fcce4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.139+0000 7fccea7fc700 1 -- 172.21.15.129:0/370285670 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fccf002d3f0 con 0x7fccf4107b40 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.335+0000 7fccfb5be700 1 -- 172.21.15.129:0/370285670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fccf407bf00 con 0x7fccf4107b40 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.339+0000 7fccea7fc700 1 -- 172.21.15.129:0/370285670 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7fccf001a9f0 con 0x7fccf4107b40 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.341+0000 7fccfb5be700 1 -- 172.21.15.129:0/370285670 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcce0039160 msgr2=0x7fcce003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.341+0000 7fccfb5be700 1 --2- 172.21.15.129:0/370285670 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcce0039160 0x7fcce003b670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcce4006fd0 tx=0x7fcce4006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.341+0000 7fccfb5be700 1 -- 172.21.15.129:0/370285670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 msgr2=0x7fccf419bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.341+0000 7fccfb5be700 1 --2- 172.21.15.129:0/370285670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf419bee0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fccf00046e0 tx=0x7fccf0011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.341+0000 7fccfb5be700 1 -- 172.21.15.129:0/370285670 shutdown_connections 2024-05-20T17:49:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.341+0000 7fccfb5be700 1 --2- 172.21.15.129:0/370285670 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fcce0039160 0x7fcce003b670 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.341+0000 7fccfb5be700 1 --2- 172.21.15.129:0/370285670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fccf4107b40 0x7fccf419bee0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.341+0000 7fccfb5be700 1 -- 172.21.15.129:0/370285670 >> 172.21.15.129:0/370285670 conn(0x7fccf4103180 msgr2=0x7fccf4103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.341+0000 7fccfb5be700 1 -- 172.21.15.129:0/370285670 shutdown_connections 2024-05-20T17:49:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:37.341+0000 7fccfb5be700 1 -- 172.21.15.129:0/370285670 wait complete. 2024-05-20T17:49:38.565 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:38 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/370285670' entity='client.admin' 2024-05-20T17:49:39.337 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.353+0000 7f2147c96700 1 Processor -- start 2024-05-20T17:49:39.337 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.353+0000 7f2147c96700 1 -- start start 2024-05-20T17:49:39.337 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.353+0000 7f2147c96700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214010db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:39.337 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.353+0000 7f2147c96700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f214010e0a0 con 0x7f214007bc60 2024-05-20T17:49:39.337 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.354+0000 7f2145a32700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214010db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:39.337 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.354+0000 7f2145a32700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214010db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51954/0 (socket says 172.21.15.129:51954) 2024-05-20T17:49:39.337 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.354+0000 7f2145a32700 1 -- 172.21.15.129:0/2750718590 learned_addr learned my addr 172.21.15.129:0/2750718590 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:39.338 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.354+0000 7f2145a32700 1 -- 172.21.15.129:0/2750718590 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f214010e220 con 0x7f214007bc60 2024-05-20T17:49:39.338 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.354+0000 7f2145a32700 1 --2- 172.21.15.129:0/2750718590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214010db60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f213c009e80 tx=0x7f213c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669a98b7c4d7bd08 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:39.338 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.354+0000 7f2144a30700 1 -- 172.21.15.129:0/2750718590 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f213c017070 con 0x7f214007bc60 2024-05-20T17:49:39.338 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.354+0000 7f2144a30700 1 -- 172.21.15.129:0/2750718590 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f213c00b8c0 con 0x7f214007bc60 2024-05-20T17:49:39.338 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.354+0000 7f2144a30700 1 -- 172.21.15.129:0/2750718590 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f213c0045c0 con 0x7f214007bc60 2024-05-20T17:49:39.338 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.355+0000 7f2147c96700 1 -- 172.21.15.129:0/2750718590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 msgr2=0x7f214010db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:39.338 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.355+0000 7f2147c96700 1 --2- 172.21.15.129:0/2750718590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214010db60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f213c009e80 tx=0x7f213c00b330 comp rx=0 tx=0).stop 2024-05-20T17:49:39.338 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.355+0000 7f2147c96700 1 -- 172.21.15.129:0/2750718590 shutdown_connections 2024-05-20T17:49:39.338 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.355+0000 7f2147c96700 1 --2- 172.21.15.129:0/2750718590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214010db60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:39.338 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.355+0000 7f2147c96700 1 -- 172.21.15.129:0/2750718590 >> 172.21.15.129:0/2750718590 conn(0x7f21401031d0 msgr2=0x7f2140105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:39.338 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.355+0000 7f2147c96700 1 -- 172.21.15.129:0/2750718590 shutdown_connections 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.355+0000 7f2147c96700 1 -- 172.21.15.129:0/2750718590 wait complete. 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.356+0000 7f2147c96700 1 Processor -- start 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.356+0000 7f2147c96700 1 -- start start 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.356+0000 7f2147c96700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.356+0000 7f2147c96700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f214010f1e0 con 0x7f214007bc60 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.356+0000 7f2145a32700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.356+0000 7f2145a32700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214019f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51964/0 (socket says 172.21.15.129:51964) 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.356+0000 7f2145a32700 1 -- 172.21.15.129:0/4099040096 learned_addr learned my addr 172.21.15.129:0/4099040096 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.356+0000 7f2145a32700 1 -- 172.21.15.129:0/4099040096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f214019f600 con 0x7f214007bc60 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.357+0000 7f2145a32700 1 --2- 172.21.15.129:0/4099040096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214019f0c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f213c009e50 tx=0x7f213c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.357+0000 7f2136ffd700 1 -- 172.21.15.129:0/4099040096 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f213c020020 con 0x7f214007bc60 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.357+0000 7f2136ffd700 1 -- 172.21.15.129:0/4099040096 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f213c003b50 con 0x7f214007bc60 2024-05-20T17:49:39.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.357+0000 7f2147c96700 1 -- 172.21.15.129:0/4099040096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f214019f890 con 0x7f214007bc60 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.357+0000 7f2136ffd700 1 -- 172.21.15.129:0/4099040096 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f213c017040 con 0x7f214007bc60 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.357+0000 7f2147c96700 1 -- 172.21.15.129:0/4099040096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21401a24a0 con 0x7f214007bc60 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.358+0000 7f2147c96700 1 -- 172.21.15.129:0/4099040096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2140063ca0 con 0x7f214007bc60 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.358+0000 7f2136ffd700 1 -- 172.21.15.129:0/4099040096 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f213c023020 con 0x7f214007bc60 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.358+0000 7f2136ffd700 1 --2- 172.21.15.129:0/4099040096 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f212c039110 0x7f212c03b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.358+0000 7f2136ffd700 1 -- 172.21.15.129:0/4099040096 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f213c00f050 con 0x7f214007bc60 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.358+0000 7f2145231700 1 --2- 172.21.15.129:0/4099040096 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f212c039110 0x7f212c03b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.359+0000 7f2145231700 1 --2- 172.21.15.129:0/4099040096 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f212c039110 0x7f212c03b620 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2130006fd0 tx=0x7f2130006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.363+0000 7f2136ffd700 1 -- 172.21.15.129:0/4099040096 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f213c02a020 con 0x7f214007bc60 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.559+0000 7f2147c96700 1 -- 172.21.15.129:0/4099040096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f21401a2b90 con 0x7f214007bc60 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.562+0000 7f2136ffd700 1 -- 172.21.15.129:0/4099040096 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v7)=0 v7) ==== 130+0+0 (secure 0 0 0) 0x7f213c0247b0 con 0x7f214007bc60 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.564+0000 7f2147c96700 1 -- 172.21.15.129:0/4099040096 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f212c039110 msgr2=0x7f212c03b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:39.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.564+0000 7f2147c96700 1 --2- 172.21.15.129:0/4099040096 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f212c039110 0x7f212c03b620 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2130006fd0 tx=0x7f2130006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:39.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.564+0000 7f2147c96700 1 -- 172.21.15.129:0/4099040096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 msgr2=0x7f214019f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:39.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.564+0000 7f2147c96700 1 --2- 172.21.15.129:0/4099040096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214019f0c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f213c009e50 tx=0x7f213c00bd30 comp rx=0 tx=0).stop 2024-05-20T17:49:39.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.564+0000 7f2147c96700 1 -- 172.21.15.129:0/4099040096 shutdown_connections 2024-05-20T17:49:39.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.564+0000 7f2147c96700 1 --2- 172.21.15.129:0/4099040096 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f212c039110 0x7f212c03b620 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:39.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.564+0000 7f2147c96700 1 --2- 172.21.15.129:0/4099040096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214007bc60 0x7f214019f0c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:39.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.564+0000 7f2147c96700 1 -- 172.21.15.129:0/4099040096 >> 172.21.15.129:0/4099040096 conn(0x7f21401031d0 msgr2=0x7f2140104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:39.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.565+0000 7f2147c96700 1 -- 172.21.15.129:0/4099040096 shutdown_connections 2024-05-20T17:49:39.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:38.565+0000 7f2147c96700 1 -- 172.21.15.129:0/4099040096 wait complete. 2024-05-20T17:49:39.341 INFO:teuthology.orchestra.run.smithi129.stdout:Enabling the dashboard module... 2024-05-20T17:49:39.682 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:39 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4099040096' entity='client.admin' 2024-05-20T17:49:39.682 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:39 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:39.682 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:39 smithi129 ceph-mon[32029]: from='mgr.14118 172.21.15.129:0/3537133830' entity='mgr.x' 2024-05-20T17:49:40.603 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:40 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4196603511' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-20T17:49:40.881 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:40 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: ignoring --setuser ceph since I am not root 2024-05-20T17:49:40.881 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:40 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: ignoring --setgroup ceph since I am not root 2024-05-20T17:49:40.881 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:40 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:40.782+0000 7f0d0696a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-20T17:49:41.164 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:40 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:40.985+0000 7f0d0696a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:49:41.164 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:41 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:41.096+0000 7f0d0696a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-20T17:49:41.197 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.797+0000 7fe8bf228700 1 Processor -- start 2024-05-20T17:49:41.197 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.797+0000 7fe8bf228700 1 -- start start 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.797+0000 7fe8bf228700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b810dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.797+0000 7fe8bf228700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b810e0e0 con 0x7fe8b807bc20 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.797+0000 7fe8bcfc4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b810dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.797+0000 7fe8bcfc4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b810dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41760/0 (socket says 172.21.15.129:41760) 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.797+0000 7fe8bcfc4700 1 -- 172.21.15.129:0/3783770737 learned_addr learned my addr 172.21.15.129:0/3783770737 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.797+0000 7fe8bcfc4700 1 -- 172.21.15.129:0/3783770737 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b810e260 con 0x7fe8b807bc20 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.798+0000 7fe8bcfc4700 1 --2- 172.21.15.129:0/3783770737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b810dba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe8b40098a0 tx=0x7fe8b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda8737205c88b47 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.798+0000 7fe8af7fe700 1 -- 172.21.15.129:0/3783770737 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8b4004160 con 0x7fe8b807bc20 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.798+0000 7fe8af7fe700 1 -- 172.21.15.129:0/3783770737 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8b4004300 con 0x7fe8b807bc20 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.798+0000 7fe8af7fe700 1 -- 172.21.15.129:0/3783770737 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8b4011b20 con 0x7fe8b807bc20 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.799+0000 7fe8bf228700 1 -- 172.21.15.129:0/3783770737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 msgr2=0x7fe8b810dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:41.198 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.799+0000 7fe8bf228700 1 --2- 172.21.15.129:0/3783770737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b810dba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe8b40098a0 tx=0x7fe8b4009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.799+0000 7fe8bf228700 1 -- 172.21.15.129:0/3783770737 shutdown_connections 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.799+0000 7fe8bf228700 1 --2- 172.21.15.129:0/3783770737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b810dba0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.799+0000 7fe8bf228700 1 -- 172.21.15.129:0/3783770737 >> 172.21.15.129:0/3783770737 conn(0x7fe8b81031d0 msgr2=0x7fe8b8105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.799+0000 7fe8bf228700 1 -- 172.21.15.129:0/3783770737 shutdown_connections 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.799+0000 7fe8bf228700 1 -- 172.21.15.129:0/3783770737 wait complete. 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.800+0000 7fe8bf228700 1 Processor -- start 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.800+0000 7fe8bf228700 1 -- start start 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.800+0000 7fe8bf228700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.800+0000 7fe8bf228700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b810f210 con 0x7fe8b807bc20 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.800+0000 7fe8bcfc4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.800+0000 7fe8bcfc4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b819f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41772/0 (socket says 172.21.15.129:41772) 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.800+0000 7fe8bcfc4700 1 -- 172.21.15.129:0/4196603511 learned_addr learned my addr 172.21.15.129:0/4196603511 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:41.199 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.800+0000 7fe8bcfc4700 1 -- 172.21.15.129:0/4196603511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b819f600 con 0x7fe8b807bc20 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.801+0000 7fe8bcfc4700 1 --2- 172.21.15.129:0/4196603511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b819f0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4011740 tx=0x7fe8b4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.801+0000 7fe8adffb700 1 -- 172.21.15.129:0/4196603511 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8b4003a40 con 0x7fe8b807bc20 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.801+0000 7fe8bf228700 1 -- 172.21.15.129:0/4196603511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8b819f890 con 0x7fe8b807bc20 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.801+0000 7fe8adffb700 1 -- 172.21.15.129:0/4196603511 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8b4003be0 con 0x7fe8b807bc20 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.801+0000 7fe8adffb700 1 -- 172.21.15.129:0/4196603511 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8b401a720 con 0x7fe8b807bc20 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.801+0000 7fe8bf228700 1 -- 172.21.15.129:0/4196603511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8b81a24a0 con 0x7fe8b807bc20 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.802+0000 7fe8bf228700 1 -- 172.21.15.129:0/4196603511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8b8063ca0 con 0x7fe8b807bc20 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.802+0000 7fe8adffb700 1 -- 172.21.15.129:0/4196603511 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fe8b401aa20 con 0x7fe8b807bc20 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.802+0000 7fe8adffb700 1 --2- 172.21.15.129:0/4196603511 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe8a00390c0 0x7fe8a003b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.802+0000 7fe8adffb700 1 -- 172.21.15.129:0/4196603511 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe8b404d710 con 0x7fe8b807bc20 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.803+0000 7fe8affff700 1 --2- 172.21.15.129:0/4196603511 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe8a00390c0 0x7fe8a003b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:41.200 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.803+0000 7fe8affff700 1 --2- 172.21.15.129:0/4196603511 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe8a00390c0 0x7fe8a003b5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8006fd0 tx=0x7fe8a8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:39.812+0000 7fe8adffb700 1 -- 172.21.15.129:0/4196603511 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe8b4011e20 con 0x7fe8b807bc20 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.089+0000 7fe8bf228700 1 -- 172.21.15.129:0/4196603511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fe8b81a2670 con 0x7fe8b807bc20 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.565+0000 7fe8adffb700 1 -- 172.21.15.129:0/4196603511 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fe8b4019c70 con 0x7fe8b807bc20 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.567+0000 7fe8adffb700 1 -- 172.21.15.129:0/4196603511 <== mon.0 v2:172.21.15.129:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fe8b402d910 con 0x7fe8b807bc20 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.568+0000 7fe8bf228700 1 -- 172.21.15.129:0/4196603511 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe8a00390c0 msgr2=0x7fe8a003b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.568+0000 7fe8bf228700 1 --2- 172.21.15.129:0/4196603511 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe8a00390c0 0x7fe8a003b5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8006fd0 tx=0x7fe8a8006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.568+0000 7fe8bf228700 1 -- 172.21.15.129:0/4196603511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 msgr2=0x7fe8b819f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.568+0000 7fe8bf228700 1 --2- 172.21.15.129:0/4196603511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b819f0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4011740 tx=0x7fe8b4011ba0 comp rx=0 tx=0).stop 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.569+0000 7fe8bf228700 1 -- 172.21.15.129:0/4196603511 shutdown_connections 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.569+0000 7fe8bf228700 1 --2- 172.21.15.129:0/4196603511 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7fe8a00390c0 0x7fe8a003b5d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.569+0000 7fe8bf228700 1 --2- 172.21.15.129:0/4196603511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8b807bc20 0x7fe8b819f0c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.569+0000 7fe8bf228700 1 -- 172.21.15.129:0/4196603511 >> 172.21.15.129:0/4196603511 conn(0x7fe8b81031d0 msgr2=0x7fe8b8104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.569+0000 7fe8bf228700 1 -- 172.21.15.129:0/4196603511 shutdown_connections 2024-05-20T17:49:41.201 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:40.569+0000 7fe8bf228700 1 -- 172.21.15.129:0/4196603511 wait complete. 2024-05-20T17:49:41.517 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:41 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:41.426+0000 7f0d0696a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-20T17:49:41.517 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:41 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:41.516+0000 7f0d0696a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-20T17:49:41.864 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:41 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4196603511' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-20T17:49:41.865 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:41 smithi129 ceph-mon[32029]: mgrmap e9: x(active, since 18s) 2024-05-20T17:49:42.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.657+0000 7f0beb14c700 1 Processor -- start 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.657+0000 7f0beb14c700 1 -- start start 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.657+0000 7f0beb14c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be410b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.657+0000 7f0beb14c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0be410bd40 con 0x7f0be407c6b0 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.658+0000 7f0be8ee8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be410b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.658+0000 7f0be8ee8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be410b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41814/0 (socket says 172.21.15.129:41814) 2024-05-20T17:49:42.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.658+0000 7f0be8ee8700 1 -- 172.21.15.129:0/1597433454 learned_addr learned my addr 172.21.15.129:0/1597433454 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.658+0000 7f0be8ee8700 1 -- 172.21.15.129:0/1597433454 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be410bec0 con 0x7f0be407c6b0 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.658+0000 7f0be8ee8700 1 --2- 172.21.15.129:0/1597433454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be410b730 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80098a0 tx=0x7f0bd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe587438c930328 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.658+0000 7f0be37fe700 1 -- 172.21.15.129:0/1597433454 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bd8004160 con 0x7f0be407c6b0 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.658+0000 7f0be37fe700 1 -- 172.21.15.129:0/1597433454 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bd8004300 con 0x7f0be407c6b0 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.658+0000 7f0be37fe700 1 -- 172.21.15.129:0/1597433454 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bd8011b20 con 0x7f0be407c6b0 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.659+0000 7f0beb14c700 1 -- 172.21.15.129:0/1597433454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 msgr2=0x7f0be410b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.659+0000 7f0beb14c700 1 --2- 172.21.15.129:0/1597433454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be410b730 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80098a0 tx=0x7f0bd8009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.659+0000 7f0beb14c700 1 -- 172.21.15.129:0/1597433454 shutdown_connections 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.659+0000 7f0beb14c700 1 --2- 172.21.15.129:0/1597433454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be410b730 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.659+0000 7f0beb14c700 1 -- 172.21.15.129:0/1597433454 >> 172.21.15.129:0/1597433454 conn(0x7f0be4103120 msgr2=0x7f0be41055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.659+0000 7f0beb14c700 1 -- 172.21.15.129:0/1597433454 shutdown_connections 2024-05-20T17:49:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.659+0000 7f0beb14c700 1 -- 172.21.15.129:0/1597433454 wait complete. 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.660+0000 7f0beb14c700 1 Processor -- start 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.660+0000 7f0beb14c700 1 -- start start 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.660+0000 7f0beb14c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be4108ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.660+0000 7f0beb14c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0be410cfe0 con 0x7f0be407c6b0 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.660+0000 7f0be8ee8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be4108ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.660+0000 7f0be8ee8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be4108ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41820/0 (socket says 172.21.15.129:41820) 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.660+0000 7f0be8ee8700 1 -- 172.21.15.129:0/2922113133 learned_addr learned my addr 172.21.15.129:0/2922113133 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.661+0000 7f0be8ee8700 1 -- 172.21.15.129:0/2922113133 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be41093e0 con 0x7f0be407c6b0 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.661+0000 7f0be8ee8700 1 --2- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be4108ea0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8011740 tx=0x7f0bd8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.661+0000 7f0be1ffb700 1 -- 172.21.15.129:0/2922113133 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bd8003a40 con 0x7f0be407c6b0 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.661+0000 7f0be1ffb700 1 -- 172.21.15.129:0/2922113133 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bd8003be0 con 0x7f0be407c6b0 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.661+0000 7f0beb14c700 1 -- 172.21.15.129:0/2922113133 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be4107630 con 0x7f0be407c6b0 2024-05-20T17:49:42.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.661+0000 7f0be1ffb700 1 -- 172.21.15.129:0/2922113133 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bd801a720 con 0x7f0be407c6b0 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.661+0000 7f0beb14c700 1 -- 172.21.15.129:0/2922113133 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be4107a90 con 0x7f0be407c6b0 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.662+0000 7f0beb14c700 1 -- 172.21.15.129:0/2922113133 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be4063ca0 con 0x7f0be407c6b0 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.662+0000 7f0be1ffb700 1 -- 172.21.15.129:0/2922113133 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f0bd801aa20 con 0x7f0be407c6b0 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.663+0000 7f0be1ffb700 1 --2- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f0bd4039110 0x7f0bd403b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.663+0000 7f0be1ffb700 1 -- 172.21.15.129:0/2922113133 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0bd804d700 con 0x7f0be407c6b0 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.663+0000 7f0be3fff700 1 -- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f0bd4039110 msgr2=0x7f0bd403b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/1916241944 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.663+0000 7f0be3fff700 1 --2- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f0bd4039110 0x7f0bd403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.668+0000 7f0be1ffb700 1 -- 172.21.15.129:0/2922113133 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0bd8011e20 con 0x7f0be407c6b0 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.864+0000 7f0be3fff700 1 -- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f0bd4039110 msgr2=0x7f0bd403b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/1916241944 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.864+0000 7f0be3fff700 1 --2- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f0bd4039110 0x7f0bd403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.944+0000 7f0beb14c700 1 -- 172.21.15.129:0/2922113133 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f0be4108180 con 0x7f0be407c6b0 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.944+0000 7f0be1ffb700 1 -- 172.21.15.129:0/2922113133 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f0bd8019c70 con 0x7f0be407c6b0 2024-05-20T17:49:42.457 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.947+0000 7f0beb14c700 1 -- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f0bd4039110 msgr2=0x7f0bd403b620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:49:42.458 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.947+0000 7f0beb14c700 1 --2- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f0bd4039110 0x7f0bd403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:42.458 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.947+0000 7f0beb14c700 1 -- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 msgr2=0x7f0be4108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:42.458 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.947+0000 7f0beb14c700 1 --2- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be4108ea0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8011740 tx=0x7f0bd8011ba0 comp rx=0 tx=0).stop 2024-05-20T17:49:42.458 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.951+0000 7f0beb14c700 1 -- 172.21.15.129:0/2922113133 shutdown_connections 2024-05-20T17:49:42.458 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.951+0000 7f0beb14c700 1 --2- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f0bd4039110 0x7f0bd403b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:42.458 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.951+0000 7f0beb14c700 1 --2- 172.21.15.129:0/2922113133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0be407c6b0 0x7f0be4108ea0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:42.458 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.951+0000 7f0beb14c700 1 -- 172.21.15.129:0/2922113133 >> 172.21.15.129:0/2922113133 conn(0x7f0be4103120 msgr2=0x7f0be41038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:42.458 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.951+0000 7f0beb14c700 1 -- 172.21.15.129:0/2922113133 shutdown_connections 2024-05-20T17:49:42.458 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:41.951+0000 7f0beb14c700 1 -- 172.21.15.129:0/2922113133 wait complete. 2024-05-20T17:49:42.458 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for the mgr to restart... 2024-05-20T17:49:42.458 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mgr epoch 9... 2024-05-20T17:49:42.871 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:42 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2922113133' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-20T17:49:42.871 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:42 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:42.739+0000 7f0d0696a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-20T17:49:43.164 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:42 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:42.923+0000 7f0d0696a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-20T17:49:43.164 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:43.093+0000 7f0d0696a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-20T17:49:43.664 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:43.382+0000 7f0d0696a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-20T17:49:43.664 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:43.475+0000 7f0d0696a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-20T17:49:44.164 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:43.679+0000 7f0d0696a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-20T17:49:44.164 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:43.872+0000 7f0d0696a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:49:44.664 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:44 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:44.494+0000 7f0d0696a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-20T17:49:45.237 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:44 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:44.929+0000 7f0d0696a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-20T17:49:45.237 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:45 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:45.027+0000 7f0d0696a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-20T17:49:45.237 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:45 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:45.236+0000 7f0d0696a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-20T17:49:46.117 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:45 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:45.774+0000 7f0d0696a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-20T17:49:46.117 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:45 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:45.867+0000 7f0d0696a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-20T17:49:46.414 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:46.116+0000 7f0d0696a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-20T17:49:46.414 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:46.253+0000 7f0d0696a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-20T17:49:46.414 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:46.352+0000 7f0d0696a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-20T17:49:46.791 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:46.471+0000 7f0d0696a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-20T17:49:46.791 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:46.790+0000 7f0d0696a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-20T17:49:47.664 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:47 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:47.396+0000 7f0d0696a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-20T17:49:47.664 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:49:47 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:49:47.616+0000 7f0d0696a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-20T17:49:48.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:47 smithi129 ceph-mon[32029]: Active manager daemon x restarted 2024-05-20T17:49:48.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:47 smithi129 ceph-mon[32029]: Activating manager daemon x 2024-05-20T17:49:48.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:47 smithi129 ceph-mon[32029]: osdmap e3: 0 total, 0 up, 0 in 2024-05-20T17:49:48.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:47 smithi129 ceph-mon[32029]: mgrmap e10: x(active, starting, since 0.00386439s) 2024-05-20T17:49:48.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:49:48.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-20T17:49:48.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-20T17:49:48.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-20T17:49:48.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-20T17:49:48.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:47 smithi129 ceph-mon[32029]: Manager daemon x is now available 2024-05-20T17:49:48.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:49:48.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-20T17:49:48.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-20T17:49:48.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:48 smithi129 ceph-mon[32029]: [20/May/2024:17:49:48] ENGINE Bus STARTING 2024-05-20T17:49:48.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:48 smithi129 ceph-mon[32029]: [20/May/2024:17:49:48] ENGINE Serving on https://172.21.15.129:7150 2024-05-20T17:49:48.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:48 smithi129 ceph-mon[32029]: [20/May/2024:17:49:48] ENGINE Serving on http://172.21.15.129:8765 2024-05-20T17:49:48.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:48 smithi129 ceph-mon[32029]: [20/May/2024:17:49:48] ENGINE Bus STARTED 2024-05-20T17:49:48.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:48 smithi129 ceph-mon[32029]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-20T17:49:48.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:48 smithi129 ceph-mon[32029]: mgrmap e11: x(active, since 1.0064s) 2024-05-20T17:49:48.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:48 smithi129 ceph-mon[32029]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-20T17:49:49.180 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-05-20T17:49:49.180 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-20T17:49:49.180 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-20T17:49:49.180 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-05-20T17:49:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.870+0000 7f8b2a557700 1 Processor -- start 2024-05-20T17:49:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.870+0000 7f8b2a557700 1 -- start start 2024-05-20T17:49:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.870+0000 7f8b2a557700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b24108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.870+0000 7f8b2a557700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b24109560 con 0x7f8b24108b60 2024-05-20T17:49:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.870+0000 7f8b23fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b24108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.870+0000 7f8b23fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b24108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41834/0 (socket says 172.21.15.129:41834) 2024-05-20T17:49:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.870+0000 7f8b23fff700 1 -- 172.21.15.129:0/1476743294 learned_addr learned my addr 172.21.15.129:0/1476743294 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.871+0000 7f8b23fff700 1 -- 172.21.15.129:0/1476743294 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b24109c90 con 0x7f8b24108b60 2024-05-20T17:49:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.871+0000 7f8b23fff700 1 --2- 172.21.15.129:0/1476743294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b24108f70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8b140098a0 tx=0x7f8b14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfde3d4b6f9ed072 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.871+0000 7f8b22ffd700 1 -- 172.21.15.129:0/1476743294 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b14004160 con 0x7f8b24108b60 2024-05-20T17:49:49.181 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.871+0000 7f8b22ffd700 1 -- 172.21.15.129:0/1476743294 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b14004300 con 0x7f8b24108b60 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.871+0000 7f8b22ffd700 1 -- 172.21.15.129:0/1476743294 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b14011b20 con 0x7f8b24108b60 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.872+0000 7f8b2a557700 1 -- 172.21.15.129:0/1476743294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 msgr2=0x7f8b24108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.872+0000 7f8b2a557700 1 --2- 172.21.15.129:0/1476743294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b24108f70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8b140098a0 tx=0x7f8b14009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.872+0000 7f8b2a557700 1 -- 172.21.15.129:0/1476743294 shutdown_connections 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.872+0000 7f8b2a557700 1 --2- 172.21.15.129:0/1476743294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b24108f70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.872+0000 7f8b2a557700 1 -- 172.21.15.129:0/1476743294 >> 172.21.15.129:0/1476743294 conn(0x7f8b24104220 msgr2=0x7f8b24106670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.872+0000 7f8b2a557700 1 -- 172.21.15.129:0/1476743294 shutdown_connections 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.872+0000 7f8b2a557700 1 -- 172.21.15.129:0/1476743294 wait complete. 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.873+0000 7f8b2a557700 1 Processor -- start 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.873+0000 7f8b2a557700 1 -- start start 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.873+0000 7f8b2a557700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b2419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:49.182 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.874+0000 7f8b2a557700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2410d040 con 0x7f8b24108b60 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.874+0000 7f8b23fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b2419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.874+0000 7f8b23fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b2419cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41844/0 (socket says 172.21.15.129:41844) 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.874+0000 7f8b23fff700 1 -- 172.21.15.129:0/799749131 learned_addr learned my addr 172.21.15.129:0/799749131 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.874+0000 7f8b23fff700 1 -- 172.21.15.129:0/799749131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2419d400 con 0x7f8b24108b60 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.874+0000 7f8b23fff700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b2419cec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009fb0 tx=0x7f8b14011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.874+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b14003930 con 0x7f8b24108b60 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.874+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b14003ad0 con 0x7f8b24108b60 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.874+0000 7f8b2a557700 1 -- 172.21.15.129:0/799749131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b2419d690 con 0x7f8b24108b60 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.874+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b14019670 con 0x7f8b24108b60 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.874+0000 7f8b2a557700 1 -- 172.21.15.129:0/799749131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b241a02a0 con 0x7f8b24108b60 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.875+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f8b14019920 con 0x7f8b24108b60 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.876+0000 7f8b217fa700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 0x7f8b0c03b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:49.183 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.876+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 --> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8b0c03bd90 con 0x7f8b0c039170 2024-05-20T17:49:49.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.876+0000 7f8b237fe700 1 -- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 msgr2=0x7f8b0c03b680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/1916241944 2024-05-20T17:49:49.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.876+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8b1404d6b0 con 0x7f8b24108b60 2024-05-20T17:49:49.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:42.876+0000 7f8b237fe700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 0x7f8b0c03b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:49:49.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:43.077+0000 7f8b237fe700 1 -- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 msgr2=0x7f8b0c03b680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/1916241944 2024-05-20T17:49:49.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:43.077+0000 7f8b237fe700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 0x7f8b0c03b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-20T17:49:49.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:43.478+0000 7f8b237fe700 1 -- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 msgr2=0x7f8b0c03b680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/1916241944 2024-05-20T17:49:49.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:43.478+0000 7f8b237fe700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 0x7f8b0c03b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-20T17:49:49.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:44.279+0000 7f8b237fe700 1 -- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 msgr2=0x7f8b0c03b680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/1916241944 2024-05-20T17:49:49.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:44.279+0000 7f8b237fe700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 0x7f8b0c03b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-20T17:49:49.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:45.880+0000 7f8b237fe700 1 -- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 msgr2=0x7f8b0c03b680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/1916241944 2024-05-20T17:49:49.184 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:45.880+0000 7f8b237fe700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 0x7f8b0c03b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:47.620+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f8b1402d960 con 0x7f8b24108b60 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:47.620+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 msgr2=0x7f8b0c03b680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:47.620+0000 7f8b217fa700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 0x7f8b0c03b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.621+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f8b1404f190 con 0x7f8b24108b60 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.621+0000 7f8b217fa700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8b1404d240 0x7f8b0c03ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.621+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8b14007da0 con 0x7f8b1404d240 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.622+0000 7f8b237fe700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8b1404d240 0x7f8b0c03ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.623+0000 7f8b237fe700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8b1404d240 0x7f8b0c03ec90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8b18003a10 tx=0x7f8b18009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.623+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f8b14007da0 con 0x7f8b1404d240 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.637+0000 7f8b2a557700 1 -- 172.21.15.129:0/799749131 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f8b24063ca0 con 0x7f8b1404d240 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.637+0000 7f8b217fa700 1 -- 172.21.15.129:0/799749131 <== mgr.14150 v2:172.21.15.129:6800/2870089245 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f8b24063ca0 con 0x7f8b1404d240 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.638+0000 7f8b2a557700 1 -- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8b1404d240 msgr2=0x7f8b0c03ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:49.185 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.638+0000 7f8b2a557700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8b1404d240 0x7f8b0c03ec90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8b18003a10 tx=0x7f8b18009270 comp rx=0 tx=0).stop 2024-05-20T17:49:49.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.638+0000 7f8b2a557700 1 -- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 msgr2=0x7f8b2419cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:49.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.638+0000 7f8b2a557700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b2419cec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009fb0 tx=0x7f8b14011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:49.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.638+0000 7f8b2a557700 1 -- 172.21.15.129:0/799749131 shutdown_connections 2024-05-20T17:49:49.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.638+0000 7f8b2a557700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8b1404d240 0x7f8b0c03ec90 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:49.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.638+0000 7f8b2a557700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b24108b60 0x7f8b2419cec0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:49.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.638+0000 7f8b2a557700 1 --2- 172.21.15.129:0/799749131 >> [v2:172.21.15.129:6800/1916241944,v1:172.21.15.129:6801/1916241944] conn(0x7f8b0c039170 0x7f8b0c03b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:49.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.638+0000 7f8b2a557700 1 -- 172.21.15.129:0/799749131 >> 172.21.15.129:0/799749131 conn(0x7f8b24104220 msgr2=0x7f8b241059c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:49.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.638+0000 7f8b2a557700 1 -- 172.21.15.129:0/799749131 shutdown_connections 2024-05-20T17:49:49.186 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:48.638+0000 7f8b2a557700 1 -- 172.21.15.129:0/799749131 wait complete. 2024-05-20T17:49:49.186 INFO:teuthology.orchestra.run.smithi129.stdout:mgr epoch 9 is available 2024-05-20T17:49:49.186 INFO:teuthology.orchestra.run.smithi129.stdout:Generating a dashboard self-signed certificate... 2024-05-20T17:49:50.494 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-20T17:49:50.494 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.549+0000 7f3cf5423700 1 Processor -- start 2024-05-20T17:49:50.494 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.549+0000 7f3cf5423700 1 -- start start 2024-05-20T17:49:50.494 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.549+0000 7f3cf5423700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf0107f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:50.494 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.549+0000 7f3cf5423700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf0108510 con 0x7f3cf0107b10 2024-05-20T17:49:50.494 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.550+0000 7f3ceeffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf0107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:50.494 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.550+0000 7f3ceeffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf0107f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41908/0 (socket says 172.21.15.129:41908) 2024-05-20T17:49:50.494 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.550+0000 7f3ceeffd700 1 -- 172.21.15.129:0/816533958 learned_addr learned my addr 172.21.15.129:0/816533958 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:50.494 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.550+0000 7f3ceeffd700 1 -- 172.21.15.129:0/816533958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf0108c40 con 0x7f3cf0107b10 2024-05-20T17:49:50.494 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.550+0000 7f3ceeffd700 1 --2- 172.21.15.129:0/816533958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf0107f20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009e80 tx=0x7f3ce000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6941046cb2108112 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.550+0000 7f3cee7fc700 1 -- 172.21.15.129:0/816533958 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce0017070 con 0x7f3cf0107b10 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.551+0000 7f3cee7fc700 1 -- 172.21.15.129:0/816533958 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ce000b8c0 con 0x7f3cf0107b10 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.551+0000 7f3cee7fc700 1 -- 172.21.15.129:0/816533958 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce00045c0 con 0x7f3cf0107b10 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.551+0000 7f3cf5423700 1 -- 172.21.15.129:0/816533958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 msgr2=0x7f3cf0107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.551+0000 7f3cf5423700 1 --2- 172.21.15.129:0/816533958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf0107f20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009e80 tx=0x7f3ce000b330 comp rx=0 tx=0).stop 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.551+0000 7f3cf5423700 1 -- 172.21.15.129:0/816533958 shutdown_connections 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.551+0000 7f3cf5423700 1 --2- 172.21.15.129:0/816533958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf0107f20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.551+0000 7f3cf5423700 1 -- 172.21.15.129:0/816533958 >> 172.21.15.129:0/816533958 conn(0x7f3cf01031d0 msgr2=0x7f3cf0105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.551+0000 7f3cf5423700 1 -- 172.21.15.129:0/816533958 shutdown_connections 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.551+0000 7f3cf5423700 1 -- 172.21.15.129:0/816533958 wait complete. 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.552+0000 7f3cf5423700 1 Processor -- start 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.552+0000 7f3cf5423700 1 -- start start 2024-05-20T17:49:50.495 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.552+0000 7f3cf5423700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf007b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.552+0000 7f3cf5423700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf010bff0 con 0x7f3cf0107b10 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.552+0000 7f3ceeffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf007b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.552+0000 7f3ceeffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf007b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41916/0 (socket says 172.21.15.129:41916) 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.552+0000 7f3ceeffd700 1 -- 172.21.15.129:0/179735578 learned_addr learned my addr 172.21.15.129:0/179735578 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.553+0000 7f3ceeffd700 1 -- 172.21.15.129:0/179735578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf007be40 con 0x7f3cf0107b10 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.553+0000 7f3ceeffd700 1 --2- 172.21.15.129:0/179735578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf007b900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009fd0 tx=0x7f3ce0004050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.553+0000 7f3cecff9700 1 -- 172.21.15.129:0/179735578 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce0020020 con 0x7f3cf0107b10 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.553+0000 7f3cecff9700 1 -- 172.21.15.129:0/179735578 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ce0012430 con 0x7f3cf0107b10 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.553+0000 7f3cf5423700 1 -- 172.21.15.129:0/179735578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf019fcc0 con 0x7f3cf0107b10 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.553+0000 7f3cecff9700 1 -- 172.21.15.129:0/179735578 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce0017040 con 0x7f3cf0107b10 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.553+0000 7f3cf5423700 1 -- 172.21.15.129:0/179735578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf01a01a0 con 0x7f3cf0107b10 2024-05-20T17:49:50.496 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.554+0000 7f3cecff9700 1 -- 172.21.15.129:0/179735578 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f3ce0023020 con 0x7f3cf0107b10 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.554+0000 7f3cf5423700 1 -- 172.21.15.129:0/179735578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf0063ca0 con 0x7f3cf0107b10 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.554+0000 7f3cecff9700 1 --2- 172.21.15.129:0/179735578 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3cd8039050 0x7f3cd803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.554+0000 7f3cecff9700 1 -- 172.21.15.129:0/179735578 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3ce000f050 con 0x7f3cf0107b10 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.554+0000 7f3cdffff700 1 --2- 172.21.15.129:0/179735578 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3cd8039050 0x7f3cd803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.555+0000 7f3cdffff700 1 --2- 172.21.15.129:0/179735578 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3cd8039050 0x7f3cd803b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4006fd0 tx=0x7f3ce4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.560+0000 7f3cecff9700 1 -- 172.21.15.129:0/179735578 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3ce002a020 con 0x7f3cf0107b10 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.673+0000 7f3cecff9700 1 -- 172.21.15.129:0/179735578 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f3ce0025400 con 0x7f3cf0107b10 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.766+0000 7f3cf5423700 1 -- 172.21.15.129:0/179735578 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f3cf01a0430 con 0x7f3cd8039050 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.923+0000 7f3cecff9700 1 -- 172.21.15.129:0/179735578 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f3cf01a0430 con 0x7f3cd8039050 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.925+0000 7f3cf5423700 1 -- 172.21.15.129:0/179735578 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3cd8039050 msgr2=0x7f3cd803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.925+0000 7f3cf5423700 1 --2- 172.21.15.129:0/179735578 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3cd8039050 0x7f3cd803b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4006fd0 tx=0x7f3ce4006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.925+0000 7f3cf5423700 1 -- 172.21.15.129:0/179735578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 msgr2=0x7f3cf007b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:50.497 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.925+0000 7f3cf5423700 1 --2- 172.21.15.129:0/179735578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf007b900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009fd0 tx=0x7f3ce0004050 comp rx=0 tx=0).stop 2024-05-20T17:49:50.498 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.925+0000 7f3cf5423700 1 -- 172.21.15.129:0/179735578 shutdown_connections 2024-05-20T17:49:50.498 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.925+0000 7f3cf5423700 1 --2- 172.21.15.129:0/179735578 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3cd8039050 0x7f3cd803b560 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:50.498 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.925+0000 7f3cf5423700 1 --2- 172.21.15.129:0/179735578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cf0107b10 0x7f3cf007b900 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:50.498 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.925+0000 7f3cf5423700 1 -- 172.21.15.129:0/179735578 >> 172.21.15.129:0/179735578 conn(0x7f3cf01031d0 msgr2=0x7f3cf0104950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:50.498 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.925+0000 7f3cf5423700 1 -- 172.21.15.129:0/179735578 shutdown_connections 2024-05-20T17:49:50.498 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:49.925+0000 7f3cf5423700 1 -- 172.21.15.129:0/179735578 wait complete. 2024-05-20T17:49:50.498 INFO:teuthology.orchestra.run.smithi129.stdout:Creating initial admin user... 2024-05-20T17:49:50.751 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:50 smithi129 ceph-mon[32029]: mgrmap e12: x(active, since 2s) 2024-05-20T17:49:50.752 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:50 smithi129 ceph-mon[32029]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:49:50.752 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:49:50.752 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:49:50.752 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:49:52.043 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$IalQrenoyYnzZTDVwQrqaem3uVG7ca2jgG7llZket7gk.IuLU083q", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1716227391, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-20T17:49:52.043 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.959+0000 7f76e108b700 1 Processor -- start 2024-05-20T17:49:52.043 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.959+0000 7f76e108b700 1 -- start start 2024-05-20T17:49:52.043 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.959+0000 7f76e108b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc10b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.959+0000 7f76e108b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f76dc10bd50 con 0x7f76dc07bc60 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.959+0000 7f76dad9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc10b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.959+0000 7f76dad9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc10b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41084/0 (socket says 172.21.15.129:41084) 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.959+0000 7f76dad9d700 1 -- 172.21.15.129:0/1287249574 learned_addr learned my addr 172.21.15.129:0/1287249574 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.960+0000 7f76dad9d700 1 -- 172.21.15.129:0/1287249574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76dc10bed0 con 0x7f76dc07bc60 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.960+0000 7f76dad9d700 1 --2- 172.21.15.129:0/1287249574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc10b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0098a0 tx=0x7f76cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9fb1ccde259adc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.960+0000 7f76d9d9b700 1 -- 172.21.15.129:0/1287249574 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76cc004160 con 0x7f76dc07bc60 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.960+0000 7f76d9d9b700 1 -- 172.21.15.129:0/1287249574 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76cc004300 con 0x7f76dc07bc60 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.960+0000 7f76d9d9b700 1 -- 172.21.15.129:0/1287249574 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76cc011b20 con 0x7f76dc07bc60 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.961+0000 7f76e108b700 1 -- 172.21.15.129:0/1287249574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 msgr2=0x7f76dc10b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.961+0000 7f76e108b700 1 --2- 172.21.15.129:0/1287249574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc10b760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0098a0 tx=0x7f76cc009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.961+0000 7f76e108b700 1 -- 172.21.15.129:0/1287249574 shutdown_connections 2024-05-20T17:49:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.961+0000 7f76e108b700 1 --2- 172.21.15.129:0/1287249574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc10b760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.961+0000 7f76e108b700 1 -- 172.21.15.129:0/1287249574 >> 172.21.15.129:0/1287249574 conn(0x7f76dc1031d0 msgr2=0x7f76dc105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.961+0000 7f76e108b700 1 -- 172.21.15.129:0/1287249574 shutdown_connections 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.961+0000 7f76e108b700 1 -- 172.21.15.129:0/1287249574 wait complete. 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.962+0000 7f76e108b700 1 Processor -- start 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.963+0000 7f76e108b700 1 -- start start 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.963+0000 7f76e108b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.963+0000 7f76e108b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f76dc10cff0 con 0x7f76dc07bc60 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.963+0000 7f76dad9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.963+0000 7f76dad9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc19ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41098/0 (socket says 172.21.15.129:41098) 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.963+0000 7f76dad9d700 1 -- 172.21.15.129:0/1136480040 learned_addr learned my addr 172.21.15.129:0/1136480040 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.963+0000 7f76dad9d700 1 -- 172.21.15.129:0/1136480040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76dc19d3f0 con 0x7f76dc07bc60 2024-05-20T17:49:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.963+0000 7f76dad9d700 1 --2- 172.21.15.129:0/1136480040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc19ceb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f76cc011740 tx=0x7f76cc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:52.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.964+0000 7f76c3fff700 1 -- 172.21.15.129:0/1136480040 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76cc003a40 con 0x7f76dc07bc60 2024-05-20T17:49:52.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.964+0000 7f76c3fff700 1 -- 172.21.15.129:0/1136480040 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76cc003be0 con 0x7f76dc07bc60 2024-05-20T17:49:52.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.964+0000 7f76e108b700 1 -- 172.21.15.129:0/1136480040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76dc19d680 con 0x7f76dc07bc60 2024-05-20T17:49:52.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.964+0000 7f76c3fff700 1 -- 172.21.15.129:0/1136480040 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76cc01a720 con 0x7f76dc07bc60 2024-05-20T17:49:52.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.964+0000 7f76e108b700 1 -- 172.21.15.129:0/1136480040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76dc1a0290 con 0x7f76dc07bc60 2024-05-20T17:49:52.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.965+0000 7f76e108b700 1 -- 172.21.15.129:0/1136480040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76dc063ca0 con 0x7f76dc07bc60 2024-05-20T17:49:52.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.965+0000 7f76c3fff700 1 -- 172.21.15.129:0/1136480040 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f76cc01aa20 con 0x7f76dc07bc60 2024-05-20T17:49:52.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.965+0000 7f76c3fff700 1 --2- 172.21.15.129:0/1136480040 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f76c4039110 0x7f76c403b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:52.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.965+0000 7f76c3fff700 1 -- 172.21.15.129:0/1136480040 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f76cc04d7c0 con 0x7f76dc07bc60 2024-05-20T17:49:52.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.965+0000 7f76da59c700 1 --2- 172.21.15.129:0/1136480040 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f76c4039110 0x7f76c403b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:52.046 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.966+0000 7f76da59c700 1 --2- 172.21.15.129:0/1136480040 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f76c4039110 0x7f76c403b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f76d0006fd0 tx=0x7f76d0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:50.970+0000 7f76c3fff700 1 -- 172.21.15.129:0/1136480040 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f76cc011e20 con 0x7f76dc07bc60 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.195+0000 7f76e108b700 1 -- 172.21.15.129:0/1136480040 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f76dc1a0460 con 0x7f76c4039110 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.428+0000 7f76c3fff700 1 -- 172.21.15.129:0/1136480040 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f76dc1a0460 con 0x7f76c4039110 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.430+0000 7f76e108b700 1 -- 172.21.15.129:0/1136480040 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f76c4039110 msgr2=0x7f76c403b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.430+0000 7f76e108b700 1 --2- 172.21.15.129:0/1136480040 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f76c4039110 0x7f76c403b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f76d0006fd0 tx=0x7f76d0006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.430+0000 7f76e108b700 1 -- 172.21.15.129:0/1136480040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 msgr2=0x7f76dc19ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.430+0000 7f76e108b700 1 --2- 172.21.15.129:0/1136480040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc19ceb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f76cc011740 tx=0x7f76cc011ba0 comp rx=0 tx=0).stop 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.430+0000 7f76e108b700 1 -- 172.21.15.129:0/1136480040 shutdown_connections 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.430+0000 7f76e108b700 1 --2- 172.21.15.129:0/1136480040 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f76c4039110 0x7f76c403b620 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.430+0000 7f76e108b700 1 --2- 172.21.15.129:0/1136480040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc07bc60 0x7f76dc19ceb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.430+0000 7f76e108b700 1 -- 172.21.15.129:0/1136480040 >> 172.21.15.129:0/1136480040 conn(0x7f76dc1031d0 msgr2=0x7f76dc104990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.430+0000 7f76e108b700 1 -- 172.21.15.129:0/1136480040 shutdown_connections 2024-05-20T17:49:52.047 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:51.430+0000 7f76e108b700 1 -- 172.21.15.129:0/1136480040 wait complete. 2024-05-20T17:49:52.048 INFO:teuthology.orchestra.run.smithi129.stdout:Fetching dashboard port number... 2024-05-20T17:49:52.296 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:52 smithi129 ceph-mon[32029]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:49:52.296 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:52 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:49:52.296 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:52 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:49:53.226 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 8443 2024-05-20T17:49:53.226 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.409+0000 7f8d8271f700 1 Processor -- start 2024-05-20T17:49:53.226 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.409+0000 7f8d8271f700 1 -- start start 2024-05-20T17:49:53.226 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.409+0000 7f8d8271f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c10b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:53.226 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.409+0000 7f8d8271f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7c10bd40 con 0x7f8d7c07c690 2024-05-20T17:49:53.226 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.410+0000 7f8d7bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c10b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:53.226 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.410+0000 7f8d7bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c10b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41100/0 (socket says 172.21.15.129:41100) 2024-05-20T17:49:53.226 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.410+0000 7f8d7bfff700 1 -- 172.21.15.129:0/3359704950 learned_addr learned my addr 172.21.15.129:0/3359704950 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.410+0000 7f8d7bfff700 1 -- 172.21.15.129:0/3359704950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d7c10bec0 con 0x7f8d7c07c690 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.410+0000 7f8d7bfff700 1 --2- 172.21.15.129:0/3359704950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c10b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0098a0 tx=0x7f8d6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336343ca2662ae66 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.410+0000 7f8d7affd700 1 -- 172.21.15.129:0/3359704950 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d6c004160 con 0x7f8d7c07c690 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.410+0000 7f8d7affd700 1 -- 172.21.15.129:0/3359704950 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d6c004300 con 0x7f8d7c07c690 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.410+0000 7f8d7affd700 1 -- 172.21.15.129:0/3359704950 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d6c011b20 con 0x7f8d7c07c690 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.411+0000 7f8d8271f700 1 -- 172.21.15.129:0/3359704950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 msgr2=0x7f8d7c10b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.411+0000 7f8d8271f700 1 --2- 172.21.15.129:0/3359704950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c10b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0098a0 tx=0x7f8d6c009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.411+0000 7f8d8271f700 1 -- 172.21.15.129:0/3359704950 shutdown_connections 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.411+0000 7f8d8271f700 1 --2- 172.21.15.129:0/3359704950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c10b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.411+0000 7f8d8271f700 1 -- 172.21.15.129:0/3359704950 >> 172.21.15.129:0/3359704950 conn(0x7f8d7c103120 msgr2=0x7f8d7c1055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.411+0000 7f8d8271f700 1 -- 172.21.15.129:0/3359704950 shutdown_connections 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.411+0000 7f8d8271f700 1 -- 172.21.15.129:0/3359704950 wait complete. 2024-05-20T17:49:53.227 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.412+0000 7f8d8271f700 1 Processor -- start 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.412+0000 7f8d8271f700 1 -- start start 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.412+0000 7f8d8271f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.412+0000 7f8d8271f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7c10cfe0 con 0x7f8d7c07c690 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.413+0000 7f8d7bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.413+0000 7f8d7bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c19ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41116/0 (socket says 172.21.15.129:41116) 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.413+0000 7f8d7bfff700 1 -- 172.21.15.129:0/3632180974 learned_addr learned my addr 172.21.15.129:0/3632180974 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.413+0000 7f8d7bfff700 1 -- 172.21.15.129:0/3632180974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d7c19d3b0 con 0x7f8d7c07c690 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.413+0000 7f8d7bfff700 1 --2- 172.21.15.129:0/3632180974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c19ce70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c011740 tx=0x7f8d6c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.413+0000 7f8d797fa700 1 -- 172.21.15.129:0/3632180974 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d6c003a40 con 0x7f8d7c07c690 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.413+0000 7f8d797fa700 1 -- 172.21.15.129:0/3632180974 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d6c003be0 con 0x7f8d7c07c690 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.413+0000 7f8d797fa700 1 -- 172.21.15.129:0/3632180974 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d6c01a720 con 0x7f8d7c07c690 2024-05-20T17:49:53.228 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.413+0000 7f8d8271f700 1 -- 172.21.15.129:0/3632180974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d7c19d640 con 0x7f8d7c07c690 2024-05-20T17:49:53.229 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.414+0000 7f8d8271f700 1 -- 172.21.15.129:0/3632180974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d7c1a0250 con 0x7f8d7c07c690 2024-05-20T17:49:53.229 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.415+0000 7f8d797fa700 1 -- 172.21.15.129:0/3632180974 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f8d6c01aa20 con 0x7f8d7c07c690 2024-05-20T17:49:53.229 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.415+0000 7f8d8271f700 1 -- 172.21.15.129:0/3632180974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d7c19daa0 con 0x7f8d7c07c690 2024-05-20T17:49:53.229 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.415+0000 7f8d797fa700 1 --2- 172.21.15.129:0/3632180974 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8d68039110 0x7f8d6803b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:53.229 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.415+0000 7f8d797fa700 1 -- 172.21.15.129:0/3632180974 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8d6c04e770 con 0x7f8d7c07c690 2024-05-20T17:49:53.229 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.415+0000 7f8d7b7fe700 1 --2- 172.21.15.129:0/3632180974 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8d68039110 0x7f8d6803b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:53.229 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.416+0000 7f8d7b7fe700 1 --2- 172.21.15.129:0/3632180974 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8d68039110 0x7f8d6803b620 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8d64006fd0 tx=0x7f8d64006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:53.229 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.420+0000 7f8d797fa700 1 -- 172.21.15.129:0/3632180974 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8d6c011e20 con 0x7f8d7c07c690 2024-05-20T17:49:53.229 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.623+0000 7f8d8271f700 1 -- 172.21.15.129:0/3632180974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f8d7c063ca0 con 0x7f8d7c07c690 2024-05-20T17:49:53.229 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.624+0000 7f8d797fa700 1 -- 172.21.15.129:0/3632180974 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f8d6c019c70 con 0x7f8d7c07c690 2024-05-20T17:49:53.229 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.626+0000 7f8d8271f700 1 -- 172.21.15.129:0/3632180974 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8d68039110 msgr2=0x7f8d6803b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:53.230 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.626+0000 7f8d8271f700 1 --2- 172.21.15.129:0/3632180974 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8d68039110 0x7f8d6803b620 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8d64006fd0 tx=0x7f8d64006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:53.230 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.626+0000 7f8d8271f700 1 -- 172.21.15.129:0/3632180974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 msgr2=0x7f8d7c19ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:53.230 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.626+0000 7f8d8271f700 1 --2- 172.21.15.129:0/3632180974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c19ce70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c011740 tx=0x7f8d6c011ba0 comp rx=0 tx=0).stop 2024-05-20T17:49:53.230 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.627+0000 7f8d8271f700 1 -- 172.21.15.129:0/3632180974 shutdown_connections 2024-05-20T17:49:53.230 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.627+0000 7f8d8271f700 1 --2- 172.21.15.129:0/3632180974 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8d68039110 0x7f8d6803b620 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:53.230 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.627+0000 7f8d8271f700 1 --2- 172.21.15.129:0/3632180974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7c07c690 0x7f8d7c19ce70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:53.230 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.627+0000 7f8d8271f700 1 -- 172.21.15.129:0/3632180974 >> 172.21.15.129:0/3632180974 conn(0x7f8d7c103120 msgr2=0x7f8d7c1038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:53.230 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.627+0000 7f8d8271f700 1 -- 172.21.15.129:0/3632180974 shutdown_connections 2024-05-20T17:49:53.230 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:52.627+0000 7f8d8271f700 1 -- 172.21.15.129:0/3632180974 wait complete. 2024-05-20T17:49:53.232 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-20T17:49:53.232 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout disabled 2024-05-20T17:49:53.240 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-20T17:49:53.240 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-05-20T17:49:53.240 INFO:teuthology.orchestra.run.smithi129.stdout:firewalld.service is not enabled 2024-05-20T17:49:53.240 INFO:teuthology.orchestra.run.smithi129.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-20T17:49:53.242 INFO:teuthology.orchestra.run.smithi129.stdout:Ceph Dashboard is now available at: 2024-05-20T17:49:53.242 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:49:53.242 INFO:teuthology.orchestra.run.smithi129.stdout: URL: https://smithi129.front.sepia.ceph.com:8443/ 2024-05-20T17:49:53.242 INFO:teuthology.orchestra.run.smithi129.stdout: User: admin 2024-05-20T17:49:53.243 INFO:teuthology.orchestra.run.smithi129.stdout: Password: eznnwc47cr 2024-05-20T17:49:53.243 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:49:53.243 INFO:teuthology.orchestra.run.smithi129.stdout:Saving cluster configuration to /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config directory 2024-05-20T17:49:53.243 INFO:teuthology.orchestra.run.smithi129.stdout:Enabling autotune for osd_memory_target 2024-05-20T17:49:53.530 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:53 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3632180974' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-20T17:49:54.558 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.641+0000 7f78f6af2700 1 Processor -- start 2024-05-20T17:49:54.559 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.641+0000 7f78f6af2700 1 -- start start 2024-05-20T17:49:54.559 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.641+0000 7f78f6af2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f0109090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:54.559 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.641+0000 7f78f6af2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78f0109680 con 0x7f78f0108c80 2024-05-20T17:49:54.559 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.642+0000 7f78f488e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f0109090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:54.559 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.642+0000 7f78f488e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f0109090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41122/0 (socket says 172.21.15.129:41122) 2024-05-20T17:49:54.559 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.642+0000 7f78f488e700 1 -- 172.21.15.129:0/2027165783 learned_addr learned my addr 172.21.15.129:0/2027165783 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:54.559 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.642+0000 7f78f488e700 1 -- 172.21.15.129:0/2027165783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f0109d60 con 0x7f78f0108c80 2024-05-20T17:49:54.559 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.642+0000 7f78f488e700 1 --2- 172.21.15.129:0/2027165783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f0109090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f78e40098a0 tx=0x7f78e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbac3852e09e9548 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:54.559 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.642+0000 7f78ef7fe700 1 -- 172.21.15.129:0/2027165783 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78e4004160 con 0x7f78f0108c80 2024-05-20T17:49:54.559 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.642+0000 7f78ef7fe700 1 -- 172.21.15.129:0/2027165783 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e4004300 con 0x7f78f0108c80 2024-05-20T17:49:54.559 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.642+0000 7f78ef7fe700 1 -- 172.21.15.129:0/2027165783 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78e4011b20 con 0x7f78f0108c80 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.643+0000 7f78f6af2700 1 -- 172.21.15.129:0/2027165783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 msgr2=0x7f78f0109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.643+0000 7f78f6af2700 1 --2- 172.21.15.129:0/2027165783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f0109090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f78e40098a0 tx=0x7f78e4009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.643+0000 7f78f6af2700 1 -- 172.21.15.129:0/2027165783 shutdown_connections 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.643+0000 7f78f6af2700 1 --2- 172.21.15.129:0/2027165783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f0109090 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.643+0000 7f78f6af2700 1 -- 172.21.15.129:0/2027165783 >> 172.21.15.129:0/2027165783 conn(0x7f78f01042e0 msgr2=0x7f78f0106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.643+0000 7f78f6af2700 1 -- 172.21.15.129:0/2027165783 shutdown_connections 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.643+0000 7f78f6af2700 1 -- 172.21.15.129:0/2027165783 wait complete. 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.644+0000 7f78f6af2700 1 Processor -- start 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.644+0000 7f78f6af2700 1 -- start start 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.644+0000 7f78f6af2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f019cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.644+0000 7f78f6af2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78f010d110 con 0x7f78f0108c80 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.644+0000 7f78f488e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f019cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:54.560 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.644+0000 7f78f488e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f019cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41132/0 (socket says 172.21.15.129:41132) 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.644+0000 7f78f488e700 1 -- 172.21.15.129:0/2255735274 learned_addr learned my addr 172.21.15.129:0/2255735274 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.644+0000 7f78f488e700 1 -- 172.21.15.129:0/2255735274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f019d4a0 con 0x7f78f0108c80 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.645+0000 7f78f488e700 1 --2- 172.21.15.129:0/2255735274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f019cf60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009fb0 tx=0x7f78e4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.645+0000 7f78edffb700 1 -- 172.21.15.129:0/2255735274 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78e4003930 con 0x7f78f0108c80 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.645+0000 7f78edffb700 1 -- 172.21.15.129:0/2255735274 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e4003ad0 con 0x7f78f0108c80 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.645+0000 7f78f6af2700 1 -- 172.21.15.129:0/2255735274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f019d730 con 0x7f78f0108c80 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.645+0000 7f78edffb700 1 -- 172.21.15.129:0/2255735274 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78e4019670 con 0x7f78f0108c80 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.645+0000 7f78f6af2700 1 -- 172.21.15.129:0/2255735274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f01a0340 con 0x7f78f0108c80 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.646+0000 7f78f6af2700 1 -- 172.21.15.129:0/2255735274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f0063ca0 con 0x7f78f0108c80 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.646+0000 7f78edffb700 1 -- 172.21.15.129:0/2255735274 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f78e4019920 con 0x7f78f0108c80 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.646+0000 7f78edffb700 1 --2- 172.21.15.129:0/2255735274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78e0039170 0x7f78e003b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.646+0000 7f78edffb700 1 -- 172.21.15.129:0/2255735274 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f78e404d790 con 0x7f78f0108c80 2024-05-20T17:49:54.561 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.646+0000 7f78effff700 1 --2- 172.21.15.129:0/2255735274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78e0039170 0x7f78e003b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.647+0000 7f78effff700 1 --2- 172.21.15.129:0/2255735274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78e0039170 0x7f78e003b680 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f78dc006fd0 tx=0x7f78dc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.652+0000 7f78edffb700 1 -- 172.21.15.129:0/2255735274 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f78e4011e20 con 0x7f78f0108c80 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.853+0000 7f78f6af2700 1 -- 172.21.15.129:0/2255735274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f78f01a0510 con 0x7f78f0108c80 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.853+0000 7f78edffb700 1 -- 172.21.15.129:0/2255735274 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f78e4019d60 con 0x7f78f0108c80 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.855+0000 7f78f6af2700 1 -- 172.21.15.129:0/2255735274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78e0039170 msgr2=0x7f78e003b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.855+0000 7f78f6af2700 1 --2- 172.21.15.129:0/2255735274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78e0039170 0x7f78e003b680 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f78dc006fd0 tx=0x7f78dc006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.855+0000 7f78f6af2700 1 -- 172.21.15.129:0/2255735274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 msgr2=0x7f78f019cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.855+0000 7f78f6af2700 1 --2- 172.21.15.129:0/2255735274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f019cf60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f78e4009fb0 tx=0x7f78e4011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.855+0000 7f78f6af2700 1 -- 172.21.15.129:0/2255735274 shutdown_connections 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.855+0000 7f78f6af2700 1 --2- 172.21.15.129:0/2255735274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78e0039170 0x7f78e003b680 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.855+0000 7f78f6af2700 1 --2- 172.21.15.129:0/2255735274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0108c80 0x7f78f019cf60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:54.562 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.855+0000 7f78f6af2700 1 -- 172.21.15.129:0/2255735274 >> 172.21.15.129:0/2255735274 conn(0x7f78f01042e0 msgr2=0x7f78f0104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:54.563 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.856+0000 7f78f6af2700 1 -- 172.21.15.129:0/2255735274 shutdown_connections 2024-05-20T17:49:54.563 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:53.856+0000 7f78f6af2700 1 -- 172.21.15.129:0/2255735274 wait complete. 2024-05-20T17:49:55.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:55 smithi129 ceph-mon[32029]: mgrmap e13: x(active, since 6s) 2024-05-20T17:49:56.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.992+0000 7f79cc130700 1 Processor -- start 2024-05-20T17:49:56.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.992+0000 7f79cc130700 1 -- start start 2024-05-20T17:49:56.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.993+0000 7f79cc130700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c410b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:56.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.993+0000 7f79cc130700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79c410bdb0 con 0x7f79c407bbe0 2024-05-20T17:49:56.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.993+0000 7f79c9ecc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c410b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:56.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.993+0000 7f79c9ecc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c410b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41142/0 (socket says 172.21.15.129:41142) 2024-05-20T17:49:56.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.993+0000 7f79c9ecc700 1 -- 172.21.15.129:0/373582402 learned_addr learned my addr 172.21.15.129:0/373582402 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:56.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.993+0000 7f79c9ecc700 1 -- 172.21.15.129:0/373582402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c410bf30 con 0x7f79c407bbe0 2024-05-20T17:49:56.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.993+0000 7f79c9ecc700 1 --2- 172.21.15.129:0/373582402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c410b7e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f79c00098a0 tx=0x7f79c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e7b10f0e952aaa server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:56.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.993+0000 7f79c8eca700 1 -- 172.21.15.129:0/373582402 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c0004160 con 0x7f79c407bbe0 2024-05-20T17:49:56.001 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.993+0000 7f79c8eca700 1 -- 172.21.15.129:0/373582402 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c0004300 con 0x7f79c407bbe0 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.993+0000 7f79c8eca700 1 -- 172.21.15.129:0/373582402 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c0011b20 con 0x7f79c407bbe0 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.994+0000 7f79cc130700 1 -- 172.21.15.129:0/373582402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 msgr2=0x7f79c410b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.994+0000 7f79cc130700 1 --2- 172.21.15.129:0/373582402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c410b7e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f79c00098a0 tx=0x7f79c0009bb0 comp rx=0 tx=0).stop 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.994+0000 7f79cc130700 1 -- 172.21.15.129:0/373582402 shutdown_connections 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.994+0000 7f79cc130700 1 --2- 172.21.15.129:0/373582402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c410b7e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.994+0000 7f79cc130700 1 -- 172.21.15.129:0/373582402 >> 172.21.15.129:0/373582402 conn(0x7f79c41031d0 msgr2=0x7f79c4105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.994+0000 7f79cc130700 1 -- 172.21.15.129:0/373582402 shutdown_connections 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.995+0000 7f79cc130700 1 -- 172.21.15.129:0/373582402 wait complete. 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.995+0000 7f79cc130700 1 Processor -- start 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.995+0000 7f79cc130700 1 -- start start 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.995+0000 7f79cc130700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.995+0000 7f79cc130700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79c410cff0 con 0x7f79c407bbe0 2024-05-20T17:49:56.002 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.995+0000 7f79c9ecc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.996+0000 7f79c9ecc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c419cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41144/0 (socket says 172.21.15.129:41144) 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.996+0000 7f79c9ecc700 1 -- 172.21.15.129:0/3069090275 learned_addr learned my addr 172.21.15.129:0/3069090275 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.996+0000 7f79c9ecc700 1 -- 172.21.15.129:0/3069090275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c419d3e0 con 0x7f79c407bbe0 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.996+0000 7f79c9ecc700 1 --2- 172.21.15.129:0/3069090275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c419cea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f79c0000c00 tx=0x7f79c0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.996+0000 7f79b6ffd700 1 -- 172.21.15.129:0/3069090275 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c0003b60 con 0x7f79c407bbe0 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.996+0000 7f79cc130700 1 -- 172.21.15.129:0/3069090275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c419d670 con 0x7f79c407bbe0 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.996+0000 7f79b6ffd700 1 -- 172.21.15.129:0/3069090275 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c001a910 con 0x7f79c407bbe0 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.996+0000 7f79b6ffd700 1 -- 172.21.15.129:0/3069090275 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c0024930 con 0x7f79c407bbe0 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.996+0000 7f79cc130700 1 -- 172.21.15.129:0/3069090275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c41a0280 con 0x7f79c407bbe0 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.997+0000 7f79b6ffd700 1 -- 172.21.15.129:0/3069090275 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f79c0022020 con 0x7f79c407bbe0 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.997+0000 7f79cc130700 1 -- 172.21.15.129:0/3069090275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79c419dad0 con 0x7f79c407bbe0 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.997+0000 7f79b6ffd700 1 --2- 172.21.15.129:0/3069090275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79b0039200 0x7f79b003b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:56.003 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.998+0000 7f79b6ffd700 1 -- 172.21.15.129:0/3069090275 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f79c004d760 con 0x7f79c407bbe0 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.998+0000 7f79c96cb700 1 --2- 172.21.15.129:0/3069090275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79b0039200 0x7f79b003b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:54.998+0000 7f79c96cb700 1 --2- 172.21.15.129:0/3069090275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79b0039200 0x7f79b003b710 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f79b8006fd0 tx=0x7f79b8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.003+0000 7f79b6ffd700 1 -- 172.21.15.129:0/3069090275 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f79c000b3e0 con 0x7f79c407bbe0 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.317+0000 7f79cc130700 1 -- 172.21.15.129:0/3069090275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f79c4063ca0 con 0x7f79c407bbe0 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.320+0000 7f79b6ffd700 1 -- 172.21.15.129:0/3069090275 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) ==== 153+0+0 (secure 0 0 0) 0x7f79c00172b0 con 0x7f79c407bbe0 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.323+0000 7f79cc130700 1 -- 172.21.15.129:0/3069090275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79b0039200 msgr2=0x7f79b003b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.323+0000 7f79cc130700 1 --2- 172.21.15.129:0/3069090275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79b0039200 0x7f79b003b710 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f79b8006fd0 tx=0x7f79b8006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.323+0000 7f79cc130700 1 -- 172.21.15.129:0/3069090275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 msgr2=0x7f79c419cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.323+0000 7f79cc130700 1 --2- 172.21.15.129:0/3069090275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c419cea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f79c0000c00 tx=0x7f79c0011c10 comp rx=0 tx=0).stop 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.323+0000 7f79cc130700 1 -- 172.21.15.129:0/3069090275 shutdown_connections 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.323+0000 7f79cc130700 1 --2- 172.21.15.129:0/3069090275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79b0039200 0x7f79b003b710 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.323+0000 7f79cc130700 1 --2- 172.21.15.129:0/3069090275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79c407bbe0 0x7f79c419cea0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.323+0000 7f79cc130700 1 -- 172.21.15.129:0/3069090275 >> 172.21.15.129:0/3069090275 conn(0x7f79c41031d0 msgr2=0x7f79c4104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:56.004 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.323+0000 7f79cc130700 1 -- 172.21.15.129:0/3069090275 shutdown_connections 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-05-20T17:49:55.323+0000 7f79cc130700 1 -- 172.21.15.129:0/3069090275 wait complete. 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout:Or, if you are only running a single cluster on this host: 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:49:56.005 INFO:teuthology.orchestra.run.smithi129.stdout: ceph telemetry on 2024-05-20T17:49:56.006 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:49:56.006 INFO:teuthology.orchestra.run.smithi129.stdout:For more information see: 2024-05-20T17:49:56.006 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:49:56.006 INFO:teuthology.orchestra.run.smithi129.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-20T17:49:56.006 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:49:56.006 INFO:teuthology.orchestra.run.smithi129.stdout:Bootstrap complete. 2024-05-20T17:49:56.041 INFO:tasks.cephadm:Fetching config... 2024-05-20T17:49:56.042 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:49:56.042 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-20T17:49:56.058 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-20T17:49:56.058 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:49:56.058 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-20T17:49:56.114 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-20T17:49:56.114 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:49:56.114 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-20T17:49:56.179 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-20T17:49:56.179 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:49:56.179 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-20T17:49:56.235 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-20T17:49:56.235 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHGwKD8K6rp9Hhw2NEccATSZ12j5Oga2cFfFtVTSjlWwnilwx/4DuXALXS3PeQ7HMMSxqqJ+QR7IQK2gRNZ6rkOUjkFs8LsImwI4K4tD5+r4fTqqDLBNH1P577LFEc68U/pnfrVdb0xd95dEALFplq5uNUZizD6HA8QviW5OyELNeaFVKafw2dEMK+33I4djdKpLY59Tuw59HMDG8c6NSjpcL4h02JYGd6r/LBSOaFhObjIC5Hdr3l0/0Ajlz73/Is9Jc3tZGnqj2sh1LGfkzm/KMA7VR8HfCYETDyU63zEGTDusNGT8VkQQneS60jbUfu3+3vlVSIlocM+s6P4c/bElrBln0LPzA76uy9bzZ3TAZvgpESBPw8u1pIhlItkVtGwDls2yluXnWsj8BqNOiTbfg38cSABedeodTMJqdlK21AQ/p7fzJPx4gG9MZTRJ2QnBSUdPNxXTquVL4as+Ubvpe4mEGO5HUlGVIVW+SKYkWR0pMGZ4FbCybtIA9BiH8= ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-20T17:49:56.310 INFO:teuthology.orchestra.run.smithi129.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHGwKD8K6rp9Hhw2NEccATSZ12j5Oga2cFfFtVTSjlWwnilwx/4DuXALXS3PeQ7HMMSxqqJ+QR7IQK2gRNZ6rkOUjkFs8LsImwI4K4tD5+r4fTqqDLBNH1P577LFEc68U/pnfrVdb0xd95dEALFplq5uNUZizD6HA8QviW5OyELNeaFVKafw2dEMK+33I4djdKpLY59Tuw59HMDG8c6NSjpcL4h02JYGd6r/LBSOaFhObjIC5Hdr3l0/0Ajlz73/Is9Jc3tZGnqj2sh1LGfkzm/KMA7VR8HfCYETDyU63zEGTDusNGT8VkQQneS60jbUfu3+3vlVSIlocM+s6P4c/bElrBln0LPzA76uy9bzZ3TAZvgpESBPw8u1pIhlItkVtGwDls2yluXnWsj8BqNOiTbfg38cSABedeodTMJqdlK21AQ/p7fzJPx4gG9MZTRJ2QnBSUdPNxXTquVL4as+Ubvpe4mEGO5HUlGVIVW+SKYkWR0pMGZ4FbCybtIA9BiH8= ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:49:56.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHGwKD8K6rp9Hhw2NEccATSZ12j5Oga2cFfFtVTSjlWwnilwx/4DuXALXS3PeQ7HMMSxqqJ+QR7IQK2gRNZ6rkOUjkFs8LsImwI4K4tD5+r4fTqqDLBNH1P577LFEc68U/pnfrVdb0xd95dEALFplq5uNUZizD6HA8QviW5OyELNeaFVKafw2dEMK+33I4djdKpLY59Tuw59HMDG8c6NSjpcL4h02JYGd6r/LBSOaFhObjIC5Hdr3l0/0Ajlz73/Is9Jc3tZGnqj2sh1LGfkzm/KMA7VR8HfCYETDyU63zEGTDusNGT8VkQQneS60jbUfu3+3vlVSIlocM+s6P4c/bElrBln0LPzA76uy9bzZ3TAZvgpESBPw8u1pIhlItkVtGwDls2yluXnWsj8BqNOiTbfg38cSABedeodTMJqdlK21AQ/p7fzJPx4gG9MZTRJ2QnBSUdPNxXTquVL4as+Ubvpe4mEGO5HUlGVIVW+SKYkWR0pMGZ4FbCybtIA9BiH8= ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-20T17:49:56.361 INFO:teuthology.orchestra.run.smithi154.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHGwKD8K6rp9Hhw2NEccATSZ12j5Oga2cFfFtVTSjlWwnilwx/4DuXALXS3PeQ7HMMSxqqJ+QR7IQK2gRNZ6rkOUjkFs8LsImwI4K4tD5+r4fTqqDLBNH1P577LFEc68U/pnfrVdb0xd95dEALFplq5uNUZizD6HA8QviW5OyELNeaFVKafw2dEMK+33I4djdKpLY59Tuw59HMDG8c6NSjpcL4h02JYGd6r/LBSOaFhObjIC5Hdr3l0/0Ajlz73/Is9Jc3tZGnqj2sh1LGfkzm/KMA7VR8HfCYETDyU63zEGTDusNGT8VkQQneS60jbUfu3+3vlVSIlocM+s6P4c/bElrBln0LPzA76uy9bzZ3TAZvgpESBPw8u1pIhlItkVtGwDls2yluXnWsj8BqNOiTbfg38cSABedeodTMJqdlK21AQ/p7fzJPx4gG9MZTRJ2QnBSUdPNxXTquVL4as+Ubvpe4mEGO5HUlGVIVW+SKYkWR0pMGZ4FbCybtIA9BiH8= ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:49:56.373 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHGwKD8K6rp9Hhw2NEccATSZ12j5Oga2cFfFtVTSjlWwnilwx/4DuXALXS3PeQ7HMMSxqqJ+QR7IQK2gRNZ6rkOUjkFs8LsImwI4K4tD5+r4fTqqDLBNH1P577LFEc68U/pnfrVdb0xd95dEALFplq5uNUZizD6HA8QviW5OyELNeaFVKafw2dEMK+33I4djdKpLY59Tuw59HMDG8c6NSjpcL4h02JYGd6r/LBSOaFhObjIC5Hdr3l0/0Ajlz73/Is9Jc3tZGnqj2sh1LGfkzm/KMA7VR8HfCYETDyU63zEGTDusNGT8VkQQneS60jbUfu3+3vlVSIlocM+s6P4c/bElrBln0LPzA76uy9bzZ3TAZvgpESBPw8u1pIhlItkVtGwDls2yluXnWsj8BqNOiTbfg38cSABedeodTMJqdlK21AQ/p7fzJPx4gG9MZTRJ2QnBSUdPNxXTquVL4as+Ubvpe4mEGO5HUlGVIVW+SKYkWR0pMGZ4FbCybtIA9BiH8= ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-20T17:49:56.411 INFO:teuthology.orchestra.run.smithi204.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHGwKD8K6rp9Hhw2NEccATSZ12j5Oga2cFfFtVTSjlWwnilwx/4DuXALXS3PeQ7HMMSxqqJ+QR7IQK2gRNZ6rkOUjkFs8LsImwI4K4tD5+r4fTqqDLBNH1P577LFEc68U/pnfrVdb0xd95dEALFplq5uNUZizD6HA8QviW5OyELNeaFVKafw2dEMK+33I4djdKpLY59Tuw59HMDG8c6NSjpcL4h02JYGd6r/LBSOaFhObjIC5Hdr3l0/0Ajlz73/Is9Jc3tZGnqj2sh1LGfkzm/KMA7VR8HfCYETDyU63zEGTDusNGT8VkQQneS60jbUfu3+3vlVSIlocM+s6P4c/bElrBln0LPzA76uy9bzZ3TAZvgpESBPw8u1pIhlItkVtGwDls2yluXnWsj8BqNOiTbfg38cSABedeodTMJqdlK21AQ/p7fzJPx4gG9MZTRJ2QnBSUdPNxXTquVL4as+Ubvpe4mEGO5HUlGVIVW+SKYkWR0pMGZ4FbCybtIA9BiH8= ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:49:56.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-20T17:49:56.659 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:49:56.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:56 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3069090275' entity='client.admin' 2024-05-20T17:49:58.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.834+0000 7ffa1a28b700 1 -- 172.21.15.129:0/2206891401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa14076600 msgr2=0x7ffa14074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.834+0000 7ffa1a28b700 1 --2- 172.21.15.129:0/2206891401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa14076600 0x7ffa14074a30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009910 tx=0x7ffa04009c20 comp rx=0 tx=0).stop 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.834+0000 7ffa1a28b700 1 -- 172.21.15.129:0/2206891401 shutdown_connections 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.834+0000 7ffa1a28b700 1 --2- 172.21.15.129:0/2206891401 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa14076600 0x7ffa14074a30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.834+0000 7ffa1a28b700 1 -- 172.21.15.129:0/2206891401 >> 172.21.15.129:0/2206891401 conn(0x7ffa140fe090 msgr2=0x7ffa14100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.834+0000 7ffa1a28b700 1 -- 172.21.15.129:0/2206891401 shutdown_connections 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.834+0000 7ffa1a28b700 1 -- 172.21.15.129:0/2206891401 wait complete. 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.834+0000 7ffa1a28b700 1 Processor -- start 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.835+0000 7ffa1a28b700 1 -- start start 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.835+0000 7ffa1a28b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa14076600 0x7ffa1419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.835+0000 7ffa1a28b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1410f220 con 0x7ffa14076600 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.835+0000 7ffa13fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa14076600 0x7ffa1419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:58.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.835+0000 7ffa13fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa14076600 0x7ffa1419ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41164/0 (socket says 172.21.15.129:41164) 2024-05-20T17:49:58.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.835+0000 7ffa13fff700 1 -- 172.21.15.129:0/1793658466 learned_addr learned my addr 172.21.15.129:0/1793658466 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:49:58.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.835+0000 7ffa13fff700 1 -- 172.21.15.129:0/1793658466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1419f190 con 0x7ffa14076600 2024-05-20T17:49:58.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.835+0000 7ffa13fff700 1 --2- 172.21.15.129:0/1793658466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa14076600 0x7ffa1419ec50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffa04004fb0 tx=0x7ffa04005090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:58.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.836+0000 7ffa117fa700 1 -- 172.21.15.129:0/1793658466 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa04027070 con 0x7ffa14076600 2024-05-20T17:49:58.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.836+0000 7ffa1a28b700 1 -- 172.21.15.129:0/1793658466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1419f420 con 0x7ffa14076600 2024-05-20T17:49:58.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.836+0000 7ffa1a28b700 1 -- 172.21.15.129:0/1793658466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa141a2030 con 0x7ffa14076600 2024-05-20T17:49:58.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.836+0000 7ffa117fa700 1 -- 172.21.15.129:0/1793658466 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa04022470 con 0x7ffa14076600 2024-05-20T17:49:58.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.836+0000 7ffa117fa700 1 -- 172.21.15.129:0/1793658466 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa0400f5b0 con 0x7ffa14076600 2024-05-20T17:49:58.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.837+0000 7ffa117fa700 1 -- 172.21.15.129:0/1793658466 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7ffa0402a070 con 0x7ffa14076600 2024-05-20T17:49:58.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.837+0000 7ffa117fa700 1 --2- 172.21.15.129:0/1793658466 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff9fc039150 0x7ff9fc03b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:49:58.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.837+0000 7ffa117fa700 1 -- 172.21.15.129:0/1793658466 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ffa0401e030 con 0x7ffa14076600 2024-05-20T17:49:58.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.837+0000 7ffa1a28b700 1 -- 172.21.15.129:0/1793658466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa00005360 con 0x7ffa14076600 2024-05-20T17:49:58.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.837+0000 7ffa137fe700 1 --2- 172.21.15.129:0/1793658466 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff9fc039150 0x7ff9fc03b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:49:58.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.837+0000 7ffa137fe700 1 --2- 172.21.15.129:0/1793658466 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff9fc039150 0x7ff9fc03b660 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ffa08006fd0 tx=0x7ffa08006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:49:58.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:58.842+0000 7ffa117fa700 1 -- 172.21.15.129:0/1793658466 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffa04013070 con 0x7ffa14076600 2024-05-20T17:49:59.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.038+0000 7ffa1a28b700 1 -- 172.21.15.129:0/1793658466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7ffa00005650 con 0x7ffa14076600 2024-05-20T17:49:59.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.040+0000 7ffa117fa700 1 -- 172.21.15.129:0/1793658466 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7ffa040292c0 con 0x7ffa14076600 2024-05-20T17:49:59.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.043+0000 7ffa1a28b700 1 -- 172.21.15.129:0/1793658466 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff9fc039150 msgr2=0x7ff9fc03b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:59.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.043+0000 7ffa1a28b700 1 --2- 172.21.15.129:0/1793658466 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff9fc039150 0x7ff9fc03b660 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ffa08006fd0 tx=0x7ffa08006e40 comp rx=0 tx=0).stop 2024-05-20T17:49:59.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.043+0000 7ffa1a28b700 1 -- 172.21.15.129:0/1793658466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa14076600 msgr2=0x7ffa1419ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:49:59.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.043+0000 7ffa1a28b700 1 --2- 172.21.15.129:0/1793658466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa14076600 0x7ffa1419ec50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffa04004fb0 tx=0x7ffa04005090 comp rx=0 tx=0).stop 2024-05-20T17:49:59.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.043+0000 7ffa1a28b700 1 -- 172.21.15.129:0/1793658466 shutdown_connections 2024-05-20T17:49:59.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.043+0000 7ffa1a28b700 1 --2- 172.21.15.129:0/1793658466 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff9fc039150 0x7ff9fc03b660 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:59.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.043+0000 7ffa1a28b700 1 --2- 172.21.15.129:0/1793658466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa14076600 0x7ffa1419ec50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:49:59.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.043+0000 7ffa1a28b700 1 -- 172.21.15.129:0/1793658466 >> 172.21.15.129:0/1793658466 conn(0x7ffa140fe090 msgr2=0x7ffa140fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:49:59.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.043+0000 7ffa1a28b700 1 -- 172.21.15.129:0/1793658466 shutdown_connections 2024-05-20T17:49:59.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:49:59.043+0000 7ffa1a28b700 1 -- 172.21.15.129:0/1793658466 wait complete. 2024-05-20T17:49:59.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:49:59.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:49:59.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:49:59.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:49:59.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:49:59.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:49:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:49:59.550 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-20T17:49:59.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-20T17:49:59.786 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:50:00.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:00 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1793658466' entity='client.admin' 2024-05-20T17:50:00.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:00.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:01.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.272+0000 7fb0aff02700 1 -- 172.21.15.129:0/1802412250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0a8101f20 msgr2=0x7fb0a8102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:01.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.272+0000 7fb0aff02700 1 --2- 172.21.15.129:0/1802412250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0a8101f20 0x7fb0a8102330 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009910 tx=0x7fb09c009c20 comp rx=0 tx=0).stop 2024-05-20T17:50:01.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.272+0000 7fb0aff02700 1 -- 172.21.15.129:0/1802412250 shutdown_connections 2024-05-20T17:50:01.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.272+0000 7fb0aff02700 1 --2- 172.21.15.129:0/1802412250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0a8101f20 0x7fb0a8102330 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:01.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.272+0000 7fb0aff02700 1 -- 172.21.15.129:0/1802412250 >> 172.21.15.129:0/1802412250 conn(0x7fb0a80fc710 msgr2=0x7fb0a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:01.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.273+0000 7fb0aff02700 1 -- 172.21.15.129:0/1802412250 shutdown_connections 2024-05-20T17:50:01.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.273+0000 7fb0aff02700 1 -- 172.21.15.129:0/1802412250 wait complete. 2024-05-20T17:50:01.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.273+0000 7fb0aff02700 1 Processor -- start 2024-05-20T17:50:01.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.273+0000 7fb0aff02700 1 -- start start 2024-05-20T17:50:01.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.273+0000 7fb0aff02700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0a8101f20 0x7fb0a81986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:01.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.273+0000 7fb0aff02700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a8107a60 con 0x7fb0a8101f20 2024-05-20T17:50:01.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.274+0000 7fb0adc9e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0a8101f20 0x7fb0a81986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:01.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.274+0000 7fb0adc9e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0a8101f20 0x7fb0a81986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34930/0 (socket says 172.21.15.129:34930) 2024-05-20T17:50:01.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.274+0000 7fb0adc9e700 1 -- 172.21.15.129:0/2959907073 learned_addr learned my addr 172.21.15.129:0/2959907073 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:50:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.274+0000 7fb0adc9e700 1 -- 172.21.15.129:0/2959907073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a8198bf0 con 0x7fb0a8101f20 2024-05-20T17:50:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.274+0000 7fb0adc9e700 1 --2- 172.21.15.129:0/2959907073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0a8101f20 0x7fb0a81986b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00b5c0 tx=0x7fb09c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.274+0000 7fb09affd700 1 -- 172.21.15.129:0/2959907073 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb09c027070 con 0x7fb0a8101f20 2024-05-20T17:50:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.274+0000 7fb09affd700 1 -- 172.21.15.129:0/2959907073 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb09c022470 con 0x7fb0a8101f20 2024-05-20T17:50:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.274+0000 7fb09affd700 1 -- 172.21.15.129:0/2959907073 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb09c00f5d0 con 0x7fb0a8101f20 2024-05-20T17:50:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.274+0000 7fb0aff02700 1 -- 172.21.15.129:0/2959907073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0a8198e80 con 0x7fb0a8101f20 2024-05-20T17:50:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.274+0000 7fb0aff02700 1 -- 172.21.15.129:0/2959907073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0a819ba90 con 0x7fb0a8101f20 2024-05-20T17:50:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.275+0000 7fb0aff02700 1 -- 172.21.15.129:0/2959907073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0a81081e0 con 0x7fb0a8101f20 2024-05-20T17:50:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.275+0000 7fb09affd700 1 -- 172.21.15.129:0/2959907073 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fb09c02a070 con 0x7fb0a8101f20 2024-05-20T17:50:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.275+0000 7fb09affd700 1 --2- 172.21.15.129:0/2959907073 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb0940390f0 0x7fb09403b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.275+0000 7fb09affd700 1 -- 172.21.15.129:0/2959907073 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb09c01e030 con 0x7fb0a8101f20 2024-05-20T17:50:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.276+0000 7fb0ad49d700 1 --2- 172.21.15.129:0/2959907073 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb0940390f0 0x7fb09403b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.276+0000 7fb0ad49d700 1 --2- 172.21.15.129:0/2959907073 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb0940390f0 0x7fb09403b600 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4006fd0 tx=0x7fb0a4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:01.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.281+0000 7fb09affd700 1 -- 172.21.15.129:0/2959907073 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb09c013070 con 0x7fb0a8101f20 2024-05-20T17:50:01.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.478+0000 7fb0aff02700 1 -- 172.21.15.129:0/2959907073 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fb0a802dd70 con 0x7fb0940390f0 2024-05-20T17:50:01.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.482+0000 7fb09affd700 1 -- 172.21.15.129:0/2959907073 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb0a802dd70 con 0x7fb0940390f0 2024-05-20T17:50:01.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.484+0000 7fb0aff02700 1 -- 172.21.15.129:0/2959907073 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb0940390f0 msgr2=0x7fb09403b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:01.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.484+0000 7fb0aff02700 1 --2- 172.21.15.129:0/2959907073 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb0940390f0 0x7fb09403b600 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4006fd0 tx=0x7fb0a4006e40 comp rx=0 tx=0).stop 2024-05-20T17:50:01.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.484+0000 7fb0aff02700 1 -- 172.21.15.129:0/2959907073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0a8101f20 msgr2=0x7fb0a81986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:01.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.484+0000 7fb0aff02700 1 --2- 172.21.15.129:0/2959907073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0a8101f20 0x7fb0a81986b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00b5c0 tx=0x7fb09c005050 comp rx=0 tx=0).stop 2024-05-20T17:50:01.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.484+0000 7fb0aff02700 1 -- 172.21.15.129:0/2959907073 shutdown_connections 2024-05-20T17:50:01.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.484+0000 7fb0aff02700 1 --2- 172.21.15.129:0/2959907073 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb0940390f0 0x7fb09403b600 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:01.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.484+0000 7fb0aff02700 1 --2- 172.21.15.129:0/2959907073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0a8101f20 0x7fb0a81986b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:01.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.484+0000 7fb0aff02700 1 -- 172.21.15.129:0/2959907073 >> 172.21.15.129:0/2959907073 conn(0x7fb0a80fc710 msgr2=0x7fb0a8101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:01.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.484+0000 7fb0aff02700 1 -- 172.21.15.129:0/2959907073 shutdown_connections 2024-05-20T17:50:01.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:01.484+0000 7fb0aff02700 1 -- 172.21.15.129:0/2959907073 wait complete. 2024-05-20T17:50:02.234 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi154 2024-05-20T17:50:02.235 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-05-20T17:50:02.235 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.conf 2024-05-20T17:50:02.254 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-05-20T17:50:02.255 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:02.312 INFO:tasks.cephadm:Adding host smithi154 to orchestrator... 2024-05-20T17:50:02.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch host add smithi154 2024-05-20T17:50:02.553 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:50:02.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:02 smithi129 ceph-mon[32029]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:02.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:02.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:02.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:02.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:02.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:02 smithi129 ceph-mon[32029]: Updating smithi129:/etc/ceph/ceph.conf 2024-05-20T17:50:02.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:02.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:02.771 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:03.595 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:03 smithi129 ceph-mon[32029]: Updating smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:03.595 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:03 smithi129 ceph-mon[32029]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:03.595 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:03 smithi129 ceph-mon[32029]: Updating smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:50:04.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.045+0000 7fa02deb4700 1 -- 172.21.15.129:0/3174979056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0281013d0 msgr2=0x7fa0281017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:04.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.045+0000 7fa02deb4700 1 --2- 172.21.15.129:0/3174979056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0281013d0 0x7fa0281017e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa01800aa20 tx=0x7fa01800ad30 comp rx=0 tx=0).stop 2024-05-20T17:50:04.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.045+0000 7fa02deb4700 1 -- 172.21.15.129:0/3174979056 shutdown_connections 2024-05-20T17:50:04.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.045+0000 7fa02deb4700 1 --2- 172.21.15.129:0/3174979056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0281013d0 0x7fa0281017e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:04.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.045+0000 7fa02deb4700 1 -- 172.21.15.129:0/3174979056 >> 172.21.15.129:0/3174979056 conn(0x7fa028077be0 msgr2=0x7fa028078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:04.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.045+0000 7fa02deb4700 1 -- 172.21.15.129:0/3174979056 shutdown_connections 2024-05-20T17:50:04.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.045+0000 7fa02deb4700 1 -- 172.21.15.129:0/3174979056 wait complete. 2024-05-20T17:50:04.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.045+0000 7fa02deb4700 1 Processor -- start 2024-05-20T17:50:04.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.046+0000 7fa02deb4700 1 -- start start 2024-05-20T17:50:04.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.046+0000 7fa02deb4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0281013d0 0x7fa028103e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:04.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.046+0000 7fa02deb4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa02810aff0 con 0x7fa0281013d0 2024-05-20T17:50:04.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.046+0000 7fa0277fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0281013d0 0x7fa028103e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:04.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.046+0000 7fa0277fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0281013d0 0x7fa028103e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34956/0 (socket says 172.21.15.129:34956) 2024-05-20T17:50:04.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.046+0000 7fa0277fe700 1 -- 172.21.15.129:0/3278176452 learned_addr learned my addr 172.21.15.129:0/3278176452 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:50:04.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.046+0000 7fa0277fe700 1 -- 172.21.15.129:0/3278176452 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa028105cc0 con 0x7fa0281013d0 2024-05-20T17:50:04.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.046+0000 7fa0277fe700 1 --2- 172.21.15.129:0/3278176452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0281013d0 0x7fa028103e70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa01800c6d0 tx=0x7fa018004d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:04.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.046+0000 7fa024ff9700 1 -- 172.21.15.129:0/3278176452 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0180053a0 con 0x7fa0281013d0 2024-05-20T17:50:04.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.047+0000 7fa02deb4700 1 -- 172.21.15.129:0/3278176452 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0281043b0 con 0x7fa0281013d0 2024-05-20T17:50:04.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.047+0000 7fa024ff9700 1 -- 172.21.15.129:0/3278176452 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa018005540 con 0x7fa0281013d0 2024-05-20T17:50:04.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.047+0000 7fa024ff9700 1 -- 172.21.15.129:0/3278176452 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa01800e6e0 con 0x7fa0281013d0 2024-05-20T17:50:04.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.047+0000 7fa02deb4700 1 -- 172.21.15.129:0/3278176452 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0281047b0 con 0x7fa0281013d0 2024-05-20T17:50:04.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.047+0000 7fa024ff9700 1 -- 172.21.15.129:0/3278176452 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fa01800e930 con 0x7fa0281013d0 2024-05-20T17:50:04.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.047+0000 7fa02deb4700 1 -- 172.21.15.129:0/3278176452 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0281027a0 con 0x7fa0281013d0 2024-05-20T17:50:04.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.048+0000 7fa024ff9700 1 --2- 172.21.15.129:0/3278176452 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa010039150 0x7fa01003b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:04.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.048+0000 7fa024ff9700 1 -- 172.21.15.129:0/3278176452 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa01801e070 con 0x7fa0281013d0 2024-05-20T17:50:04.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.048+0000 7fa026ffd700 1 --2- 172.21.15.129:0/3278176452 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa010039150 0x7fa01003b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:04.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.048+0000 7fa026ffd700 1 --2- 172.21.15.129:0/3278176452 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa010039150 0x7fa01003b660 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa01c006fd0 tx=0x7fa01c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.053+0000 7fa024ff9700 1 -- 172.21.15.129:0/3278176452 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa018015e80 con 0x7fa0281013d0 2024-05-20T17:50:04.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:04.252+0000 7fa02deb4700 1 -- 172.21.15.129:0/3278176452 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}) -- 0x7fa0281053a0 con 0x7fa010039150 2024-05-20T17:50:05.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:04 smithi129 ceph-mon[32029]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:06.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:05 smithi129 ceph-mon[32029]: Deploying cephadm binary to smithi154 2024-05-20T17:50:08.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:08.406+0000 7fa024ff9700 1 -- 172.21.15.129:0/3278176452 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fa0281053a0 con 0x7fa010039150 2024-05-20T17:50:08.408 INFO:teuthology.orchestra.run.smithi129.stdout:Added host 'smithi154' with addr '172.21.15.154' 2024-05-20T17:50:08.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:08.409+0000 7fa02deb4700 1 -- 172.21.15.129:0/3278176452 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa010039150 msgr2=0x7fa01003b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:08.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:08.409+0000 7fa02deb4700 1 --2- 172.21.15.129:0/3278176452 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa010039150 0x7fa01003b660 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa01c006fd0 tx=0x7fa01c006e40 comp rx=0 tx=0).stop 2024-05-20T17:50:08.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:08.409+0000 7fa02deb4700 1 -- 172.21.15.129:0/3278176452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0281013d0 msgr2=0x7fa028103e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:08.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:08.409+0000 7fa02deb4700 1 --2- 172.21.15.129:0/3278176452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0281013d0 0x7fa028103e70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa01800c6d0 tx=0x7fa018004d80 comp rx=0 tx=0).stop 2024-05-20T17:50:08.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:08.409+0000 7fa02deb4700 1 -- 172.21.15.129:0/3278176452 shutdown_connections 2024-05-20T17:50:08.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:08.409+0000 7fa02deb4700 1 --2- 172.21.15.129:0/3278176452 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa010039150 0x7fa01003b660 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:08.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:08.409+0000 7fa02deb4700 1 --2- 172.21.15.129:0/3278176452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0281013d0 0x7fa028103e70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:08.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:08.409+0000 7fa02deb4700 1 -- 172.21.15.129:0/3278176452 >> 172.21.15.129:0/3278176452 conn(0x7fa028077be0 msgr2=0x7fa028108e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:08.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:08.410+0000 7fa02deb4700 1 -- 172.21.15.129:0/3278176452 shutdown_connections 2024-05-20T17:50:08.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:08.410+0000 7fa02deb4700 1 -- 172.21.15.129:0/3278176452 wait complete. 2024-05-20T17:50:08.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:08 smithi129 ceph-mon[32029]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:08.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:08.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:08.998 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch host ls --format=json 2024-05-20T17:50:09.232 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:50:09.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:09 smithi129 ceph-mon[32029]: Added host smithi154 2024-05-20T17:50:09.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:09.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:10.702 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:10 smithi129 ceph-mon[32029]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:10.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.701+0000 7fa71698c700 1 -- 172.21.15.129:0/988000451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa710107900 msgr2=0x7fa710107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:10.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.701+0000 7fa71698c700 1 --2- 172.21.15.129:0/988000451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa710107900 0x7fa710107d10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa700009910 tx=0x7fa700009c20 comp rx=0 tx=0).stop 2024-05-20T17:50:10.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.702+0000 7fa71698c700 1 -- 172.21.15.129:0/988000451 shutdown_connections 2024-05-20T17:50:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.702+0000 7fa71698c700 1 --2- 172.21.15.129:0/988000451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa710107900 0x7fa710107d10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.702+0000 7fa71698c700 1 -- 172.21.15.129:0/988000451 >> 172.21.15.129:0/988000451 conn(0x7fa710077430 msgr2=0x7fa710077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.702+0000 7fa71698c700 1 -- 172.21.15.129:0/988000451 shutdown_connections 2024-05-20T17:50:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.702+0000 7fa71698c700 1 -- 172.21.15.129:0/988000451 wait complete. 2024-05-20T17:50:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.703+0000 7fa71698c700 1 Processor -- start 2024-05-20T17:50:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.703+0000 7fa71698c700 1 -- start start 2024-05-20T17:50:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.703+0000 7fa71698c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa710107900 0x7fa7101977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.703+0000 7fa71698c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa71010bfa0 con 0x7fa710107900 2024-05-20T17:50:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.703+0000 7fa70ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa710107900 0x7fa7101977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.703+0000 7fa70ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa710107900 0x7fa7101977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55778/0 (socket says 172.21.15.129:55778) 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.703+0000 7fa70ffff700 1 -- 172.21.15.129:0/1278788919 learned_addr learned my addr 172.21.15.129:0/1278788919 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.704+0000 7fa70ffff700 1 -- 172.21.15.129:0/1278788919 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa710197ce0 con 0x7fa710107900 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.704+0000 7fa70ffff700 1 --2- 172.21.15.129:0/1278788919 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa710107900 0x7fa7101977a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa700004f80 tx=0x7fa700005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.704+0000 7fa70d7fa700 1 -- 172.21.15.129:0/1278788919 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa700026070 con 0x7fa710107900 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.704+0000 7fa70d7fa700 1 -- 172.21.15.129:0/1278788919 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa70000b6f0 con 0x7fa710107900 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.704+0000 7fa71698c700 1 -- 172.21.15.129:0/1278788919 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa710197fd0 con 0x7fa710107900 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.704+0000 7fa70d7fa700 1 -- 172.21.15.129:0/1278788919 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa700003910 con 0x7fa710107900 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.704+0000 7fa71698c700 1 -- 172.21.15.129:0/1278788919 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa71019bfd0 con 0x7fa710107900 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.705+0000 7fa70d7fa700 1 -- 172.21.15.129:0/1278788919 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fa700029070 con 0x7fa710107900 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.705+0000 7fa71698c700 1 -- 172.21.15.129:0/1278788919 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa710103010 con 0x7fa710107900 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.705+0000 7fa70d7fa700 1 --2- 172.21.15.129:0/1278788919 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa6f8039150 0x7fa6f803b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:10.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.705+0000 7fa70d7fa700 1 -- 172.21.15.129:0/1278788919 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa70001e070 con 0x7fa710107900 2024-05-20T17:50:10.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.706+0000 7fa70f7fe700 1 --2- 172.21.15.129:0/1278788919 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa6f8039150 0x7fa6f803b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:10.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.706+0000 7fa70f7fe700 1 --2- 172.21.15.129:0/1278788919 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa6f8039150 0x7fa6f803b660 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa704006fd0 tx=0x7fa704006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:10.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.711+0000 7fa70d7fa700 1 -- 172.21.15.129:0/1278788919 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa700013070 con 0x7fa710107900 2024-05-20T17:50:10.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.920+0000 7fa71698c700 1 -- 172.21.15.129:0/1278788919 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa710198b00 con 0x7fa6f8039150 2024-05-20T17:50:10.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.921+0000 7fa70d7fa700 1 -- 172.21.15.129:0/1278788919 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7fa710198b00 con 0x7fa6f8039150 2024-05-20T17:50:10.922 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:50:10.922 INFO:teuthology.orchestra.run.smithi129.stdout:[{"addr": "172.21.15.129", "hostname": "smithi129", "labels": [], "status": ""}, {"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}] 2024-05-20T17:50:10.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.924+0000 7fa71698c700 1 -- 172.21.15.129:0/1278788919 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa6f8039150 msgr2=0x7fa6f803b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:10.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.924+0000 7fa71698c700 1 --2- 172.21.15.129:0/1278788919 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa6f8039150 0x7fa6f803b660 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa704006fd0 tx=0x7fa704006e40 comp rx=0 tx=0).stop 2024-05-20T17:50:10.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.924+0000 7fa71698c700 1 -- 172.21.15.129:0/1278788919 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa710107900 msgr2=0x7fa7101977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:10.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.924+0000 7fa71698c700 1 --2- 172.21.15.129:0/1278788919 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa710107900 0x7fa7101977a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa700004f80 tx=0x7fa700005060 comp rx=0 tx=0).stop 2024-05-20T17:50:10.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.924+0000 7fa71698c700 1 -- 172.21.15.129:0/1278788919 shutdown_connections 2024-05-20T17:50:10.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.924+0000 7fa71698c700 1 --2- 172.21.15.129:0/1278788919 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa6f8039150 0x7fa6f803b660 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:10.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.924+0000 7fa71698c700 1 --2- 172.21.15.129:0/1278788919 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa710107900 0x7fa7101977a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:10.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.924+0000 7fa71698c700 1 -- 172.21.15.129:0/1278788919 >> 172.21.15.129:0/1278788919 conn(0x7fa710077430 msgr2=0x7fa710100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:10.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.924+0000 7fa71698c700 1 -- 172.21.15.129:0/1278788919 shutdown_connections 2024-05-20T17:50:10.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:10.924+0000 7fa71698c700 1 -- 172.21.15.129:0/1278788919 wait complete. 2024-05-20T17:50:11.531 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi204 2024-05-20T17:50:11.531 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-20T17:50:11.531 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.conf 2024-05-20T17:50:11.551 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-20T17:50:11.551 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:11.608 INFO:tasks.cephadm:Adding host smithi204 to orchestrator... 2024-05-20T17:50:11.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch host add smithi204 2024-05-20T17:50:11.858 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:50:12.292 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:12 smithi129 ceph-mon[32029]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:50:12.292 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:12.292 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:12 smithi129 ceph-mon[32029]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.380+0000 7ff38aaac700 1 -- 172.21.15.129:0/4069547608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3841078d0 msgr2=0x7ff384107ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.380+0000 7ff38aaac700 1 --2- 172.21.15.129:0/4069547608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3841078d0 0x7ff384107ce0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff378009910 tx=0x7ff378009c20 comp rx=0 tx=0).stop 2024-05-20T17:50:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.380+0000 7ff38aaac700 1 -- 172.21.15.129:0/4069547608 shutdown_connections 2024-05-20T17:50:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.380+0000 7ff38aaac700 1 --2- 172.21.15.129:0/4069547608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3841078d0 0x7ff384107ce0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.380+0000 7ff38aaac700 1 -- 172.21.15.129:0/4069547608 >> 172.21.15.129:0/4069547608 conn(0x7ff3840fd3e0 msgr2=0x7ff3840ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.380+0000 7ff38aaac700 1 -- 172.21.15.129:0/4069547608 shutdown_connections 2024-05-20T17:50:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.380+0000 7ff38aaac700 1 -- 172.21.15.129:0/4069547608 wait complete. 2024-05-20T17:50:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.381+0000 7ff38aaac700 1 Processor -- start 2024-05-20T17:50:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.381+0000 7ff38aaac700 1 -- start start 2024-05-20T17:50:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.381+0000 7ff38aaac700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3841078d0 0x7ff38419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.381+0000 7ff38aaac700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff38410bf70 con 0x7ff3841078d0 2024-05-20T17:50:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.381+0000 7ff388848700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3841078d0 0x7ff38419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:13.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.381+0000 7ff388848700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3841078d0 0x7ff38419e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55792/0 (socket says 172.21.15.129:55792) 2024-05-20T17:50:13.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.381+0000 7ff388848700 1 -- 172.21.15.129:0/947430337 learned_addr learned my addr 172.21.15.129:0/947430337 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:50:13.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.381+0000 7ff388848700 1 -- 172.21.15.129:0/947430337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38419eaf0 con 0x7ff3841078d0 2024-05-20T17:50:13.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.381+0000 7ff388848700 1 --2- 172.21.15.129:0/947430337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3841078d0 0x7ff38419e5b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff378004f80 tx=0x7ff378005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:13.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.382+0000 7ff381ffb700 1 -- 172.21.15.129:0/947430337 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff378026070 con 0x7ff3841078d0 2024-05-20T17:50:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.382+0000 7ff381ffb700 1 -- 172.21.15.129:0/947430337 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff37800b6f0 con 0x7ff3841078d0 2024-05-20T17:50:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.382+0000 7ff38aaac700 1 -- 172.21.15.129:0/947430337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff38419ede0 con 0x7ff3841078d0 2024-05-20T17:50:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.382+0000 7ff381ffb700 1 -- 172.21.15.129:0/947430337 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff378003910 con 0x7ff3841078d0 2024-05-20T17:50:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.382+0000 7ff38aaac700 1 -- 172.21.15.129:0/947430337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff384198910 con 0x7ff3841078d0 2024-05-20T17:50:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.383+0000 7ff381ffb700 1 -- 172.21.15.129:0/947430337 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7ff378029070 con 0x7ff3841078d0 2024-05-20T17:50:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.383+0000 7ff381ffb700 1 --2- 172.21.15.129:0/947430337 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3740390f0 0x7ff37403b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.383+0000 7ff381ffb700 1 -- 172.21.15.129:0/947430337 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff37801e070 con 0x7ff3841078d0 2024-05-20T17:50:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.383+0000 7ff383fff700 1 --2- 172.21.15.129:0/947430337 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3740390f0 0x7ff37403b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.383+0000 7ff38aaac700 1 -- 172.21.15.129:0/947430337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff384102fe0 con 0x7ff3841078d0 2024-05-20T17:50:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.383+0000 7ff383fff700 1 --2- 172.21.15.129:0/947430337 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3740390f0 0x7ff37403b600 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff370006fd0 tx=0x7ff370006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:13.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.388+0000 7ff381ffb700 1 -- 172.21.15.129:0/947430337 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff378030020 con 0x7ff3841078d0 2024-05-20T17:50:13.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:13.587+0000 7ff38aaac700 1 -- 172.21.15.129:0/947430337 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}) -- 0x7ff38419f010 con 0x7ff3740390f0 2024-05-20T17:50:13.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:13 smithi129 ceph-mon[32029]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:13.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:13 smithi129 ceph-mon[32029]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:15 smithi129 ceph-mon[32029]: Deploying cephadm binary to smithi204 2024-05-20T17:50:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:15.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:15.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:15.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:50:15.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:15.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:15.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:15 smithi129 ceph-mon[32029]: Updating smithi154:/etc/ceph/ceph.conf 2024-05-20T17:50:15.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:15 smithi129 ceph-mon[32029]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:16.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:16 smithi129 ceph-mon[32029]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:17.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:17.709+0000 7ff381ffb700 1 -- 172.21.15.129:0/947430337 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7ff38419f010 con 0x7ff3740390f0 2024-05-20T17:50:17.710 INFO:teuthology.orchestra.run.smithi129.stdout:Added host 'smithi204' with addr '172.21.15.204' 2024-05-20T17:50:17.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:17.712+0000 7ff38aaac700 1 -- 172.21.15.129:0/947430337 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3740390f0 msgr2=0x7ff37403b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:17.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:17.712+0000 7ff38aaac700 1 --2- 172.21.15.129:0/947430337 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3740390f0 0x7ff37403b600 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff370006fd0 tx=0x7ff370006e40 comp rx=0 tx=0).stop 2024-05-20T17:50:17.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:17.712+0000 7ff38aaac700 1 -- 172.21.15.129:0/947430337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3841078d0 msgr2=0x7ff38419e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:17.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:17.712+0000 7ff38aaac700 1 --2- 172.21.15.129:0/947430337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3841078d0 0x7ff38419e5b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff378004f80 tx=0x7ff378005060 comp rx=0 tx=0).stop 2024-05-20T17:50:17.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:17.712+0000 7ff38aaac700 1 -- 172.21.15.129:0/947430337 shutdown_connections 2024-05-20T17:50:17.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:17.712+0000 7ff38aaac700 1 --2- 172.21.15.129:0/947430337 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3740390f0 0x7ff37403b600 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:17.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:17.712+0000 7ff38aaac700 1 --2- 172.21.15.129:0/947430337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3841078d0 0x7ff38419e5b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:17.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:17.712+0000 7ff38aaac700 1 -- 172.21.15.129:0/947430337 >> 172.21.15.129:0/947430337 conn(0x7ff3840fd3e0 msgr2=0x7ff3841053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:17.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:17.712+0000 7ff38aaac700 1 -- 172.21.15.129:0/947430337 shutdown_connections 2024-05-20T17:50:17.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:17.712+0000 7ff38aaac700 1 -- 172.21.15.129:0/947430337 wait complete. 2024-05-20T17:50:18.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch host ls --format=json 2024-05-20T17:50:18.550 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:50:18.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:18 smithi129 ceph-mon[32029]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:18.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:18 smithi129 ceph-mon[32029]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:50:18.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:18.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:18.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:18.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:18 smithi129 ceph-mon[32029]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:18.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:18.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:18.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:19.508 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:19 smithi129 ceph-mon[32029]: Added host smithi204 2024-05-20T17:50:19.508 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:19 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:20.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:19.999+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3044283864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38108a10 msgr2=0x7f0a38108e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:20.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:19.999+0000 7f0a3cb37700 1 --2- 172.21.15.129:0/3044283864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38108a10 0x7f0a38108e20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009910 tx=0x7f0a28009c20 comp rx=0 tx=0).stop 2024-05-20T17:50:20.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:19.999+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3044283864 shutdown_connections 2024-05-20T17:50:20.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:19.999+0000 7f0a3cb37700 1 --2- 172.21.15.129:0/3044283864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38108a10 0x7f0a38108e20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:20.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:19.999+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3044283864 >> 172.21.15.129:0/3044283864 conn(0x7f0a380fe520 msgr2=0x7f0a38100970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:20.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:19.999+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3044283864 shutdown_connections 2024-05-20T17:50:20.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:19.999+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3044283864 wait complete. 2024-05-20T17:50:20.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.000+0000 7f0a3cb37700 1 Processor -- start 2024-05-20T17:50:20.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.000+0000 7f0a3cb37700 1 -- start start 2024-05-20T17:50:20.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.000+0000 7f0a3cb37700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38108a10 0x7f0a3819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:20.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.000+0000 7f0a3cb37700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3810d0b0 con 0x7f0a38108a10 2024-05-20T17:50:20.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.000+0000 7f0a3659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38108a10 0x7f0a3819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:20.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.000+0000 7f0a3659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38108a10 0x7f0a3819e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59018/0 (socket says 172.21.15.129:59018) 2024-05-20T17:50:20.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.000+0000 7f0a3659c700 1 -- 172.21.15.129:0/3386111165 learned_addr learned my addr 172.21.15.129:0/3386111165 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:50:20.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.000+0000 7f0a3659c700 1 -- 172.21.15.129:0/3386111165 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3819eab0 con 0x7f0a38108a10 2024-05-20T17:50:20.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.001+0000 7f0a3659c700 1 --2- 172.21.15.129:0/3386111165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38108a10 0x7f0a3819e570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a28004f80 tx=0x7f0a28005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:20.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.001+0000 7f0a277fe700 1 -- 172.21.15.129:0/3386111165 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a28026070 con 0x7f0a38108a10 2024-05-20T17:50:20.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.001+0000 7f0a277fe700 1 -- 172.21.15.129:0/3386111165 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a2800b6f0 con 0x7f0a38108a10 2024-05-20T17:50:20.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.001+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3386111165 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3819eda0 con 0x7f0a38108a10 2024-05-20T17:50:20.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.001+0000 7f0a277fe700 1 -- 172.21.15.129:0/3386111165 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a28003910 con 0x7f0a38108a10 2024-05-20T17:50:20.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.001+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3386111165 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a381988d0 con 0x7f0a38108a10 2024-05-20T17:50:20.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.002+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3386111165 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a38104120 con 0x7f0a38108a10 2024-05-20T17:50:20.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.002+0000 7f0a277fe700 1 -- 172.21.15.129:0/3386111165 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f0a28029070 con 0x7f0a38108a10 2024-05-20T17:50:20.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.002+0000 7f0a277fe700 1 --2- 172.21.15.129:0/3386111165 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f0a200390f0 0x7f0a2003b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:20.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.002+0000 7f0a277fe700 1 -- 172.21.15.129:0/3386111165 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0a2801e070 con 0x7f0a38108a10 2024-05-20T17:50:20.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.002+0000 7f0a35d9b700 1 --2- 172.21.15.129:0/3386111165 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f0a200390f0 0x7f0a2003b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:20.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.003+0000 7f0a35d9b700 1 --2- 172.21.15.129:0/3386111165 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f0a200390f0 0x7f0a2003b600 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c006fd0 tx=0x7f0a2c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:20.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.007+0000 7f0a277fe700 1 -- 172.21.15.129:0/3386111165 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0a28013070 con 0x7f0a38108a10 2024-05-20T17:50:20.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.207+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3386111165 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0a38199c10 con 0x7f0a200390f0 2024-05-20T17:50:20.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.208+0000 7f0a277fe700 1 -- 172.21.15.129:0/3386111165 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f0a38199c10 con 0x7f0a200390f0 2024-05-20T17:50:20.208 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:50:20.209 INFO:teuthology.orchestra.run.smithi129.stdout:[{"addr": "172.21.15.129", "hostname": "smithi129", "labels": [], "status": ""}, {"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}, {"addr": "172.21.15.204", "hostname": "smithi204", "labels": [], "status": ""}] 2024-05-20T17:50:20.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.210+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3386111165 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f0a200390f0 msgr2=0x7f0a2003b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:20.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.210+0000 7f0a3cb37700 1 --2- 172.21.15.129:0/3386111165 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f0a200390f0 0x7f0a2003b600 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c006fd0 tx=0x7f0a2c006e40 comp rx=0 tx=0).stop 2024-05-20T17:50:20.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.210+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3386111165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38108a10 msgr2=0x7f0a3819e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:20.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.210+0000 7f0a3cb37700 1 --2- 172.21.15.129:0/3386111165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38108a10 0x7f0a3819e570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a28004f80 tx=0x7f0a28005060 comp rx=0 tx=0).stop 2024-05-20T17:50:20.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.210+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3386111165 shutdown_connections 2024-05-20T17:50:20.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.210+0000 7f0a3cb37700 1 --2- 172.21.15.129:0/3386111165 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f0a200390f0 0x7f0a2003b600 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:20.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.210+0000 7f0a3cb37700 1 --2- 172.21.15.129:0/3386111165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a38108a10 0x7f0a3819e570 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:20.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.210+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3386111165 >> 172.21.15.129:0/3386111165 conn(0x7f0a380fe520 msgr2=0x7f0a380ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:20.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.210+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3386111165 shutdown_connections 2024-05-20T17:50:20.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:20.210+0000 7f0a3cb37700 1 -- 172.21.15.129:0/3386111165 wait complete. 2024-05-20T17:50:20.505 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:20 smithi129 ceph-mon[32029]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:20.505 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:20 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:20.820 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-20T17:50:20.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd crush tunables default 2024-05-20T17:50:21.054 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:50:21.775 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:21 smithi129 ceph-mon[32029]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:50:22.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.506+0000 7f376418f700 1 -- 172.21.15.129:0/1003925020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375c107900 msgr2=0x7f375c107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:22.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.506+0000 7f376418f700 1 --2- 172.21.15.129:0/1003925020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375c107900 0x7f375c107d10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f375800b0a0 tx=0x7f375800b3b0 comp rx=0 tx=0).stop 2024-05-20T17:50:22.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.506+0000 7f376418f700 1 -- 172.21.15.129:0/1003925020 shutdown_connections 2024-05-20T17:50:22.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.506+0000 7f376418f700 1 --2- 172.21.15.129:0/1003925020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375c107900 0x7f375c107d10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:22.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.506+0000 7f376418f700 1 -- 172.21.15.129:0/1003925020 >> 172.21.15.129:0/1003925020 conn(0x7f375c077430 msgr2=0x7f375c077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:22.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.506+0000 7f376418f700 1 -- 172.21.15.129:0/1003925020 shutdown_connections 2024-05-20T17:50:22.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.506+0000 7f376418f700 1 -- 172.21.15.129:0/1003925020 wait complete. 2024-05-20T17:50:22.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.507+0000 7f376418f700 1 Processor -- start 2024-05-20T17:50:22.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.507+0000 7f376418f700 1 -- start start 2024-05-20T17:50:22.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.507+0000 7f376418f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375c107900 0x7f375c1976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:22.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.507+0000 7f376418f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f375c10bfa0 con 0x7f375c107900 2024-05-20T17:50:22.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.507+0000 7f3761f2b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375c107900 0x7f375c1976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:22.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.507+0000 7f3761f2b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375c107900 0x7f375c1976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59038/0 (socket says 172.21.15.129:59038) 2024-05-20T17:50:22.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.507+0000 7f3761f2b700 1 -- 172.21.15.129:0/3094845761 learned_addr learned my addr 172.21.15.129:0/3094845761 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:50:22.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.508+0000 7f3761f2b700 1 -- 172.21.15.129:0/3094845761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375c197c30 con 0x7f375c107900 2024-05-20T17:50:22.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.508+0000 7f3761f2b700 1 --2- 172.21.15.129:0/3094845761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375c107900 0x7f375c1976f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f37580062a0 tx=0x7f3758003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:22.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.508+0000 7f3752ffd700 1 -- 172.21.15.129:0/3094845761 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3758017040 con 0x7f375c107900 2024-05-20T17:50:22.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.508+0000 7f3752ffd700 1 -- 172.21.15.129:0/3094845761 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3758022070 con 0x7f375c107900 2024-05-20T17:50:22.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.508+0000 7f3752ffd700 1 -- 172.21.15.129:0/3094845761 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f375801d440 con 0x7f375c107900 2024-05-20T17:50:22.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.508+0000 7f376418f700 1 -- 172.21.15.129:0/3094845761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f375c197ec0 con 0x7f375c107900 2024-05-20T17:50:22.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.508+0000 7f376418f700 1 -- 172.21.15.129:0/3094845761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f375c19bf20 con 0x7f375c107900 2024-05-20T17:50:22.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.509+0000 7f3752ffd700 1 -- 172.21.15.129:0/3094845761 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f375801d6f0 con 0x7f375c107900 2024-05-20T17:50:22.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.509+0000 7f3752ffd700 1 --2- 172.21.15.129:0/3094845761 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3748039150 0x7f374803b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:22.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.509+0000 7f3752ffd700 1 -- 172.21.15.129:0/3094845761 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f375800f050 con 0x7f375c107900 2024-05-20T17:50:22.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.509+0000 7f376172a700 1 --2- 172.21.15.129:0/3094845761 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3748039150 0x7f374803b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:22.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.509+0000 7f376418f700 1 -- 172.21.15.129:0/3094845761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f375c103010 con 0x7f375c107900 2024-05-20T17:50:22.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.510+0000 7f376172a700 1 --2- 172.21.15.129:0/3094845761 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3748039150 0x7f374803b660 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f374c006fd0 tx=0x7f374c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:22.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.515+0000 7f3752ffd700 1 -- 172.21.15.129:0/3094845761 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3758007e20 con 0x7f375c107900 2024-05-20T17:50:22.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:22 smithi129 ceph-mon[32029]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:22.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:22.715+0000 7f376418f700 1 -- 172.21.15.129:0/3094845761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f375c063ca0 con 0x7f375c107900 2024-05-20T17:50:23.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:23.429+0000 7f3752ffd700 1 -- 172.21.15.129:0/3094845761 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f3758025e40 con 0x7f375c107900 2024-05-20T17:50:23.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:23.432+0000 7f376418f700 1 -- 172.21.15.129:0/3094845761 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3748039150 msgr2=0x7f374803b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:23.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:23.432+0000 7f376418f700 1 --2- 172.21.15.129:0/3094845761 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3748039150 0x7f374803b660 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f374c006fd0 tx=0x7f374c006e40 comp rx=0 tx=0).stop 2024-05-20T17:50:23.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:23.432+0000 7f376418f700 1 -- 172.21.15.129:0/3094845761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375c107900 msgr2=0x7f375c1976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:23.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:23.432+0000 7f376418f700 1 --2- 172.21.15.129:0/3094845761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375c107900 0x7f375c1976f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f37580062a0 tx=0x7f3758003ce0 comp rx=0 tx=0).stop 2024-05-20T17:50:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:23.432+0000 7f376418f700 1 -- 172.21.15.129:0/3094845761 shutdown_connections 2024-05-20T17:50:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:23.432+0000 7f376418f700 1 --2- 172.21.15.129:0/3094845761 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3748039150 0x7f374803b660 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:23.432+0000 7f376418f700 1 --2- 172.21.15.129:0/3094845761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f375c107900 0x7f375c1976f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:23.432+0000 7f376418f700 1 -- 172.21.15.129:0/3094845761 >> 172.21.15.129:0/3094845761 conn(0x7f375c077430 msgr2=0x7f375c100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:23.432+0000 7f376418f700 1 -- 172.21.15.129:0/3094845761 shutdown_connections 2024-05-20T17:50:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:23.432+0000 7f376418f700 1 -- 172.21.15.129:0/3094845761 wait complete. 2024-05-20T17:50:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:adjusted tunables profile to default 2024-05-20T17:50:23.795 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:23 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3094845761' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-20T17:50:24.082 INFO:tasks.cephadm:Adding mon.a on smithi129 2024-05-20T17:50:24.082 INFO:tasks.cephadm:Adding mon.b on smithi154 2024-05-20T17:50:24.082 INFO:tasks.cephadm:Adding mon.c on smithi204 2024-05-20T17:50:24.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch apply mon '3;smithi129:172.21.15.129=a;smithi154:172.21.15.154=b;smithi204:172.21.15.204=c' 2024-05-20T17:50:24.319 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:24.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3094845761' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-20T17:50:24.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: Updating smithi204:/etc/ceph/ceph.conf 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:24.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:25.747 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.747+0000 7f1bdf544700 1 -- 172.21.15.204:0/738050839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bd81022c0 msgr2=0x7f1bd81026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:25.747 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.747+0000 7f1bdf544700 1 --2- 172.21.15.204:0/738050839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bd81022c0 0x7f1bd81026d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1bd400b0a0 tx=0x7f1bd400b3b0 comp rx=0 tx=0).stop 2024-05-20T17:50:25.747 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.747+0000 7f1bdf544700 1 -- 172.21.15.204:0/738050839 shutdown_connections 2024-05-20T17:50:25.748 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.747+0000 7f1bdf544700 1 --2- 172.21.15.204:0/738050839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bd81022c0 0x7f1bd81026d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:25.748 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.747+0000 7f1bdf544700 1 -- 172.21.15.204:0/738050839 >> 172.21.15.204:0/738050839 conn(0x7f1bd8076bd0 msgr2=0x7f1bd8077030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:25.748 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.747+0000 7f1bdf544700 1 -- 172.21.15.204:0/738050839 shutdown_connections 2024-05-20T17:50:25.748 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.747+0000 7f1bdf544700 1 -- 172.21.15.204:0/738050839 wait complete. 2024-05-20T17:50:25.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.748+0000 7f1bdf544700 1 Processor -- start 2024-05-20T17:50:25.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.748+0000 7f1bdf544700 1 -- start start 2024-05-20T17:50:25.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.748+0000 7f1bdf544700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bd81022c0 0x7f1bd819d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:25.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.748+0000 7f1bdf544700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd810c1e0 con 0x7f1bd81022c0 2024-05-20T17:50:25.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.748+0000 7f1bdd2e0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bd81022c0 0x7f1bd819d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:25.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.748+0000 7f1bdd2e0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bd81022c0 0x7f1bd819d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.204:37050/0 (socket says 172.21.15.204:37050) 2024-05-20T17:50:25.749 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.748+0000 7f1bdd2e0700 1 -- 172.21.15.204:0/999176177 learned_addr learned my addr 172.21.15.204:0/999176177 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-20T17:50:25.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.749+0000 7f1bdd2e0700 1 -- 172.21.15.204:0/999176177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd819db10 con 0x7f1bd81022c0 2024-05-20T17:50:25.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.749+0000 7f1bdd2e0700 1 --2- 172.21.15.204:0/999176177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bd81022c0 0x7f1bd819d5d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4015040 tx=0x7f1bd4003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:25.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.749+0000 7f1bce7fc700 1 -- 172.21.15.204:0/999176177 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1bd4017040 con 0x7f1bd81022c0 2024-05-20T17:50:25.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.749+0000 7f1bce7fc700 1 -- 172.21.15.204:0/999176177 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bd4022070 con 0x7f1bd81022c0 2024-05-20T17:50:25.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.749+0000 7f1bdf544700 1 -- 172.21.15.204:0/999176177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd819dda0 con 0x7f1bd81022c0 2024-05-20T17:50:25.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.749+0000 7f1bce7fc700 1 -- 172.21.15.204:0/999176177 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1bd401d5d0 con 0x7f1bd81022c0 2024-05-20T17:50:25.750 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.749+0000 7f1bdf544700 1 -- 172.21.15.204:0/999176177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd8197760 con 0x7f1bd81022c0 2024-05-20T17:50:25.751 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.750+0000 7f1bce7fc700 1 -- 172.21.15.204:0/999176177 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f1bd401d8d0 con 0x7f1bd81022c0 2024-05-20T17:50:25.751 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.750+0000 7f1bce7fc700 1 --2- 172.21.15.204:0/999176177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1bc40390f0 0x7f1bc403b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:25.751 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.750+0000 7f1bdf544700 1 -- 172.21.15.204:0/999176177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd81032e0 con 0x7f1bd81022c0 2024-05-20T17:50:25.751 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.750+0000 7f1bce7fc700 1 -- 172.21.15.204:0/999176177 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1bd400f050 con 0x7f1bd81022c0 2024-05-20T17:50:25.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.751+0000 7f1bdcadf700 1 --2- 172.21.15.204:0/999176177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1bc40390f0 0x7f1bc403b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:25.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.751+0000 7f1bdcadf700 1 --2- 172.21.15.204:0/999176177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1bc40390f0 0x7f1bc403b600 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8006fd0 tx=0x7f1bc8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:25.756 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.756+0000 7f1bce7fc700 1 -- 172.21.15.204:0/999176177 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1bd4007e20 con 0x7f1bd81022c0 2024-05-20T17:50:25.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:25 smithi129 ceph-mon[32029]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:25.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:25 smithi129 ceph-mon[32029]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:25.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:25 smithi129 ceph-mon[32029]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:50:25.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.963+0000 7f1bdf544700 1 -- 172.21.15.204:0/999176177 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi129:172.21.15.129=a;smithi154:172.21.15.154=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}) -- 0x7f1bd806a6f0 con 0x7f1bc40390f0 2024-05-20T17:50:25.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.971+0000 7f1bce7fc700 1 -- 172.21.15.204:0/999176177 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1bd806a6f0 con 0x7f1bc40390f0 2024-05-20T17:50:25.971 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mon update... 2024-05-20T17:50:25.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.973+0000 7f1bdf544700 1 -- 172.21.15.204:0/999176177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1bc40390f0 msgr2=0x7f1bc403b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:25.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.973+0000 7f1bdf544700 1 --2- 172.21.15.204:0/999176177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1bc40390f0 0x7f1bc403b600 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8006fd0 tx=0x7f1bc8006e40 comp rx=0 tx=0).stop 2024-05-20T17:50:25.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.973+0000 7f1bdf544700 1 -- 172.21.15.204:0/999176177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bd81022c0 msgr2=0x7f1bd819d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:25.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.973+0000 7f1bdf544700 1 --2- 172.21.15.204:0/999176177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bd81022c0 0x7f1bd819d5d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4015040 tx=0x7f1bd4003d50 comp rx=0 tx=0).stop 2024-05-20T17:50:25.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.973+0000 7f1bdf544700 1 -- 172.21.15.204:0/999176177 shutdown_connections 2024-05-20T17:50:25.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.973+0000 7f1bdf544700 1 --2- 172.21.15.204:0/999176177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1bc40390f0 0x7f1bc403b600 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:25.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.973+0000 7f1bdf544700 1 --2- 172.21.15.204:0/999176177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bd81022c0 0x7f1bd819d5d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:25.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.973+0000 7f1bdf544700 1 -- 172.21.15.204:0/999176177 >> 172.21.15.204:0/999176177 conn(0x7f1bd8076bd0 msgr2=0x7f1bd81098e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:25.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.973+0000 7f1bdf544700 1 -- 172.21.15.204:0/999176177 shutdown_connections 2024-05-20T17:50:25.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:25.973+0000 7f1bdf544700 1 -- 172.21.15.204:0/999176177 wait complete. 2024-05-20T17:50:26.574 DEBUG:teuthology.orchestra.run.smithi154:mon.b> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.b.service 2024-05-20T17:50:26.577 DEBUG:teuthology.orchestra.run.smithi204:mon.c> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.c.service 2024-05-20T17:50:26.580 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-20T17:50:26.580 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph mon dump -f json 2024-05-20T17:50:26.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:26 smithi129 ceph-mon[32029]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:26.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:26.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:26.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:26.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:26.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:26.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:26.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:27.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:27 smithi129 ceph-mon[32029]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi129:172.21.15.129=a;smithi154:172.21.15.154=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:27.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:27 smithi129 ceph-mon[32029]: Saving service mon spec with placement smithi129:172.21.15.129=a;smithi154:172.21.15.154=b;smithi204:172.21.15.204=c;count:3 2024-05-20T17:50:27.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:27 smithi129 ceph-mon[32029]: Deploying daemon mon.c on smithi204 2024-05-20T17:50:28.687 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:28.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:28 smithi129 ceph-mon[32029]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:29.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:29 smithi204 systemd[1]: Starting Ceph mon.c for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:50:30.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:29 smithi129 ceph-mon[32029]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:30.522 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.500+0000 7f03b13fe700 1 -- 172.21.15.204:0/3617288827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03ac103bb0 msgr2=0x7f03ac103fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:30.522 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.500+0000 7f03b13fe700 1 --2- 172.21.15.204:0/3617288827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03ac103bb0 0x7f03ac103fc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f039c009910 tx=0x7f039c009c20 comp rx=0 tx=0).stop 2024-05-20T17:50:30.522 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.500+0000 7f03b13fe700 1 -- 172.21.15.204:0/3617288827 shutdown_connections 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.500+0000 7f03b13fe700 1 --2- 172.21.15.204:0/3617288827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03ac103bb0 0x7f03ac103fc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.500+0000 7f03b13fe700 1 -- 172.21.15.204:0/3617288827 >> 172.21.15.204:0/3617288827 conn(0x7f03ac0fd3e0 msgr2=0x7f03ac0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.500+0000 7f03b13fe700 1 -- 172.21.15.204:0/3617288827 shutdown_connections 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.500+0000 7f03b13fe700 1 -- 172.21.15.204:0/3617288827 wait complete. 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.501+0000 7f03b13fe700 1 Processor -- start 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.501+0000 7f03b13fe700 1 -- start start 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.501+0000 7f03b13fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03ac103bb0 0x7f03ac19e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.501+0000 7f03b13fe700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac10d0f0 con 0x7f03ac103bb0 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.502+0000 7f03aaffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03ac103bb0 0x7f03ac19e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.502+0000 7f03aaffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03ac103bb0 0x7f03ac19e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.204:37068/0 (socket says 172.21.15.204:37068) 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.502+0000 7f03aaffd700 1 -- 172.21.15.204:0/1188257378 learned_addr learned my addr 172.21.15.204:0/1188257378 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.502+0000 7f03aaffd700 1 -- 172.21.15.204:0/1188257378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03ac19ead0 con 0x7f03ac103bb0 2024-05-20T17:50:30.523 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.502+0000 7f03aaffd700 1 --2- 172.21.15.204:0/1188257378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03ac103bb0 0x7f03ac19e590 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f039c006010 tx=0x7f039c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.502+0000 7f0393fff700 1 -- 172.21.15.204:0/1188257378 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f039c026070 con 0x7f03ac103bb0 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.502+0000 7f03b13fe700 1 -- 172.21.15.204:0/1188257378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03ac19ed60 con 0x7f03ac103bb0 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.502+0000 7f0393fff700 1 -- 172.21.15.204:0/1188257378 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f039c00bc30 con 0x7f03ac103bb0 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.502+0000 7f0393fff700 1 -- 172.21.15.204:0/1188257378 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f039c003ab0 con 0x7f03ac103bb0 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.502+0000 7f03b13fe700 1 -- 172.21.15.204:0/1188257378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03ac198720 con 0x7f03ac103bb0 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.503+0000 7f03b13fe700 1 -- 172.21.15.204:0/1188257378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03ac06b1f0 con 0x7f03ac103bb0 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.503+0000 7f0393fff700 1 -- 172.21.15.204:0/1188257378 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f039c029070 con 0x7f03ac103bb0 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.504+0000 7f0393fff700 1 --2- 172.21.15.204:0/1188257378 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f03940390f0 0x7f039403b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.504+0000 7f0393fff700 1 -- 172.21.15.204:0/1188257378 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f039c01e070 con 0x7f03ac103bb0 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.504+0000 7f03aa7fc700 1 --2- 172.21.15.204:0/1188257378 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f03940390f0 0x7f039403b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.504+0000 7f03aa7fc700 1 --2- 172.21.15.204:0/1188257378 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f03940390f0 0x7f039403b600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f03a0006fd0 tx=0x7f03a0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:30.524 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.509+0000 7f0393fff700 1 -- 172.21.15.204:0/1188257378 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f039c013070 con 0x7f03ac103bb0 2024-05-20T17:50:30.802 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.800+0000 7f03b13fe700 1 -- 172.21.15.204:0/1188257378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f03ac199c30 con 0x7f03ac103bb0 2024-05-20T17:50:30.802 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.801+0000 7f0393fff700 1 -- 172.21.15.204:0/1188257378 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f039c030090 con 0x7f03ac103bb0 2024-05-20T17:50:30.802 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:50:30.802 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":1,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","modified":"2024-05-20T17:48:46.628733Z","created":"2024-05-20T17:48:46.628733Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-20T17:50:30.804 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.803+0000 7f03b13fe700 1 -- 172.21.15.204:0/1188257378 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f03940390f0 msgr2=0x7f039403b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:30.804 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.803+0000 7f03b13fe700 1 --2- 172.21.15.204:0/1188257378 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f03940390f0 0x7f039403b600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f03a0006fd0 tx=0x7f03a0006e40 comp rx=0 tx=0).stop 2024-05-20T17:50:30.804 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.803+0000 7f03b13fe700 1 -- 172.21.15.204:0/1188257378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03ac103bb0 msgr2=0x7f03ac19e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:30.804 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.803+0000 7f03b13fe700 1 --2- 172.21.15.204:0/1188257378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03ac103bb0 0x7f03ac19e590 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f039c006010 tx=0x7f039c005050 comp rx=0 tx=0).stop 2024-05-20T17:50:30.804 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.804+0000 7f03b13fe700 1 -- 172.21.15.204:0/1188257378 shutdown_connections 2024-05-20T17:50:30.804 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.804+0000 7f03b13fe700 1 --2- 172.21.15.204:0/1188257378 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f03940390f0 0x7f039403b600 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:30.805 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.804+0000 7f03b13fe700 1 --2- 172.21.15.204:0/1188257378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03ac103bb0 0x7f03ac19e590 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:30.805 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.804+0000 7f03b13fe700 1 -- 172.21.15.204:0/1188257378 >> 172.21.15.204:0/1188257378 conn(0x7f03ac0fd3e0 msgr2=0x7f03ac104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:30.805 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.804+0000 7f03b13fe700 1 -- 172.21.15.204:0/1188257378 shutdown_connections 2024-05-20T17:50:30.805 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:30.804+0000 7f03b13fe700 1 -- 172.21.15.204:0/1188257378 wait complete. 2024-05-20T17:50:30.805 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 1 2024-05-20T17:50:30.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 podman[37045]: 2024-05-20 17:50:30.632050485 +0000 UTC m=+0.093181076 container create 9a5d5ce86e415ed75bde17d5e1a4226712af369d99856cd846d0e19805eccaa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS) 2024-05-20T17:50:30.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 podman[37045]: 2024-05-20 17:50:30.559372293 +0000 UTC m=+0.020502910 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:50:30.882 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 podman[37045]: 2024-05-20 17:50:30.835014238 +0000 UTC m=+0.296144837 container init 9a5d5ce86e415ed75bde17d5e1a4226712af369d99856cd846d0e19805eccaa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 podman[37045]: 2024-05-20 17:50:30.838805444 +0000 UTC m=+0.299936036 container start 9a5d5ce86e415ed75bde17d5e1a4226712af369d99856cd846d0e19805eccaa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: set uid:gid to 167:167 (ceph:ceph) 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev), process ceph-mon, pid 2 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: pidfile_write: ignore empty --pid-file 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: load: jerasure load: lrc 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: RocksDB version: 7.9.2 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Git sha 0 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Compile date 2024-05-17 21:21:31 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: DB SUMMARY 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: DB Session ID: 9Y8XUM3Q36AKY8S1T58D 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: CURRENT file: CURRENT 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: IDENTITY file: IDENTITY 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-20T17:50:30.883 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-20T17:50:30.884 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-20T17:50:30.884 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.error_if_exists: 0 2024-05-20T17:50:30.884 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.create_if_missing: 0 2024-05-20T17:50:30.884 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.paranoid_checks: 1 2024-05-20T17:50:30.884 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.env: 0x55df63b554a0 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.fs: PosixFileSystem 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.info_log: 0x55df64d57d40 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_file_opening_threads: 16 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.statistics: (nil) 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.use_fsync: 0 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_log_file_size: 0 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.keep_log_file_num: 1000 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.recycle_log_file_num: 0 2024-05-20T17:50:30.885 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.allow_fallocate: 1 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.allow_mmap_reads: 0 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.allow_mmap_writes: 0 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.use_direct_reads: 0 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.create_missing_column_families: 0 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.db_log_dir: 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.wal_dir: 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-20T17:50:30.886 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.advise_random_on_open: 1 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.db_write_buffer_size: 0 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.write_buffer_manager: 0x55df64d2b2c0 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.rate_limiter: (nil) 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.wal_recovery_mode: 2 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.enable_thread_tracking: 0 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.enable_pipelined_write: 0 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.unordered_write: 0 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-20T17:50:30.887 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.row_cache: None 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.wal_filter: None 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.allow_ingest_behind: 0 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.two_write_queues: 0 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.manual_wal_flush: 0 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.wal_compression: 0 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.atomic_flush: 0 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.log_readahead_size: 0 2024-05-20T17:50:30.888 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.best_efforts_recovery: 0 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.allow_data_in_errors: 0 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.db_host_id: __hostname__ 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_background_jobs: 2 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_background_compactions: -1 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_subcompactions: 1 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_total_wal_size: 0 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-20T17:50:30.889 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_open_files: -1 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bytes_per_sync: 0 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_readahead_size: 0 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_background_flushes: -1 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Compression algorithms supported: 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: kZSTD supported: 0 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: kXpressCompression supported: 0 2024-05-20T17:50:30.890 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: kLZ4HCCompression supported: 1 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: kZlibCompression supported: 1 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: kSnappyCompression supported: 1 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: kLZ4Compression supported: 1 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: kBZip2Compression supported: 0 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.merge_operator: 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_filter: None 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_filter_factory: None 2024-05-20T17:50:30.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.sst_partitioner_factory: None 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55df64d57e40) 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: cache_index_and_filter_blocks: 1 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: pin_top_level_index_and_filter: 1 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_type: 0 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: data_block_index_type: 0 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_shortening: 1 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: checksum: 4 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: no_block_cache: 0 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache: 0x55df64d5b090 2024-05-20T17:50:30.892 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_name: BinnedLRUCache 2024-05-20T17:50:30.893 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_options: 2024-05-20T17:50:30.893 INFO:journalctl@ceph.mon.c.smithi204.stdout: capacity : 536870912 2024-05-20T17:50:30.893 INFO:journalctl@ceph.mon.c.smithi204.stdout: num_shard_bits : 4 2024-05-20T17:50:30.893 INFO:journalctl@ceph.mon.c.smithi204.stdout: strict_capacity_limit : 0 2024-05-20T17:50:30.893 INFO:journalctl@ceph.mon.c.smithi204.stdout: high_pri_pool_ratio: 0.000 2024-05-20T17:50:30.893 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_compressed: (nil) 2024-05-20T17:50:30.893 INFO:journalctl@ceph.mon.c.smithi204.stdout: persistent_cache: (nil) 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_size: 4096 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_size_deviation: 10 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_restart_interval: 16 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_block_restart_interval: 1 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: metadata_block_size: 4096 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: partition_filters: 0 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: use_delta_encoding: 1 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: filter_policy: bloomfilter 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: whole_key_filtering: 1 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: verify_compression: 0 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: read_amp_bytes_per_bit: 0 2024-05-20T17:50:30.894 INFO:journalctl@ceph.mon.c.smithi204.stdout: format_version: 5 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout: enable_index_compression: 1 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_align: 0 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout: max_auto_readahead_size: 262144 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout: prepopulate_block_cache: 0 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout: initial_auto_readahead_size: 8192 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout: num_file_reads_for_auto_readahead: 2 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.write_buffer_size: 33554432 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_write_buffer_number: 2 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compression: NoCompression 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bottommost_compression: Disabled 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.prefix_extractor: nullptr 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.num_levels: 7 2024-05-20T17:50:30.895 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compression_opts.level: 32767 2024-05-20T17:50:30.896 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compression_opts.strategy: 0 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compression_opts.enabled: false 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.target_file_size_base: 67108864 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-20T17:50:30.897 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.arena_block_size: 1048576 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-20T17:50:30.898 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.disable_auto_compactions: 0 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.table_properties_collectors: 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.inplace_update_support: 0 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-20T17:50:30.899 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.bloom_locality: 0 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.max_successive_merges: 0 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.paranoid_file_checks: 0 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.force_consistency_checks: 1 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.report_bg_io_stats: 0 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.ttl: 2592000 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.enable_blob_files: false 2024-05-20T17:50:30.900 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.min_blob_size: 0 2024-05-20T17:50:30.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.blob_file_size: 268435456 2024-05-20T17:50:30.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-20T17:50:30.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-20T17:50:30.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-20T17:50:30.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-20T17:50:30.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-20T17:50:30.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.blob_file_starting_level: 0 2024-05-20T17:50:30.901 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-20T17:50:31.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:30 smithi129 ceph-mon[32029]: from='client.? 172.21.15.204:0/1188257378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-20T17:50:31.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-20T17:50:31.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-20T17:50:31.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b4d5ab1a-b96a-4382-a98d-8496241fab53 2024-05-20T17:50:31.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716227430889108, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716227430889615, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716227430, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b4d5ab1a-b96a-4382-a98d-8496241fab53", "db_session_id": "9Y8XUM3Q36AKY8S1T58D", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716227430889711, "job": 1, "event": "recovery_finished"} 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55df64d41c00 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: DB pointer 0x55df69b2e000 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** DB Stats ** 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-20T17:50:31.182 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** Compaction Stats [default] ** 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-20T17:50:31.183 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** Compaction Stats [default] ** 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-20T17:50:31.184 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-20T17:50:31.185 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-20T17:50:31.185 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-20T17:50:31.185 INFO:journalctl@ceph.mon.c.smithi204.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-20T17:50:31.185 INFO:journalctl@ceph.mon.c.smithi204.stdout: Block cache BinnedLRUCache@0x55df64d5b090#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-05-20T17:50:31.185 INFO:journalctl@ceph.mon.c.smithi204.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-20T17:50:31.185 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-05-20T17:50:31.185 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-20T17:50:31.185 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-20T17:50:31.186 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: using public_addr v2:172.21.15.204:0/0 -> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-05-20T17:50:31.186 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: starting mon.c rank -1 at public addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] at bind addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:31.186 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(???) e0 preinit fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:31.186 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).mds e1 new map 2024-05-20T17:50:31.186 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-20T17:50:31.186 INFO:journalctl@ceph.mon.c.smithi204.stdout: e1 2024-05-20T17:50:31.186 INFO:journalctl@ceph.mon.c.smithi204.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-20T17:50:31.186 INFO:journalctl@ceph.mon.c.smithi204.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-20T17:50:31.186 INFO:journalctl@ceph.mon.c.smithi204.stdout: legacy client fscid: -1 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout: No filesystems configured 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:50:31.187 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Deploying cephadm binary to smithi204 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Updating smithi154:/etc/ceph/ceph.conf 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:31.188 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Added host smithi204 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:31.189 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3094845761' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3094845761' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Updating smithi204:/etc/ceph/ceph.conf 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:31.191 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi129:172.21.15.129=a;smithi154:172.21.15.154=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:31.192 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Saving service mon spec with placement smithi129:172.21.15.129=a;smithi154:172.21.15.154=b;smithi204:172.21.15.204=c;count:3 2024-05-20T17:50:31.192 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: Deploying daemon mon.c on smithi204 2024-05-20T17:50:31.192 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:31.192 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:31.192 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: from='client.? 172.21.15.204:0/1188257378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-20T17:50:31.192 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:30 smithi204 ceph-mon[37076]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-20T17:50:31.630 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:31 smithi204 bash[37045]: 9a5d5ce86e415ed75bde17d5e1a4226712af369d99856cd846d0e19805eccaa1 2024-05-20T17:50:31.630 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:31 smithi204 systemd[1]: Started Ceph mon.c for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:50:32.656 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-20T17:50:32.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph mon dump -f json 2024-05-20T17:50:32.899 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:50:36.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:50:36.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: mon.a calling monitor election 2024-05-20T17:50:36.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:36.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: mon.c calling monitor election 2024-05-20T17:50:36.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:36.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:36.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-20T17:50:36.416 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: monmap epoch 2 2024-05-20T17:50:36.416 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:36.416 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: last_changed 2024-05-20T17:50:30.913844+0000 2024-05-20T17:50:36.416 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: created 2024-05-20T17:48:46.628733+0000 2024-05-20T17:50:36.416 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: min_mon_release 19 (squid) 2024-05-20T17:50:36.416 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: election_strategy: 1 2024-05-20T17:50:36.416 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-05-20T17:50:36.416 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-05-20T17:50:36.416 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: fsmap 2024-05-20T17:50:36.416 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:36.417 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: mgrmap e13: x(active, since 48s) 2024-05-20T17:50:36.417 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: overall HEALTH_OK 2024-05-20T17:50:36.417 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:36.417 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:36.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:50:36.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: mon.a calling monitor election 2024-05-20T17:50:36.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:36.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: mon.c calling monitor election 2024-05-20T17:50:36.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:36.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:36.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:36.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-20T17:50:36.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: monmap epoch 2 2024-05-20T17:50:36.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:36.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: last_changed 2024-05-20T17:50:30.913844+0000 2024-05-20T17:50:36.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: created 2024-05-20T17:48:46.628733+0000 2024-05-20T17:50:36.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: min_mon_release 19 (squid) 2024-05-20T17:50:36.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: election_strategy: 1 2024-05-20T17:50:36.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-05-20T17:50:36.433 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-05-20T17:50:36.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: fsmap 2024-05-20T17:50:36.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:36.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: mgrmap e13: x(active, since 48s) 2024-05-20T17:50:36.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: overall HEALTH_OK 2024-05-20T17:50:36.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:36.434 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:37.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:36 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:37.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:36 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:37.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:36 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:37.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:36 smithi204 ceph-mon[37076]: Deploying daemon mon.b on smithi154 2024-05-20T17:50:37.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:36 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:37.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.376+0000 7f62d0167700 1 -- 172.21.15.204:0/836825931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b40035c0 msgr2=0x7f62b4005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:37.377 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.376+0000 7f62d0167700 1 --2- 172.21.15.204:0/836825931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b40035c0 0x7f62b4005aa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f62c81029f0 tx=0x7f62c400feb0 comp rx=0 tx=0).stop 2024-05-20T17:50:37.377 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.376+0000 7f62d0167700 1 -- 172.21.15.204:0/836825931 shutdown_connections 2024-05-20T17:50:37.377 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.376+0000 7f62d0167700 1 --2- 172.21.15.204:0/836825931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b40035c0 0x7f62b4005aa0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:37.377 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.376+0000 7f62d0167700 1 -- 172.21.15.204:0/836825931 >> 172.21.15.204:0/836825931 conn(0x7f62c80fe620 msgr2=0x7f62c8100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:37.377 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.376+0000 7f62d0167700 1 -- 172.21.15.204:0/836825931 shutdown_connections 2024-05-20T17:50:37.377 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.376+0000 7f62d0167700 1 -- 172.21.15.204:0/836825931 wait complete. 2024-05-20T17:50:37.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.377+0000 7f62d0167700 1 Processor -- start 2024-05-20T17:50:37.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.377+0000 7f62d0167700 1 -- start start 2024-05-20T17:50:37.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.377+0000 7f62d0167700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b40035c0 0x7f62c819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:37.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.377+0000 7f62d0167700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62c819eba0 0x7f62c8198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:37.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.377+0000 7f62d0167700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f62c810abf0 con 0x7f62b40035c0 2024-05-20T17:50:37.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.377+0000 7f62d0167700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f62c810aa70 con 0x7f62c819eba0 2024-05-20T17:50:37.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.377+0000 7f62cd702700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62c819eba0 0x7f62c8198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:37.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62cd702700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62c819eba0 0x7f62c8198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:35496/0 (socket says 172.21.15.204:35496) 2024-05-20T17:50:37.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62cd702700 1 -- 172.21.15.204:0/4284219616 learned_addr learned my addr 172.21.15.204:0/4284219616 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-20T17:50:37.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62cdf03700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b40035c0 0x7f62c819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:37.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62cd702700 1 -- 172.21.15.204:0/4284219616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62c819eba0 msgr2=0x7f62c8198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-20T17:50:37.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62cd702700 1 -- 172.21.15.204:0/4284219616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62c819eba0 msgr2=0x7f62c8198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-20T17:50:37.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62cd702700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62c819eba0 0x7f62c8198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-20T17:50:37.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62cd702700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62c819eba0 0x7f62c8198880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:50:37.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62cdf03700 1 -- 172.21.15.204:0/4284219616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62c819eba0 msgr2=0x7f62c8198880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:50:37.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62cdf03700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62c819eba0 0x7f62c8198880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:37.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62cdf03700 1 -- 172.21.15.204:0/4284219616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c819f170 con 0x7f62b40035c0 2024-05-20T17:50:37.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62cdf03700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b40035c0 0x7f62c819e660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f62c4010010 tx=0x7f62c4005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:37.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62beffd700 1 -- 172.21.15.204:0/4284219616 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f62c4004aa0 con 0x7f62b40035c0 2024-05-20T17:50:37.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62beffd700 1 -- 172.21.15.204:0/4284219616 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c4013e40 con 0x7f62b40035c0 2024-05-20T17:50:37.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62d0167700 1 -- 172.21.15.204:0/4284219616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c8198e20 con 0x7f62b40035c0 2024-05-20T17:50:37.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.378+0000 7f62beffd700 1 -- 172.21.15.204:0/4284219616 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f62c4011c70 con 0x7f62b40035c0 2024-05-20T17:50:37.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.379+0000 7f62d0167700 1 -- 172.21.15.204:0/4284219616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c8199380 con 0x7f62b40035c0 2024-05-20T17:50:37.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.379+0000 7f62beffd700 1 -- 172.21.15.204:0/4284219616 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f62c4023070 con 0x7f62b40035c0 2024-05-20T17:50:37.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.380+0000 7f62d0167700 1 -- 172.21.15.204:0/4284219616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62c8104200 con 0x7f62b40035c0 2024-05-20T17:50:37.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.380+0000 7f62beffd700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62ac038ee0 0x7f62ac03b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:37.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.380+0000 7f62beffd700 1 -- 172.21.15.204:0/4284219616 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f62c4020050 con 0x7f62b40035c0 2024-05-20T17:50:37.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.380+0000 7f62cd702700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62ac038ee0 0x7f62ac03b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:37.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.380+0000 7f62cd702700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62ac038ee0 0x7f62ac03b3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f62c80fee10 tx=0x7f62b8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:37.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.385+0000 7f62beffd700 1 -- 172.21.15.204:0/4284219616 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f62c4031020 con 0x7f62b40035c0 2024-05-20T17:50:37.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:36 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:37.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:36 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:37.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:36 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:37.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:36 smithi129 ceph-mon[32029]: Deploying daemon mon.b on smithi154 2024-05-20T17:50:37.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:36 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:37.663 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.663+0000 7f62d0167700 1 -- 172.21.15.204:0/4284219616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f62c8199db0 con 0x7f62b40035c0 2024-05-20T17:50:37.664 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.664+0000 7f62beffd700 1 -- 172.21.15.204:0/4284219616 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f62c4028030 con 0x7f62b40035c0 2024-05-20T17:50:37.664 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:50:37.664 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":2,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","modified":"2024-05-20T17:50:30.913844Z","created":"2024-05-20T17:48:46.628733Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-20T17:50:37.667 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.667+0000 7f62d0167700 1 -- 172.21.15.204:0/4284219616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62ac038ee0 msgr2=0x7f62ac03b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:37.668 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.667+0000 7f62d0167700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62ac038ee0 0x7f62ac03b3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f62c80fee10 tx=0x7f62b8008040 comp rx=0 tx=0).stop 2024-05-20T17:50:37.668 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.667+0000 7f62d0167700 1 -- 172.21.15.204:0/4284219616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b40035c0 msgr2=0x7f62c819e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:37.668 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.667+0000 7f62d0167700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b40035c0 0x7f62c819e660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f62c4010010 tx=0x7f62c4005570 comp rx=0 tx=0).stop 2024-05-20T17:50:37.668 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.667+0000 7f62d0167700 1 -- 172.21.15.204:0/4284219616 shutdown_connections 2024-05-20T17:50:37.668 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.667+0000 7f62d0167700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62ac038ee0 0x7f62ac03b3f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:37.668 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.667+0000 7f62d0167700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b40035c0 0x7f62c819e660 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:37.668 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.667+0000 7f62d0167700 1 --2- 172.21.15.204:0/4284219616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62c819eba0 0x7f62c8198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:37.668 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.667+0000 7f62d0167700 1 -- 172.21.15.204:0/4284219616 >> 172.21.15.204:0/4284219616 conn(0x7f62c80fe620 msgr2=0x7f62c8105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:37.668 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.667+0000 7f62d0167700 1 -- 172.21.15.204:0/4284219616 shutdown_connections 2024-05-20T17:50:37.668 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:37.667+0000 7f62d0167700 1 -- 172.21.15.204:0/4284219616 wait complete. 2024-05-20T17:50:37.669 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 2 2024-05-20T17:50:38.376 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:38 smithi204 ceph-mon[37076]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:38.376 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:38 smithi204 ceph-mon[37076]: from='client.? 172.21.15.204:0/4284219616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-20T17:50:38.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:38 smithi129 ceph-mon[32029]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:38.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:38 smithi129 ceph-mon[32029]: from='client.? 172.21.15.204:0/4284219616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-20T17:50:39.404 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-20T17:50:39.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph mon dump -f json 2024-05-20T17:50:39.643 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:50:39.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(???) e0 preinit fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:40.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).mds e1 new map 2024-05-20T17:50:40.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).mds e1 print_map 2024-05-20T17:50:40.145 INFO:journalctl@ceph.mon.b.smithi154.stdout: e1 2024-05-20T17:50:40.146 INFO:journalctl@ceph.mon.b.smithi154.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-20T17:50:40.146 INFO:journalctl@ceph.mon.b.smithi154.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-20T17:50:40.146 INFO:journalctl@ceph.mon.b.smithi154.stdout: legacy client fscid: -1 2024-05-20T17:50:40.146 INFO:journalctl@ceph.mon.b.smithi154.stdout: 2024-05-20T17:50:40.146 INFO:journalctl@ceph.mon.b.smithi154.stdout: No filesystems configured 2024-05-20T17:50:40.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-20T17:50:40.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-20T17:50:40.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-20T17:50:40.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-20T17:50:40.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-20T17:50:40.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-20T17:50:40.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-20T17:50:40.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-20T17:50:40.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-20T17:50:40.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-20T17:50:40.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:50:40.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:40.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Deploying cephadm binary to smithi204 2024-05-20T17:50:40.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Updating smithi154:/etc/ceph/ceph.conf 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:50:40.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Added host smithi204 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3094845761' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3094845761' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Updating smithi204:/etc/ceph/ceph.conf 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.150 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.151 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.151 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.151 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:40.151 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:40.151 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:50:40.151 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi129:172.21.15.129=a;smithi154:172.21.15.154=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Saving service mon spec with placement smithi129:172.21.15.129=a;smithi154:172.21.15.154=b;smithi204:172.21.15.204=c;count:3 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Deploying daemon mon.c on smithi204 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.152 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='client.? 172.21.15.204:0/1188257378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.a calling monitor election 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.c calling monitor election 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:40.153 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: monmap epoch 2 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: last_changed 2024-05-20T17:50:30.913844+0000 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: created 2024-05-20T17:48:46.628733+0000 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: min_mon_release 19 (squid) 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: election_strategy: 1 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: fsmap 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:40.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mgrmap e13: x(active, since 48s) 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: overall HEALTH_OK 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: Deploying daemon mon.b on smithi154 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: from='client.? 172.21.15.204:0/4284219616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 ceph-mon[36482]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 bash[36463]: d52372016c01d90b428e254277ef200845bb0fc7bd9b5b61523704eaa2c3b22a 2024-05-20T17:50:40.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:39 smithi154 systemd[1]: Started Ceph mon.b for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:50:45.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:45.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:50:45.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: mon.c calling monitor election 2024-05-20T17:50:45.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: mon.a calling monitor election 2024-05-20T17:50:45.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:45.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:45.166 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.166 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.166 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:45.166 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.166 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.166 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-20T17:50:45.166 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: monmap epoch 3 2024-05-20T17:50:45.167 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:45.167 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: last_changed 2024-05-20T17:50:39.659477+0000 2024-05-20T17:50:45.167 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: created 2024-05-20T17:48:46.628733+0000 2024-05-20T17:50:45.167 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: min_mon_release 19 (squid) 2024-05-20T17:50:45.167 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: election_strategy: 1 2024-05-20T17:50:45.167 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-05-20T17:50:45.167 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-05-20T17:50:45.168 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: 2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-05-20T17:50:45.168 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: fsmap 2024-05-20T17:50:45.168 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:45.168 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: mgrmap e13: x(active, since 57s) 2024-05-20T17:50:45.168 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: overall HEALTH_OK 2024-05-20T17:50:45.168 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:45.168 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:45.169 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:45.169 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:45.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:45.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:50:45.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: mon.c calling monitor election 2024-05-20T17:50:45.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: mon.a calling monitor election 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: monmap epoch 3 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:45.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: last_changed 2024-05-20T17:50:39.659477+0000 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: created 2024-05-20T17:48:46.628733+0000 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: min_mon_release 19 (squid) 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: election_strategy: 1 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: 2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: fsmap 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: mgrmap e13: x(active, since 57s) 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: overall HEALTH_OK 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:45.183 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:45.367 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.367+0000 7f01249a0700 1 -- 172.21.15.204:0/4184733517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01080035c0 msgr2=0x7f0108005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:45.367 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.367+0000 7f01249a0700 1 --2- 172.21.15.204:0/4184733517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01080035c0 0x7f0108005aa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f012006ac40 tx=0x7f011000fe60 comp rx=0 tx=0).stop 2024-05-20T17:50:45.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.367+0000 7f01249a0700 1 -- 172.21.15.204:0/4184733517 shutdown_connections 2024-05-20T17:50:45.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.367+0000 7f01249a0700 1 --2- 172.21.15.204:0/4184733517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01080035c0 0x7f0108005aa0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:45.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.367+0000 7f01249a0700 1 -- 172.21.15.204:0/4184733517 >> 172.21.15.204:0/4184733517 conn(0x7f01200fd340 msgr2=0x7f01200ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:45.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.367+0000 7f01249a0700 1 -- 172.21.15.204:0/4184733517 shutdown_connections 2024-05-20T17:50:45.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.367+0000 7f01249a0700 1 -- 172.21.15.204:0/4184733517 wait complete. 2024-05-20T17:50:45.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.367+0000 7f01249a0700 1 Processor -- start 2024-05-20T17:50:45.369 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f01249a0700 1 -- start start 2024-05-20T17:50:45.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f01249a0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01080035c0 0x7f0120101720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:45.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f01249a0700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0120101c60 0x7f0120102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:45.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f01249a0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01201a4bc0 0x7f01201a6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:45.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f01249a0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f012010a9d0 con 0x7f01201a4bc0 2024-05-20T17:50:45.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f01249a0700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f012010a850 con 0x7f01080035c0 2024-05-20T17:50:45.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f01249a0700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f012010ab50 con 0x7f0120101c60 2024-05-20T17:50:45.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011effd700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0120101c60 0x7f0120102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:45.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01201a4bc0 0x7f01201a6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:45.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011effd700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0120101c60 0x7f0120102070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:35504/0 (socket says 172.21.15.204:35504) 2024-05-20T17:50:45.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011f7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01080035c0 0x7f0120101720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:45.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011effd700 1 -- 172.21.15.204:0/2365412465 learned_addr learned my addr 172.21.15.204:0/2365412465 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011effd700 1 -- 172.21.15.204:0/2365412465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0120101c60 msgr2=0x7f0120102070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011effd700 1 -- 172.21.15.204:0/2365412465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0120101c60 msgr2=0x7f0120102070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011effd700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0120101c60 0x7f0120102070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011f7fe700 1 -- 172.21.15.204:0/2365412465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01080035c0 msgr2=0x7f0120101720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011f7fe700 1 -- 172.21.15.204:0/2365412465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01080035c0 msgr2=0x7f0120101720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011f7fe700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01080035c0 0x7f0120101720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011f7fe700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01080035c0 0x7f0120101720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011effd700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0120101c60 0x7f0120102070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011ffff700 1 -- 172.21.15.204:0/2365412465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0120101c60 msgr2=0x7f0120102070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011ffff700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0120101c60 0x7f0120102070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011ffff700 1 -- 172.21.15.204:0/2365412465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01080035c0 msgr2=0x7f0120101720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011ffff700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01080035c0 0x7f0120101720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.368+0000 7f011ffff700 1 -- 172.21.15.204:0/2365412465 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01201a7660 con 0x7f01201a4bc0 2024-05-20T17:50:45.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.369+0000 7f011ffff700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01201a4bc0 0x7f01201a6fe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f011800bd50 tx=0x7f011800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:45.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.369+0000 7f011cff9700 1 -- 172.21.15.204:0/2365412465 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118019070 con 0x7f01201a4bc0 2024-05-20T17:50:45.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.369+0000 7f011cff9700 1 -- 172.21.15.204:0/2365412465 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f011800cd10 con 0x7f01201a4bc0 2024-05-20T17:50:45.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.369+0000 7f011cff9700 1 -- 172.21.15.204:0/2365412465 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01180127e0 con 0x7f01201a4bc0 2024-05-20T17:50:45.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.369+0000 7f01249a0700 1 -- 172.21.15.204:0/2365412465 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01201a7950 con 0x7f01201a4bc0 2024-05-20T17:50:45.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.369+0000 7f01249a0700 1 -- 172.21.15.204:0/2365412465 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01201a7e90 con 0x7f01201a4bc0 2024-05-20T17:50:45.373 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.370+0000 7f011cff9700 1 -- 172.21.15.204:0/2365412465 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f011801c070 con 0x7f01201a4bc0 2024-05-20T17:50:45.373 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.370+0000 7f01249a0700 1 -- 172.21.15.204:0/2365412465 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0120063ca0 con 0x7f01201a4bc0 2024-05-20T17:50:45.373 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.370+0000 7f011cff9700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f010c039580 0x7f010c03ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:45.373 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.370+0000 7f011cff9700 1 -- 172.21.15.204:0/2365412465 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f011804cb10 con 0x7f01201a4bc0 2024-05-20T17:50:45.373 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.370+0000 7f011f7fe700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f010c039580 0x7f010c03ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:45.373 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.371+0000 7f011f7fe700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f010c039580 0x7f010c03ba90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f011000f540 tx=0x7f0110001e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:45.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.376+0000 7f011cff9700 1 -- 172.21.15.204:0/2365412465 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0118013c30 con 0x7f01201a4bc0 2024-05-20T17:50:45.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.680+0000 7f01249a0700 1 -- 172.21.15.204:0/2365412465 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f012002dc60 con 0x7f01201a4bc0 2024-05-20T17:50:45.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.680+0000 7f011cff9700 1 -- 172.21.15.204:0/2365412465 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7f011801d730 con 0x7f01201a4bc0 2024-05-20T17:50:45.681 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-05-20T17:50:45.681 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":3,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","modified":"2024-05-20T17:50:39.659477Z","created":"2024-05-20T17:48:46.628733Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-20T17:50:45.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 -- 172.21.15.204:0/2365412465 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f010c039580 msgr2=0x7f010c03ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:45.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f010c039580 0x7f010c03ba90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f011000f540 tx=0x7f0110001e70 comp rx=0 tx=0).stop 2024-05-20T17:50:45.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 -- 172.21.15.204:0/2365412465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01201a4bc0 msgr2=0x7f01201a6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:45.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01201a4bc0 0x7f01201a6fe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f011800bd50 tx=0x7f011800be30 comp rx=0 tx=0).stop 2024-05-20T17:50:45.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 -- 172.21.15.204:0/2365412465 shutdown_connections 2024-05-20T17:50:45.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f010c039580 0x7f010c03ba90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:45.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01080035c0 0x7f0120101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:45.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01201a4bc0 0x7f01201a6fe0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:45.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 --2- 172.21.15.204:0/2365412465 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0120101c60 0x7f0120102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:45.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 -- 172.21.15.204:0/2365412465 >> 172.21.15.204:0/2365412465 conn(0x7f01200fd340 msgr2=0x7f01201002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:45.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 -- 172.21.15.204:0/2365412465 shutdown_connections 2024-05-20T17:50:45.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:45.683+0000 7f01249a0700 1 -- 172.21.15.204:0/2365412465 wait complete. 2024-05-20T17:50:45.684 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 3 2024-05-20T17:50:45.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:45.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:45 smithi204 ceph-mon[37076]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:45.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:45.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:45 smithi204 ceph-mon[37076]: from='client.? 172.21.15.204:0/2365412465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-20T17:50:46.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:46.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:45 smithi129 ceph-mon[32029]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:46.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:46.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:45 smithi129 ceph-mon[32029]: from='client.? 172.21.15.204:0/2365412465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-20T17:50:46.250 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-20T17:50:46.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph config generate-minimal-conf 2024-05-20T17:50:46.489 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:50:47.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:47.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:50:47.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: mon.c calling monitor election 2024-05-20T17:50:47.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: mon.a calling monitor election 2024-05-20T17:50:47.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:47.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: monmap epoch 3 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: last_changed 2024-05-20T17:50:39.659477+0000 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: created 2024-05-20T17:48:46.628733+0000 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: min_mon_release 19 (squid) 2024-05-20T17:50:47.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: election_strategy: 1 2024-05-20T17:50:47.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-05-20T17:50:47.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-05-20T17:50:47.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: 2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-05-20T17:50:47.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: fsmap 2024-05-20T17:50:47.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:47.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: mgrmap e13: x(active, since 57s) 2024-05-20T17:50:47.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: overall HEALTH_OK 2024-05-20T17:50:47.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:47.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:47.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:47.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:46 smithi154 ceph-mon[36482]: from='client.? 172.21.15.204:0/2365412465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-20T17:50:47.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: mon.b calling monitor election 2024-05-20T17:50:47.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: mon.b calling monitor election 2024-05-20T17:50:47.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: mon.c calling monitor election 2024-05-20T17:50:47.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: mon.a calling monitor election 2024-05-20T17:50:47.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-20T17:50:47.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: monmap epoch 3 2024-05-20T17:50:47.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:47.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: last_changed 2024-05-20T17:50:39.659477+0000 2024-05-20T17:50:47.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: created 2024-05-20T17:48:46.628733+0000 2024-05-20T17:50:47.793 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: min_mon_release 19 (squid) 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: election_strategy: 1 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: 2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: fsmap 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: mgrmap e13: x(active, since 59s) 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: overall HEALTH_OK 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.794 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.795 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.795 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.795 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.795 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.795 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.795 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.795 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:47.795 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-20T17:50:47.795 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:47.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: mon.b calling monitor election 2024-05-20T17:50:47.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: mon.b calling monitor election 2024-05-20T17:50:47.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: mon.c calling monitor election 2024-05-20T17:50:47.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: mon.a calling monitor election 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: monmap epoch 3 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: last_changed 2024-05-20T17:50:39.659477+0000 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: created 2024-05-20T17:48:46.628733+0000 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: min_mon_release 19 (squid) 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: election_strategy: 1 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: 2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: fsmap 2024-05-20T17:50:47.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: mgrmap e13: x(active, since 59s) 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: overall HEALTH_OK 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.897 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.898 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:47.898 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-20T17:50:47.898 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:47.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: mon.b calling monitor election 2024-05-20T17:50:47.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: mon.b calling monitor election 2024-05-20T17:50:47.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: mon.c calling monitor election 2024-05-20T17:50:47.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: mon.a calling monitor election 2024-05-20T17:50:47.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: monmap epoch 3 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: last_changed 2024-05-20T17:50:39.659477+0000 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: created 2024-05-20T17:48:46.628733+0000 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: min_mon_release 19 (squid) 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: election_strategy: 1 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: 0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: 2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: fsmap 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: osdmap e4: 0 total, 0 up, 0 in 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: mgrmap e13: x(active, since 59s) 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: overall HEALTH_OK 2024-05-20T17:50:47.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-20T17:50:47.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.009+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3471802810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb781078d0 msgr2=0x7ffb78107ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.009+0000 7ffb7cfc7700 1 --2- 172.21.15.129:0/3471802810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb781078d0 0x7ffb78107ce0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ffb68009910 tx=0x7ffb68009c20 comp rx=0 tx=0).stop 2024-05-20T17:50:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.009+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3471802810 shutdown_connections 2024-05-20T17:50:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.009+0000 7ffb7cfc7700 1 --2- 172.21.15.129:0/3471802810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb781078d0 0x7ffb78107ce0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.009+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3471802810 >> 172.21.15.129:0/3471802810 conn(0x7ffb780fd3e0 msgr2=0x7ffb780ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.009+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3471802810 shutdown_connections 2024-05-20T17:50:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.009+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3471802810 wait complete. 2024-05-20T17:50:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.010+0000 7ffb7cfc7700 1 Processor -- start 2024-05-20T17:50:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.010+0000 7ffb7cfc7700 1 -- start start 2024-05-20T17:50:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.010+0000 7ffb7cfc7700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb781078d0 0x7ffb7819e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.010+0000 7ffb7cfc7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb7819ecd0 0x7ffb781989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb7cfc7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb78198ef0 0x7ffb78199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb7cfc7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7810bf30 con 0x7ffb7819ecd0 2024-05-20T17:50:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb7cfc7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7810bdb0 con 0x7ffb78198ef0 2024-05-20T17:50:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb7cfc7700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7810c0b0 con 0x7ffb781078d0 2024-05-20T17:50:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb75d9b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb7819ecd0 0x7ffb781989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb75d9b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb7819ecd0 0x7ffb781989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42038/0 (socket says 172.21.15.129:42038) 2024-05-20T17:50:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb75d9b700 1 -- 172.21.15.129:0/3708495539 learned_addr learned my addr 172.21.15.129:0/3708495539 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb7659c700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb781078d0 0x7ffb7819e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb76d9d700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb78198ef0 0x7ffb78199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb75d9b700 1 -- 172.21.15.129:0/3708495539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb781078d0 msgr2=0x7ffb7819e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb75d9b700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb781078d0 0x7ffb7819e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb75d9b700 1 -- 172.21.15.129:0/3708495539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb78198ef0 msgr2=0x7ffb78199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb75d9b700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb78198ef0 0x7ffb78199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb75d9b700 1 -- 172.21.15.129:0/3708495539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb781a5620 con 0x7ffb7819ecd0 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb76d9d700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb78198ef0 0x7ffb78199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb75d9b700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb7819ecd0 0x7ffb781989b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c00b780 tx=0x7ffb6c00bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb677fe700 1 -- 172.21.15.129:0/3708495539 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c004230 con 0x7ffb7819ecd0 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb677fe700 1 -- 172.21.15.129:0/3708495539 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb6c0043d0 con 0x7ffb7819ecd0 2024-05-20T17:50:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.011+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3708495539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb781a58b0 con 0x7ffb7819ecd0 2024-05-20T17:50:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.012+0000 7ffb677fe700 1 -- 172.21.15.129:0/3708495539 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c011c60 con 0x7ffb7819ecd0 2024-05-20T17:50:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.012+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3708495539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb781a5e10 con 0x7ffb7819ecd0 2024-05-20T17:50:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.012+0000 7ffb677fe700 1 -- 172.21.15.129:0/3708495539 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7ffb6c01a400 con 0x7ffb7819ecd0 2024-05-20T17:50:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.013+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3708495539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb78063ca0 con 0x7ffb7819ecd0 2024-05-20T17:50:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.013+0000 7ffb677fe700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffb60039580 0x7ffb6003ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.013+0000 7ffb677fe700 1 -- 172.21.15.129:0/3708495539 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ffb6c04bdd0 con 0x7ffb7819ecd0 2024-05-20T17:50:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.013+0000 7ffb7659c700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffb60039580 0x7ffb6003ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.013+0000 7ffb7659c700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffb60039580 0x7ffb6003ba90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800b5c0 tx=0x7ffb6801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:48.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.019+0000 7ffb677fe700 1 -- 172.21.15.129:0/3708495539 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffb6c04f030 con 0x7ffb7819ecd0 2024-05-20T17:50:48.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.221+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3708495539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7ffb781a6100 con 0x7ffb7819ecd0 2024-05-20T17:50:48.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.221+0000 7ffb677fe700 1 -- 172.21.15.129:0/3708495539 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7ffb6c01a7f0 con 0x7ffb7819ecd0 2024-05-20T17:50:48.222 INFO:teuthology.orchestra.run.smithi129.stdout:# minimal ceph.conf for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:48.222 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-05-20T17:50:48.222 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:50:48.222 INFO:teuthology.orchestra.run.smithi129.stdout: mon_host = [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-05-20T17:50:48.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3708495539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffb60039580 msgr2=0x7ffb6003ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:48.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffb60039580 0x7ffb6003ba90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800b5c0 tx=0x7ffb6801a040 comp rx=0 tx=0).stop 2024-05-20T17:50:48.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3708495539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb7819ecd0 msgr2=0x7ffb781989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:48.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb7819ecd0 0x7ffb781989b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c00b780 tx=0x7ffb6c00bb40 comp rx=0 tx=0).stop 2024-05-20T17:50:48.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3708495539 shutdown_connections 2024-05-20T17:50:48.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffb60039580 0x7ffb6003ba90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:48.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb78198ef0 0x7ffb78199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:48.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffb781078d0 0x7ffb7819e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:48.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 --2- 172.21.15.129:0/3708495539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb7819ecd0 0x7ffb781989b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:48.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3708495539 >> 172.21.15.129:0/3708495539 conn(0x7ffb780fd3e0 msgr2=0x7ffb781053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:48.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3708495539 shutdown_connections 2024-05-20T17:50:48.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:48.224+0000 7ffb7cfc7700 1 -- 172.21.15.129:0/3708495539 wait complete. 2024-05-20T17:50:48.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:48 smithi129 ceph-mon[32029]: Updating smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:48.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:48 smithi129 ceph-mon[32029]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:48.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:48 smithi129 ceph-mon[32029]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:48.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:48 smithi129 ceph-mon[32029]: Reconfiguring mon.a (unknown last config time)... 2024-05-20T17:50:48.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:48 smithi129 ceph-mon[32029]: Reconfiguring daemon mon.a on smithi129 2024-05-20T17:50:48.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:48 smithi129 ceph-mon[32029]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:48.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:48.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:48.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:48 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3708495539' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:48.822 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-20T17:50:48.822 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:50:48.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.conf 2024-05-20T17:50:48.850 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:50:48.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:48.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:48 smithi154 ceph-mon[36482]: Updating smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:48.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:48 smithi154 ceph-mon[36482]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:48.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:48 smithi154 ceph-mon[36482]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:48.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:48 smithi154 ceph-mon[36482]: Reconfiguring mon.a (unknown last config time)... 2024-05-20T17:50:48.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:48 smithi154 ceph-mon[36482]: Reconfiguring daemon mon.a on smithi129 2024-05-20T17:50:48.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:48 smithi154 ceph-mon[36482]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:48.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:48.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:48.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:48 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3708495539' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:48.915 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-05-20T17:50:48.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.conf 2024-05-20T17:50:48.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:48 smithi204 ceph-mon[37076]: Updating smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:48.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:48 smithi204 ceph-mon[37076]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:48.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:48 smithi204 ceph-mon[37076]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:50:48.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:48 smithi204 ceph-mon[37076]: Reconfiguring mon.a (unknown last config time)... 2024-05-20T17:50:48.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:48 smithi204 ceph-mon[37076]: Reconfiguring daemon mon.a on smithi129 2024-05-20T17:50:48.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:48 smithi204 ceph-mon[37076]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:48.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:50:48.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:48.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:48 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3708495539' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:48.947 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-05-20T17:50:48.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:49.016 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-20T17:50:49.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.conf 2024-05-20T17:50:49.047 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-20T17:50:49.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:50:49.117 INFO:tasks.cephadm:Adding mgr.x on smithi129 2024-05-20T17:50:49.117 INFO:tasks.cephadm:Adding mgr.y on smithi154 2024-05-20T17:50:49.117 INFO:tasks.cephadm:Adding mgr.z on smithi204 2024-05-20T17:50:49.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch apply mgr '3;smithi129=x;smithi154=y;smithi204=z' 2024-05-20T17:50:49.392 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:50:49.771 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:49 smithi204 ceph-mon[37076]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:50.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:49 smithi154 ceph-mon[36482]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:50.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:49 smithi129 ceph-mon[32029]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:50.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:50.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:50.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:50 smithi154 ceph-mon[36482]: Reconfiguring mon.b (monmap changed)... 2024-05-20T17:50:50.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:50.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-20T17:50:50.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:50.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:50 smithi154 ceph-mon[36482]: Reconfiguring daemon mon.b on smithi154 2024-05-20T17:50:50.928 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.924+0000 7fc61e4d4700 1 -- 172.21.15.204:0/1303482172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618101a70 msgr2=0x7fc618101e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.924+0000 7fc61e4d4700 1 --2- 172.21.15.204:0/1303482172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618101a70 0x7fc618101e80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc608009910 tx=0x7fc608009c20 comp rx=0 tx=0).stop 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.924+0000 7fc61e4d4700 1 -- 172.21.15.204:0/1303482172 shutdown_connections 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.924+0000 7fc61e4d4700 1 --2- 172.21.15.204:0/1303482172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618101a70 0x7fc618101e80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.924+0000 7fc61e4d4700 1 -- 172.21.15.204:0/1303482172 >> 172.21.15.204:0/1303482172 conn(0x7fc6180fd0d0 msgr2=0x7fc6180ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.924+0000 7fc61e4d4700 1 -- 172.21.15.204:0/1303482172 shutdown_connections 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.924+0000 7fc61e4d4700 1 -- 172.21.15.204:0/1303482172 wait complete. 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.925+0000 7fc61e4d4700 1 Processor -- start 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.925+0000 7fc61e4d4700 1 -- start start 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.925+0000 7fc61e4d4700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc618101a70 0x7fc618198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.925+0000 7fc61e4d4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618198dc0 0x7fc61819dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.925+0000 7fc61e4d4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61819e290 0x7fc6181a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:50.929 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.925+0000 7fc61e4d4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc618107b40 con 0x7fc618198dc0 2024-05-20T17:50:50.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.925+0000 7fc61e4d4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6181079c0 con 0x7fc61819e290 2024-05-20T17:50:50.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.925+0000 7fc61e4d4700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc618107cc0 con 0x7fc618101a70 2024-05-20T17:50:50.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc617fff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc618101a70 0x7fc618198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:50.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc617fff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc618101a70 0x7fc618198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:36978/0 (socket says 172.21.15.204:36978) 2024-05-20T17:50:50.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc617fff700 1 -- 172.21.15.204:0/2181649665 learned_addr learned my addr 172.21.15.204:0/2181649665 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-20T17:50:50.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc6177fe700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618198dc0 0x7fc61819dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:50.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc61ca71700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61819e290 0x7fc6181a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:50.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc617fff700 1 -- 172.21.15.204:0/2181649665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61819e290 msgr2=0x7fc6181a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:50.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc617fff700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61819e290 0x7fc6181a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:50.930 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc617fff700 1 -- 172.21.15.204:0/2181649665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618198dc0 msgr2=0x7fc61819dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc617fff700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618198dc0 0x7fc61819dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc617fff700 1 -- 172.21.15.204:0/2181649665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6181a0c20 con 0x7fc618101a70 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc61ca71700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61819e290 0x7fc6181a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc617fff700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc618101a70 0x7fc618198880 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc608004f80 tx=0x7fc608004fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc6157fa700 1 -- 172.21.15.204:0/2181649665 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc608027070 con 0x7fc618101a70 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc6157fa700 1 -- 172.21.15.204:0/2181649665 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc608022470 con 0x7fc618101a70 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc6157fa700 1 -- 172.21.15.204:0/2181649665 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60800f6a0 con 0x7fc618101a70 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc61e4d4700 1 -- 172.21.15.204:0/2181649665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6181a0eb0 con 0x7fc618101a70 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.926+0000 7fc61e4d4700 1 -- 172.21.15.204:0/2181649665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6181a1380 con 0x7fc618101a70 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.927+0000 7fc6157fa700 1 -- 172.21.15.204:0/2181649665 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fc60802a070 con 0x7fc618101a70 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.927+0000 7fc6157fa700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc600039580 0x7fc60003ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.927+0000 7fc6157fa700 1 -- 172.21.15.204:0/2181649665 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc60804e1d0 con 0x7fc618101a70 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.928+0000 7fc61e4d4700 1 -- 172.21.15.204:0/2181649665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc618063ca0 con 0x7fc618101a70 2024-05-20T17:50:50.931 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.928+0000 7fc6177fe700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc600039580 0x7fc60003ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:50.932 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.928+0000 7fc6177fe700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc600039580 0x7fc60003ba90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc618199930 tx=0x7fc60c009210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:50.933 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:50.933+0000 7fc6157fa700 1 -- 172.21.15.204:0/2181649665 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc608031020 con 0x7fc618101a70 2024-05-20T17:50:51.144 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.143+0000 7fc61e4d4700 1 -- 172.21.15.204:0/2181649665 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi129=x;smithi154=y;smithi204=z", "target": ["mon-mgr", ""]}) -- 0x7fc618062a80 con 0x7fc600039580 2024-05-20T17:50:51.144 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:51.144 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:51.144 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:50 smithi204 ceph-mon[37076]: Reconfiguring mon.b (monmap changed)... 2024-05-20T17:50:51.145 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:51.145 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-20T17:50:51.145 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:51.145 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:50 smithi204 ceph-mon[37076]: Reconfiguring daemon mon.b on smithi154 2024-05-20T17:50:51.151 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.151+0000 7fc6157fa700 1 -- 172.21.15.204:0/2181649665 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc618062a80 con 0x7fc600039580 2024-05-20T17:50:51.151 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mgr update... 2024-05-20T17:50:51.153 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 -- 172.21.15.204:0/2181649665 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc600039580 msgr2=0x7fc60003ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:51.153 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc600039580 0x7fc60003ba90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc618199930 tx=0x7fc60c009210 comp rx=0 tx=0).stop 2024-05-20T17:50:51.153 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 -- 172.21.15.204:0/2181649665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc618101a70 msgr2=0x7fc618198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:51.153 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc618101a70 0x7fc618198880 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc608004f80 tx=0x7fc608004fb0 comp rx=0 tx=0).stop 2024-05-20T17:50:51.153 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 -- 172.21.15.204:0/2181649665 shutdown_connections 2024-05-20T17:50:51.153 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc600039580 0x7fc60003ba90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:51.153 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61819e290 0x7fc6181a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:51.154 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc618101a70 0x7fc618198880 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:51.154 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 --2- 172.21.15.204:0/2181649665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618198dc0 0x7fc61819dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:51.154 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 -- 172.21.15.204:0/2181649665 >> 172.21.15.204:0/2181649665 conn(0x7fc6180fd0d0 msgr2=0x7fc61806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:51.154 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 -- 172.21.15.204:0/2181649665 shutdown_connections 2024-05-20T17:50:51.154 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:50:51.153+0000 7fc61e4d4700 1 -- 172.21.15.204:0/2181649665 wait complete. 2024-05-20T17:50:51.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:51.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:51.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:50 smithi129 ceph-mon[32029]: Reconfiguring mon.b (monmap changed)... 2024-05-20T17:50:51.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:51.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-20T17:50:51.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:51.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:50 smithi129 ceph-mon[32029]: Reconfiguring daemon mon.b on smithi154 2024-05-20T17:50:51.654 DEBUG:teuthology.orchestra.run.smithi154:mgr.y> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.y.service 2024-05-20T17:50:51.657 DEBUG:teuthology.orchestra.run.smithi204:mgr.z> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.z.service 2024-05-20T17:50:51.659 INFO:tasks.cephadm:Deploying OSDs... 2024-05-20T17:50:51.659 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:50:51.659 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-05-20T17:50:51.679 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-20T17:50:51.679 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-05-20T17:50:51.736 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-20T17:50:51.736 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:51.736 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-20T17:50:51.736 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:51.737 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:51.737 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-05-20 17:49:56.939445988 +0000 2024-05-20T17:50:51.737 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-05-20 17:44:30.990175987 +0000 2024-05-20T17:50:51.737 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-05-20 17:44:30.990175987 +0000 2024-05-20T17:50:51.737 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-05-20 17:44:30.990175987 +0000 2024-05-20T17:50:51.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-20T17:50:51.804 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-05-20T17:50:51.804 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-05-20T17:50:51.804 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.00022672 s, 2.3 MB/s 2024-05-20T17:50:51.806 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-20T17:50:51.865 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-05-20T17:50:51.922 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-20T17:50:51.922 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:51.922 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-05-20T17:50:51.922 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:51.923 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:51.923 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-05-20 17:49:56.971445317 +0000 2024-05-20T17:50:51.923 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-05-20 17:44:31.311168638 +0000 2024-05-20T17:50:51.923 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-05-20 17:44:31.311168638 +0000 2024-05-20T17:50:51.923 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-05-20 17:44:31.311168638 +0000 2024-05-20T17:50:51.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-20T17:50:51.988 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-05-20T17:50:51.988 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-05-20T17:50:51.988 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000231798 s, 2.2 MB/s 2024-05-20T17:50:51.989 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-20T17:50:52.048 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-05-20T17:50:52.105 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-20T17:50:52.106 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:52.106 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-20T17:50:52.106 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:52.106 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:52.106 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-05-20 17:49:57.002444667 +0000 2024-05-20T17:50:52.106 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-05-20 17:44:31.647160946 +0000 2024-05-20T17:50:52.106 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-05-20 17:44:31.647160946 +0000 2024-05-20T17:50:52.106 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-05-20 17:44:31.647160946 +0000 2024-05-20T17:50:52.106 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-20T17:50:52.171 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-05-20T17:50:52.171 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-05-20T17:50:52.171 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000245388 s, 2.1 MB/s 2024-05-20T17:50:52.172 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-20T17:50:52.230 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-05-20T17:50:52.286 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-20T17:50:52.286 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:52.286 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-20T17:50:52.286 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:52.286 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:52.287 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-05-20 17:49:57.030444080 +0000 2024-05-20T17:50:52.287 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-05-20 17:44:32.000152864 +0000 2024-05-20T17:50:52.287 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-05-20 17:44:32.000152864 +0000 2024-05-20T17:50:52.287 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-05-20 17:44:32.000152864 +0000 2024-05-20T17:50:52.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-20T17:50:52.350 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-05-20T17:50:52.350 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-05-20T17:50:52.351 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000235475 s, 2.2 MB/s 2024-05-20T17:50:52.352 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-20T17:50:52.409 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-05-20T17:50:52.409 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-05-20T17:50:52.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:52 smithi129 ceph-mon[32029]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi129=x;smithi154=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:52.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:52 smithi129 ceph-mon[32029]: Saving service mgr spec with placement smithi129=x;smithi154=y;smithi204=z;count:3 2024-05-20T17:50:52.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:52 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:52.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:52 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:52.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:52 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:52.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:52 smithi129 ceph-mon[32029]: Reconfiguring mon.c (monmap changed)... 2024-05-20T17:50:52.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:52 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:52.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:52 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-20T17:50:52.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:52 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:52.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:52 smithi129 ceph-mon[32029]: Reconfiguring daemon mon.c on smithi204 2024-05-20T17:50:52.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:52 smithi129 ceph-mon[32029]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:52.428 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-20T17:50:52.428 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-05-20T17:50:52.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:52 smithi204 ceph-mon[37076]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi129=x;smithi154=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:52.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:52 smithi204 ceph-mon[37076]: Saving service mgr spec with placement smithi129=x;smithi154=y;smithi204=z;count:3 2024-05-20T17:50:52.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:52 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:52.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:52 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:52.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:52 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:52.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:52 smithi204 ceph-mon[37076]: Reconfiguring mon.c (monmap changed)... 2024-05-20T17:50:52.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:52 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:52.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:52 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-20T17:50:52.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:52 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:52.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:52 smithi204 ceph-mon[37076]: Reconfiguring daemon mon.c on smithi204 2024-05-20T17:50:52.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:52 smithi204 ceph-mon[37076]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-20T17:50:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:52.486 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-20T17:50:52.486 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:52.486 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:52.486 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-05-20 17:50:39.096164199 +0000 2024-05-20T17:50:52.486 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-05-20 17:44:51.008237580 +0000 2024-05-20T17:50:52.486 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-05-20 17:44:51.008237580 +0000 2024-05-20T17:50:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-05-20 17:44:51.008237580 +0000 2024-05-20T17:50:52.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-20T17:50:52.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:52 smithi154 ceph-mon[36482]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi129=x;smithi154=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:52.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:52 smithi154 ceph-mon[36482]: Saving service mgr spec with placement smithi129=x;smithi154=y;smithi204=z;count:3 2024-05-20T17:50:52.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:52 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:52.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:52 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:52.550 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:52 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:52.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:52 smithi154 ceph-mon[36482]: Reconfiguring mon.c (monmap changed)... 2024-05-20T17:50:52.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:52 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-20T17:50:52.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:52 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-20T17:50:52.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:52 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:52.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:52 smithi154 ceph-mon[36482]: Reconfiguring daemon mon.c on smithi204 2024-05-20T17:50:52.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:52 smithi154 ceph-mon[36482]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:52.552 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-05-20T17:50:52.552 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-05-20T17:50:52.553 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000280356 s, 1.8 MB/s 2024-05-20T17:50:52.554 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-20T17:50:52.612 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-05-20T17:50:52.668 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-20T17:50:52.669 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:52.669 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-20T17:50:52.669 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:52.669 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:52.669 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-05-20 17:50:39.096164199 +0000 2024-05-20T17:50:52.669 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-05-20 17:44:51.333228522 +0000 2024-05-20T17:50:52.669 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-05-20 17:44:51.333228522 +0000 2024-05-20T17:50:52.669 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-05-20 17:44:51.333228522 +0000 2024-05-20T17:50:52.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-20T17:50:52.733 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-05-20T17:50:52.734 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-05-20T17:50:52.734 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000199122 s, 2.6 MB/s 2024-05-20T17:50:52.735 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-20T17:50:52.792 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-05-20T17:50:52.847 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-20T17:50:52.847 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:52.847 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-20T17:50:52.848 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:52.848 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:52.848 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-05-20 17:50:39.097164173 +0000 2024-05-20T17:50:52.848 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-05-20 17:44:51.662219352 +0000 2024-05-20T17:50:52.848 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-05-20 17:44:51.662219352 +0000 2024-05-20T17:50:52.848 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-05-20 17:44:51.662219352 +0000 2024-05-20T17:50:52.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-20T17:50:52.911 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-05-20T17:50:52.911 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-05-20T17:50:52.911 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000198782 s, 2.6 MB/s 2024-05-20T17:50:52.913 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-20T17:50:52.969 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-05-20T17:50:53.024 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-20T17:50:53.024 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:53.024 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-20T17:50:53.024 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:53.025 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:53.025 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-05-20 17:50:39.097164173 +0000 2024-05-20T17:50:53.025 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-05-20 17:44:51.999209960 +0000 2024-05-20T17:50:53.025 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-05-20 17:44:51.999209960 +0000 2024-05-20T17:50:53.025 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-05-20 17:44:51.999209960 +0000 2024-05-20T17:50:53.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-20T17:50:53.088 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-05-20T17:50:53.088 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-05-20T17:50:53.088 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000194515 s, 2.6 MB/s 2024-05-20T17:50:53.089 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-20T17:50:53.146 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-05-20T17:50:53.146 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-05-20T17:50:53.163 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-20T17:50:53.163 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-05-20T17:50:53.219 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-20T17:50:53.219 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:53.219 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-20T17:50:53.219 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:53.219 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:53.219 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-05-20 17:50:29.819435293 +0000 2024-05-20T17:50:53.219 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-05-20 17:45:03.519958701 +0000 2024-05-20T17:50:53.219 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-05-20 17:45:03.519958701 +0000 2024-05-20T17:50:53.219 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-05-20 17:45:03.519958701 +0000 2024-05-20T17:50:53.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-20T17:50:53.283 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-05-20T17:50:53.283 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-05-20T17:50:53.283 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.00027824 s, 1.8 MB/s 2024-05-20T17:50:53.285 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-20T17:50:53.341 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-05-20T17:50:53.396 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-20T17:50:53.396 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:53.396 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-20T17:50:53.396 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:53.396 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:53.396 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-05-20 17:50:29.820435281 +0000 2024-05-20T17:50:53.396 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-05-20 17:45:03.853954164 +0000 2024-05-20T17:50:53.396 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-05-20 17:45:03.853954164 +0000 2024-05-20T17:50:53.396 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-05-20 17:45:03.853954164 +0000 2024-05-20T17:50:53.397 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-20T17:50:53.458 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-05-20T17:50:53.458 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-05-20T17:50:53.458 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000241892 s, 2.1 MB/s 2024-05-20T17:50:53.460 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-20T17:50:53.514 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-05-20T17:50:53.569 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-20T17:50:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:53.570 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-20T17:50:53.570 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:53.570 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:53.570 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-05-20 17:50:29.820435281 +0000 2024-05-20T17:50:53.570 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-05-20 17:45:04.195949520 +0000 2024-05-20T17:50:53.570 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-05-20 17:45:04.195949520 +0000 2024-05-20T17:50:53.570 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-05-20 17:45:04.195949520 +0000 2024-05-20T17:50:53.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-20T17:50:53.632 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-05-20T17:50:53.632 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-05-20T17:50:53.632 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000219364 s, 2.3 MB/s 2024-05-20T17:50:53.633 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-20T17:50:53.689 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-05-20T17:50:53.743 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-20T17:50:53.744 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-20T17:50:53.744 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-20T17:50:53.744 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:50:53.744 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-05-20T17:50:53.744 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-05-20 17:50:29.820435281 +0000 2024-05-20T17:50:53.744 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-05-20 17:45:04.527945011 +0000 2024-05-20T17:50:53.744 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-05-20 17:45:04.527945011 +0000 2024-05-20T17:50:53.744 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-05-20 17:45:04.527945011 +0000 2024-05-20T17:50:53.744 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-20T17:50:53.806 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-05-20T17:50:53.806 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-05-20T17:50:53.806 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000180596 s, 2.8 MB/s 2024-05-20T17:50:53.807 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-20T17:50:53.863 INFO:tasks.cephadm:Deploying osd.0 on smithi129 with /dev/vg_nvme/lv_4... 2024-05-20T17:50:53.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-20T17:50:53.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:53.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:53.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:53.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:53.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:53.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:53.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-20T17:50:53.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-20T17:50:53.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-20T17:50:53.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:53.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: Deploying daemon mgr.z on smithi204 2024-05-20T17:50:53.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:53 smithi204 ceph-mon[37076]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:54.093 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:50:54.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:54.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:54.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:54.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:54.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:54.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:54.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-20T17:50:54.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-20T17:50:54.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-20T17:50:54.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:54.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: Deploying daemon mgr.z on smithi204 2024-05-20T17:50:54.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:53 smithi154 ceph-mon[36482]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:54.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:54.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:54.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:54.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:54.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:50:54.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:54.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-20T17:50:54.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-20T17:50:54.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-20T17:50:54.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:54.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: Deploying daemon mgr.z on smithi204 2024-05-20T17:50:54.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:53 smithi129 ceph-mon[32029]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:55.589 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:55 smithi204 systemd[1]: Starting Ceph mgr.z for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:50:55.844 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:55 smithi204 ceph-mon[37076]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:55.844 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:55 smithi204 podman[38887]: 2024-05-20 17:50:55.589199881 +0000 UTC m=+0.092077420 container create bd030ed21de2a015e08e5965ab244e7b16996c1c7dd0eb5400edb526c3ac92d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-20T17:50:55.844 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:55 smithi204 podman[38887]: 2024-05-20 17:50:55.51421941 +0000 UTC m=+0.017096975 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:50:55.845 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:55 smithi204 podman[38887]: 2024-05-20 17:50:55.808274113 +0000 UTC m=+0.311151659 container init bd030ed21de2a015e08e5965ab244e7b16996c1c7dd0eb5400edb526c3ac92d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-20T17:50:55.845 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:55 smithi204 podman[38887]: 2024-05-20 17:50:55.810940002 +0000 UTC m=+0.313817549 container start bd030ed21de2a015e08e5965ab244e7b16996c1c7dd0eb5400edb526c3ac92d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True) 2024-05-20T17:50:55.854 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-20T17:50:55.855 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-20T17:50:55.855 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-05-20T17:50:55.855 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-05-20T17:50:55.855 INFO:teuthology.orchestra.run.smithi129.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114748 s, 914 MB/s 2024-05-20T17:50:55.855 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 2024-05-20T17:50:55.855 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-05-20T17:50:56.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:55 smithi154 ceph-mon[36482]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:56.159 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:55 smithi204 ceph-mgr[38906]: -- 172.21.15.204:0/2893315909 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c3e413e5a0 con 0x55c3e427c900 2024-05-20T17:50:56.159 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:55 smithi204 bash[38887]: bd030ed21de2a015e08e5965ab244e7b16996c1c7dd0eb5400edb526c3ac92d7 2024-05-20T17:50:56.159 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:55 smithi204 systemd[1]: Started Ceph mgr.z for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:50:56.159 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:55 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:55.974+0000 7f2f1ce0e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-20T17:50:56.159 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:56 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:56.159+0000 7f2f1ce0e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:50:56.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:55 smithi129 ceph-mon[32029]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:56.431 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:56 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:56.262+0000 7f2f1ce0e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-20T17:50:56.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi129:vg_nvme/lv_4 2024-05-20T17:50:56.712 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:50:56.931 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:56 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:56.567+0000 7f2f1ce0e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-20T17:50:56.931 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:56 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:56.652+0000 7f2f1ce0e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-20T17:50:57.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:57.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:57.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:57.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-20T17:50:57.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-20T17:50:57.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-20T17:50:57.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:57.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:56 smithi129 ceph-mon[32029]: Deploying daemon mgr.y on smithi154 2024-05-20T17:50:57.340 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:57.340 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:57.340 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:57.341 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-20T17:50:57.341 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-20T17:50:57.341 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-20T17:50:57.341 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:57.341 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:56 smithi154 ceph-mon[36482]: Deploying daemon mgr.y on smithi154 2024-05-20T17:50:57.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:57.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:57.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:57.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-20T17:50:57.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-20T17:50:57.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-20T17:50:57.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:57.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:56 smithi204 ceph-mon[37076]: Deploying daemon mgr.y on smithi154 2024-05-20T17:50:57.895 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:57 smithi154 systemd[1]: Starting Ceph mgr.y for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:50:58.073 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:57 smithi129 ceph-mon[32029]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:58.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:57 smithi204 ceph-mon[37076]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:58.181 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:57 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:57.885+0000 7f2f1ce0e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-20T17:50:58.181 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:58 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:58.059+0000 7f2f1ce0e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-20T17:50:58.219 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:57 smithi154 ceph-mon[36482]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:50:58.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:58 smithi154 podman[37838]: 2024-05-20 17:50:58.001373596 +0000 UTC m=+0.103655743 container create f3b4f8606f46b99edb408c628588263e44f0446995edc78e3899700acd42ba62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-20T17:50:58.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:58 smithi154 podman[37838]: 2024-05-20 17:50:57.915006959 +0000 UTC m=+0.017289113 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:50:58.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:58 smithi154 podman[37838]: 2024-05-20 17:50:58.220301404 +0000 UTC m=+0.322583554 container init f3b4f8606f46b99edb408c628588263e44f0446995edc78e3899700acd42ba62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-20T17:50:58.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.322+0000 7f637127e700 1 -- 172.21.15.129:0/598353301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f636c06bb80 msgr2=0x7f636c105d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:58.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.322+0000 7f637127e700 1 --2- 172.21.15.129:0/598353301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f636c06bb80 0x7f636c105d20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f635c009990 tx=0x7f635c009ca0 comp rx=0 tx=0).stop 2024-05-20T17:50:58.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.322+0000 7f637127e700 1 -- 172.21.15.129:0/598353301 shutdown_connections 2024-05-20T17:50:58.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.322+0000 7f637127e700 1 --2- 172.21.15.129:0/598353301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f636c108e60 0x7f636c10b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:58.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.322+0000 7f637127e700 1 --2- 172.21.15.129:0/598353301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f636c06bb80 0x7f636c105d20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:58.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.322+0000 7f637127e700 1 --2- 172.21.15.129:0/598353301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f636c1062f0 0x7f636c1087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:58.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.322+0000 7f637127e700 1 -- 172.21.15.129:0/598353301 >> 172.21.15.129:0/598353301 conn(0x7f636c0fd6f0 msgr2=0x7f636c0ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:50:58.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.322+0000 7f637127e700 1 -- 172.21.15.129:0/598353301 shutdown_connections 2024-05-20T17:50:58.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.322+0000 7f637127e700 1 -- 172.21.15.129:0/598353301 wait complete. 2024-05-20T17:50:58.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.322+0000 7f637127e700 1 Processor -- start 2024-05-20T17:50:58.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f637127e700 1 -- start start 2024-05-20T17:50:58.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f637127e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f636c06bb80 0x7f636c199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:58.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f637127e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f636c1062f0 0x7f636c199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:58.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f637127e700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f636c108e60 0x7f636c1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:58.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f637127e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f636c10ed50 con 0x7f636c1062f0 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f637127e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f636c10ebd0 con 0x7f636c06bb80 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f637127e700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f636c10eed0 con 0x7f636c108e60 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f636b7fe700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f636c108e60 0x7f636c1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f636b7fe700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f636c108e60 0x7f636c1a0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:42882/0 (socket says 172.21.15.129:42882) 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f636b7fe700 1 -- 172.21.15.129:0/2073819610 learned_addr learned my addr 172.21.15.129:0/2073819610 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f636affd700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f636c06bb80 0x7f636c199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f636a7fc700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f636c1062f0 0x7f636c199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f636b7fe700 1 -- 172.21.15.129:0/2073819610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f636c06bb80 msgr2=0x7f636c199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f636b7fe700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f636c06bb80 0x7f636c199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f636b7fe700 1 -- 172.21.15.129:0/2073819610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f636c1062f0 msgr2=0x7f636c199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f636b7fe700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f636c1062f0 0x7f636c199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:50:58.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.323+0000 7f636b7fe700 1 -- 172.21.15.129:0/2073819610 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636c1a1380 con 0x7f636c108e60 2024-05-20T17:50:58.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.324+0000 7f636b7fe700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f636c108e60 0x7f636c1a0d00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f636400ef30 tx=0x7f636400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:58.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.324+0000 7f6353fff700 1 -- 172.21.15.129:0/2073819610 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63640092e0 con 0x7f636c108e60 2024-05-20T17:50:58.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.324+0000 7f6353fff700 1 -- 172.21.15.129:0/2073819610 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6364004bb0 con 0x7f636c108e60 2024-05-20T17:50:58.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.324+0000 7f6353fff700 1 -- 172.21.15.129:0/2073819610 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6364004280 con 0x7f636c108e60 2024-05-20T17:50:58.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.324+0000 7f637127e700 1 -- 172.21.15.129:0/2073819610 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f636c1a1610 con 0x7f636c108e60 2024-05-20T17:50:58.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.324+0000 7f637127e700 1 -- 172.21.15.129:0/2073819610 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f636c1a1b00 con 0x7f636c108e60 2024-05-20T17:50:58.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.325+0000 7f637127e700 1 -- 172.21.15.129:0/2073819610 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f636c050170 con 0x7f636c108e60 2024-05-20T17:50:58.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.325+0000 7f6353fff700 1 -- 172.21.15.129:0/2073819610 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f636401b030 con 0x7f636c108e60 2024-05-20T17:50:58.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.326+0000 7f6353fff700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6354039330 0x7f635403b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:50:58.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.326+0000 7f6353fff700 1 -- 172.21.15.129:0/2073819610 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f636404be40 con 0x7f636c108e60 2024-05-20T17:50:58.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.326+0000 7f636affd700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6354039330 0x7f635403b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:50:58.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.326+0000 7f636affd700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6354039330 0x7f635403b840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f635c00b2e0 tx=0x7f635c00b880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:50:58.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.336+0000 7f6353fff700 1 -- 172.21.15.129:0/2073819610 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6364022020 con 0x7f636c108e60 2024-05-20T17:50:58.529 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:58 smithi154 podman[37838]: 2024-05-20 17:50:58.223502425 +0000 UTC m=+0.325784576 container start f3b4f8606f46b99edb408c628588263e44f0446995edc78e3899700acd42ba62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514) 2024-05-20T17:50:58.530 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:58 smithi154 ceph-mgr[37858]: -- 172.21.15.154:0/3603816576 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aef89f05a0 con 0x55aef8b2e900 2024-05-20T17:50:58.530 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:58 smithi154 bash[37838]: f3b4f8606f46b99edb408c628588263e44f0446995edc78e3899700acd42ba62 2024-05-20T17:50:58.530 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:58 smithi154 systemd[1]: Started Ceph mgr.y for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:50:58.530 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:50:58.400+0000 7f4aedd48200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-20T17:50:58.531 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:58 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:58.235+0000 7f2f1ce0e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-20T17:50:58.531 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:58 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:58.530+0000 7f2f1ce0e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-20T17:50:58.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:50:58.549+0000 7f637127e700 1 -- 172.21.15.129:0/2073819610 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f636c062a80 con 0x7f6354039330 2024-05-20T17:50:58.837 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:58 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:58.626+0000 7f2f1ce0e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-20T17:50:58.838 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:58 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:58.837+0000 7f2f1ce0e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-20T17:50:58.894 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:50:58.597+0000 7f4aedd48200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:50:58.894 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:50:58.709+0000 7f4aedd48200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-20T17:50:59.181 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:59 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:59.035+0000 7f2f1ce0e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:50:59.351 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:59 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:50:59.039+0000 7f4aedd48200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-20T17:50:59.351 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:50:59 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:50:59.147+0000 7f4aedd48200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-20T17:50:59.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:59 smithi129 ceph-mon[32029]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:59.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:50:59.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:50:59.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:59.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:50:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:59.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:59 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:59 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:59 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:59 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:59 smithi154 ceph-mon[36482]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:59.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:59 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:50:59.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:59 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:50:59.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:59 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:59.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:50:59 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:50:59.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:59 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:59 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:59 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:59 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:50:59.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:59 smithi204 ceph-mon[37076]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:50:59.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:59 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:50:59.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:59 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:50:59.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:59 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:50:59.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:50:59 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:00.123 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:50:59 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:50:59.687+0000 7f2f1ce0e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-20T17:51:00.378 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:00 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:00.122+0000 7f2f1ce0e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-20T17:51:00.378 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:00 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:00.220+0000 7f2f1ce0e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-20T17:51:00.645 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:00 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:00.397+0000 7f4aedd48200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-20T17:51:00.645 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:00 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:00.567+0000 7f4aedd48200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-20T17:51:00.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:00 smithi154 ceph-mon[36482]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:00.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.646 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.646 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.646 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:00.646 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:00.646 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.646 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-20T17:51:00.646 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-20T17:51:00.646 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:00.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:00 smithi129 ceph-mon[32029]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:00.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:00.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:00.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-20T17:51:00.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-20T17:51:00.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:00.681 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:00 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:00.422+0000 7f2f1ce0e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-20T17:51:00.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:00 smithi204 ceph-mon[37076]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:00.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:00.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:00.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:00.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-20T17:51:00.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-20T17:51:00.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:01.026 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:00 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:00.738+0000 7f4aedd48200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-20T17:51:01.026 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:01 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:01.025+0000 7f4aedd48200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-20T17:51:01.290 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:00 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:00.952+0000 7f2f1ce0e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-20T17:51:01.290 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:01.046+0000 7f2f1ce0e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-20T17:51:01.321 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:01 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:01.117+0000 7f4aedd48200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-20T17:51:01.321 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:01 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:01.321+0000 7f4aedd48200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-20T17:51:01.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:01 smithi204 ceph-mon[37076]: Reconfiguring mgr.x (unknown last config time)... 2024-05-20T17:51:01.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:01 smithi204 ceph-mon[37076]: Reconfiguring daemon mgr.x on smithi129 2024-05-20T17:51:01.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:01 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3987834527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb"}]: dispatch 2024-05-20T17:51:01.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:01 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3987834527' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb"}]': finished 2024-05-20T17:51:01.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:01 smithi204 ceph-mon[37076]: osdmap e5: 1 total, 0 up, 1 in 2024-05-20T17:51:01.633 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:01 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:01.634 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:01.289+0000 7f2f1ce0e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-20T17:51:01.634 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:01.423+0000 7f2f1ce0e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-20T17:51:01.634 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:01.517+0000 7f2f1ce0e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-20T17:51:01.634 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:01.632+0000 7f2f1ce0e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-20T17:51:01.644 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:01 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:01.514+0000 7f4aedd48200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:51:01.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:01 smithi154 ceph-mon[36482]: Reconfiguring mgr.x (unknown last config time)... 2024-05-20T17:51:01.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:01 smithi154 ceph-mon[36482]: Reconfiguring daemon mgr.x on smithi129 2024-05-20T17:51:01.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:01 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3987834527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb"}]: dispatch 2024-05-20T17:51:01.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:01 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3987834527' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb"}]': finished 2024-05-20T17:51:01.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:01 smithi154 ceph-mon[36482]: osdmap e5: 1 total, 0 up, 1 in 2024-05-20T17:51:01.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:01 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:01.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:01 smithi129 ceph-mon[32029]: Reconfiguring mgr.x (unknown last config time)... 2024-05-20T17:51:01.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:01 smithi129 ceph-mon[32029]: Reconfiguring daemon mgr.x on smithi129 2024-05-20T17:51:01.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:01 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3987834527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb"}]: dispatch 2024-05-20T17:51:01.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:01 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3987834527' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb"}]': finished 2024-05-20T17:51:01.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:01 smithi129 ceph-mon[32029]: osdmap e5: 1 total, 0 up, 1 in 2024-05-20T17:51:01.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:01 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:02.383 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:01.934+0000 7f2f1ce0e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-20T17:51:02.384 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:02 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:02.120+0000 7f4aedd48200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-20T17:51:02.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:02 smithi154 ceph-mon[36482]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:02.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:02 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3065231646' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:51:02.645 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:02 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:02.537+0000 7f4aedd48200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-20T17:51:02.645 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:02 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:02.632+0000 7f4aedd48200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-20T17:51:02.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:02 smithi129 ceph-mon[32029]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:02.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:02 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3065231646' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:51:02.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:02 smithi204 ceph-mon[37076]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:02.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:02 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3065231646' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:51:02.681 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:02 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:02.531+0000 7f2f1ce0e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-20T17:51:03.144 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:02 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:02.833+0000 7f4aedd48200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-20T17:51:03.181 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:51:02 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:51:02.749+0000 7f2f1ce0e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-20T17:51:03.644 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:03.361+0000 7f4aedd48200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-20T17:51:03.645 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:03.452+0000 7f4aedd48200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-20T17:51:03.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:03.682+0000 7f6353fff700 1 -- 172.21.15.129:0/2073819610 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mgrmap(e 14) ==== 92061+0+0 (secure 0 0 0) 0x7f6364012050 con 0x7f636c108e60 2024-05-20T17:51:03.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:03 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:03.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:03 smithi204 ceph-mon[37076]: Standby manager daemon z started 2024-05-20T17:51:03.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:03 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-20T17:51:03.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:03 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:51:03.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:03 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-20T17:51:03.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:03 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:51:03.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:03 smithi204 ceph-mon[37076]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:04.036 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:03.694+0000 7f4aedd48200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-20T17:51:04.036 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:03.826+0000 7f4aedd48200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-20T17:51:04.036 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:03.920+0000 7f4aedd48200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-20T17:51:04.036 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:04 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:04.036+0000 7f4aedd48200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-20T17:51:04.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:03 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:04.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:03 smithi154 ceph-mon[36482]: Standby manager daemon z started 2024-05-20T17:51:04.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:03 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-20T17:51:04.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:03 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:51:04.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:03 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-20T17:51:04.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:03 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:51:04.037 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:03 smithi154 ceph-mon[36482]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:04.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:03 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:04.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:03 smithi129 ceph-mon[32029]: Standby manager daemon z started 2024-05-20T17:51:04.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:03 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-20T17:51:04.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:03 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:51:04.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:03 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-20T17:51:04.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:03 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.204:0/639693740' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:51:04.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:03 smithi129 ceph-mon[32029]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:04.644 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:04 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:04.337+0000 7f4aedd48200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-20T17:51:04.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:04 smithi204 ceph-mon[37076]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-20T17:51:04.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:04 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-20T17:51:04.934 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:04 smithi154 ceph-mon[36482]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-20T17:51:04.934 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:04 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-20T17:51:04.953 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:04 smithi129 ceph-mon[32029]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-20T17:51:04.953 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:04 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-20T17:51:05.394 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:04 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:04.934+0000 7f4aedd48200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-20T17:51:05.395 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:51:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:51:05.148+0000 7f4aedd48200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-20T17:51:05.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:05.691+0000 7f6353fff700 1 -- 172.21.15.129:0/2073819610 <== mon.1 v2:172.21.15.204:3300/0 8 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f6364007a20 con 0x7f636c108e60 2024-05-20T17:51:05.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:05 smithi129 ceph-mon[32029]: Standby manager daemon y started 2024-05-20T17:51:05.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:05 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-20T17:51:05.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:05 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:51:05.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:05 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-20T17:51:05.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:05 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:51:05.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:05 smithi129 ceph-mon[32029]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:05.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:05 smithi204 ceph-mon[37076]: Standby manager daemon y started 2024-05-20T17:51:05.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:05 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-20T17:51:05.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:05 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:51:05.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:05 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-20T17:51:05.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:05 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:51:05.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:05 smithi204 ceph-mon[37076]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:06.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:05 smithi154 ceph-mon[36482]: Standby manager daemon y started 2024-05-20T17:51:06.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:05 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-20T17:51:06.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:05 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:51:06.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:05 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-20T17:51:06.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:05 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.154:0/2089539350' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:51:06.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:05 smithi154 ceph-mon[36482]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:06.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:06 smithi129 ceph-mon[32029]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-05-20T17:51:06.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-20T17:51:06.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:06.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:06.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:06.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:06.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:06.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:07.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:06 smithi154 ceph-mon[36482]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-05-20T17:51:07.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-20T17:51:07.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:07.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:07.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:07.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:07.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:07.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:07.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:06 smithi204 ceph-mon[37076]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-05-20T17:51:07.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-20T17:51:07.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:07.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:07.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:07.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:07.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:07.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:07.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:07 smithi129 ceph-mon[32029]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:08.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:07 smithi154 ceph-mon[36482]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:08.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:07 smithi204 ceph-mon[37076]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:09.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-20T17:51:09.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:09.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-20T17:51:09.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:09.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-20T17:51:09.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:10.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:09 smithi154 ceph-mon[36482]: Deploying daemon osd.0 on smithi129 2024-05-20T17:51:10.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:09 smithi154 ceph-mon[36482]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:10.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:09 smithi129 ceph-mon[32029]: Deploying daemon osd.0 on smithi129 2024-05-20T17:51:10.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:09 smithi129 ceph-mon[32029]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:10.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:09 smithi204 ceph-mon[37076]: Deploying daemon osd.0 on smithi129 2024-05-20T17:51:10.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:09 smithi204 ceph-mon[37076]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:12.130 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:11 smithi129 ceph-mon[32029]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:12.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:11 smithi154 ceph-mon[36482]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:12.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:11 smithi204 ceph-mon[37076]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:13.889 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:13 smithi129 ceph-mon[32029]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:14.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:13 smithi154 ceph-mon[36482]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:14.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:13 smithi204 ceph-mon[37076]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:15.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:14 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:15.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:14 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:15.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:14 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:15.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:14 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:15.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:14 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:15.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:14 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:14 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:14 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:14 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:16.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:15 smithi129 ceph-mon[32029]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:16.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:15 smithi154 ceph-mon[36482]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:16.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:15 smithi204 ceph-mon[37076]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:17.573 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:17 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:17.573 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:17 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:17.573 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:17 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:17.573 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:17 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:17.573 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:17 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:17.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.655+0000 7f6353fff700 1 -- 172.21.15.129:0/2073819610 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f636c062a80 con 0x7f6354039330 2024-05-20T17:51:17.656 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 0 on host 'smithi129' 2024-05-20T17:51:17.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 -- 172.21.15.129:0/2073819610 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6354039330 msgr2=0x7f635403b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:51:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6354039330 0x7f635403b840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f635c00b2e0 tx=0x7f635c00b880 comp rx=0 tx=0).stop 2024-05-20T17:51:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 -- 172.21.15.129:0/2073819610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f636c108e60 msgr2=0x7f636c1a0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:51:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f636c108e60 0x7f636c1a0d00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f636400ef30 tx=0x7f636400c3a0 comp rx=0 tx=0).stop 2024-05-20T17:51:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 -- 172.21.15.129:0/2073819610 shutdown_connections 2024-05-20T17:51:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6354039330 0x7f635403b840 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f636c06bb80 0x7f636c199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f636c108e60 0x7f636c1a0d00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 --2- 172.21.15.129:0/2073819610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f636c1062f0 0x7f636c199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 -- 172.21.15.129:0/2073819610 >> 172.21.15.129:0/2073819610 conn(0x7f636c0fd6f0 msgr2=0x7f636c1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:51:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 -- 172.21.15.129:0/2073819610 shutdown_connections 2024-05-20T17:51:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:51:17.658+0000 7f637127e700 1 -- 172.21.15.129:0/2073819610 wait complete. 2024-05-20T17:51:17.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:17 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:17.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:17 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:17.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:17 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:17.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:17 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:17.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:17 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:17.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:17 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:17.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:17 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:17.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:17 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:17.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:17 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:17.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:17 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:18.320 DEBUG:teuthology.orchestra.run.smithi129:osd.0> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.0.service 2024-05-20T17:51:18.322 INFO:tasks.cephadm:Deploying osd.1 on smithi154 with /dev/vg_nvme/lv_4... 2024-05-20T17:51:18.323 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-20T17:51:18.565 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T17:51:18.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:18 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:18.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:18 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:18.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:18 smithi154 ceph-mon[36482]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:18.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:18 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:18.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:18 smithi154 ceph-mon[36482]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-20T17:51:18.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:18.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:18.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:18 smithi129 ceph-mon[32029]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:18.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:18.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:18 smithi129 ceph-mon[32029]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-20T17:51:18.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:18 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:18.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:18 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:18.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:18 smithi204 ceph-mon[37076]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:18.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:18 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:18.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:18 smithi204 ceph-mon[37076]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-20T17:51:19.746 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:19 smithi154 ceph-mon[36482]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-20T17:51:19.746 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:19 smithi154 ceph-mon[36482]: osdmap e6: 1 total, 0 up, 1 in 2024-05-20T17:51:19.746 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:19 smithi154 ceph-mon[36482]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:51:19.747 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:19 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:19.766 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:19 smithi129 ceph-mon[32029]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-20T17:51:19.766 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:19 smithi129 ceph-mon[32029]: osdmap e6: 1 total, 0 up, 1 in 2024-05-20T17:51:19.766 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:19 smithi129 ceph-mon[32029]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:51:19.766 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:19 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:19.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:19 smithi204 ceph-mon[37076]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-20T17:51:19.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:19 smithi204 ceph-mon[37076]: osdmap e6: 1 total, 0 up, 1 in 2024-05-20T17:51:19.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:19 smithi204 ceph-mon[37076]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:51:19.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:19 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:20.318 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-20T17:51:20.318 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-20T17:51:20.318 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-05-20T17:51:20.318 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-05-20T17:51:20.318 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112182 s, 935 MB/s 2024-05-20T17:51:20.318 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-05-20T17:51:20.414 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 17:51:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0[44123]: 2024-05-20T17:51:20.097+0000 7f4258e04700 -1 osd.0 0 waiting for initial osdmap 2024-05-20T17:51:20.414 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 17:51:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0[44123]: 2024-05-20T17:51:20.102+0000 7f4250bb1700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:51:20.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:20 smithi154 ceph-mon[36482]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:51:20.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:20 smithi154 ceph-mon[36482]: osdmap e7: 1 total, 0 up, 1 in 2024-05-20T17:51:20.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:20 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:20.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:20 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:20.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:20 smithi154 ceph-mon[36482]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:20.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:20 smithi154 ceph-mon[36482]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' 2024-05-20T17:51:20.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:20 smithi129 ceph-mon[32029]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:51:20.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:20 smithi129 ceph-mon[32029]: osdmap e7: 1 total, 0 up, 1 in 2024-05-20T17:51:20.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:20 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:20.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:20 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:20.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:20 smithi129 ceph-mon[32029]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:20.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:20 smithi129 ceph-mon[32029]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' 2024-05-20T17:51:20.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:20 smithi204 ceph-mon[37076]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:51:20.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:20 smithi204 ceph-mon[37076]: osdmap e7: 1 total, 0 up, 1 in 2024-05-20T17:51:20.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:20 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:20.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:20 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:20.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:20 smithi204 ceph-mon[37076]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-20T17:51:20.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:20 smithi204 ceph-mon[37076]: from='osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746]' entity='osd.0' 2024-05-20T17:51:20.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi154:vg_nvme/lv_4 2024-05-20T17:51:21.193 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T17:51:21.850 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:51:21.850 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:51:21.850 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:21.850 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746] boot 2024-05-20T17:51:21.850 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: osdmap e8: 1 total, 1 up, 1 in 2024-05-20T17:51:21.850 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:21.850 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:21.850 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:21.850 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:51:21.851 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:21.851 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:21.851 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:21.851 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:21.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:51:21.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:51:21.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:21.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746] boot 2024-05-20T17:51:21.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: osdmap e8: 1 total, 1 up, 1 in 2024-05-20T17:51:21.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:21.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:21.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:21.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:51:21.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:21.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:21.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:21.915 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:21.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:51:21.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:51:21.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:21.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: osd.0 [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746] boot 2024-05-20T17:51:21.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: osdmap e8: 1 total, 1 up, 1 in 2024-05-20T17:51:21.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:51:21.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:21.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:21.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:51:21.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:21.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:21.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:21.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:22.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.559+0000 7f95a19f9700 1 -- 172.21.15.154:0/2431606535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f959c1031f0 msgr2=0x7f959c103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:51:22.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.559+0000 7f95a19f9700 1 --2- 172.21.15.154:0/2431606535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f959c1031f0 0x7f959c103600 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f958c009990 tx=0x7f958c009ca0 comp rx=0 tx=0).stop 2024-05-20T17:51:22.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.559+0000 7f95a19f9700 1 -- 172.21.15.154:0/2431606535 shutdown_connections 2024-05-20T17:51:22.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.559+0000 7f95a19f9700 1 --2- 172.21.15.154:0/2431606535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f959c1031f0 0x7f959c103600 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:22.560 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.559+0000 7f95a19f9700 1 --2- 172.21.15.154:0/2431606535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f959c104cc0 0x7f959c10b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:22.560 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.559+0000 7f95a19f9700 1 --2- 172.21.15.154:0/2431606535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f959c104320 0x7f959c104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:22.560 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.559+0000 7f95a19f9700 1 -- 172.21.15.154:0/2431606535 >> 172.21.15.154:0/2431606535 conn(0x7f959c0fe850 msgr2=0x7f959c100ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:51:22.560 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.559+0000 7f95a19f9700 1 -- 172.21.15.154:0/2431606535 shutdown_connections 2024-05-20T17:51:22.560 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.559+0000 7f95a19f9700 1 -- 172.21.15.154:0/2431606535 wait complete. 2024-05-20T17:51:22.560 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.559+0000 7f95a19f9700 1 Processor -- start 2024-05-20T17:51:22.560 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.559+0000 7f95a19f9700 1 -- start start 2024-05-20T17:51:22.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f95a19f9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f959c1031f0 0x7f959c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:51:22.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f95a19f9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f959c104320 0x7f959c199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:51:22.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f95a19f9700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f959c104cc0 0x7f959c1a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:51:22.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f95a19f9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f959c10ee50 con 0x7f959c104320 2024-05-20T17:51:22.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f95a19f9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f959c10ecd0 con 0x7f959c1031f0 2024-05-20T17:51:22.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f95a19f9700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f959c10efd0 con 0x7f959c104cc0 2024-05-20T17:51:22.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959affd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f959c1031f0 0x7f959c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:51:22.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959affd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f959c1031f0 0x7f959c199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:46646/0 (socket says 172.21.15.154:46646) 2024-05-20T17:51:22.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959affd700 1 -- 172.21.15.154:0/2997793767 learned_addr learned my addr 172.21.15.154:0/2997793767 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-05-20T17:51:22.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959a7fc700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f959c104320 0x7f959c199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:51:22.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959b7fe700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f959c104cc0 0x7f959c1a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:51:22.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959affd700 1 -- 172.21.15.154:0/2997793767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f959c104cc0 msgr2=0x7f959c1a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:51:22.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959affd700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f959c104cc0 0x7f959c1a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:22.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959affd700 1 -- 172.21.15.154:0/2997793767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f959c104320 msgr2=0x7f959c199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:51:22.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959affd700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f959c104320 0x7f959c199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:22.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959affd700 1 -- 172.21.15.154:0/2997793767 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959c075980 con 0x7f959c1031f0 2024-05-20T17:51:22.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959a7fc700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f959c104320 0x7f959c199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:51:22.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959b7fe700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f959c104cc0 0x7f959c1a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:51:22.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.560+0000 7f959affd700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f959c1031f0 0x7f959c199930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f958c000c00 tx=0x7f958c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:51:22.564 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.561+0000 7f95a09f7700 1 -- 172.21.15.154:0/2997793767 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c027070 con 0x7f959c1031f0 2024-05-20T17:51:22.564 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.561+0000 7f95a09f7700 1 -- 172.21.15.154:0/2997793767 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f958c022470 con 0x7f959c1031f0 2024-05-20T17:51:22.564 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.561+0000 7f95a19f9700 1 -- 172.21.15.154:0/2997793767 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959c075c10 con 0x7f959c1031f0 2024-05-20T17:51:22.564 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.561+0000 7f95a09f7700 1 -- 172.21.15.154:0/2997793767 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c00f6c0 con 0x7f959c1031f0 2024-05-20T17:51:22.564 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.561+0000 7f95a19f9700 1 -- 172.21.15.154:0/2997793767 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f959c076170 con 0x7f959c1031f0 2024-05-20T17:51:22.564 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.562+0000 7f95a19f9700 1 -- 172.21.15.154:0/2997793767 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f959c050170 con 0x7f959c1031f0 2024-05-20T17:51:22.565 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.563+0000 7f95a09f7700 1 -- 172.21.15.154:0/2997793767 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f958c02a070 con 0x7f959c1031f0 2024-05-20T17:51:22.565 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.564+0000 7f95a09f7700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f95840a2b90 0x7f95840a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:51:22.565 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.564+0000 7f95a09f7700 1 -- 172.21.15.154:0/2997793767 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7f958c0cef40 con 0x7f959c1031f0 2024-05-20T17:51:22.565 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.564+0000 7f959a7fc700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f95840a2b90 0x7f95840a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:51:22.565 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.564+0000 7f959a7fc700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f95840a2b90 0x7f95840a50a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9590005fd0 tx=0x7f9590005e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:51:22.569 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.568+0000 7f95a09f7700 1 -- 172.21.15.154:0/2997793767 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f958c031070 con 0x7f959c1031f0 2024-05-20T17:51:22.774 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:22.774+0000 7f95a19f9700 1 -- 172.21.15.154:0/2997793767 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f959c062a80 con 0x7f95840a2b90 2024-05-20T17:51:22.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:22 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi129 2024-05-20T17:51:22.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:22 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi129 to 17088M 2024-05-20T17:51:22.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:22 smithi154 ceph-mon[36482]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:22.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:22 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi129 2024-05-20T17:51:22.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:22 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi129 to 17088M 2024-05-20T17:51:22.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:22 smithi129 ceph-mon[32029]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:22.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:22 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi129 2024-05-20T17:51:22.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:22 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi129 to 17088M 2024-05-20T17:51:22.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:22 smithi204 ceph-mon[37076]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:23.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:23 smithi154 ceph-mon[36482]: osdmap e9: 1 total, 1 up, 1 in 2024-05-20T17:51:23.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:23 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:51:23.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:23 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:51:23.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:23 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:23.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:23 smithi129 ceph-mon[32029]: osdmap e9: 1 total, 1 up, 1 in 2024-05-20T17:51:23.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:23 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:51:23.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:23 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:51:23.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:23 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:23.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:23 smithi204 ceph-mon[37076]: osdmap e9: 1 total, 1 up, 1 in 2024-05-20T17:51:23.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:23 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:51:23.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:23 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:51:23.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:23 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:24.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:24 smithi154 ceph-mon[36482]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:51:24.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:24 smithi154 ceph-mon[36482]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:24.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:24 smithi129 ceph-mon[32029]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:51:24.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:24 smithi129 ceph-mon[32029]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:24.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:24 smithi204 ceph-mon[37076]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:51:24.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:24 smithi204 ceph-mon[37076]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:25.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:25 smithi154 ceph-mon[36482]: from='client.? 172.21.15.154:0/2720775527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b41acf7-4a61-4152-9ec6-a0c980cb29c3"}]: dispatch 2024-05-20T17:51:25.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:25 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b41acf7-4a61-4152-9ec6-a0c980cb29c3"}]: dispatch 2024-05-20T17:51:25.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:25 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b41acf7-4a61-4152-9ec6-a0c980cb29c3"}]': finished 2024-05-20T17:51:25.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:25 smithi154 ceph-mon[36482]: osdmap e10: 2 total, 1 up, 2 in 2024-05-20T17:51:25.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:25 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:25.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:25 smithi129 ceph-mon[32029]: from='client.? 172.21.15.154:0/2720775527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b41acf7-4a61-4152-9ec6-a0c980cb29c3"}]: dispatch 2024-05-20T17:51:25.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:25 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b41acf7-4a61-4152-9ec6-a0c980cb29c3"}]: dispatch 2024-05-20T17:51:25.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:25 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b41acf7-4a61-4152-9ec6-a0c980cb29c3"}]': finished 2024-05-20T17:51:25.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:25 smithi129 ceph-mon[32029]: osdmap e10: 2 total, 1 up, 2 in 2024-05-20T17:51:25.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:25 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:25.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:25 smithi204 ceph-mon[37076]: from='client.? 172.21.15.154:0/2720775527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b41acf7-4a61-4152-9ec6-a0c980cb29c3"}]: dispatch 2024-05-20T17:51:25.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:25 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b41acf7-4a61-4152-9ec6-a0c980cb29c3"}]: dispatch 2024-05-20T17:51:25.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:25 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b41acf7-4a61-4152-9ec6-a0c980cb29c3"}]': finished 2024-05-20T17:51:25.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:25 smithi204 ceph-mon[37076]: osdmap e10: 2 total, 1 up, 2 in 2024-05-20T17:51:25.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:25 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:26.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:26 smithi129 ceph-mon[32029]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:26.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:26 smithi129 ceph-mon[32029]: from='client.? 172.21.15.154:0/665121414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:51:26.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:26 smithi204 ceph-mon[37076]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:26.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:26 smithi204 ceph-mon[37076]: from='client.? 172.21.15.154:0/665121414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:51:27.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:26 smithi154 ceph-mon[36482]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:27.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:26 smithi154 ceph-mon[36482]: from='client.? 172.21.15.154:0/665121414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:51:28.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:28 smithi129 ceph-mon[32029]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:28.924 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:28 smithi154 ceph-mon[36482]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:28 smithi204 ceph-mon[37076]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:30.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:30 smithi129 ceph-mon[32029]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:30.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:30 smithi204 ceph-mon[37076]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:30.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:30 smithi154 ceph-mon[36482]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:32.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:32 smithi129 ceph-mon[32029]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:32.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-20T17:51:32.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:32.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:32 smithi204 ceph-mon[37076]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:32.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-20T17:51:32.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:32.967 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:32 smithi154 ceph-mon[36482]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:32.967 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-20T17:51:32.967 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:33.882 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:33 smithi154 ceph-mon[36482]: Deploying daemon osd.1 on smithi154 2024-05-20T17:51:33.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:33 smithi204 ceph-mon[37076]: Deploying daemon osd.1 on smithi154 2024-05-20T17:51:34.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:33 smithi129 ceph-mon[32029]: Deploying daemon osd.1 on smithi154 2024-05-20T17:51:34.926 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:34 smithi154 ceph-mon[36482]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:34.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:34 smithi204 ceph-mon[37076]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:35.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:34 smithi129 ceph-mon[32029]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:36.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:36 smithi154 ceph-mon[36482]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:36.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:36 smithi204 ceph-mon[37076]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:37.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:36 smithi129 ceph-mon[32029]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:38.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:38 smithi154 ceph-mon[36482]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:38.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:38.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:38.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:38.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:38 smithi129 ceph-mon[32029]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:38.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:38.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:38.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:38.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:38 smithi204 ceph-mon[37076]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:40.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:40 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:40.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:40 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:40.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:40 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:40.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:40 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:40.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:40 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:40.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:40 smithi154 ceph-mon[36482]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:40.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:40 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:40.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:40 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:40.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:40 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:40.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:40 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:40.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:40 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:40.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:40 smithi129 ceph-mon[32029]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:40.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:40 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:40.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:40 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:40.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:40 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:40.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:40 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:40.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:40 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:40.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:40 smithi204 ceph-mon[37076]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:41.086 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 1 on host 'smithi154' 2024-05-20T17:51:41.087 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.086+0000 7f95a09f7700 1 -- 172.21.15.154:0/2997793767 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f959c062a80 con 0x7f95840a2b90 2024-05-20T17:51:41.088 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.088+0000 7f95a19f9700 1 -- 172.21.15.154:0/2997793767 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f95840a2b90 msgr2=0x7f95840a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:51:41.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.088+0000 7f95a19f9700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f95840a2b90 0x7f95840a50a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9590005fd0 tx=0x7f9590005e40 comp rx=0 tx=0).stop 2024-05-20T17:51:41.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.088+0000 7f95a19f9700 1 -- 172.21.15.154:0/2997793767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f959c1031f0 msgr2=0x7f959c199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:51:41.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.088+0000 7f95a19f9700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f959c1031f0 0x7f959c199930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f958c000c00 tx=0x7f958c009f90 comp rx=0 tx=0).stop 2024-05-20T17:51:41.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.088+0000 7f95a19f9700 1 -- 172.21.15.154:0/2997793767 shutdown_connections 2024-05-20T17:51:41.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.088+0000 7f95a19f9700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f95840a2b90 0x7f95840a50a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:41.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.088+0000 7f95a19f9700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f959c1031f0 0x7f959c199930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:41.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.088+0000 7f95a19f9700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f959c104cc0 0x7f959c1a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:41.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.088+0000 7f95a19f9700 1 --2- 172.21.15.154:0/2997793767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f959c104320 0x7f959c199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:41.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.088+0000 7f95a19f9700 1 -- 172.21.15.154:0/2997793767 >> 172.21.15.154:0/2997793767 conn(0x7f959c0fe850 msgr2=0x7f959c107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:51:41.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.089+0000 7f95a19f9700 1 -- 172.21.15.154:0/2997793767 shutdown_connections 2024-05-20T17:51:41.089 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:51:41.089+0000 7f95a19f9700 1 -- 172.21.15.154:0/2997793767 wait complete. 2024-05-20T17:51:41.735 DEBUG:teuthology.orchestra.run.smithi154:osd.1> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.1.service 2024-05-20T17:51:41.738 INFO:tasks.cephadm:Deploying osd.2 on smithi204 with /dev/vg_nvme/lv_4... 2024-05-20T17:51:41.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-20T17:51:41.982 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:51:42.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:42.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:42.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:42.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:42 smithi204 ceph-mon[37076]: from='osd.1 [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-20T17:51:42.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:42 smithi204 ceph-mon[37076]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-20T17:51:42.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:42 smithi204 ceph-mon[37076]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:42.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:42.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:42.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:42.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:42 smithi154 ceph-mon[36482]: from='osd.1 [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-20T17:51:42.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:42 smithi154 ceph-mon[36482]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-20T17:51:42.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:42 smithi154 ceph-mon[36482]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:42.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:42.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:42.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:51:42.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:42 smithi129 ceph-mon[32029]: from='osd.1 [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-20T17:51:42.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:42 smithi129 ceph-mon[32029]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-20T17:51:42.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:42 smithi129 ceph-mon[32029]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:43.184 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:43 smithi154 ceph-mon[36482]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-20T17:51:43.185 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:43 smithi154 ceph-mon[36482]: from='osd.1 [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:51:43.185 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:43 smithi154 ceph-mon[36482]: osdmap e11: 2 total, 1 up, 2 in 2024-05-20T17:51:43.185 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:43 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:43.185 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:43 smithi154 ceph-mon[36482]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:51:43.185 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 17:51:43 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1[41040]: 2024-05-20T17:51:43.055+0000 7f0e61c6c700 -1 osd.1 0 waiting for initial osdmap 2024-05-20T17:51:43.185 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 17:51:43 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1[41040]: 2024-05-20T17:51:43.061+0000 7f0e5da21700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:51:43.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:43 smithi204 ceph-mon[37076]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-20T17:51:43.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:43 smithi204 ceph-mon[37076]: from='osd.1 [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:51:43.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:43 smithi204 ceph-mon[37076]: osdmap e11: 2 total, 1 up, 2 in 2024-05-20T17:51:43.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:43 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:43.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:43 smithi204 ceph-mon[37076]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:51:43.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:43 smithi129 ceph-mon[32029]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-20T17:51:43.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:43 smithi129 ceph-mon[32029]: from='osd.1 [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:51:43.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:43 smithi129 ceph-mon[32029]: osdmap e11: 2 total, 1 up, 2 in 2024-05-20T17:51:43.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:43 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:43.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:43 smithi129 ceph-mon[32029]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:51:43.755 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-20T17:51:43.755 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-20T17:51:43.755 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-05-20T17:51:43.755 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-05-20T17:51:43.756 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128081 s, 819 MB/s 2024-05-20T17:51:43.756 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-05-20T17:51:44.314 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:44 smithi154 ceph-mon[36482]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:51:44.314 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:44 smithi154 ceph-mon[36482]: osdmap e12: 2 total, 1 up, 2 in 2024-05-20T17:51:44.314 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:44 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:44.314 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:44 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:44.314 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:44 smithi154 ceph-mon[36482]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:44.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:44 smithi204 ceph-mon[37076]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:51:44.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:44 smithi204 ceph-mon[37076]: osdmap e12: 2 total, 1 up, 2 in 2024-05-20T17:51:44.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:44.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:44.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:44 smithi204 ceph-mon[37076]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:44.356 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_4 2024-05-20T17:51:44.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:44 smithi129 ceph-mon[32029]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:51:44.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:44 smithi129 ceph-mon[32029]: osdmap e12: 2 total, 1 up, 2 in 2024-05-20T17:51:44.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:44.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:44.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:44 smithi129 ceph-mon[32029]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-20T17:51:44.598 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:51:45.286 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:51:45.286 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: osd.1 [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596] boot 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: osdmap e13: 2 total, 2 up, 2 in 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi154 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi154 to 17088M 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:45.287 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:45.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:51:45.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:51:45.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: osd.1 [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596] boot 2024-05-20T17:51:45.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: osdmap e13: 2 total, 2 up, 2 in 2024-05-20T17:51:45.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:45.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi154 2024-05-20T17:51:45.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:45.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:45.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:51:45.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi154 to 17088M 2024-05-20T17:51:45.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:45.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:45.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:45.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:45.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:51:45.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:51:45.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: osd.1 [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596] boot 2024-05-20T17:51:45.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: osdmap e13: 2 total, 2 up, 2 in 2024-05-20T17:51:45.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:51:45.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi154 2024-05-20T17:51:45.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:45.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:45.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:51:45.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi154 to 17088M 2024-05-20T17:51:45.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:45.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:45.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:51:45.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:51:46.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.028+0000 7f48a350c700 1 -- 172.21.15.204:0/2662015164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f489c1020d0 msgr2=0x7f489c1024e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:51:46.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.028+0000 7f48a350c700 1 --2- 172.21.15.204:0/2662015164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f489c1020d0 0x7f489c1024e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4898009990 tx=0x7f4898009ca0 comp rx=0 tx=0).stop 2024-05-20T17:51:46.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.028+0000 7f48a350c700 1 -- 172.21.15.204:0/2662015164 shutdown_connections 2024-05-20T17:51:46.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.028+0000 7f48a350c700 1 --2- 172.21.15.204:0/2662015164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f489c103ba0 0x7f489c10a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:46.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.028+0000 7f48a350c700 1 --2- 172.21.15.204:0/2662015164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f489c1020d0 0x7f489c1024e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:46.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.028+0000 7f48a350c700 1 --2- 172.21.15.204:0/2662015164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f489c103200 0x7f489c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.028+0000 7f48a350c700 1 -- 172.21.15.204:0/2662015164 >> 172.21.15.204:0/2662015164 conn(0x7f489c0fd710 msgr2=0x7f489c0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.028+0000 7f48a350c700 1 -- 172.21.15.204:0/2662015164 shutdown_connections 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.028+0000 7f48a350c700 1 -- 172.21.15.204:0/2662015164 wait complete. 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a350c700 1 Processor -- start 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a350c700 1 -- start start 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a350c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f489c1020d0 0x7f489c1998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a350c700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f489c103200 0x7f489c199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a350c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f489c103ba0 0x7f489c1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a350c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f489c10eda0 con 0x7f489c103ba0 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a350c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f489c10ec20 con 0x7f489c1020d0 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a350c700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f489c10ef20 con 0x7f489c103200 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a0aa7700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f489c103200 0x7f489c199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:51:46.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a0aa7700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f489c103200 0x7f489c199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:49398/0 (socket says 172.21.15.204:49398) 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a0aa7700 1 -- 172.21.15.204:0/591161174 learned_addr learned my addr 172.21.15.204:0/591161174 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.029+0000 7f48a12a8700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f489c1020d0 0x7f489c1998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48a1aa9700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f489c103ba0 0x7f489c1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48a0aa7700 1 -- 172.21.15.204:0/591161174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f489c1020d0 msgr2=0x7f489c1998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48a0aa7700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f489c1020d0 0x7f489c1998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48a0aa7700 1 -- 172.21.15.204:0/591161174 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f489c103ba0 msgr2=0x7f489c1a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48a0aa7700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f489c103ba0 0x7f489c1a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48a0aa7700 1 -- 172.21.15.204:0/591161174 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f489c1a1380 con 0x7f489c103200 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48a12a8700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f489c1020d0 0x7f489c1998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48a1aa9700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f489c103ba0 0x7f489c1a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48a0aa7700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f489c103200 0x7f489c199df0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f488c00c9b0 tx=0x7f488c00cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:51:46.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48927fc700 1 -- 172.21.15.204:0/591161174 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c007b00 con 0x7f489c103200 2024-05-20T17:51:46.033 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48a350c700 1 -- 172.21.15.204:0/591161174 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f489c1a1670 con 0x7f489c103200 2024-05-20T17:51:46.033 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48927fc700 1 -- 172.21.15.204:0/591161174 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f488c007ca0 con 0x7f489c103200 2024-05-20T17:51:46.033 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48927fc700 1 -- 172.21.15.204:0/591161174 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c018780 con 0x7f489c103200 2024-05-20T17:51:46.033 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.030+0000 7f48a350c700 1 -- 172.21.15.204:0/591161174 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f489c1a1bb0 con 0x7f489c103200 2024-05-20T17:51:46.034 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.031+0000 7f48a350c700 1 -- 172.21.15.204:0/591161174 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f489c050170 con 0x7f489c103200 2024-05-20T17:51:46.034 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.032+0000 7f48927fc700 1 -- 172.21.15.204:0/591161174 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f488c00f450 con 0x7f489c103200 2024-05-20T17:51:46.034 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.033+0000 7f48927fc700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f48880a2b90 0x7f48880a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:51:46.034 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.033+0000 7f48927fc700 1 -- 172.21.15.204:0/591161174 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f488c0cdae0 con 0x7f489c103200 2024-05-20T17:51:46.034 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.033+0000 7f48a12a8700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f48880a2b90 0x7f48880a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:51:46.034 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.034+0000 7f48a12a8700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f48880a2b90 0x7f48880a50a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f48980039c0 tx=0x7f489801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:51:46.037 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.037+0000 7f48927fc700 1 -- 172.21.15.204:0/591161174 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f488c014030 con 0x7f489c103200 2024-05-20T17:51:46.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:46 smithi204 ceph-mon[37076]: osdmap e14: 2 total, 2 up, 2 in 2024-05-20T17:51:46.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:46 smithi204 ceph-mon[37076]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:46.246 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:51:46.247+0000 7f48a350c700 1 -- 172.21.15.204:0/591161174 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f489c062a80 con 0x7f48880a2b90 2024-05-20T17:51:46.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:46 smithi154 ceph-mon[36482]: osdmap e14: 2 total, 2 up, 2 in 2024-05-20T17:51:46.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:46 smithi154 ceph-mon[36482]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:46.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:46 smithi129 ceph-mon[32029]: osdmap e14: 2 total, 2 up, 2 in 2024-05-20T17:51:46.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:46 smithi129 ceph-mon[32029]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:47.126 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:47 smithi204 ceph-mon[37076]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:51:47.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:51:47.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:51:47.127 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:47.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:47 smithi154 ceph-mon[36482]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:51:47.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:51:47.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:51:47.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:47.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:47 smithi129 ceph-mon[32029]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:51:47.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:51:47.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:51:47.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:48.124 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:48 smithi204 ceph-mon[37076]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:48.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:48 smithi154 ceph-mon[36482]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:48.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:48 smithi129 ceph-mon[32029]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:49 smithi154 ceph-mon[36482]: from='client.? 172.21.15.204:0/1571859530' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f8c70f-8cd2-46ee-93da-e1ac0b73de83"}]: dispatch 2024-05-20T17:51:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:49 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f8c70f-8cd2-46ee-93da-e1ac0b73de83"}]: dispatch 2024-05-20T17:51:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:49 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6f8c70f-8cd2-46ee-93da-e1ac0b73de83"}]': finished 2024-05-20T17:51:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:49 smithi154 ceph-mon[36482]: osdmap e15: 3 total, 2 up, 3 in 2024-05-20T17:51:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:49 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:51:49.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:49 smithi129 ceph-mon[32029]: from='client.? 172.21.15.204:0/1571859530' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f8c70f-8cd2-46ee-93da-e1ac0b73de83"}]: dispatch 2024-05-20T17:51:49.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:49 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f8c70f-8cd2-46ee-93da-e1ac0b73de83"}]: dispatch 2024-05-20T17:51:49.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:49 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6f8c70f-8cd2-46ee-93da-e1ac0b73de83"}]': finished 2024-05-20T17:51:49.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:49 smithi129 ceph-mon[32029]: osdmap e15: 3 total, 2 up, 3 in 2024-05-20T17:51:49.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:49 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:51:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:49 smithi204 ceph-mon[37076]: from='client.? 172.21.15.204:0/1571859530' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f8c70f-8cd2-46ee-93da-e1ac0b73de83"}]: dispatch 2024-05-20T17:51:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:49 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6f8c70f-8cd2-46ee-93da-e1ac0b73de83"}]: dispatch 2024-05-20T17:51:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:49 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6f8c70f-8cd2-46ee-93da-e1ac0b73de83"}]': finished 2024-05-20T17:51:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:49 smithi204 ceph-mon[37076]: osdmap e15: 3 total, 2 up, 3 in 2024-05-20T17:51:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:49 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:51:50.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:50 smithi154 ceph-mon[36482]: from='client.? 172.21.15.204:0/1525843783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:51:50.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:50 smithi154 ceph-mon[36482]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:50.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:50 smithi129 ceph-mon[32029]: from='client.? 172.21.15.204:0/1525843783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:51:50.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:50 smithi129 ceph-mon[32029]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:50.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:50 smithi204 ceph-mon[37076]: from='client.? 172.21.15.204:0/1525843783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:51:50.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:50 smithi204 ceph-mon[37076]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:52.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:51 smithi154 ceph-mon[36482]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:52.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:51 smithi129 ceph-mon[32029]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:52.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:51 smithi204 ceph-mon[37076]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:53.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:53 smithi204 ceph-mon[37076]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:54.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:53 smithi154 ceph-mon[36482]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:54.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:53 smithi129 ceph-mon[32029]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:55.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:55 smithi204 ceph-mon[37076]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:56.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:55 smithi154 ceph-mon[36482]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:56.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:55 smithi129 ceph-mon[32029]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:56.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-20T17:51:56.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:56.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:56 smithi204 ceph-mon[37076]: Deploying daemon osd.2 on smithi204 2024-05-20T17:51:57.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-20T17:51:57.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:57.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:56 smithi154 ceph-mon[36482]: Deploying daemon osd.2 on smithi204 2024-05-20T17:51:57.166 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-20T17:51:57.166 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:51:57.166 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:56 smithi129 ceph-mon[32029]: Deploying daemon osd.2 on smithi204 2024-05-20T17:51:58.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:57 smithi154 ceph-mon[36482]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:58.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:57 smithi129 ceph-mon[32029]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:51:58.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:57 smithi204 ceph-mon[37076]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:00.016 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:51:59 smithi204 ceph-mon[37076]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:00.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:51:59 smithi154 ceph-mon[36482]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:00.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:51:59 smithi129 ceph-mon[32029]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:02.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:02 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:02.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:02 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:02.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:02 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:02.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:02 smithi154 ceph-mon[36482]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:02.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:02.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:02.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:02.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:02 smithi129 ceph-mon[32029]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:02.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:02 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:02.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:02 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:02.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:02 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:02.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:02 smithi204 ceph-mon[37076]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:04.545 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:04 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:04.545 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:04 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:04.545 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:04 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:04.545 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:04 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:04.545 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:04 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:04.545 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:04 smithi204 ceph-mon[37076]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:04.611 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.611+0000 7f48927fc700 1 -- 172.21.15.204:0/591161174 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f489c062a80 con 0x7f48880a2b90 2024-05-20T17:52:04.611 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 2 on host 'smithi204' 2024-05-20T17:52:04.613 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 -- 172.21.15.204:0/591161174 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f48880a2b90 msgr2=0x7f48880a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:04.613 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f48880a2b90 0x7f48880a50a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f48980039c0 tx=0x7f489801a040 comp rx=0 tx=0).stop 2024-05-20T17:52:04.613 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 -- 172.21.15.204:0/591161174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f489c103200 msgr2=0x7f489c199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:04.613 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f489c103200 0x7f489c199df0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f488c00c9b0 tx=0x7f488c00cd70 comp rx=0 tx=0).stop 2024-05-20T17:52:04.613 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 -- 172.21.15.204:0/591161174 shutdown_connections 2024-05-20T17:52:04.613 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f48880a2b90 0x7f48880a50a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:04.613 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f489c1020d0 0x7f489c1998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:04.613 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f489c103ba0 0x7f489c1a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:04.613 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 --2- 172.21.15.204:0/591161174 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f489c103200 0x7f489c199df0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:04.614 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 -- 172.21.15.204:0/591161174 >> 172.21.15.204:0/591161174 conn(0x7f489c0fd710 msgr2=0x7f489c106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:52:04.614 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 -- 172.21.15.204:0/591161174 shutdown_connections 2024-05-20T17:52:04.614 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:04.613+0000 7f48a350c700 1 -- 172.21.15.204:0/591161174 wait complete. 2024-05-20T17:52:04.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:04 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:04.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:04 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:04.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:04 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:04.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:04 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:04.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:04 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:04.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:04 smithi154 ceph-mon[36482]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:04.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:04 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:04.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:04 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:04.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:04 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:04.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:04 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:04.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:04 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:04.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:04 smithi129 ceph-mon[32029]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:05.196 DEBUG:teuthology.orchestra.run.smithi204:osd.2> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.2.service 2024-05-20T17:52:05.198 INFO:tasks.cephadm:Deploying osd.3 on smithi129 with /dev/vg_nvme/lv_3... 2024-05-20T17:52:05.199 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-20T17:52:05.447 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:52:05.890 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:05 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:05.890 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:05 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:05.890 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:05 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:05.890 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:05 smithi129 ceph-mon[32029]: from='osd.2 [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-20T17:52:05.890 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:05 smithi129 ceph-mon[32029]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-20T17:52:05.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:05 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:05.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:05 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:05.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:05 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:05.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:05 smithi154 ceph-mon[36482]: from='osd.2 [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-20T17:52:05.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:05 smithi154 ceph-mon[36482]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-20T17:52:05.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:05 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:05.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:05 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:05.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:05 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:05.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:05 smithi204 ceph-mon[37076]: from='osd.2 [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-20T17:52:05.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:05 smithi204 ceph-mon[37076]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-20T17:52:06.624 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:06 smithi204 ceph-mon[37076]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-20T17:52:06.624 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:06 smithi204 ceph-mon[37076]: osdmap e16: 3 total, 2 up, 3 in 2024-05-20T17:52:06.624 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:06.624 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:06 smithi204 ceph-mon[37076]: from='osd.2 [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:52:06.624 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:06 smithi204 ceph-mon[37076]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:52:06.625 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:06 smithi204 ceph-mon[37076]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:06.625 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 17:52:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2[42150]: 2024-05-20T17:52:06.596+0000 7fa3ebeee700 -1 osd.2 0 waiting for initial osdmap 2024-05-20T17:52:06.625 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 17:52:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2[42150]: 2024-05-20T17:52:06.599+0000 7fa3e7ca3700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:52:06.823 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:06 smithi129 ceph-mon[32029]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-20T17:52:06.823 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:06 smithi129 ceph-mon[32029]: osdmap e16: 3 total, 2 up, 3 in 2024-05-20T17:52:06.823 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:06.823 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:06 smithi129 ceph-mon[32029]: from='osd.2 [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:52:06.823 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:06 smithi129 ceph-mon[32029]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:52:06.824 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:06 smithi129 ceph-mon[32029]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:06.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:06 smithi154 ceph-mon[36482]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-20T17:52:06.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:06 smithi154 ceph-mon[36482]: osdmap e16: 3 total, 2 up, 3 in 2024-05-20T17:52:06.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:06.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:06 smithi154 ceph-mon[36482]: from='osd.2 [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:52:06.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:06 smithi154 ceph-mon[36482]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:52:06.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:06 smithi154 ceph-mon[36482]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:07.399 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-20T17:52:07.399 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-20T17:52:07.399 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-05-20T17:52:07.399 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-05-20T17:52:07.399 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111135 s, 944 MB/s 2024-05-20T17:52:07.399 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-05-20T17:52:07.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:07 smithi204 ceph-mon[37076]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:52:07.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:07 smithi204 ceph-mon[37076]: osdmap e17: 3 total, 2 up, 3 in 2024-05-20T17:52:07.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:07.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:07.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:07 smithi129 ceph-mon[32029]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:52:07.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:07 smithi129 ceph-mon[32029]: osdmap e17: 3 total, 2 up, 3 in 2024-05-20T17:52:07.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:07.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:07.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:07 smithi154 ceph-mon[36482]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:52:07.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:07 smithi154 ceph-mon[36482]: osdmap e17: 3 total, 2 up, 3 in 2024-05-20T17:52:07.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:07.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:08.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi129:vg_nvme/lv_3 2024-05-20T17:52:08.298 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:52:08.741 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:52:08.741 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:52:08.741 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:08.741 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: osd.2 [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720] boot 2024-05-20T17:52:08.741 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: osdmap e18: 3 total, 3 up, 3 in 2024-05-20T17:52:08.741 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:08.741 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:08.742 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-20T17:52:08.742 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:08.742 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:08.742 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:08.742 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:08.742 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:08.742 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:08.742 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:08 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:08.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:52:08.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:52:08.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:08.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: osd.2 [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720] boot 2024-05-20T17:52:08.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: osdmap e18: 3 total, 3 up, 3 in 2024-05-20T17:52:08.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:08.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:08.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-20T17:52:08.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:08.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:08.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:08.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:08.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:08.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:08.896 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:08 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:08.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:52:08.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:52:08.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:08.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: osd.2 [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720] boot 2024-05-20T17:52:08.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: osdmap e18: 3 total, 3 up, 3 in 2024-05-20T17:52:08.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:52:08.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-20T17:52:08.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-20T17:52:08.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:08.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:08.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:08.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:08.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:08.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:08.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:08 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:09.766 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:09 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi204 2024-05-20T17:52:09.766 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:09 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi204 to 17088M 2024-05-20T17:52:09.766 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-20T17:52:09.766 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:09 smithi129 ceph-mon[32029]: osdmap e19: 3 total, 3 up, 3 in 2024-05-20T17:52:09.767 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-20T17:52:09.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.884+0000 7fa2b82c9700 1 -- 172.21.15.129:0/3145560759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2b01035a0 msgr2=0x7fa2b01039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:09.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.884+0000 7fa2b82c9700 1 --2- 172.21.15.129:0/3145560759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2b01035a0 0x7fa2b01039d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009990 tx=0x7fa2a0009ca0 comp rx=0 tx=0).stop 2024-05-20T17:52:09.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.884+0000 7fa2b82c9700 1 -- 172.21.15.129:0/3145560759 shutdown_connections 2024-05-20T17:52:09.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.884+0000 7fa2b82c9700 1 --2- 172.21.15.129:0/3145560759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2b0102c00 0x7fa2b010a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:09.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.884+0000 7fa2b82c9700 1 --2- 172.21.15.129:0/3145560759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2b01035a0 0x7fa2b01039d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:09.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.884+0000 7fa2b82c9700 1 --2- 172.21.15.129:0/3145560759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2b01022b0 0x7fa2b01026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:09.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.884+0000 7fa2b82c9700 1 -- 172.21.15.129:0/3145560759 >> 172.21.15.129:0/3145560759 conn(0x7fa2b00fce90 msgr2=0x7fa2b00ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:52:09.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.884+0000 7fa2b82c9700 1 -- 172.21.15.129:0/3145560759 shutdown_connections 2024-05-20T17:52:09.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.884+0000 7fa2b82c9700 1 -- 172.21.15.129:0/3145560759 wait complete. 2024-05-20T17:52:09.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.885+0000 7fa2b82c9700 1 Processor -- start 2024-05-20T17:52:09.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.885+0000 7fa2b82c9700 1 -- start start 2024-05-20T17:52:09.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.885+0000 7fa2b82c9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2b01022b0 0x7fa2b0101c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.885+0000 7fa2b82c9700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2b0102c00 0x7fa2b019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.885+0000 7fa2b82c9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2b01035a0 0x7fa2b01a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.885+0000 7fa2b82c9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b010dc70 con 0x7fa2b01022b0 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.885+0000 7fa2b82c9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b010daf0 con 0x7fa2b01035a0 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.885+0000 7fa2b82c9700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b010ddf0 con 0x7fa2b0102c00 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b6065700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2b01022b0 0x7fa2b0101c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b5864700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2b0102c00 0x7fa2b019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b6065700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2b01022b0 0x7fa2b0101c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48516/0 (socket says 172.21.15.129:48516) 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b6065700 1 -- 172.21.15.129:0/2491683210 learned_addr learned my addr 172.21.15.129:0/2491683210 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b6866700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2b01035a0 0x7fa2b01a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b6065700 1 -- 172.21.15.129:0/2491683210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2b0102c00 msgr2=0x7fa2b019e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:09.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b6065700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2b0102c00 0x7fa2b019e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b6065700 1 -- 172.21.15.129:0/2491683210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2b01035a0 msgr2=0x7fa2b01a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b6065700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2b01035a0 0x7fa2b01a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b6065700 1 -- 172.21.15.129:0/2491683210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b01983e0 con 0x7fa2b01022b0 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b6866700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2b01035a0 0x7fa2b01a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b5864700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2b0102c00 0x7fa2b019e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b6065700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2b01022b0 0x7fa2b0101c00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00098e0 tx=0x7fa2a00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2a77fe700 1 -- 172.21.15.129:0/2491683210 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0003c70 con 0x7fa2b01022b0 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2a77fe700 1 -- 172.21.15.129:0/2491683210 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2a0004830 con 0x7fa2b01022b0 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b82c9700 1 -- 172.21.15.129:0/2491683210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b0198670 con 0x7fa2b01022b0 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2a77fe700 1 -- 172.21.15.129:0/2491683210 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0005330 con 0x7fa2b01022b0 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.886+0000 7fa2b82c9700 1 -- 172.21.15.129:0/2491683210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b0198b40 con 0x7fa2b01022b0 2024-05-20T17:52:09.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.887+0000 7fa2b82c9700 1 -- 172.21.15.129:0/2491683210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b0050170 con 0x7fa2b01022b0 2024-05-20T17:52:09.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.889+0000 7fa2a77fe700 1 -- 172.21.15.129:0/2491683210 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7fa2a00054d0 con 0x7fa2b01022b0 2024-05-20T17:52:09.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.890+0000 7fa2a77fe700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa29c0a2c10 0x7fa29c0a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:09.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.890+0000 7fa2b5864700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa29c0a2c10 0x7fa29c0a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:09.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.890+0000 7fa2a77fe700 1 -- 172.21.15.129:0/2491683210 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7fa2a00cf580 con 0x7fa2b01022b0 2024-05-20T17:52:09.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.890+0000 7fa2b5864700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa29c0a2c10 0x7fa29c0a5120 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa2b01015b0 tx=0x7fa2ac00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:52:09.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:09 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi204 2024-05-20T17:52:09.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:09 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi204 to 17088M 2024-05-20T17:52:09.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-20T17:52:09.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:09 smithi154 ceph-mon[36482]: osdmap e19: 3 total, 3 up, 3 in 2024-05-20T17:52:09.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-20T17:52:09.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:09.897+0000 7fa2a77fe700 1 -- 172.21.15.129:0/2491683210 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa2a0030070 con 0x7fa2b01022b0 2024-05-20T17:52:09.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:09 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi204 2024-05-20T17:52:09.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:09 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi204 to 17088M 2024-05-20T17:52:09.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-20T17:52:09.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:09 smithi204 ceph-mon[37076]: osdmap e19: 3 total, 3 up, 3 in 2024-05-20T17:52:09.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-20T17:52:10.023 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 17:52:10 smithi129 sudo[48796]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-20T17:52:10.023 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 17:52:10 smithi129 sudo[48796]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:10.023 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 17:52:10 smithi129 sudo[48796]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:10.023 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 17:52:10 smithi129 sudo[48796]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:10.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:10.110+0000 7fa2b82c9700 1 -- 172.21.15.129:0/2491683210 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa2b0062a80 con 0x7fa29c0a2c10 2024-05-20T17:52:10.617 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 17:52:10 smithi129 sudo[48866]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-20T17:52:10.617 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 17:52:10 smithi129 sudo[48866]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:10.617 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 17:52:10 smithi129 sudo[48866]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:10.617 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 17:52:10 smithi129 sudo[48866]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:10.798 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 17:52:10 smithi154 sudo[44858]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-20T17:52:10.798 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 17:52:10 smithi154 sudo[44858]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:10.798 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 17:52:10 smithi154 sudo[44858]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:10.798 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 17:52:10 smithi154 sudo[44858]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:10.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:10 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-20T17:52:10.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:10 smithi154 ceph-mon[36482]: osdmap e20: 3 total, 3 up, 3 in 2024-05-20T17:52:10.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:10 smithi154 ceph-mon[36482]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:10.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:10 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:52:10.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:10 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:52:10.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:10 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:10.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:10 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:10.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:10 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-20T17:52:10.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:10 smithi129 ceph-mon[32029]: osdmap e20: 3 total, 3 up, 3 in 2024-05-20T17:52:10.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:10 smithi129 ceph-mon[32029]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:10.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:10 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:52:10.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:10 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:52:10.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:10 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:10.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:10 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:10.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:10 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-20T17:52:10.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:10 smithi204 ceph-mon[37076]: osdmap e20: 3 total, 3 up, 3 in 2024-05-20T17:52:10.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:10 smithi204 ceph-mon[37076]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:10.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:10 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:52:10.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:10 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:52:10.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:10 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:10.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:10 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:11.145 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 17:52:10 smithi154 sudo[44861]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-20T17:52:11.146 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 17:52:10 smithi154 sudo[44861]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:11.146 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 17:52:10 smithi154 sudo[44861]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:11.146 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 17:52:10 smithi154 sudo[44861]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:11.318 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 17:52:11 smithi204 sudo[46197]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-20T17:52:11.318 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 17:52:11 smithi204 sudo[46197]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:11.318 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 17:52:11 smithi204 sudo[46197]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:11.318 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 17:52:11 smithi204 sudo[46197]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:11.648 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 sudo[48942]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-20T17:52:11.648 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 sudo[48942]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:11.648 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 sudo[48942]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:11.648 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 sudo[48942]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:11.680 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 17:52:11 smithi204 sudo[46200]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-20T17:52:11.680 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 17:52:11 smithi204 sudo[46200]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:11.680 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 17:52:11 smithi204 sudo[46200]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:11.681 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 17:52:11 smithi204 sudo[46200]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:11.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 sudo[48966]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-20T17:52:11.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 sudo[48966]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:11.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 sudo[48966]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:11.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 sudo[48966]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:11.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 ceph-mon[32029]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:52:11.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 ceph-mon[32029]: osdmap e21: 3 total, 3 up, 3 in 2024-05-20T17:52:11.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:11.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:11.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:11 smithi129 ceph-mon[32029]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-20T17:52:12.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:11 smithi204 ceph-mon[37076]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:52:12.106 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:11 smithi204 ceph-mon[37076]: osdmap e21: 3 total, 3 up, 3 in 2024-05-20T17:52:12.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:11 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:12.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:11 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:12.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:11 smithi204 ceph-mon[37076]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-20T17:52:12.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 sudo[46203]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-20T17:52:12.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 sudo[46203]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:12.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 sudo[46203]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:12.107 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 sudo[46203]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:12.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:11 smithi154 ceph-mon[36482]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:52:12.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:11 smithi154 ceph-mon[36482]: osdmap e21: 3 total, 3 up, 3 in 2024-05-20T17:52:12.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:11 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:12.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:11 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:12.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:11 smithi154 ceph-mon[36482]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-20T17:52:12.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:12.352+0000 7fa2a77fe700 1 -- 172.21.15.129:0/2491683210 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa2a0099990 con 0x7fa2b01022b0 2024-05-20T17:52:12.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 sudo[46206]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-20T17:52:12.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 sudo[46206]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:12.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 sudo[46206]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:12.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 sudo[46206]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:12.837 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 sudo[44864]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-20T17:52:12.838 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 sudo[44864]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:12.838 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 sudo[44864]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:12.838 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 sudo[44864]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:13.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 sudo[44867]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 sudo[44867]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 sudo[44867]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 sudo[44867]: pam_unix(sudo:session): session closed for user root 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:52:13.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-20T17:52:13.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-20T17:52:13.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-20T17:52:13.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:52:13.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:52:13.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:52:13.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2359693903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fbfcecf-5f49-429e-a60f-2aae1d18eb16"}]: dispatch 2024-05-20T17:52:13.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-20T17:52:13.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2359693903' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3fbfcecf-5f49-429e-a60f-2aae1d18eb16"}]': finished 2024-05-20T17:52:13.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: osdmap e22: 4 total, 3 up, 4 in 2024-05-20T17:52:13.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:13.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:52:13.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:52:13.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:52:13.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2359693903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fbfcecf-5f49-429e-a60f-2aae1d18eb16"}]: dispatch 2024-05-20T17:52:13.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-20T17:52:13.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2359693903' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3fbfcecf-5f49-429e-a60f-2aae1d18eb16"}]': finished 2024-05-20T17:52:13.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: osdmap e22: 4 total, 3 up, 4 in 2024-05-20T17:52:13.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:13.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-20T17:52:13.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:52:13.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:52:13.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:52:13.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2359693903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3fbfcecf-5f49-429e-a60f-2aae1d18eb16"}]: dispatch 2024-05-20T17:52:13.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-20T17:52:13.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2359693903' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3fbfcecf-5f49-429e-a60f-2aae1d18eb16"}]': finished 2024-05-20T17:52:13.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: osdmap e22: 4 total, 3 up, 4 in 2024-05-20T17:52:13.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:14.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:13 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3956525853' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:52:14.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:13 smithi154 ceph-mon[36482]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:14.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:13 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3956525853' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:52:14.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:13 smithi129 ceph-mon[32029]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:14.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:13 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3956525853' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:52:14.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:13 smithi204 ceph-mon[37076]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:15 smithi154 ceph-mon[36482]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.0 KiB/s rd, 364 KiB/s wr, 25 op/s 2024-05-20T17:52:16.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:15 smithi129 ceph-mon[32029]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.0 KiB/s rd, 364 KiB/s wr, 25 op/s 2024-05-20T17:52:16.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:15 smithi204 ceph-mon[37076]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 8.0 KiB/s rd, 364 KiB/s wr, 25 op/s 2024-05-20T17:52:17.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:17 smithi129 ceph-mon[32029]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.7 KiB/s rd, 260 KiB/s wr, 17 op/s 2024-05-20T17:52:18.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:17 smithi154 ceph-mon[36482]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.7 KiB/s rd, 260 KiB/s wr, 17 op/s 2024-05-20T17:52:18.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:17 smithi204 ceph-mon[37076]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.7 KiB/s rd, 260 KiB/s wr, 17 op/s 2024-05-20T17:52:19.900 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:19 smithi129 ceph-mon[32029]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.4 KiB/s rd, 203 KiB/s wr, 13 op/s 2024-05-20T17:52:20.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:19 smithi154 ceph-mon[36482]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.4 KiB/s rd, 203 KiB/s wr, 13 op/s 2024-05-20T17:52:20.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:19 smithi204 ceph-mon[37076]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.4 KiB/s rd, 203 KiB/s wr, 13 op/s 2024-05-20T17:52:21.047 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:20 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-20T17:52:21.047 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:20 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:21.047 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:20 smithi129 ceph-mon[32029]: Deploying daemon osd.3 on smithi129 2024-05-20T17:52:21.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:20 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-20T17:52:21.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:20 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:21.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:20 smithi154 ceph-mon[36482]: Deploying daemon osd.3 on smithi129 2024-05-20T17:52:21.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:20 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-20T17:52:21.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:20 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:21.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:20 smithi204 ceph-mon[37076]: Deploying daemon osd.3 on smithi129 2024-05-20T17:52:21.777 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:21 smithi129 ceph-mon[32029]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-05-20T17:52:22.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:21 smithi154 ceph-mon[36482]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-05-20T17:52:22.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:21 smithi204 ceph-mon[37076]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-05-20T17:52:23.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:23 smithi129 ceph-mon[32029]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 169 KiB/s wr, 11 op/s 2024-05-20T17:52:24.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:23 smithi154 ceph-mon[36482]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 169 KiB/s wr, 11 op/s 2024-05-20T17:52:24.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:23 smithi204 ceph-mon[37076]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 169 KiB/s wr, 11 op/s 2024-05-20T17:52:25.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:25 smithi129 ceph-mon[32029]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-05-20T17:52:26.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:25 smithi154 ceph-mon[36482]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-05-20T17:52:26.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:25 smithi204 ceph-mon[37076]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-05-20T17:52:27.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:26 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:27.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:26 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:27.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:26 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:27.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:26 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:27.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:26 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:27.082 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:26 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:27.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:27.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:27.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:28.023 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:27 smithi129 ceph-mon[32029]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:28.061 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:27 smithi154 ceph-mon[36482]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:28.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:27 smithi204 ceph-mon[37076]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:29.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.266+0000 7fa2a77fe700 1 -- 172.21.15.129:0/2491683210 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa2b0062a80 con 0x7fa29c0a2c10 2024-05-20T17:52:29.266 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 3 on host 'smithi129' 2024-05-20T17:52:29.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 -- 172.21.15.129:0/2491683210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa29c0a2c10 msgr2=0x7fa29c0a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:29.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa29c0a2c10 0x7fa29c0a5120 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa2b01015b0 tx=0x7fa2ac00b410 comp rx=0 tx=0).stop 2024-05-20T17:52:29.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 -- 172.21.15.129:0/2491683210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2b01022b0 msgr2=0x7fa2b0101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:29.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2b01022b0 0x7fa2b0101c00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00098e0 tx=0x7fa2a00047b0 comp rx=0 tx=0).stop 2024-05-20T17:52:29.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 -- 172.21.15.129:0/2491683210 shutdown_connections 2024-05-20T17:52:29.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa29c0a2c10 0x7fa29c0a5120 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:29.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2b01035a0 0x7fa2b01a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:29.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2b01022b0 0x7fa2b0101c00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:29.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 --2- 172.21.15.129:0/2491683210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa2b0102c00 0x7fa2b019e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:29.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 -- 172.21.15.129:0/2491683210 >> 172.21.15.129:0/2491683210 conn(0x7fa2b00fce90 msgr2=0x7fa2b01064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:52:29.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 -- 172.21.15.129:0/2491683210 shutdown_connections 2024-05-20T17:52:29.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:52:29.268+0000 7fa2b82c9700 1 -- 172.21.15.129:0/2491683210 wait complete. 2024-05-20T17:52:29.945 DEBUG:teuthology.orchestra.run.smithi129:osd.3> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.3.service 2024-05-20T17:52:29.947 INFO:tasks.cephadm:Deploying osd.4 on smithi154 with /dev/vg_nvme/lv_3... 2024-05-20T17:52:29.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-20T17:52:30.193 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T17:52:30.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:30 smithi129 ceph-mon[32029]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:30.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:30 smithi129 ceph-mon[32029]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-20T17:52:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:30 smithi154 ceph-mon[36482]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:30.513 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:30 smithi154 ceph-mon[36482]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-20T17:52:30.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:30.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:30 smithi204 ceph-mon[37076]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:30.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:30 smithi204 ceph-mon[37076]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-20T17:52:31.355 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:31 smithi129 ceph-mon[32029]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-20T17:52:31.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:31 smithi129 ceph-mon[32029]: osdmap e23: 4 total, 3 up, 4 in 2024-05-20T17:52:31.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:31 smithi129 ceph-mon[32029]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:52:31.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:31 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:31.357 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 17:52:31 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3[51041]: 2024-05-20T17:52:31.349+0000 7f5e98926700 -1 osd.3 0 waiting for initial osdmap 2024-05-20T17:52:31.357 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 17:52:31 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3[51041]: 2024-05-20T17:52:31.354+0000 7f5e946db700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:52:31.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:31 smithi154 ceph-mon[36482]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-20T17:52:31.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:31 smithi154 ceph-mon[36482]: osdmap e23: 4 total, 3 up, 4 in 2024-05-20T17:52:31.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:31 smithi154 ceph-mon[36482]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:52:31.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:31 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:31.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:31 smithi204 ceph-mon[37076]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-20T17:52:31.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:31 smithi204 ceph-mon[37076]: osdmap e23: 4 total, 3 up, 4 in 2024-05-20T17:52:31.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:31 smithi204 ceph-mon[37076]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:52:31.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:31 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:32.215 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-20T17:52:32.216 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-20T17:52:32.216 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-05-20T17:52:32.216 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-05-20T17:52:32.216 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114648 s, 915 MB/s 2024-05-20T17:52:32.216 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-05-20T17:52:32.566 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:32 smithi129 ceph-mon[32029]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:52:32.566 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:32 smithi129 ceph-mon[32029]: osdmap e24: 4 total, 3 up, 4 in 2024-05-20T17:52:32.566 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:32.566 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:32.566 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:32 smithi129 ceph-mon[32029]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:32.607 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:32 smithi154 ceph-mon[36482]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:52:32.608 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:32 smithi154 ceph-mon[36482]: osdmap e24: 4 total, 3 up, 4 in 2024-05-20T17:52:32.608 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:32.608 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:32.608 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:32 smithi154 ceph-mon[36482]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:32.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:32 smithi204 ceph-mon[37076]: from='osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:52:32.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:32 smithi204 ceph-mon[37076]: osdmap e24: 4 total, 3 up, 4 in 2024-05-20T17:52:32.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:32.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:32.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:32 smithi204 ceph-mon[37076]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-20T17:52:32.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi154:vg_nvme/lv_3 2024-05-20T17:52:33.095 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T17:52:33.486 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:52:33.486 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415] boot 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: osdmap e25: 4 total, 4 up, 4 in 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.487 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:33.488 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:33.488 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:33.488 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:52:33.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:52:33.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:33.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415] boot 2024-05-20T17:52:33.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: osdmap e25: 4 total, 4 up, 4 in 2024-05-20T17:52:33.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:33.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:33.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:33.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:33.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:33.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:33.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:33.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:33.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:52:33.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: osd.3 [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415] boot 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: osdmap e25: 4 total, 4 up, 4 in 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:33.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:33.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:33.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:33.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:34.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:34 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi129 2024-05-20T17:52:34.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:34 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi129 to 8544M 2024-05-20T17:52:34.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:34 smithi154 ceph-mon[36482]: osdmap e26: 4 total, 4 up, 4 in 2024-05-20T17:52:34.544 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:34 smithi154 ceph-mon[36482]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:34.660 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.659+0000 7eff8f19e700 1 -- 172.21.15.154:0/4207443915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff88103030 msgr2=0x7eff88103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:34.661 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.659+0000 7eff8f19e700 1 --2- 172.21.15.154:0/4207443915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff88103030 0x7eff88103500 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7eff74009990 tx=0x7eff74009ca0 comp rx=0 tx=0).stop 2024-05-20T17:52:34.661 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.659+0000 7eff8f19e700 1 -- 172.21.15.154:0/4207443915 shutdown_connections 2024-05-20T17:52:34.661 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.659+0000 7eff8f19e700 1 --2- 172.21.15.154:0/4207443915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff88103b90 0x7eff8810a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:34.661 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.659+0000 7eff8f19e700 1 --2- 172.21.15.154:0/4207443915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88102650 0x7eff88102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:34.661 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.659+0000 7eff8f19e700 1 --2- 172.21.15.154:0/4207443915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff88103030 0x7eff88103500 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:34.661 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.659+0000 7eff8f19e700 1 -- 172.21.15.154:0/4207443915 >> 172.21.15.154:0/4207443915 conn(0x7eff880fcea0 msgr2=0x7eff880ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:52:34.661 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.659+0000 7eff8f19e700 1 -- 172.21.15.154:0/4207443915 shutdown_connections 2024-05-20T17:52:34.661 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.659+0000 7eff8f19e700 1 -- 172.21.15.154:0/4207443915 wait complete. 2024-05-20T17:52:34.661 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.659+0000 7eff8f19e700 1 Processor -- start 2024-05-20T17:52:34.661 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff8f19e700 1 -- start start 2024-05-20T17:52:34.661 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff8f19e700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff88102650 0x7eff88106060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff8f19e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff88103030 0x7eff88104710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff8f19e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88103b90 0x7eff88104c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff8f19e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810dcb0 con 0x7eff88103b90 2024-05-20T17:52:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff8f19e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810db30 con 0x7eff88103030 2024-05-20T17:52:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff8f19e700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810de30 con 0x7eff88102650 2024-05-20T17:52:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff87fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff88103030 0x7eff88104710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff87fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff88103030 0x7eff88104710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:37216/0 (socket says 172.21.15.154:37216) 2024-05-20T17:52:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff87fff700 1 -- 172.21.15.154:0/1502054121 learned_addr learned my addr 172.21.15.154:0/1502054121 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-05-20T17:52:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff8d73b700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88103b90 0x7eff88104c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff8cf3a700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff88102650 0x7eff88106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff87fff700 1 -- 172.21.15.154:0/1502054121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff88102650 msgr2=0x7eff88106060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:34.663 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff87fff700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff88102650 0x7eff88106060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:34.663 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff87fff700 1 -- 172.21.15.154:0/1502054121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88103b90 msgr2=0x7eff88104c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:34.663 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff87fff700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88103b90 0x7eff88104c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:34.663 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff87fff700 1 -- 172.21.15.154:0/1502054121 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff881053b0 con 0x7eff88103030 2024-05-20T17:52:34.663 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.660+0000 7eff87fff700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff88103030 0x7eff88104710 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff740057c0 tx=0x7eff740057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:52:34.663 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.661+0000 7eff85ffb700 1 -- 172.21.15.154:0/1502054121 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff74027070 con 0x7eff88103030 2024-05-20T17:52:34.663 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.661+0000 7eff8f19e700 1 -- 172.21.15.154:0/1502054121 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff881a3940 con 0x7eff88103030 2024-05-20T17:52:34.663 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.661+0000 7eff85ffb700 1 -- 172.21.15.154:0/1502054121 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff74004150 con 0x7eff88103030 2024-05-20T17:52:34.663 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.661+0000 7eff85ffb700 1 -- 172.21.15.154:0/1502054121 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7400f460 con 0x7eff88103030 2024-05-20T17:52:34.663 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.661+0000 7eff8f19e700 1 -- 172.21.15.154:0/1502054121 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff881a3ce0 con 0x7eff88103030 2024-05-20T17:52:34.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:34 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi129 2024-05-20T17:52:34.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:34 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi129 to 8544M 2024-05-20T17:52:34.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:34 smithi129 ceph-mon[32029]: osdmap e26: 4 total, 4 up, 4 in 2024-05-20T17:52:34.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:34 smithi129 ceph-mon[32029]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:34.665 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.662+0000 7eff8f19e700 1 -- 172.21.15.154:0/1502054121 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff88050170 con 0x7eff88103030 2024-05-20T17:52:34.665 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.663+0000 7eff85ffb700 1 -- 172.21.15.154:0/1502054121 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7eff7402a070 con 0x7eff88103030 2024-05-20T17:52:34.665 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.664+0000 7eff85ffb700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff780a2c20 0x7eff780a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:34.665 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.664+0000 7eff85ffb700 1 -- 172.21.15.154:0/1502054121 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7eff740cf540 con 0x7eff88103030 2024-05-20T17:52:34.665 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.664+0000 7eff8cf3a700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff780a2c20 0x7eff780a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:34.665 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.664+0000 7eff8cf3a700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff780a2c20 0x7eff780a5130 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7eff8806aee0 tx=0x7eff7c00b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:52:34.667 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.668+0000 7eff85ffb700 1 -- 172.21.15.154:0/1502054121 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7eff74031080 con 0x7eff88103030 2024-05-20T17:52:34.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:34 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi129 2024-05-20T17:52:34.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:34 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi129 to 8544M 2024-05-20T17:52:34.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:34 smithi204 ceph-mon[37076]: osdmap e26: 4 total, 4 up, 4 in 2024-05-20T17:52:34.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:34 smithi204 ceph-mon[37076]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:34.876 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:34.876+0000 7eff8f19e700 1 -- 172.21.15.154:0/1502054121 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7eff880feea0 con 0x7eff780a2c20 2024-05-20T17:52:35.626 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:35 smithi154 ceph-mon[36482]: osdmap e27: 4 total, 4 up, 4 in 2024-05-20T17:52:35.626 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:35 smithi154 ceph-mon[36482]: osdmap e28: 4 total, 4 up, 4 in 2024-05-20T17:52:35.626 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:52:35.626 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:52:35.626 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:35.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:35 smithi129 ceph-mon[32029]: osdmap e27: 4 total, 4 up, 4 in 2024-05-20T17:52:35.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:35 smithi129 ceph-mon[32029]: osdmap e28: 4 total, 4 up, 4 in 2024-05-20T17:52:35.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:52:35.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:52:35.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:35.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:35 smithi204 ceph-mon[37076]: osdmap e27: 4 total, 4 up, 4 in 2024-05-20T17:52:35.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:35 smithi204 ceph-mon[37076]: osdmap e28: 4 total, 4 up, 4 in 2024-05-20T17:52:35.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:52:35.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:52:35.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:36.461 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:36 smithi154 ceph-mon[36482]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:52:36.462 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:36 smithi154 ceph-mon[36482]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:52:36.462 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:36 smithi154 ceph-mon[36482]: osdmap e29: 4 total, 4 up, 4 in 2024-05-20T17:52:36.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:36 smithi129 ceph-mon[32029]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:52:36.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:36 smithi129 ceph-mon[32029]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:52:36.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:36 smithi129 ceph-mon[32029]: osdmap e29: 4 total, 4 up, 4 in 2024-05-20T17:52:36.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:36 smithi204 ceph-mon[37076]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:52:36.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:36 smithi204 ceph-mon[37076]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:52:36.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:36 smithi204 ceph-mon[37076]: osdmap e29: 4 total, 4 up, 4 in 2024-05-20T17:52:37.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:37 smithi154 ceph-mon[36482]: from='client.? 172.21.15.154:0/4278331743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393945c-4ccc-4f2c-ad95-5e884497d5d5"}]: dispatch 2024-05-20T17:52:37.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:37 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393945c-4ccc-4f2c-ad95-5e884497d5d5"}]: dispatch 2024-05-20T17:52:37.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:37 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3393945c-4ccc-4f2c-ad95-5e884497d5d5"}]': finished 2024-05-20T17:52:37.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:37 smithi154 ceph-mon[36482]: osdmap e30: 5 total, 4 up, 5 in 2024-05-20T17:52:37.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:37 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:37.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:37 smithi129 ceph-mon[32029]: from='client.? 172.21.15.154:0/4278331743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393945c-4ccc-4f2c-ad95-5e884497d5d5"}]: dispatch 2024-05-20T17:52:37.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:37 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393945c-4ccc-4f2c-ad95-5e884497d5d5"}]: dispatch 2024-05-20T17:52:37.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:37 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3393945c-4ccc-4f2c-ad95-5e884497d5d5"}]': finished 2024-05-20T17:52:37.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:37 smithi129 ceph-mon[32029]: osdmap e30: 5 total, 4 up, 5 in 2024-05-20T17:52:37.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:37 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:37.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:37 smithi204 ceph-mon[37076]: from='client.? 172.21.15.154:0/4278331743' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393945c-4ccc-4f2c-ad95-5e884497d5d5"}]: dispatch 2024-05-20T17:52:37.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:37 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3393945c-4ccc-4f2c-ad95-5e884497d5d5"}]: dispatch 2024-05-20T17:52:37.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:37 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3393945c-4ccc-4f2c-ad95-5e884497d5d5"}]': finished 2024-05-20T17:52:37.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:37 smithi204 ceph-mon[37076]: osdmap e30: 5 total, 4 up, 5 in 2024-05-20T17:52:37.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:37 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:38.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:38 smithi154 ceph-mon[36482]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:52:38.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:38 smithi154 ceph-mon[36482]: from='client.? 172.21.15.154:0/743527740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:52:38.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:38 smithi129 ceph-mon[32029]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:52:38.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:38 smithi129 ceph-mon[32029]: from='client.? 172.21.15.154:0/743527740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:52:38.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:38 smithi204 ceph-mon[37076]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:52:38.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:38 smithi204 ceph-mon[37076]: from='client.? 172.21.15.154:0/743527740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:52:40.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:39 smithi154 ceph-mon[36482]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-20T17:52:40.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:39 smithi129 ceph-mon[32029]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-20T17:52:40.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:39 smithi204 ceph-mon[37076]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-20T17:52:41.835 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:41 smithi154 ceph-mon[36482]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-20T17:52:42.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:41 smithi129 ceph-mon[32029]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-20T17:52:42.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:41 smithi204 ceph-mon[37076]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-20T17:52:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:43 smithi154 ceph-mon[36482]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:44.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:43 smithi129 ceph-mon[32029]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:44.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:43 smithi204 ceph-mon[37076]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:45.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-20T17:52:45.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:45.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:44 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-20T17:52:45.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:44 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:45.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-20T17:52:45.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:46.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:45 smithi154 ceph-mon[36482]: Deploying daemon osd.4 on smithi154 2024-05-20T17:52:46.030 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:45 smithi154 ceph-mon[36482]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:46.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:45 smithi129 ceph-mon[32029]: Deploying daemon osd.4 on smithi154 2024-05-20T17:52:46.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:45 smithi129 ceph-mon[32029]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:45 smithi204 ceph-mon[37076]: Deploying daemon osd.4 on smithi154 2024-05-20T17:52:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:45 smithi204 ceph-mon[37076]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:47.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:47 smithi154 ceph-mon[36482]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:48.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:47 smithi129 ceph-mon[32029]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:47 smithi204 ceph-mon[37076]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:50.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:49 smithi154 ceph-mon[36482]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:50.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:49 smithi129 ceph-mon[32029]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:50.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:49 smithi204 ceph-mon[37076]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:51.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:51 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:51.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:51 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:51.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:51 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:51.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:51 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:51.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:51 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:51.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:51 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:51.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:51 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:51.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:51 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:51.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:51 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:52.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:52 smithi154 ceph-mon[36482]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:52.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:52 smithi129 ceph-mon[32029]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:52.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:52 smithi204 ceph-mon[37076]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:54.102 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:54.102 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:54.102 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:54.102 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:54.102 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:53 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:54.102 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:53 smithi154 ceph-mon[36482]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:54.144 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.145+0000 7eff85ffb700 1 -- 172.21.15.154:0/1502054121 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7eff880feea0 con 0x7eff780a2c20 2024-05-20T17:52:54.145 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 4 on host 'smithi154' 2024-05-20T17:52:54.147 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 -- 172.21.15.154:0/1502054121 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff780a2c20 msgr2=0x7eff780a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:54.147 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff780a2c20 0x7eff780a5130 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7eff8806aee0 tx=0x7eff7c00b5b0 comp rx=0 tx=0).stop 2024-05-20T17:52:54.147 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 -- 172.21.15.154:0/1502054121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff88103030 msgr2=0x7eff88104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:54.147 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff88103030 0x7eff88104710 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff740057c0 tx=0x7eff740057f0 comp rx=0 tx=0).stop 2024-05-20T17:52:54.147 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 -- 172.21.15.154:0/1502054121 shutdown_connections 2024-05-20T17:52:54.147 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff780a2c20 0x7eff780a5130 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:54.147 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88103b90 0x7eff88104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:54.147 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff88102650 0x7eff88106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:54.148 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 --2- 172.21.15.154:0/1502054121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff88103030 0x7eff88104710 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:54.148 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 -- 172.21.15.154:0/1502054121 >> 172.21.15.154:0/1502054121 conn(0x7eff880fcea0 msgr2=0x7eff880fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:52:54.148 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 -- 172.21.15.154:0/1502054121 shutdown_connections 2024-05-20T17:52:54.148 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:52:54.147+0000 7eff8f19e700 1 -- 172.21.15.154:0/1502054121 wait complete. 2024-05-20T17:52:54.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:54.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:54.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:54.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:54.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:53 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:54.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:53 smithi129 ceph-mon[32029]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:54.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:54.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:54.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:54.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:54.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:53 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:54.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:53 smithi204 ceph-mon[37076]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:54.866 DEBUG:teuthology.orchestra.run.smithi154:osd.4> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.4.service 2024-05-20T17:52:54.868 INFO:tasks.cephadm:Deploying osd.5 on smithi204 with /dev/vg_nvme/lv_3... 2024-05-20T17:52:54.869 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-20T17:52:55.113 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:52:55.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:55 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:55.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:55 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:55.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:55 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:55.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:55 smithi204 ceph-mon[37076]: from='osd.4 [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-20T17:52:55.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:55 smithi204 ceph-mon[37076]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-20T17:52:55.346 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:55 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:55.346 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:55 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:55.346 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:55 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:55.346 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:55 smithi154 ceph-mon[36482]: from='osd.4 [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-20T17:52:55.346 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:55 smithi154 ceph-mon[36482]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-20T17:52:55.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:55 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:55.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:55 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:55.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:55 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:52:55.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:55 smithi129 ceph-mon[32029]: from='osd.4 [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-20T17:52:55.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:55 smithi129 ceph-mon[32029]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-20T17:52:56.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:56 smithi154 ceph-mon[36482]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-20T17:52:56.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:56 smithi154 ceph-mon[36482]: osdmap e31: 5 total, 4 up, 5 in 2024-05-20T17:52:56.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:56.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:56 smithi154 ceph-mon[36482]: from='osd.4 [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:52:56.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:56 smithi154 ceph-mon[36482]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:52:56.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:56 smithi154 ceph-mon[36482]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:56.400 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:56 smithi204 ceph-mon[37076]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-20T17:52:56.400 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:56 smithi204 ceph-mon[37076]: osdmap e31: 5 total, 4 up, 5 in 2024-05-20T17:52:56.400 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:56.400 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:56 smithi204 ceph-mon[37076]: from='osd.4 [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:52:56.400 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:56 smithi204 ceph-mon[37076]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:52:56.400 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:56 smithi204 ceph-mon[37076]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:56.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:56 smithi129 ceph-mon[32029]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-20T17:52:56.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:56 smithi129 ceph-mon[32029]: osdmap e31: 5 total, 4 up, 5 in 2024-05-20T17:52:56.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:56.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:56 smithi129 ceph-mon[32029]: from='osd.4 [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:52:56.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:56 smithi129 ceph-mon[32029]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:52:56.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:56 smithi129 ceph-mon[32029]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:56.894 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 17:52:56 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T17:52:56.586+0000 7f5323422700 -1 osd.4 0 waiting for initial osdmap 2024-05-20T17:52:56.895 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 17:52:56 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T17:52:56.592+0000 7f531b1cf700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:52:56.975 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-20T17:52:56.975 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-20T17:52:56.976 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-05-20T17:52:56.976 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-05-20T17:52:56.976 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113609 s, 923 MB/s 2024-05-20T17:52:56.976 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-05-20T17:52:57.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:57 smithi204 ceph-mon[37076]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:52:57.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:57 smithi204 ceph-mon[37076]: osdmap e32: 5 total, 4 up, 5 in 2024-05-20T17:52:57.280 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:57 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:57.280 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:57 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:57.280 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:57 smithi204 ceph-mon[37076]: from='osd.4 ' entity='osd.4' 2024-05-20T17:52:57.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:57 smithi129 ceph-mon[32029]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:52:57.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:57 smithi129 ceph-mon[32029]: osdmap e32: 5 total, 4 up, 5 in 2024-05-20T17:52:57.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:57 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:57.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:57 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:57.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:57 smithi129 ceph-mon[32029]: from='osd.4 ' entity='osd.4' 2024-05-20T17:52:57.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:57 smithi154 ceph-mon[36482]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:52:57.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:57 smithi154 ceph-mon[36482]: osdmap e32: 5 total, 4 up, 5 in 2024-05-20T17:52:57.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:57 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:57.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:57 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:57.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:57 smithi154 ceph-mon[36482]: from='osd.4 ' entity='osd.4' 2024-05-20T17:52:57.587 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_3 2024-05-20T17:52:57.824 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:52:58.196 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:52:58.196 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:52:58.196 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:58.196 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: osd.4 [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593] boot 2024-05-20T17:52:58.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: osdmap e33: 5 total, 5 up, 5 in 2024-05-20T17:52:58.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:58.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:58.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:58.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:58.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:58.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:58.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:58.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:58.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:58.197 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:58.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:52:58.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:52:58.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:58.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: osd.4 [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593] boot 2024-05-20T17:52:58.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: osdmap e33: 5 total, 5 up, 5 in 2024-05-20T17:52:58.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:58.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:58.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:58.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:58.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:58.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:58.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:58.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:58.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:58.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:58.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:52:58.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:52:58.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:58.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: osd.4 [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593] boot 2024-05-20T17:52:58.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: osdmap e33: 5 total, 5 up, 5 in 2024-05-20T17:52:58.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:52:58.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-20T17:52:58.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:58.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:58.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:58.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:52:58.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:58.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:52:58.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:52:58.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:52:59.461 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.460+0000 7f74326bc700 1 -- 172.21.15.204:0/583266928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742c103860 msgr2=0x7f742c10b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:59.461 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.460+0000 7f74326bc700 1 --2- 172.21.15.204:0/583266928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742c103860 0x7f742c10b570 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f742400b0a0 tx=0x7f742400b3b0 comp rx=0 tx=0).stop 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.460+0000 7f74326bc700 1 -- 172.21.15.204:0/583266928 shutdown_connections 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.460+0000 7f74326bc700 1 --2- 172.21.15.204:0/583266928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c104830 0x7f742c104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.460+0000 7f74326bc700 1 --2- 172.21.15.204:0/583266928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742c103860 0x7f742c10b570 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.460+0000 7f74326bc700 1 --2- 172.21.15.204:0/583266928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f742c102de0 0x7f742c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.460+0000 7f74326bc700 1 -- 172.21.15.204:0/583266928 >> 172.21.15.204:0/583266928 conn(0x7f742c0fe850 msgr2=0x7f742c100ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.460+0000 7f74326bc700 1 -- 172.21.15.204:0/583266928 shutdown_connections 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.460+0000 7f74326bc700 1 -- 172.21.15.204:0/583266928 wait complete. 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.460+0000 7f74326bc700 1 Processor -- start 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.460+0000 7f74326bc700 1 -- start start 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f74326bc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c102de0 0x7f742c0773a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f74326bc700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f742c103860 0x7f742c075aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:59.462 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f74326bc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742c104830 0x7f742c075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f74326bc700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f742c10eec0 con 0x7f742c104830 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f74326bc700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f742c10ed40 con 0x7f742c102de0 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f74326bc700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f742c10f040 con 0x7f742c103860 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f742b7fe700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f742c103860 0x7f742c075aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f742bfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c102de0 0x7f742c0773a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f742bfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c102de0 0x7f742c0773a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.204:35334/0 (socket says 172.21.15.204:35334) 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f742b7fe700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f742c103860 0x7f742c075aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:52550/0 (socket says 172.21.15.204:52550) 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f742bfff700 1 -- 172.21.15.204:0/3175425527 learned_addr learned my addr 172.21.15.204:0/3175425527 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f7430c59700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742c104830 0x7f742c075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f742bfff700 1 -- 172.21.15.204:0/3175425527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f742c103860 msgr2=0x7f742c075aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f742bfff700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f742c103860 0x7f742c075aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f742bfff700 1 -- 172.21.15.204:0/3175425527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742c104830 msgr2=0x7f742c075fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:52:59.463 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f742bfff700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742c104830 0x7f742c075fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:52:59.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f742bfff700 1 -- 172.21.15.204:0/3175425527 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742c0766f0 con 0x7f742c102de0 2024-05-20T17:52:59.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f742bfff700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c102de0 0x7f742c0773a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f742000ea10 tx=0x7f742000ed20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:52:59.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.461+0000 7f74297fa700 1 -- 172.21.15.204:0/3175425527 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742000cbd0 con 0x7f742c102de0 2024-05-20T17:52:59.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.462+0000 7f74297fa700 1 -- 172.21.15.204:0/3175425527 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f742000cd70 con 0x7f742c102de0 2024-05-20T17:52:59.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.462+0000 7f74326bc700 1 -- 172.21.15.204:0/3175425527 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742c1a39e0 con 0x7f742c102de0 2024-05-20T17:52:59.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.462+0000 7f74297fa700 1 -- 172.21.15.204:0/3175425527 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74200186e0 con 0x7f742c102de0 2024-05-20T17:52:59.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.462+0000 7f74326bc700 1 -- 172.21.15.204:0/3175425527 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f742c1a3ec0 con 0x7f742c102de0 2024-05-20T17:52:59.465 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.462+0000 7f74326bc700 1 -- 172.21.15.204:0/3175425527 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f742c050170 con 0x7f742c102de0 2024-05-20T17:52:59.465 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.464+0000 7f74297fa700 1 -- 172.21.15.204:0/3175425527 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f742001f020 con 0x7f742c102de0 2024-05-20T17:52:59.465 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.464+0000 7f74297fa700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f74140a2ca0 0x7f74140a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:52:59.465 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.464+0000 7f74297fa700 1 -- 172.21.15.204:0/3175425527 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f74200ce4c0 con 0x7f742c102de0 2024-05-20T17:52:59.465 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.465+0000 7f742b7fe700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f74140a2ca0 0x7f74140a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:52:59.465 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.465+0000 7f742b7fe700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f74140a2ca0 0x7f74140a51b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f742c076d50 tx=0x7f741c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:52:59.467 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.468+0000 7f74297fa700 1 -- 172.21.15.204:0/3175425527 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f742009c3f0 con 0x7f742c102de0 2024-05-20T17:52:59.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:52:59.673+0000 7f74326bc700 1 -- 172.21.15.204:0/3175425527 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f742c02e010 con 0x7f74140a2ca0 2024-05-20T17:52:59.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:59 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi154 2024-05-20T17:52:59.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:59 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi154 to 8544M 2024-05-20T17:52:59.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:52:59 smithi154 ceph-mon[36482]: osdmap e34: 5 total, 5 up, 5 in 2024-05-20T17:52:59.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:59 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi154 2024-05-20T17:52:59.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:59 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi154 to 8544M 2024-05-20T17:52:59.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:52:59 smithi129 ceph-mon[32029]: osdmap e34: 5 total, 5 up, 5 in 2024-05-20T17:52:59.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:59 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi154 2024-05-20T17:52:59.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:59 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi154 to 8544M 2024-05-20T17:52:59.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:52:59 smithi204 ceph-mon[37076]: osdmap e34: 5 total, 5 up, 5 in 2024-05-20T17:53:00.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:00 smithi154 ceph-mon[36482]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:00.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:00 smithi154 ceph-mon[36482]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:53:00.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:53:00.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:53:00.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:00 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:00.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:00 smithi129 ceph-mon[32029]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:00.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:00 smithi129 ceph-mon[32029]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:53:00.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:53:00.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:53:00.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:00 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:00.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:00 smithi204 ceph-mon[37076]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:00.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:00 smithi204 ceph-mon[37076]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:53:00.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:53:00.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:53:00.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:00 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:01.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:01 smithi204 ceph-mon[37076]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:02.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:01 smithi154 ceph-mon[36482]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:02.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:01 smithi129 ceph-mon[32029]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:03.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:02 smithi154 ceph-mon[36482]: from='client.? 172.21.15.204:0/388954972' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dbdb424-101b-488e-826c-fac4b7f35dfa"}]: dispatch 2024-05-20T17:53:03.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:02 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dbdb424-101b-488e-826c-fac4b7f35dfa"}]: dispatch 2024-05-20T17:53:03.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:02 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9dbdb424-101b-488e-826c-fac4b7f35dfa"}]': finished 2024-05-20T17:53:03.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:02 smithi154 ceph-mon[36482]: osdmap e35: 6 total, 5 up, 6 in 2024-05-20T17:53:03.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:02 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:03.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:02 smithi129 ceph-mon[32029]: from='client.? 172.21.15.204:0/388954972' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dbdb424-101b-488e-826c-fac4b7f35dfa"}]: dispatch 2024-05-20T17:53:03.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:02 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dbdb424-101b-488e-826c-fac4b7f35dfa"}]: dispatch 2024-05-20T17:53:03.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:02 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9dbdb424-101b-488e-826c-fac4b7f35dfa"}]': finished 2024-05-20T17:53:03.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:02 smithi129 ceph-mon[32029]: osdmap e35: 6 total, 5 up, 6 in 2024-05-20T17:53:03.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:03.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:02 smithi204 ceph-mon[37076]: from='client.? 172.21.15.204:0/388954972' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dbdb424-101b-488e-826c-fac4b7f35dfa"}]: dispatch 2024-05-20T17:53:03.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:02 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dbdb424-101b-488e-826c-fac4b7f35dfa"}]: dispatch 2024-05-20T17:53:03.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:02 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9dbdb424-101b-488e-826c-fac4b7f35dfa"}]': finished 2024-05-20T17:53:03.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:02 smithi204 ceph-mon[37076]: osdmap e35: 6 total, 5 up, 6 in 2024-05-20T17:53:03.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:02 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:04.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:03 smithi154 ceph-mon[36482]: from='client.? 172.21.15.204:0/3977885095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:53:04.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:03 smithi154 ceph-mon[36482]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:04.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:03 smithi129 ceph-mon[32029]: from='client.? 172.21.15.204:0/3977885095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:53:04.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:03 smithi129 ceph-mon[32029]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:04.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:03 smithi204 ceph-mon[37076]: from='client.? 172.21.15.204:0/3977885095' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:53:04.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:03 smithi204 ceph-mon[37076]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:06.053 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:05 smithi204 ceph-mon[37076]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:06.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:05 smithi154 ceph-mon[36482]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:06.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:05 smithi129 ceph-mon[32029]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:08.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:07 smithi154 ceph-mon[36482]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:08.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:07 smithi129 ceph-mon[32029]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:08.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:07 smithi204 ceph-mon[37076]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:09.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:09 smithi204 ceph-mon[37076]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:10.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:09 smithi154 ceph-mon[36482]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:10.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:09 smithi129 ceph-mon[32029]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:11.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:10 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-20T17:53:11.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:10 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:11.084 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:10 smithi204 ceph-mon[37076]: Deploying daemon osd.5 on smithi204 2024-05-20T17:53:11.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:10 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-20T17:53:11.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:10 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:11.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:10 smithi154 ceph-mon[36482]: Deploying daemon osd.5 on smithi204 2024-05-20T17:53:11.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:10 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-20T17:53:11.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:10 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:11.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:10 smithi129 ceph-mon[32029]: Deploying daemon osd.5 on smithi204 2024-05-20T17:53:12.085 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:11 smithi204 ceph-mon[37076]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:12.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:11 smithi154 ceph-mon[36482]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:12.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:11 smithi129 ceph-mon[32029]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:14.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:13 smithi154 ceph-mon[36482]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:14.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:13 smithi129 ceph-mon[32029]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:14.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:13 smithi204 ceph-mon[37076]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:16.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:16 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:16.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:16 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:16.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:16 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:16.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:16 smithi204 ceph-mon[37076]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:16.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:16 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:16.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:16 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:16.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:16 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:16.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:16 smithi154 ceph-mon[36482]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:16.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:16 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:16.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:16 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:16.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:16 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:16.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:16 smithi129 ceph-mon[32029]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:17.942 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:17 smithi129 ceph-mon[32029]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:18.056 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:17 smithi204 ceph-mon[37076]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:18.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:17 smithi154 ceph-mon[36482]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:19.236 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.237+0000 7f74297fa700 1 -- 172.21.15.204:0/3175425527 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f742c02e010 con 0x7f74140a2ca0 2024-05-20T17:53:19.237 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 5 on host 'smithi204' 2024-05-20T17:53:19.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 -- 172.21.15.204:0/3175425527 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f74140a2ca0 msgr2=0x7f74140a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:53:19.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f74140a2ca0 0x7f74140a51b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f742c076d50 tx=0x7f741c009260 comp rx=0 tx=0).stop 2024-05-20T17:53:19.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 -- 172.21.15.204:0/3175425527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c102de0 msgr2=0x7f742c0773a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:53:19.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c102de0 0x7f742c0773a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f742000ea10 tx=0x7f742000ed20 comp rx=0 tx=0).stop 2024-05-20T17:53:19.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 -- 172.21.15.204:0/3175425527 shutdown_connections 2024-05-20T17:53:19.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f74140a2ca0 0x7f74140a51b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:19.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f742c102de0 0x7f742c0773a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:19.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742c104830 0x7f742c075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:19.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 --2- 172.21.15.204:0/3175425527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f742c103860 0x7f742c075aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:19.240 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 -- 172.21.15.204:0/3175425527 >> 172.21.15.204:0/3175425527 conn(0x7f742c0fe850 msgr2=0x7f742c107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:53:19.240 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 -- 172.21.15.204:0/3175425527 shutdown_connections 2024-05-20T17:53:19.240 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:53:19.239+0000 7f74326bc700 1 -- 172.21.15.204:0/3175425527 wait complete. 2024-05-20T17:53:19.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:19 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:19 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:19 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:19.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:19 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:19.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:19 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:19 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:19 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:19.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:19 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:19 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:19 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:19 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:19.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:19 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:19.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:19 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:19 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:19 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:19.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:19 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:19 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:19 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:19 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:19.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:19 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:19.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:19 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:19 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:19 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:19.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:19 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:19.908 DEBUG:teuthology.orchestra.run.smithi204:osd.5> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.5.service 2024-05-20T17:53:19.911 INFO:tasks.cephadm:Deploying osd.6 on smithi129 with /dev/vg_nvme/lv_2... 2024-05-20T17:53:19.911 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-20T17:53:20.160 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:53:20.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:20 smithi204 ceph-mon[37076]: from='osd.5 [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-20T17:53:20.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:20 smithi204 ceph-mon[37076]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-20T17:53:20.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:20 smithi204 ceph-mon[37076]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:20.594 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:20 smithi129 ceph-mon[32029]: from='osd.5 [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-20T17:53:20.594 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:20 smithi129 ceph-mon[32029]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-20T17:53:20.594 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:20 smithi129 ceph-mon[32029]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:20.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:20 smithi154 ceph-mon[36482]: from='osd.5 [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-20T17:53:20.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:20 smithi154 ceph-mon[36482]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-20T17:53:20.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:20 smithi154 ceph-mon[36482]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:21.386 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:21 smithi204 ceph-mon[37076]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-20T17:53:21.386 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:21 smithi204 ceph-mon[37076]: osdmap e36: 6 total, 5 up, 6 in 2024-05-20T17:53:21.386 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:21.386 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:21 smithi204 ceph-mon[37076]: from='osd.5 [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:53:21.386 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:21 smithi204 ceph-mon[37076]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:53:21.587 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:21 smithi129 ceph-mon[32029]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-20T17:53:21.587 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:21 smithi129 ceph-mon[32029]: osdmap e36: 6 total, 5 up, 6 in 2024-05-20T17:53:21.588 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:21.588 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:21 smithi129 ceph-mon[32029]: from='osd.5 [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:53:21.588 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:21 smithi129 ceph-mon[32029]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:53:21.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:21 smithi154 ceph-mon[36482]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-20T17:53:21.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:21 smithi154 ceph-mon[36482]: osdmap e36: 6 total, 5 up, 6 in 2024-05-20T17:53:21.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:21.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:21 smithi154 ceph-mon[36482]: from='osd.5 [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:53:21.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:21 smithi154 ceph-mon[36482]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:53:22.181 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 17:53:21 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T17:53:21.819+0000 7ff858126700 -1 osd.5 0 waiting for initial osdmap 2024-05-20T17:53:22.181 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 17:53:21 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T17:53:21.827+0000 7ff84fed3700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:53:22.327 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-20T17:53:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-20T17:53:22.328 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-05-20T17:53:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-05-20T17:53:22.328 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112292 s, 934 MB/s 2024-05-20T17:53:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-05-20T17:53:22.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:22 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:53:22.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:22 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:53:22.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:22 smithi154 ceph-mon[36482]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:53:22.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:22 smithi154 ceph-mon[36482]: osdmap e37: 6 total, 5 up, 6 in 2024-05-20T17:53:22.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:22 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:22.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:22 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:22.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:22 smithi154 ceph-mon[36482]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:22.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:22 smithi154 ceph-mon[36482]: from='osd.5 ' entity='osd.5' 2024-05-20T17:53:22.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:22 smithi154 ceph-mon[36482]: osd.5 [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578] boot 2024-05-20T17:53:22.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:22 smithi154 ceph-mon[36482]: osdmap e38: 6 total, 6 up, 6 in 2024-05-20T17:53:22.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:22 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:22.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:22 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:53:22.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:22 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:53:22.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:22 smithi129 ceph-mon[32029]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:53:22.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:22 smithi129 ceph-mon[32029]: osdmap e37: 6 total, 5 up, 6 in 2024-05-20T17:53:22.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:22 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:22.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:22 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:22.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:22 smithi129 ceph-mon[32029]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:22.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:22 smithi129 ceph-mon[32029]: from='osd.5 ' entity='osd.5' 2024-05-20T17:53:22.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:22 smithi129 ceph-mon[32029]: osd.5 [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578] boot 2024-05-20T17:53:22.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:22 smithi129 ceph-mon[32029]: osdmap e38: 6 total, 6 up, 6 in 2024-05-20T17:53:22.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:22 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:22.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:22 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:53:22.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:22 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:53:22.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:22 smithi204 ceph-mon[37076]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:53:22.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:22 smithi204 ceph-mon[37076]: osdmap e37: 6 total, 5 up, 6 in 2024-05-20T17:53:22.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:22 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:22.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:22 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:22.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:22 smithi204 ceph-mon[37076]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-20T17:53:22.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:22 smithi204 ceph-mon[37076]: from='osd.5 ' entity='osd.5' 2024-05-20T17:53:22.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:22 smithi204 ceph-mon[37076]: osd.5 [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578] boot 2024-05-20T17:53:22.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:22 smithi204 ceph-mon[37076]: osdmap e38: 6 total, 6 up, 6 in 2024-05-20T17:53:22.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:22 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:53:22.968 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi129:vg_nvme/lv_2 2024-05-20T17:53:23.208 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:53:24.610 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi204 2024-05-20T17:53:24.610 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.610 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.610 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:24.611 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:24.611 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi204 to 8544M 2024-05-20T17:53:24.611 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.611 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:24.611 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:24.611 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.611 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: osdmap e39: 6 total, 6 up, 6 in 2024-05-20T17:53:24.611 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:24 smithi129 ceph-mon[32029]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:24.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi204 2024-05-20T17:53:24.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:24.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:24.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi204 to 8544M 2024-05-20T17:53:24.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:24.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:24.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: osdmap e39: 6 total, 6 up, 6 in 2024-05-20T17:53:24.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:24 smithi154 ceph-mon[36482]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi204 2024-05-20T17:53:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:24.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:24.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi204 to 8544M 2024-05-20T17:53:24.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:24.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:24.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:24.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: osdmap e39: 6 total, 6 up, 6 in 2024-05-20T17:53:24.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:24 smithi204 ceph-mon[37076]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:24.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.867+0000 7f22f99f1700 1 -- 172.21.15.129:0/3376928572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22f4103cd0 msgr2=0x7f22f41060f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:53:24.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.867+0000 7f22f99f1700 1 --2- 172.21.15.129:0/3376928572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22f4103cd0 0x7f22f41060f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009990 tx=0x7f22e4009ca0 comp rx=0 tx=0).stop 2024-05-20T17:53:24.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.867+0000 7f22f99f1700 1 -- 172.21.15.129:0/3376928572 shutdown_connections 2024-05-20T17:53:24.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.867+0000 7f22f99f1700 1 --2- 172.21.15.129:0/3376928572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4108f90 0x7f22f410b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:24.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.868+0000 7f22f99f1700 1 --2- 172.21.15.129:0/3376928572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22f4103cd0 0x7f22f41060f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:24.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.868+0000 7f22f99f1700 1 --2- 172.21.15.129:0/3376928572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22f4106630 0x7f22f4108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:24.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.868+0000 7f22f99f1700 1 -- 172.21.15.129:0/3376928572 >> 172.21.15.129:0/3376928572 conn(0x7f22f40fd710 msgr2=0x7f22f40ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:53:24.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.868+0000 7f22f99f1700 1 -- 172.21.15.129:0/3376928572 shutdown_connections 2024-05-20T17:53:24.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.868+0000 7f22f99f1700 1 -- 172.21.15.129:0/3376928572 wait complete. 2024-05-20T17:53:24.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.868+0000 7f22f99f1700 1 Processor -- start 2024-05-20T17:53:24.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.868+0000 7f22f99f1700 1 -- start start 2024-05-20T17:53:24.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.868+0000 7f22f99f1700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22f4103cd0 0x7f22f41998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:53:24.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.868+0000 7f22f99f1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4106630 0x7f22f4199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:53:24.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.868+0000 7f22f99f1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22f4108f90 0x7f22f41a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:53:24.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.868+0000 7f22f99f1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410ede0 con 0x7f22f4108f90 2024-05-20T17:53:24.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f99f1700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410ec60 con 0x7f22f4106630 2024-05-20T17:53:24.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f99f1700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410ef60 con 0x7f22f4103cd0 2024-05-20T17:53:24.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f37fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22f4108f90 0x7f22f41a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:53:24.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f37fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22f4108f90 0x7f22f41a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57154/0 (socket says 172.21.15.129:57154) 2024-05-20T17:53:24.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f37fe700 1 -- 172.21.15.129:0/3502182656 learned_addr learned my addr 172.21.15.129:0/3502182656 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:53:24.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22ebfff700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4106630 0x7f22f4199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:53:24.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f2ffd700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22f4103cd0 0x7f22f41998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:53:24.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f37fe700 1 -- 172.21.15.129:0/3502182656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22f4103cd0 msgr2=0x7f22f41998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:53:24.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f37fe700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22f4103cd0 0x7f22f41998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:24.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f37fe700 1 -- 172.21.15.129:0/3502182656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4106630 msgr2=0x7f22f4199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:53:24.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f37fe700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4106630 0x7f22f4199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:24.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f37fe700 1 -- 172.21.15.129:0/3502182656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f41a13c0 con 0x7f22f4108f90 2024-05-20T17:53:24.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f37fe700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22f4108f90 0x7f22f41a0dd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f22ec007d70 tx=0x7f22ec00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:53:24.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f89ef700 1 -- 172.21.15.129:0/3502182656 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec016020 con 0x7f22f4108f90 2024-05-20T17:53:24.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f89ef700 1 -- 172.21.15.129:0/3502182656 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22ec00ac90 con 0x7f22f4108f90 2024-05-20T17:53:24.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f99f1700 1 -- 172.21.15.129:0/3502182656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f41a16b0 con 0x7f22f4108f90 2024-05-20T17:53:24.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f89ef700 1 -- 172.21.15.129:0/3502182656 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec0115c0 con 0x7f22f4108f90 2024-05-20T17:53:24.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.869+0000 7f22f99f1700 1 -- 172.21.15.129:0/3502182656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f41a1c20 con 0x7f22f4108f90 2024-05-20T17:53:24.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.870+0000 7f22f99f1700 1 -- 172.21.15.129:0/3502182656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f4050170 con 0x7f22f4108f90 2024-05-20T17:53:24.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.872+0000 7f22f89ef700 1 -- 172.21.15.129:0/3502182656 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f22ec019020 con 0x7f22f4108f90 2024-05-20T17:53:24.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.872+0000 7f22f89ef700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f22e00a2c20 0x7f22e00a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:53:24.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.873+0000 7f22f2ffd700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f22e00a2c20 0x7f22e00a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:53:24.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.873+0000 7f22f89ef700 1 -- 172.21.15.129:0/3502182656 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4533+0+0 (secure 0 0 0) 0x7f22ec0ce590 con 0x7f22f4108f90 2024-05-20T17:53:24.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.873+0000 7f22f2ffd700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f22e00a2c20 0x7f22e00a5130 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f22e400b5c0 tx=0x7f22e401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:53:24.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:24.876+0000 7f22f89ef700 1 -- 172.21.15.129:0/3502182656 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f22ec020020 con 0x7f22f4108f90 2024-05-20T17:53:25.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:25.084+0000 7f22f99f1700 1 -- 172.21.15.129:0/3502182656 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f22f4062a80 con 0x7f22e00a2c20 2024-05-20T17:53:25.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:25 smithi154 ceph-mon[36482]: osdmap e40: 6 total, 6 up, 6 in 2024-05-20T17:53:25.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:25 smithi154 ceph-mon[36482]: osdmap e41: 6 total, 6 up, 6 in 2024-05-20T17:53:25.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:25 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:53:25.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:25 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:53:25.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:25 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:25.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:25 smithi129 ceph-mon[32029]: osdmap e40: 6 total, 6 up, 6 in 2024-05-20T17:53:25.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:25 smithi129 ceph-mon[32029]: osdmap e41: 6 total, 6 up, 6 in 2024-05-20T17:53:25.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:25 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:53:25.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:25 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:53:25.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:25 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:25.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:25 smithi204 ceph-mon[37076]: osdmap e40: 6 total, 6 up, 6 in 2024-05-20T17:53:25.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:25 smithi204 ceph-mon[37076]: osdmap e41: 6 total, 6 up, 6 in 2024-05-20T17:53:25.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:25 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:53:25.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:25 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:53:25.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:25 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:26.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:26 smithi154 ceph-mon[36482]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:53:26.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:26 smithi154 ceph-mon[36482]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:53:26.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:26 smithi154 ceph-mon[36482]: osdmap e42: 6 total, 6 up, 6 in 2024-05-20T17:53:26.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:26 smithi129 ceph-mon[32029]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:53:26.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:26 smithi129 ceph-mon[32029]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:53:26.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:26 smithi129 ceph-mon[32029]: osdmap e42: 6 total, 6 up, 6 in 2024-05-20T17:53:26.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:26 smithi204 ceph-mon[37076]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:53:26.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:26 smithi204 ceph-mon[37076]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:53:26.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:26 smithi204 ceph-mon[37076]: osdmap e42: 6 total, 6 up, 6 in 2024-05-20T17:53:28.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:27 smithi154 ceph-mon[36482]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-05-20T17:53:28.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:27 smithi129 ceph-mon[32029]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-05-20T17:53:28.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:27 smithi204 ceph-mon[37076]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-05-20T17:53:29.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:28 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2526553951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b87d90ab-d39b-464a-b835-0a1cc50d0eb2"}]: dispatch 2024-05-20T17:53:29.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:28 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2526553951' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b87d90ab-d39b-464a-b835-0a1cc50d0eb2"}]': finished 2024-05-20T17:53:29.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:28 smithi154 ceph-mon[36482]: osdmap e43: 7 total, 6 up, 7 in 2024-05-20T17:53:29.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:28 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:29.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:28 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1738813025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:53:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:28 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2526553951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b87d90ab-d39b-464a-b835-0a1cc50d0eb2"}]: dispatch 2024-05-20T17:53:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:28 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2526553951' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b87d90ab-d39b-464a-b835-0a1cc50d0eb2"}]': finished 2024-05-20T17:53:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:28 smithi129 ceph-mon[32029]: osdmap e43: 7 total, 6 up, 7 in 2024-05-20T17:53:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:28 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:28 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1738813025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:53:29.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:28 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2526553951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b87d90ab-d39b-464a-b835-0a1cc50d0eb2"}]: dispatch 2024-05-20T17:53:29.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:28 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2526553951' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b87d90ab-d39b-464a-b835-0a1cc50d0eb2"}]': finished 2024-05-20T17:53:29.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:28 smithi204 ceph-mon[37076]: osdmap e43: 7 total, 6 up, 7 in 2024-05-20T17:53:29.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:28 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:29.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:28 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1738813025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:53:30.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:29 smithi154 ceph-mon[36482]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-20T17:53:30.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:29 smithi129 ceph-mon[32029]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-20T17:53:30.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:29 smithi204 ceph-mon[37076]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-20T17:53:31.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:31 smithi129 ceph-mon[32029]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-20T17:53:32.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:31 smithi154 ceph-mon[36482]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-20T17:53:32.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:31 smithi204 ceph-mon[37076]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-05-20T17:53:34.100 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:33 smithi129 ceph-mon[32029]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:34.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:33 smithi154 ceph-mon[36482]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:34.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:33 smithi204 ceph-mon[37076]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:35.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-20T17:53:35.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:35.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-20T17:53:35.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:35.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-20T17:53:35.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:36.569 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:36 smithi129 ceph-mon[32029]: Deploying daemon osd.6 on smithi129 2024-05-20T17:53:36.569 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:36 smithi129 ceph-mon[32029]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:36.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:36 smithi154 ceph-mon[36482]: Deploying daemon osd.6 on smithi129 2024-05-20T17:53:36.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:36 smithi154 ceph-mon[36482]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:36.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:36 smithi204 ceph-mon[37076]: Deploying daemon osd.6 on smithi129 2024-05-20T17:53:36.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:36 smithi204 ceph-mon[37076]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:38.041 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:37 smithi129 ceph-mon[32029]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:38.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:37 smithi154 ceph-mon[36482]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:38.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:37 smithi204 ceph-mon[37076]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:40.046 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:39 smithi129 ceph-mon[32029]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:40.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:39 smithi154 ceph-mon[36482]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:40.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:39 smithi204 ceph-mon[37076]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:42.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:42.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:42.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:42.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:42 smithi129 ceph-mon[32029]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:42.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:42.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:42.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:42.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:42 smithi154 ceph-mon[36482]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:42.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:42.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:42.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:42.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:42 smithi204 ceph-mon[37076]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:44.042 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:43 smithi129 ceph-mon[32029]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:44.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:43 smithi154 ceph-mon[36482]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:44.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:43 smithi204 ceph-mon[37076]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:45.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.226+0000 7f22f89ef700 1 -- 172.21.15.129:0/3502182656 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f22f4062a80 con 0x7f22e00a2c20 2024-05-20T17:53:45.225 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 6 on host 'smithi129' 2024-05-20T17:53:45.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 -- 172.21.15.129:0/3502182656 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f22e00a2c20 msgr2=0x7f22e00a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:53:45.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f22e00a2c20 0x7f22e00a5130 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f22e400b5c0 tx=0x7f22e401a040 comp rx=0 tx=0).stop 2024-05-20T17:53:45.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 -- 172.21.15.129:0/3502182656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22f4108f90 msgr2=0x7f22f41a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:53:45.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22f4108f90 0x7f22f41a0dd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f22ec007d70 tx=0x7f22ec00a340 comp rx=0 tx=0).stop 2024-05-20T17:53:45.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 -- 172.21.15.129:0/3502182656 shutdown_connections 2024-05-20T17:53:45.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f22e00a2c20 0x7f22e00a5130 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:45.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22f4108f90 0x7f22f41a0dd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:45.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22f4103cd0 0x7f22f41998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:45.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 --2- 172.21.15.129:0/3502182656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22f4106630 0x7f22f4199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:45.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 -- 172.21.15.129:0/3502182656 >> 172.21.15.129:0/3502182656 conn(0x7f22f40fd710 msgr2=0x7f22f41047c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:53:45.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 -- 172.21.15.129:0/3502182656 shutdown_connections 2024-05-20T17:53:45.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:53:45.228+0000 7f22f99f1700 1 -- 172.21.15.129:0/3502182656 wait complete. 2024-05-20T17:53:45.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:45.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:45.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:45.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:45.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:45.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:45 smithi129 ceph-mon[32029]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-20T17:53:45.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:45.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:45.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:45.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:45.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:45.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:45 smithi204 ceph-mon[37076]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-20T17:53:45.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:45.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:45.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:45.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:45.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:45.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:45 smithi154 ceph-mon[36482]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-20T17:53:46.011 DEBUG:teuthology.orchestra.run.smithi129:osd.6> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.6.service 2024-05-20T17:53:46.014 INFO:tasks.cephadm:Deploying osd.7 on smithi154 with /dev/vg_nvme/lv_2... 2024-05-20T17:53:46.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-20T17:53:46.245 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:46 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:46.245 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:46 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:46.245 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:46 smithi129 ceph-mon[32029]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-20T17:53:46.245 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:46 smithi129 ceph-mon[32029]: osdmap e44: 7 total, 6 up, 7 in 2024-05-20T17:53:46.246 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:46 smithi129 ceph-mon[32029]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:53:46.246 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:46 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:46.246 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:46 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:46.246 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:46 smithi129 ceph-mon[32029]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:46.246 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 17:53:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T17:53:46.196+0000 7f77ec2af700 -1 osd.6 0 waiting for initial osdmap 2024-05-20T17:53:46.246 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 17:53:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T17:53:46.205+0000 7f77e8064700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:53:46.260 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T17:53:46.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:46.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:46.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:46 smithi154 ceph-mon[36482]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-20T17:53:46.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:46 smithi154 ceph-mon[36482]: osdmap e44: 7 total, 6 up, 7 in 2024-05-20T17:53:46.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:46 smithi154 ceph-mon[36482]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:53:46.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:46.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:46.382 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:46 smithi154 ceph-mon[36482]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:46.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:46 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:46.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:46 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:46.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:46 smithi204 ceph-mon[37076]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-20T17:53:46.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:46 smithi204 ceph-mon[37076]: osdmap e44: 7 total, 6 up, 7 in 2024-05-20T17:53:46.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:46 smithi204 ceph-mon[37076]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:53:46.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:46 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:46.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:46 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:53:46.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:46 smithi204 ceph-mon[37076]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:47.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:47 smithi154 ceph-mon[36482]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:53:47.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:47 smithi154 ceph-mon[36482]: osdmap e45: 7 total, 6 up, 7 in 2024-05-20T17:53:47.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:47.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:47.490 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:47 smithi129 ceph-mon[32029]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:53:47.490 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:47 smithi129 ceph-mon[32029]: osdmap e45: 7 total, 6 up, 7 in 2024-05-20T17:53:47.490 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:47.490 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:47.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:47 smithi204 ceph-mon[37076]: from='osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:53:47.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:47 smithi204 ceph-mon[37076]: osdmap e45: 7 total, 6 up, 7 in 2024-05-20T17:53:47.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:47.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:48.349 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-20T17:53:48.350 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-20T17:53:48.350 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-05-20T17:53:48.350 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-05-20T17:53:48.350 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011405 s, 919 MB/s 2024-05-20T17:53:48.350 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-05-20T17:53:48.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:48 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:53:48.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:48 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:53:48.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:48.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:48 smithi154 ceph-mon[36482]: osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067] boot 2024-05-20T17:53:48.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:48 smithi154 ceph-mon[36482]: osdmap e46: 7 total, 7 up, 7 in 2024-05-20T17:53:48.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:48.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:48 smithi154 ceph-mon[36482]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:48.504 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:48 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:53:48.505 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:48 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:53:48.505 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:48.505 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:48 smithi129 ceph-mon[32029]: osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067] boot 2024-05-20T17:53:48.505 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:48 smithi129 ceph-mon[32029]: osdmap e46: 7 total, 7 up, 7 in 2024-05-20T17:53:48.505 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:48.505 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:48 smithi129 ceph-mon[32029]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:48.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:48 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:53:48.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:48 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:53:48.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:48.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:48 smithi204 ceph-mon[37076]: osd.6 [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067] boot 2024-05-20T17:53:48.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:48 smithi204 ceph-mon[37076]: osdmap e46: 7 total, 7 up, 7 in 2024-05-20T17:53:48.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:53:48.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:48 smithi204 ceph-mon[37076]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-20T17:53:49.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi154:vg_nvme/lv_2 2024-05-20T17:53:49.246 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T17:53:49.368 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:49 smithi154 ceph-mon[36482]: osdmap e47: 7 total, 7 up, 7 in 2024-05-20T17:53:49.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:49 smithi129 ceph-mon[32029]: osdmap e47: 7 total, 7 up, 7 in 2024-05-20T17:53:49.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:49 smithi204 ceph-mon[37076]: osdmap e47: 7 total, 7 up, 7 in 2024-05-20T17:53:50.383 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi129 2024-05-20T17:53:50.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:50.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:50.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:50.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi129 to 5696M 2024-05-20T17:53:50.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:50.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:50.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:50 smithi154 ceph-mon[36482]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:50.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi129 2024-05-20T17:53:50.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:50.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:50.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:50.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi129 to 5696M 2024-05-20T17:53:50.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:50.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:50.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:50 smithi129 ceph-mon[32029]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:50.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi129 2024-05-20T17:53:50.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:50.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:50.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:53:50.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi129 to 5696M 2024-05-20T17:53:50.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:50.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:53:50.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:53:50.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:50 smithi204 ceph-mon[37076]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:50.918 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.917+0000 7fa79362d700 1 -- 172.21.15.154:0/2157106432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1020c0 msgr2=0x7fa78c1024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:53:50.919 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.917+0000 7fa79362d700 1 --2- 172.21.15.154:0/2157106432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1020c0 0x7fa78c1024d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa78800b0a0 tx=0x7fa78800b3b0 comp rx=0 tx=0).stop 2024-05-20T17:53:50.919 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.917+0000 7fa79362d700 1 -- 172.21.15.154:0/2157106432 shutdown_connections 2024-05-20T17:53:50.919 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.917+0000 7fa79362d700 1 --2- 172.21.15.154:0/2157106432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1020c0 0x7fa78c1024d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:50.919 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.917+0000 7fa79362d700 1 --2- 172.21.15.154:0/2157106432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa78c103b90 0x7fa78c10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:50.919 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.917+0000 7fa79362d700 1 --2- 172.21.15.154:0/2157106432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa78c1031f0 0x7fa78c103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:50.919 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.917+0000 7fa79362d700 1 -- 172.21.15.154:0/2157106432 >> 172.21.15.154:0/2157106432 conn(0x7fa78c0fd700 msgr2=0x7fa78c0ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:53:50.919 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.918+0000 7fa79362d700 1 -- 172.21.15.154:0/2157106432 shutdown_connections 2024-05-20T17:53:50.919 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.918+0000 7fa79362d700 1 -- 172.21.15.154:0/2157106432 wait complete. 2024-05-20T17:53:50.919 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.918+0000 7fa79362d700 1 Processor -- start 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.918+0000 7fa79362d700 1 -- start start 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.918+0000 7fa79362d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1020c0 0x7fa78c1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.918+0000 7fa79362d700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa78c1031f0 0x7fa78c199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.918+0000 7fa79362d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa78c103b90 0x7fa78c1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.918+0000 7fa79362d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c10edb0 con 0x7fa78c103b90 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.918+0000 7fa79362d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c10ec30 con 0x7fa78c1020c0 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.918+0000 7fa79362d700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c10ef30 con 0x7fa78c1031f0 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.918+0000 7fa7913c9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1020c0 0x7fa78c1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa7913c9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1020c0 0x7fa78c1998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:49890/0 (socket says 172.21.15.154:49890) 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa7913c9700 1 -- 172.21.15.154:0/534036218 learned_addr learned my addr 172.21.15.154:0/534036218 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa790bc8700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa78c1031f0 0x7fa78c199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:53:50.920 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa791bca700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa78c103b90 0x7fa78c1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:53:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa7913c9700 1 -- 172.21.15.154:0/534036218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa78c1031f0 msgr2=0x7fa78c199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:53:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa7913c9700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa78c1031f0 0x7fa78c199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa7913c9700 1 -- 172.21.15.154:0/534036218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa78c103b90 msgr2=0x7fa78c1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:53:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa7913c9700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa78c103b90 0x7fa78c1a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:53:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa7913c9700 1 -- 172.21.15.154:0/534036218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78c1a1390 con 0x7fa78c1020c0 2024-05-20T17:53:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa7913c9700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1020c0 0x7fa78c1998c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa78800b070 tx=0x7fa788004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:53:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa7827fc700 1 -- 172.21.15.154:0/534036218 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa788021070 con 0x7fa78c1020c0 2024-05-20T17:53:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa7827fc700 1 -- 172.21.15.154:0/534036218 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa788007b80 con 0x7fa78c1020c0 2024-05-20T17:53:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa79362d700 1 -- 172.21.15.154:0/534036218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa78c1a1620 con 0x7fa78c1020c0 2024-05-20T17:53:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa7827fc700 1 -- 172.21.15.154:0/534036218 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78801d450 con 0x7fa78c1020c0 2024-05-20T17:53:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.919+0000 7fa79362d700 1 -- 172.21.15.154:0/534036218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa78c1a1ad0 con 0x7fa78c1020c0 2024-05-20T17:53:50.922 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.920+0000 7fa79362d700 1 -- 172.21.15.154:0/534036218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa78c050170 con 0x7fa78c1020c0 2024-05-20T17:53:50.922 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.921+0000 7fa7827fc700 1 -- 172.21.15.154:0/534036218 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa788003a00 con 0x7fa78c1020c0 2024-05-20T17:53:50.922 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.922+0000 7fa7827fc700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa7780a2c20 0x7fa7780a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:53:50.922 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.922+0000 7fa7827fc700 1 -- 172.21.15.154:0/534036218 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fa78800f020 con 0x7fa78c1020c0 2024-05-20T17:53:50.923 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.922+0000 7fa790bc8700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa7780a2c20 0x7fa7780a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:53:50.923 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.923+0000 7fa790bc8700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa7780a2c20 0x7fa7780a5130 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa78c19a970 tx=0x7fa77c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:53:50.925 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:50.926+0000 7fa7827fc700 1 -- 172.21.15.154:0/534036218 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa78802a020 con 0x7fa78c1020c0 2024-05-20T17:53:51.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:53:51.134+0000 7fa79362d700 1 -- 172.21.15.154:0/534036218 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa78c062a80 con 0x7fa7780a2c20 2024-05-20T17:53:51.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:51 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:53:51.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:51 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:53:51.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:51 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:51.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:51 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:53:51.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:51 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:53:51.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:51 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:51.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:51 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:53:51.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:51 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:53:51.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:51 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:53:52.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:52 smithi154 ceph-mon[36482]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:53:52.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:52 smithi154 ceph-mon[36482]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:52.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:52 smithi129 ceph-mon[32029]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:53:52.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:52 smithi129 ceph-mon[32029]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:52.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:52 smithi204 ceph-mon[37076]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:53:52.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:52 smithi204 ceph-mon[37076]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:54.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:53 smithi154 ceph-mon[36482]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:54.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:53 smithi129 ceph-mon[32029]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:54.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:53 smithi204 ceph-mon[37076]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:55.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:54 smithi154 ceph-mon[36482]: from='client.? 172.21.15.154:0/3713505254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e850ec9e-e2a9-4d09-86c3-9f8115e516ad"}]: dispatch 2024-05-20T17:53:55.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:54 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e850ec9e-e2a9-4d09-86c3-9f8115e516ad"}]: dispatch 2024-05-20T17:53:55.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:54 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e850ec9e-e2a9-4d09-86c3-9f8115e516ad"}]': finished 2024-05-20T17:53:55.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:54 smithi154 ceph-mon[36482]: osdmap e48: 8 total, 7 up, 8 in 2024-05-20T17:53:55.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:54 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:53:55.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:54 smithi154 ceph-mon[36482]: from='client.? 172.21.15.154:0/758035947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:53:55.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:54 smithi129 ceph-mon[32029]: from='client.? 172.21.15.154:0/3713505254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e850ec9e-e2a9-4d09-86c3-9f8115e516ad"}]: dispatch 2024-05-20T17:53:55.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:54 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e850ec9e-e2a9-4d09-86c3-9f8115e516ad"}]: dispatch 2024-05-20T17:53:55.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:54 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e850ec9e-e2a9-4d09-86c3-9f8115e516ad"}]': finished 2024-05-20T17:53:55.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:54 smithi129 ceph-mon[32029]: osdmap e48: 8 total, 7 up, 8 in 2024-05-20T17:53:55.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:54 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:53:55.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:54 smithi129 ceph-mon[32029]: from='client.? 172.21.15.154:0/758035947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:53:55.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:54 smithi204 ceph-mon[37076]: from='client.? 172.21.15.154:0/3713505254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e850ec9e-e2a9-4d09-86c3-9f8115e516ad"}]: dispatch 2024-05-20T17:53:55.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:54 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e850ec9e-e2a9-4d09-86c3-9f8115e516ad"}]: dispatch 2024-05-20T17:53:55.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:54 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e850ec9e-e2a9-4d09-86c3-9f8115e516ad"}]': finished 2024-05-20T17:53:55.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:54 smithi204 ceph-mon[37076]: osdmap e48: 8 total, 7 up, 8 in 2024-05-20T17:53:55.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:54 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:53:55.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:54 smithi204 ceph-mon[37076]: from='client.? 172.21.15.154:0/758035947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:53:56.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:55 smithi154 ceph-mon[36482]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:56.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:55 smithi129 ceph-mon[32029]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:56.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:55 smithi204 ceph-mon[37076]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:57.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:57 smithi154 ceph-mon[36482]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:58.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:57 smithi129 ceph-mon[32029]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:53:58.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:57 smithi204 ceph-mon[37076]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:00.111 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:53:59 smithi154 ceph-mon[36482]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:00.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:53:59 smithi129 ceph-mon[32029]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:00.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:53:59 smithi204 ceph-mon[37076]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:01.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:01 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-20T17:54:01.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:01 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:01.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:01 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-20T17:54:01.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:01 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:01.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:01 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-20T17:54:01.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:01 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:02.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:02 smithi154 ceph-mon[36482]: Deploying daemon osd.7 on smithi154 2024-05-20T17:54:02.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:02 smithi154 ceph-mon[36482]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:02.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:02 smithi129 ceph-mon[32029]: Deploying daemon osd.7 on smithi154 2024-05-20T17:54:02.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:02 smithi129 ceph-mon[32029]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:02.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:02 smithi204 ceph-mon[37076]: Deploying daemon osd.7 on smithi154 2024-05-20T17:54:02.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:02 smithi204 ceph-mon[37076]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:04.120 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:03 smithi154 ceph-mon[36482]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:04.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:03 smithi129 ceph-mon[32029]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:04.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:03 smithi204 ceph-mon[37076]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:06.122 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:05 smithi154 ceph-mon[36482]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:06.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:05 smithi129 ceph-mon[32029]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:06.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:05 smithi204 ceph-mon[37076]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:08.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:08.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:08.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:08.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:07 smithi154 ceph-mon[36482]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:08.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:08.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:08.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:08.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:07 smithi129 ceph-mon[32029]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:08.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:08.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:08.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:08.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:07 smithi204 ceph-mon[37076]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:10.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:09 smithi154 ceph-mon[36482]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:10.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:09 smithi129 ceph-mon[32029]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:09 smithi204 ceph-mon[37076]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:10.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.196+0000 7fa7827fc700 1 -- 172.21.15.154:0/534036218 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa78c062a80 con 0x7fa7780a2c20 2024-05-20T17:54:10.196 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 7 on host 'smithi154' 2024-05-20T17:54:10.198 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 -- 172.21.15.154:0/534036218 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa7780a2c20 msgr2=0x7fa7780a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:54:10.198 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa7780a2c20 0x7fa7780a5130 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa78c19a970 tx=0x7fa77c009260 comp rx=0 tx=0).stop 2024-05-20T17:54:10.198 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 -- 172.21.15.154:0/534036218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1020c0 msgr2=0x7fa78c1998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:54:10.199 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1020c0 0x7fa78c1998c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa78800b070 tx=0x7fa788004d00 comp rx=0 tx=0).stop 2024-05-20T17:54:10.199 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 -- 172.21.15.154:0/534036218 shutdown_connections 2024-05-20T17:54:10.199 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa7780a2c20 0x7fa7780a5130 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:10.199 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa78c1020c0 0x7fa78c1998c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:10.199 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa78c103b90 0x7fa78c1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:10.199 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 --2- 172.21.15.154:0/534036218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa78c1031f0 0x7fa78c199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:10.199 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 -- 172.21.15.154:0/534036218 >> 172.21.15.154:0/534036218 conn(0x7fa78c0fd700 msgr2=0x7fa78c106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:54:10.199 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 -- 172.21.15.154:0/534036218 shutdown_connections 2024-05-20T17:54:10.199 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:54:10.199+0000 7fa79362d700 1 -- 172.21.15.154:0/534036218 wait complete. 2024-05-20T17:54:10.782 DEBUG:teuthology.orchestra.run.smithi154:osd.7> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.7.service 2024-05-20T17:54:10.784 INFO:tasks.cephadm:Deploying osd.8 on smithi204 with /dev/vg_nvme/lv_2... 2024-05-20T17:54:10.784 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-20T17:54:11.033 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:54:11.404 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:11 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:11.405 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:11 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:11.405 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:11 smithi204 ceph-mon[37076]: from='osd.7 [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-20T17:54:11.405 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:11 smithi204 ceph-mon[37076]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-20T17:54:11.405 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:11 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:11.405 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:11 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:11 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:11.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:11 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:11 smithi154 ceph-mon[36482]: from='osd.7 [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-20T17:54:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:11 smithi154 ceph-mon[36482]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-20T17:54:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:11 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:11.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:11 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:11.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:11 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:11.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:11 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:11.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:11 smithi129 ceph-mon[32029]: from='osd.7 [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-20T17:54:11.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:11 smithi129 ceph-mon[32029]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-20T17:54:11.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:11 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:11.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:11 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:12.463 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:12 smithi204 ceph-mon[37076]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-20T17:54:12.463 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:12 smithi204 ceph-mon[37076]: osdmap e49: 8 total, 7 up, 8 in 2024-05-20T17:54:12.463 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:12.463 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:12 smithi204 ceph-mon[37076]: from='osd.7 [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:54:12.463 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:12 smithi204 ceph-mon[37076]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:54:12.463 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:12 smithi204 ceph-mon[37076]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:12.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:12 smithi154 ceph-mon[36482]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-20T17:54:12.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:12 smithi154 ceph-mon[36482]: osdmap e49: 8 total, 7 up, 8 in 2024-05-20T17:54:12.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:12.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:12 smithi154 ceph-mon[36482]: from='osd.7 [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:54:12.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:12 smithi154 ceph-mon[36482]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:54:12.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:12 smithi154 ceph-mon[36482]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:12.509 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 17:54:12 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T17:54:12.201+0000 7f2f27a17700 -1 osd.7 0 waiting for initial osdmap 2024-05-20T17:54:12.509 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 17:54:12 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T17:54:12.211+0000 7f2f237cc700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:54:12.667 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:12 smithi129 ceph-mon[32029]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-20T17:54:12.667 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:12 smithi129 ceph-mon[32029]: osdmap e49: 8 total, 7 up, 8 in 2024-05-20T17:54:12.667 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:12.667 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:12 smithi129 ceph-mon[32029]: from='osd.7 [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:54:12.668 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:12 smithi129 ceph-mon[32029]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:54:12.668 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:12 smithi129 ceph-mon[32029]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-20T17:54:13.311 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-20T17:54:13.312 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-20T17:54:13.312 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-05-20T17:54:13.312 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-05-20T17:54:13.312 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112275 s, 934 MB/s 2024-05-20T17:54:13.312 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-05-20T17:54:13.613 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:13 smithi204 ceph-mon[37076]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:54:13.613 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:13 smithi204 ceph-mon[37076]: osdmap e50: 8 total, 7 up, 8 in 2024-05-20T17:54:13.613 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:13 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:13.613 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:13 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:13.616 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:13 smithi154 ceph-mon[36482]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:54:13.616 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:13 smithi154 ceph-mon[36482]: osdmap e50: 8 total, 7 up, 8 in 2024-05-20T17:54:13.616 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:13 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:13.616 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:13 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:13.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:13 smithi129 ceph-mon[32029]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:54:13.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:13 smithi129 ceph-mon[32029]: osdmap e50: 8 total, 7 up, 8 in 2024-05-20T17:54:13.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:13 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:13.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:13 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:13.999 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_2 2024-05-20T17:54:14.240 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:54:14.362 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:14 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:54:14.362 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:14 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:54:14.362 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:14 smithi204 ceph-mon[37076]: osd.7 [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271] boot 2024-05-20T17:54:14.363 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:14 smithi204 ceph-mon[37076]: osdmap e51: 8 total, 8 up, 8 in 2024-05-20T17:54:14.363 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:14 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:14.363 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:14 smithi204 ceph-mon[37076]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:14.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:14 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:54:14.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:14 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:54:14.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:14 smithi154 ceph-mon[36482]: osd.7 [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271] boot 2024-05-20T17:54:14.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:14 smithi154 ceph-mon[36482]: osdmap e51: 8 total, 8 up, 8 in 2024-05-20T17:54:14.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:14 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:14.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:14 smithi154 ceph-mon[36482]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:14.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:14 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:54:14.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:14 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:54:14.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:14 smithi129 ceph-mon[32029]: osd.7 [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271] boot 2024-05-20T17:54:14.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:14 smithi129 ceph-mon[32029]: osdmap e51: 8 total, 8 up, 8 in 2024-05-20T17:54:14.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:14 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:54:14.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:14 smithi129 ceph-mon[32029]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:15.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: osdmap e52: 8 total, 8 up, 8 in 2024-05-20T17:54:15.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi154 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi154 to 5696M 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:15.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:15.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: osdmap e52: 8 total, 8 up, 8 in 2024-05-20T17:54:15.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi154 2024-05-20T17:54:15.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:15.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:15.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:15.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi154 to 5696M 2024-05-20T17:54:15.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:15.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:15.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:15.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:15.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:15.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: osdmap e52: 8 total, 8 up, 8 in 2024-05-20T17:54:15.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi154 2024-05-20T17:54:15.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:15.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi154 to 5696M 2024-05-20T17:54:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:15.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:15.794 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.795+0000 7f7a68d95700 1 -- 172.21.15.204:0/2431922160 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a64103ba0 msgr2=0x7f7a6410a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:54:15.794 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.795+0000 7f7a68d95700 1 --2- 172.21.15.204:0/2431922160 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a64103ba0 0x7f7a6410a450 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00b0a0 tx=0x7f7a5c00b3b0 comp rx=0 tx=0).stop 2024-05-20T17:54:15.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.795+0000 7f7a68d95700 1 -- 172.21.15.204:0/2431922160 shutdown_connections 2024-05-20T17:54:15.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.795+0000 7f7a68d95700 1 --2- 172.21.15.204:0/2431922160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a641020d0 0x7f7a641024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:15.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.795+0000 7f7a68d95700 1 --2- 172.21.15.204:0/2431922160 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a64103ba0 0x7f7a6410a450 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:15.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.795+0000 7f7a68d95700 1 --2- 172.21.15.204:0/2431922160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a64103200 0x7f7a64103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:15.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.795+0000 7f7a68d95700 1 -- 172.21.15.204:0/2431922160 >> 172.21.15.204:0/2431922160 conn(0x7f7a640fd710 msgr2=0x7f7a640ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:54:15.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.795+0000 7f7a68d95700 1 -- 172.21.15.204:0/2431922160 shutdown_connections 2024-05-20T17:54:15.795 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.795+0000 7f7a68d95700 1 -- 172.21.15.204:0/2431922160 wait complete. 2024-05-20T17:54:15.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a68d95700 1 Processor -- start 2024-05-20T17:54:15.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a68d95700 1 -- start start 2024-05-20T17:54:15.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a68d95700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a641020d0 0x7f7a640752d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:54:15.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a68d95700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a64103200 0x7f7a6419e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:54:15.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a68d95700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a64103ba0 0x7f7a641a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:54:15.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a68d95700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6410dca0 con 0x7f7a64103ba0 2024-05-20T17:54:15.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a68d95700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6410db20 con 0x7f7a641020d0 2024-05-20T17:54:15.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a68d95700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6410de20 con 0x7f7a64103200 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a61d9b700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a64103200 0x7f7a6419e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a61d9b700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a64103200 0x7f7a6419e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:45524/0 (socket says 172.21.15.204:45524) 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a61d9b700 1 -- 172.21.15.204:0/4232269137 learned_addr learned my addr 172.21.15.204:0/4232269137 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.796+0000 7f7a6259c700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a641020d0 0x7f7a640752d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a62d9d700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a64103ba0 0x7f7a641a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a61d9b700 1 -- 172.21.15.204:0/4232269137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a641020d0 msgr2=0x7f7a640752d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a61d9b700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a641020d0 0x7f7a640752d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a61d9b700 1 -- 172.21.15.204:0/4232269137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a64103ba0 msgr2=0x7f7a641a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a61d9b700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a64103ba0 0x7f7a641a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a61d9b700 1 -- 172.21.15.204:0/4232269137 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a641a1330 con 0x7f7a64103200 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a62d9d700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a64103ba0 0x7f7a641a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:54:15.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a61d9b700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a64103200 0x7f7a6419e590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400d980 tx=0x7f7a5400dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:54:15.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a537fe700 1 -- 172.21.15.204:0/4232269137 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54016070 con 0x7f7a64103200 2024-05-20T17:54:15.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a68d95700 1 -- 172.21.15.204:0/4232269137 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a641a1620 con 0x7f7a64103200 2024-05-20T17:54:15.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a537fe700 1 -- 172.21.15.204:0/4232269137 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a54004500 con 0x7f7a64103200 2024-05-20T17:54:15.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a537fe700 1 -- 172.21.15.204:0/4232269137 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54004d10 con 0x7f7a64103200 2024-05-20T17:54:15.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.797+0000 7f7a68d95700 1 -- 172.21.15.204:0/4232269137 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a641a1bb0 con 0x7f7a64103200 2024-05-20T17:54:15.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.798+0000 7f7a68d95700 1 -- 172.21.15.204:0/4232269137 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a64050170 con 0x7f7a64103200 2024-05-20T17:54:15.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.799+0000 7f7a537fe700 1 -- 172.21.15.204:0/4232269137 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7a5400bbf0 con 0x7f7a64103200 2024-05-20T17:54:15.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.800+0000 7f7a537fe700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7a4c0a2c20 0x7f7a4c0a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:54:15.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.800+0000 7f7a537fe700 1 -- 172.21.15.204:0/4232269137 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f7a540ce360 con 0x7f7a64103200 2024-05-20T17:54:15.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.800+0000 7f7a6259c700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7a4c0a2c20 0x7f7a4c0a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:54:15.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.801+0000 7f7a6259c700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7a4c0a2c20 0x7f7a4c0a5130 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a58005950 tx=0x7f7a580058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:54:15.804 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:15.805+0000 7f7a537fe700 1 -- 172.21.15.204:0/4232269137 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7a5401f020 con 0x7f7a64103200 2024-05-20T17:54:16.008 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:16.009+0000 7f7a68d95700 1 -- 172.21.15.204:0/4232269137 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7a64062a80 con 0x7f7a4c0a2c20 2024-05-20T17:54:16.564 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:16 smithi129 ceph-mon[32029]: osdmap e53: 8 total, 8 up, 8 in 2024-05-20T17:54:16.565 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:16 smithi129 ceph-mon[32029]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:16.565 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:16 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:54:16.565 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:16 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:54:16.565 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:16 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:16.616 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:16 smithi204 ceph-mon[37076]: osdmap e53: 8 total, 8 up, 8 in 2024-05-20T17:54:16.616 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:16 smithi204 ceph-mon[37076]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:16.616 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:16 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:54:16.616 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:16 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:54:16.617 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:16 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:16.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:16 smithi154 ceph-mon[36482]: osdmap e53: 8 total, 8 up, 8 in 2024-05-20T17:54:16.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:16 smithi154 ceph-mon[36482]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:16.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:16 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:54:16.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:16 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:54:16.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:16 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:17.523 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:17 smithi204 ceph-mon[37076]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:54:17.523 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:17 smithi204 ceph-mon[37076]: osdmap e54: 8 total, 8 up, 8 in 2024-05-20T17:54:17.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:17 smithi154 ceph-mon[36482]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:54:17.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:17 smithi154 ceph-mon[36482]: osdmap e54: 8 total, 8 up, 8 in 2024-05-20T17:54:17.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:17 smithi129 ceph-mon[32029]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:54:17.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:17 smithi129 ceph-mon[32029]: osdmap e54: 8 total, 8 up, 8 in 2024-05-20T17:54:18.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:18 smithi154 ceph-mon[36482]: osdmap e55: 8 total, 8 up, 8 in 2024-05-20T17:54:18.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:18 smithi154 ceph-mon[36482]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:18.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:18 smithi129 ceph-mon[32029]: osdmap e55: 8 total, 8 up, 8 in 2024-05-20T17:54:18.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:18 smithi129 ceph-mon[32029]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:18.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:18 smithi204 ceph-mon[37076]: osdmap e55: 8 total, 8 up, 8 in 2024-05-20T17:54:18.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:18 smithi204 ceph-mon[37076]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:19.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:19 smithi204 ceph-mon[37076]: from='client.? 172.21.15.204:0/3654536465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2371db0-2b8e-4a7c-86de-7c2036efca54"}]: dispatch 2024-05-20T17:54:19.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:19 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2371db0-2b8e-4a7c-86de-7c2036efca54"}]: dispatch 2024-05-20T17:54:19.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:19 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2371db0-2b8e-4a7c-86de-7c2036efca54"}]': finished 2024-05-20T17:54:19.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:19 smithi204 ceph-mon[37076]: osdmap e56: 9 total, 8 up, 9 in 2024-05-20T17:54:19.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:19 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:19.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:19 smithi154 ceph-mon[36482]: from='client.? 172.21.15.204:0/3654536465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2371db0-2b8e-4a7c-86de-7c2036efca54"}]: dispatch 2024-05-20T17:54:19.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:19 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2371db0-2b8e-4a7c-86de-7c2036efca54"}]: dispatch 2024-05-20T17:54:19.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:19 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2371db0-2b8e-4a7c-86de-7c2036efca54"}]': finished 2024-05-20T17:54:19.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:19 smithi154 ceph-mon[36482]: osdmap e56: 9 total, 8 up, 9 in 2024-05-20T17:54:19.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:19 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:19.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:19 smithi129 ceph-mon[32029]: from='client.? 172.21.15.204:0/3654536465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2371db0-2b8e-4a7c-86de-7c2036efca54"}]: dispatch 2024-05-20T17:54:19.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:19 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2371db0-2b8e-4a7c-86de-7c2036efca54"}]: dispatch 2024-05-20T17:54:19.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:19 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2371db0-2b8e-4a7c-86de-7c2036efca54"}]': finished 2024-05-20T17:54:19.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:19 smithi129 ceph-mon[32029]: osdmap e56: 9 total, 8 up, 9 in 2024-05-20T17:54:19.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:19 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:20.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:20 smithi154 ceph-mon[36482]: from='client.? 172.21.15.204:0/625390764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:54:20.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:20 smithi154 ceph-mon[36482]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:20.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:20 smithi129 ceph-mon[32029]: from='client.? 172.21.15.204:0/625390764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:54:20.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:20 smithi129 ceph-mon[32029]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:20.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:20 smithi204 ceph-mon[37076]: from='client.? 172.21.15.204:0/625390764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:54:20.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:20 smithi204 ceph-mon[37076]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:22.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:21 smithi154 ceph-mon[36482]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:22.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:21 smithi129 ceph-mon[32029]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:22.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:21 smithi204 ceph-mon[37076]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:24.042 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:23 smithi204 ceph-mon[37076]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:24.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:23 smithi154 ceph-mon[36482]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:24.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:23 smithi129 ceph-mon[32029]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:25.832 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:25 smithi204 ceph-mon[37076]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:26.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:25 smithi154 ceph-mon[36482]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:26.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:25 smithi129 ceph-mon[32029]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:27.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:26 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-20T17:54:27.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:26 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:27.086 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:26 smithi204 ceph-mon[37076]: Deploying daemon osd.8 on smithi204 2024-05-20T17:54:27.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:26 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-20T17:54:27.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:26 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:27.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:26 smithi154 ceph-mon[36482]: Deploying daemon osd.8 on smithi204 2024-05-20T17:54:27.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-20T17:54:27.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:27.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:26 smithi129 ceph-mon[32029]: Deploying daemon osd.8 on smithi204 2024-05-20T17:54:28.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:27 smithi154 ceph-mon[36482]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:28.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:27 smithi204 ceph-mon[37076]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:28.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:27 smithi129 ceph-mon[32029]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:30.063 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:29 smithi204 ceph-mon[37076]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:30.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:29 smithi154 ceph-mon[36482]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:30.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:29 smithi129 ceph-mon[32029]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:32.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:32.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:32.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:32 smithi129 ceph-mon[32029]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:32.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:32.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:32.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:32.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:32 smithi204 ceph-mon[37076]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:32.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:32 smithi154 ceph-mon[36482]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:34.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:33 smithi154 ceph-mon[36482]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:34.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:33 smithi129 ceph-mon[32029]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:34.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:33 smithi204 ceph-mon[37076]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:35.676 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:35.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:35.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:35.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:35.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:35.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:35 smithi204 ceph-mon[37076]: from='osd.8 [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-20T17:54:35.677 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:35 smithi204 ceph-mon[37076]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-20T17:54:35.677 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.677+0000 7f7a537fe700 1 -- 172.21.15.204:0/4232269137 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7a64062a80 con 0x7f7a4c0a2c20 2024-05-20T17:54:35.677 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 8 on host 'smithi204' 2024-05-20T17:54:35.679 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.679+0000 7f7a68d95700 1 -- 172.21.15.204:0/4232269137 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7a4c0a2c20 msgr2=0x7f7a4c0a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:54:35.679 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.679+0000 7f7a68d95700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7a4c0a2c20 0x7f7a4c0a5130 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a58005950 tx=0x7f7a580058e0 comp rx=0 tx=0).stop 2024-05-20T17:54:35.679 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.679+0000 7f7a68d95700 1 -- 172.21.15.204:0/4232269137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a64103200 msgr2=0x7f7a6419e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:54:35.679 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.679+0000 7f7a68d95700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a64103200 0x7f7a6419e590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400d980 tx=0x7f7a5400dd40 comp rx=0 tx=0).stop 2024-05-20T17:54:35.679 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.680+0000 7f7a68d95700 1 -- 172.21.15.204:0/4232269137 shutdown_connections 2024-05-20T17:54:35.679 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.680+0000 7f7a68d95700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7a4c0a2c20 0x7f7a4c0a5130 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:35.679 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.680+0000 7f7a68d95700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a641020d0 0x7f7a640752d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:35.679 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.680+0000 7f7a68d95700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a64103ba0 0x7f7a641a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:35.679 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.680+0000 7f7a68d95700 1 --2- 172.21.15.204:0/4232269137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7a64103200 0x7f7a6419e590 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:35.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.680+0000 7f7a68d95700 1 -- 172.21.15.204:0/4232269137 >> 172.21.15.204:0/4232269137 conn(0x7f7a640fd710 msgr2=0x7f7a64106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:54:35.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.680+0000 7f7a68d95700 1 -- 172.21.15.204:0/4232269137 shutdown_connections 2024-05-20T17:54:35.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:54:35.680+0000 7f7a68d95700 1 -- 172.21.15.204:0/4232269137 wait complete. 2024-05-20T17:54:35.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:35.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:35.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:35.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:35.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:35.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:35 smithi154 ceph-mon[36482]: from='osd.8 [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-20T17:54:35.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:35 smithi154 ceph-mon[36482]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-20T17:54:35.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:35.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:35.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:35.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:35.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:35.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:35 smithi129 ceph-mon[32029]: from='osd.8 [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-20T17:54:35.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:35 smithi129 ceph-mon[32029]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-20T17:54:36.302 DEBUG:teuthology.orchestra.run.smithi204:osd.8> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.8.service 2024-05-20T17:54:36.304 INFO:tasks.cephadm:Deploying osd.9 on smithi129 with /dev/vg_nvme/lv_1... 2024-05-20T17:54:36.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-20T17:54:36.550 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:54:36.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:36 smithi154 ceph-mon[36482]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-20T17:54:36.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:36 smithi154 ceph-mon[36482]: from='osd.8 [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:54:36.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:36 smithi154 ceph-mon[36482]: osdmap e57: 9 total, 8 up, 9 in 2024-05-20T17:54:36.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:36 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:36.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:36 smithi154 ceph-mon[36482]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:54:36.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:36 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:36.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:36 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:36.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:36 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:36.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:36 smithi154 ceph-mon[36482]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:36.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:36 smithi129 ceph-mon[32029]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-20T17:54:36.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:36 smithi129 ceph-mon[32029]: from='osd.8 [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:54:36.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:36 smithi129 ceph-mon[32029]: osdmap e57: 9 total, 8 up, 9 in 2024-05-20T17:54:36.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:36 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:36.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:36 smithi129 ceph-mon[32029]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:54:36.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:36 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:36.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:36 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:36.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:36 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:36.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:36 smithi129 ceph-mon[32029]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:36.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:36 smithi204 ceph-mon[37076]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-20T17:54:36.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:36 smithi204 ceph-mon[37076]: from='osd.8 [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:54:36.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:36 smithi204 ceph-mon[37076]: osdmap e57: 9 total, 8 up, 9 in 2024-05-20T17:54:36.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:36 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:36.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:36 smithi204 ceph-mon[37076]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:54:36.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:36 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:36.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:36 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:36.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:36 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:36.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:36 smithi204 ceph-mon[37076]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:37.413 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 17:54:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T17:54:37.128+0000 7f84acbc8700 -1 osd.8 0 waiting for initial osdmap 2024-05-20T17:54:37.414 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 17:54:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T17:54:37.140+0000 7f84a4975700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:54:37.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:37 smithi204 ceph-mon[37076]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:54:37.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:37 smithi204 ceph-mon[37076]: osdmap e58: 9 total, 8 up, 9 in 2024-05-20T17:54:37.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:37 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:37.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:37 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:37.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:37 smithi204 ceph-mon[37076]: from='osd.8 ' entity='osd.8' 2024-05-20T17:54:37.874 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:37 smithi129 ceph-mon[32029]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:54:37.874 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:37 smithi129 ceph-mon[32029]: osdmap e58: 9 total, 8 up, 9 in 2024-05-20T17:54:37.874 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:37 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:37.874 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:37 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:37.874 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:37 smithi129 ceph-mon[32029]: from='osd.8 ' entity='osd.8' 2024-05-20T17:54:37.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:37 smithi154 ceph-mon[36482]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:54:37.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:37 smithi154 ceph-mon[36482]: osdmap e58: 9 total, 8 up, 9 in 2024-05-20T17:54:37.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:37 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:37.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:37 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:37.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:37 smithi154 ceph-mon[36482]: from='osd.8 ' entity='osd.8' 2024-05-20T17:54:38.535 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-20T17:54:38.535 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-20T17:54:38.535 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-05-20T17:54:38.535 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-05-20T17:54:38.535 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127525 s, 822 MB/s 2024-05-20T17:54:38.535 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-05-20T17:54:38.846 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:38 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:54:38.846 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:38 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:54:38.847 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:38.847 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:38 smithi129 ceph-mon[32029]: osd.8 [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284] boot 2024-05-20T17:54:38.847 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:38 smithi129 ceph-mon[32029]: osdmap e59: 9 total, 9 up, 9 in 2024-05-20T17:54:38.847 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:38.847 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:38 smithi129 ceph-mon[32029]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:38.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:38 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:54:38.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:38 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:54:38.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:38.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:38 smithi154 ceph-mon[36482]: osd.8 [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284] boot 2024-05-20T17:54:38.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:38 smithi154 ceph-mon[36482]: osdmap e59: 9 total, 9 up, 9 in 2024-05-20T17:54:38.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:38.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:38 smithi154 ceph-mon[36482]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:38.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:38 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:54:38.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:38 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:54:38.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:38.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:38 smithi204 ceph-mon[37076]: osd.8 [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284] boot 2024-05-20T17:54:38.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:38 smithi204 ceph-mon[37076]: osdmap e59: 9 total, 9 up, 9 in 2024-05-20T17:54:38.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:54:38.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:38 smithi204 ceph-mon[37076]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-20T17:54:39.340 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi129:vg_nvme/lv_1 2024-05-20T17:54:39.587 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:54:39.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:39 smithi154 ceph-mon[36482]: osdmap e60: 9 total, 9 up, 9 in 2024-05-20T17:54:39.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:39.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:39.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:39.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:39.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:39.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:39.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:39.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:39.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:39 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:39.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:39 smithi129 ceph-mon[32029]: osdmap e60: 9 total, 9 up, 9 in 2024-05-20T17:54:39.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:39 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:39.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:39 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:39.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:39 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:39.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:39 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:39.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:39 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:39.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:39 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:39.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:39 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:39.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:39 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:39.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:39 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:39.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:39 smithi204 ceph-mon[37076]: osdmap e60: 9 total, 9 up, 9 in 2024-05-20T17:54:39.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:39 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:39.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:39 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:39.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:39 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:39.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:39 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:39.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:39 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:54:39.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:39 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:39.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:39 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:39.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:39 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:54:39.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:39 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:40.798 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:40 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi204 2024-05-20T17:54:40.798 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:40 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi204 to 5696M 2024-05-20T17:54:40.798 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:40 smithi129 ceph-mon[32029]: osdmap e61: 9 total, 9 up, 9 in 2024-05-20T17:54:40.798 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:40 smithi129 ceph-mon[32029]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:40.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:40 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi204 2024-05-20T17:54:40.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:40 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi204 to 5696M 2024-05-20T17:54:40.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:40 smithi154 ceph-mon[36482]: osdmap e61: 9 total, 9 up, 9 in 2024-05-20T17:54:40.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:40 smithi154 ceph-mon[36482]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:40.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:40 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi204 2024-05-20T17:54:40.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:40 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi204 to 5696M 2024-05-20T17:54:40.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:40 smithi204 ceph-mon[37076]: osdmap e61: 9 total, 9 up, 9 in 2024-05-20T17:54:40.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:40 smithi204 ceph-mon[37076]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:41.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.335+0000 7fc14a19f700 1 -- 172.21.15.129:0/3780788657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc144075530 msgr2=0x7fc144105d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:54:41.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.335+0000 7fc14a19f700 1 --2- 172.21.15.129:0/3780788657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc144075530 0x7fc144105d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc134009990 tx=0x7fc134009ca0 comp rx=0 tx=0).stop 2024-05-20T17:54:41.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.335+0000 7fc14a19f700 1 -- 172.21.15.129:0/3780788657 shutdown_connections 2024-05-20T17:54:41.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.335+0000 7fc14a19f700 1 --2- 172.21.15.129:0/3780788657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144108e80 0x7fc14410b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:41.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.335+0000 7fc14a19f700 1 --2- 172.21.15.129:0/3780788657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc144075530 0x7fc144105d40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:41.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.335+0000 7fc14a19f700 1 --2- 172.21.15.129:0/3780788657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc144106310 0x7fc1441087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:41.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.335+0000 7fc14a19f700 1 -- 172.21.15.129:0/3780788657 >> 172.21.15.129:0/3780788657 conn(0x7fc1440fd710 msgr2=0x7fc1440ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:54:41.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.335+0000 7fc14a19f700 1 -- 172.21.15.129:0/3780788657 shutdown_connections 2024-05-20T17:54:41.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.335+0000 7fc14a19f700 1 -- 172.21.15.129:0/3780788657 wait complete. 2024-05-20T17:54:41.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc14a19f700 1 Processor -- start 2024-05-20T17:54:41.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc14a19f700 1 -- start start 2024-05-20T17:54:41.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc14a19f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc144075530 0x7fc144199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:54:41.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc14a19f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144106310 0x7fc144199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:54:41.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc14a19f700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc144108e80 0x7fc1441a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:54:41.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc14a19f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc14410ed70 con 0x7fc144075530 2024-05-20T17:54:41.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc14a19f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc14410ebf0 con 0x7fc144106310 2024-05-20T17:54:41.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc14a19f700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc14410eef0 con 0x7fc144108e80 2024-05-20T17:54:41.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc1437fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc144075530 0x7fc144199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:54:41.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc1437fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc144075530 0x7fc144199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45032/0 (socket says 172.21.15.129:45032) 2024-05-20T17:54:41.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc1437fe700 1 -- 172.21.15.129:0/550932852 learned_addr learned my addr 172.21.15.129:0/550932852 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:54:41.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.336+0000 7fc143fff700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc144108e80 0x7fc1441a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:54:41.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc13bfff700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144106310 0x7fc144199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:54:41.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc1437fe700 1 -- 172.21.15.129:0/550932852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc144108e80 msgr2=0x7fc1441a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:54:41.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc1437fe700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc144108e80 0x7fc1441a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:41.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc1437fe700 1 -- 172.21.15.129:0/550932852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144106310 msgr2=0x7fc144199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:54:41.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc1437fe700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144106310 0x7fc144199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:54:41.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc1437fe700 1 -- 172.21.15.129:0/550932852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1441a1310 con 0x7fc144075530 2024-05-20T17:54:41.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc13bfff700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144106310 0x7fc144199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:54:41.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc1437fe700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc144075530 0x7fc144199840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc134005b40 tx=0x7fc13400bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:54:41.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc1417fa700 1 -- 172.21.15.129:0/550932852 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134026070 con 0x7fc144075530 2024-05-20T17:54:41.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc14a19f700 1 -- 172.21.15.129:0/550932852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1441a15a0 con 0x7fc144075530 2024-05-20T17:54:41.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc1417fa700 1 -- 172.21.15.129:0/550932852 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc13400f460 con 0x7fc144075530 2024-05-20T17:54:41.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc1417fa700 1 -- 172.21.15.129:0/550932852 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134021650 con 0x7fc144075530 2024-05-20T17:54:41.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.337+0000 7fc14a19f700 1 -- 172.21.15.129:0/550932852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1441a19f0 con 0x7fc144075530 2024-05-20T17:54:41.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.338+0000 7fc14a19f700 1 -- 172.21.15.129:0/550932852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc144050170 con 0x7fc144075530 2024-05-20T17:54:41.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.340+0000 7fc1417fa700 1 -- 172.21.15.129:0/550932852 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc134029070 con 0x7fc144075530 2024-05-20T17:54:41.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.340+0000 7fc1417fa700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc1300a2c20 0x7fc1300a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:54:41.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.340+0000 7fc13bfff700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc1300a2c20 0x7fc1300a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:54:41.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.340+0000 7fc1417fa700 1 -- 172.21.15.129:0/550932852 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7fc1340cfb80 con 0x7fc144075530 2024-05-20T17:54:41.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.341+0000 7fc13bfff700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc1300a2c20 0x7fc1300a5130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc14419a8f0 tx=0x7fc12c00a680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:54:41.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.344+0000 7fc1417fa700 1 -- 172.21.15.129:0/550932852 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc13405d850 con 0x7fc144075530 2024-05-20T17:54:41.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:54:41.550+0000 7fc14a19f700 1 -- 172.21.15.129:0/550932852 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc144062b40 con 0x7fc1300a2c20 2024-05-20T17:54:41.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:41 smithi154 ceph-mon[36482]: osdmap e62: 9 total, 9 up, 9 in 2024-05-20T17:54:41.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:41 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:54:41.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:41 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:54:41.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:41 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:41.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:41 smithi129 ceph-mon[32029]: osdmap e62: 9 total, 9 up, 9 in 2024-05-20T17:54:41.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:41 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:54:41.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:41 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:54:41.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:41 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:41.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:41 smithi204 ceph-mon[37076]: osdmap e62: 9 total, 9 up, 9 in 2024-05-20T17:54:41.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:41 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:54:41.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:41 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:54:41.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:41 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:42.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:42 smithi154 ceph-mon[36482]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:54:42.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:42 smithi154 ceph-mon[36482]: osdmap e63: 9 total, 9 up, 9 in 2024-05-20T17:54:42.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:42 smithi154 ceph-mon[36482]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:42.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:42 smithi129 ceph-mon[32029]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:54:42.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:42 smithi129 ceph-mon[32029]: osdmap e63: 9 total, 9 up, 9 in 2024-05-20T17:54:42.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:42 smithi129 ceph-mon[32029]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:42 smithi204 ceph-mon[37076]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:54:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:42 smithi204 ceph-mon[37076]: osdmap e63: 9 total, 9 up, 9 in 2024-05-20T17:54:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:42 smithi204 ceph-mon[37076]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:43.850 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:43 smithi129 ceph-mon[32029]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-05-20T17:54:44.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:43 smithi154 ceph-mon[36482]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-05-20T17:54:44.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:43 smithi204 ceph-mon[37076]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-05-20T17:54:45.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:44 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/296367803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e40eb83d-8e9f-4f69-af66-b505b01d2836"}]: dispatch 2024-05-20T17:54:45.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:44 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/296367803' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e40eb83d-8e9f-4f69-af66-b505b01d2836"}]': finished 2024-05-20T17:54:45.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:44 smithi154 ceph-mon[36482]: osdmap e64: 10 total, 9 up, 10 in 2024-05-20T17:54:45.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:44 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:54:45.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:44 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/296367803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e40eb83d-8e9f-4f69-af66-b505b01d2836"}]: dispatch 2024-05-20T17:54:45.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:44 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/296367803' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e40eb83d-8e9f-4f69-af66-b505b01d2836"}]': finished 2024-05-20T17:54:45.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:44 smithi129 ceph-mon[32029]: osdmap e64: 10 total, 9 up, 10 in 2024-05-20T17:54:45.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:54:45.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:44 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/296367803' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e40eb83d-8e9f-4f69-af66-b505b01d2836"}]: dispatch 2024-05-20T17:54:45.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:44 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/296367803' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e40eb83d-8e9f-4f69-af66-b505b01d2836"}]': finished 2024-05-20T17:54:45.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:44 smithi204 ceph-mon[37076]: osdmap e64: 10 total, 9 up, 10 in 2024-05-20T17:54:45.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:54:46.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:45 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3997142470' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:54:46.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:45 smithi129 ceph-mon[32029]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-20T17:54:46.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:45 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3997142470' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:54:46.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:45 smithi154 ceph-mon[36482]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-20T17:54:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:45 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3997142470' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:54:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:45 smithi204 ceph-mon[37076]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-20T17:54:48.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:47 smithi154 ceph-mon[36482]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-20T17:54:48.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:47 smithi129 ceph-mon[32029]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-20T17:54:48.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:47 smithi204 ceph-mon[37076]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-20T17:54:50.054 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:49 smithi129 ceph-mon[32029]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-20T17:54:50.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:49 smithi154 ceph-mon[36482]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-20T17:54:50.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:49 smithi204 ceph-mon[37076]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-20T17:54:52.042 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:51 smithi129 ceph-mon[32029]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-20T17:54:52.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:51 smithi154 ceph-mon[36482]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-20T17:54:52.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:51 smithi204 ceph-mon[37076]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-20T17:54:53.101 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:52 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-20T17:54:53.101 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:52 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:53.101 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:52 smithi129 ceph-mon[32029]: Deploying daemon osd.9 on smithi129 2024-05-20T17:54:53.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:52 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-20T17:54:53.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:52 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:53.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:52 smithi154 ceph-mon[36482]: Deploying daemon osd.9 on smithi129 2024-05-20T17:54:53.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:52 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-20T17:54:53.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:52 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:54:53.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:52 smithi204 ceph-mon[37076]: Deploying daemon osd.9 on smithi129 2024-05-20T17:54:54.141 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:53 smithi129 ceph-mon[32029]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:54.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:53 smithi154 ceph-mon[36482]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:54.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:53 smithi204 ceph-mon[37076]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:55.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:55 smithi129 ceph-mon[32029]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:56.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:55 smithi154 ceph-mon[36482]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:56.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:55 smithi204 ceph-mon[37076]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:57.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:57 smithi129 ceph-mon[32029]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:58.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:57 smithi154 ceph-mon[36482]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:58.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:57 smithi204 ceph-mon[37076]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:54:59.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:59 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:59.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:59 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:59.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:54:59 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:59.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:59.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:59.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:54:59 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:59.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:59 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:54:59.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:59 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:54:59.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:54:59 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:00.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:00 smithi129 ceph-mon[32029]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:00.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:00 smithi154 ceph-mon[36482]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:00.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:00 smithi204 ceph-mon[37076]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:01.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:01 smithi129 ceph-mon[32029]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:01.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:01 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:01.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:01 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:02.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:01 smithi154 ceph-mon[36482]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:02.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:01 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:02.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:01 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:02.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:01 smithi204 ceph-mon[37076]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:02.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:01 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:02.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:01 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:02.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.811+0000 7fc1417fa700 1 -- 172.21.15.129:0/550932852 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc144062b40 con 0x7fc1300a2c20 2024-05-20T17:55:02.811 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 9 on host 'smithi129' 2024-05-20T17:55:02.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 -- 172.21.15.129:0/550932852 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc1300a2c20 msgr2=0x7fc1300a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:02.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc1300a2c20 0x7fc1300a5130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc14419a8f0 tx=0x7fc12c00a680 comp rx=0 tx=0).stop 2024-05-20T17:55:02.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 -- 172.21.15.129:0/550932852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc144075530 msgr2=0x7fc144199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:02.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc144075530 0x7fc144199840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc134005b40 tx=0x7fc13400bbb0 comp rx=0 tx=0).stop 2024-05-20T17:55:02.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 -- 172.21.15.129:0/550932852 shutdown_connections 2024-05-20T17:55:02.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc1300a2c20 0x7fc1300a5130 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:02.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc144108e80 0x7fc1441a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:02.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc144075530 0x7fc144199840 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:02.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 --2- 172.21.15.129:0/550932852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc144106310 0x7fc144199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:02.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 -- 172.21.15.129:0/550932852 >> 172.21.15.129:0/550932852 conn(0x7fc1440fd710 msgr2=0x7fc1441043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:55:02.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 -- 172.21.15.129:0/550932852 shutdown_connections 2024-05-20T17:55:02.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:02.814+0000 7fc14a19f700 1 -- 172.21.15.129:0/550932852 wait complete. 2024-05-20T17:55:03.133 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:03.134 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:03.134 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:02 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:03.134 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:02 smithi129 ceph-mon[32029]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-20T17:55:03.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:02 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:03.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:02 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:03.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:02 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:03.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:02 smithi154 ceph-mon[36482]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-20T17:55:03.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:02 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:03.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:02 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:03.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:02 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:03.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:02 smithi204 ceph-mon[37076]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-20T17:55:03.468 DEBUG:teuthology.orchestra.run.smithi129:osd.9> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.9.service 2024-05-20T17:55:03.470 INFO:tasks.cephadm:Deploying osd.10 on smithi154 with /dev/vg_nvme/lv_1... 2024-05-20T17:55:03.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-20T17:55:03.716 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T17:55:04.090 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:03 smithi154 ceph-mon[36482]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-20T17:55:04.091 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:03 smithi154 ceph-mon[36482]: osdmap e65: 10 total, 9 up, 10 in 2024-05-20T17:55:04.091 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:03 smithi154 ceph-mon[36482]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:55:04.091 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:03 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:04.091 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:03 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:04.091 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:03 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:04.091 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:03 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:04.091 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:03 smithi154 ceph-mon[36482]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:04.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:03 smithi129 ceph-mon[32029]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-20T17:55:04.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:03 smithi129 ceph-mon[32029]: osdmap e65: 10 total, 9 up, 10 in 2024-05-20T17:55:04.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:03 smithi129 ceph-mon[32029]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:55:04.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:03 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:04.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:03 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:04.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:03 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:04.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:03 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:04.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:03 smithi129 ceph-mon[32029]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:04.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:03 smithi204 ceph-mon[37076]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-20T17:55:04.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:03 smithi204 ceph-mon[37076]: osdmap e65: 10 total, 9 up, 10 in 2024-05-20T17:55:04.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:03 smithi204 ceph-mon[37076]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-05-20T17:55:04.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:03 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:04.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:03 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:04.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:03 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:04.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:03 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:04.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:03 smithi204 ceph-mon[37076]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:04.611 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 17:55:04 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T17:55:04.192+0000 7faef84c5700 -1 osd.9 0 waiting for initial osdmap 2024-05-20T17:55:04.611 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 17:55:04 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T17:55:04.205+0000 7faef0272700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:55:04.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:04 smithi129 ceph-mon[32029]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:55:04.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:04 smithi129 ceph-mon[32029]: osdmap e66: 10 total, 9 up, 10 in 2024-05-20T17:55:04.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:04 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:04.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:04 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:04.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:04 smithi129 ceph-mon[32029]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' 2024-05-20T17:55:05.089 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:04 smithi154 ceph-mon[36482]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:55:05.089 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:04 smithi154 ceph-mon[36482]: osdmap e66: 10 total, 9 up, 10 in 2024-05-20T17:55:05.089 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:04 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:05.090 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:04 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:05.090 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:04 smithi154 ceph-mon[36482]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' 2024-05-20T17:55:05.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:04 smithi204 ceph-mon[37076]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-05-20T17:55:05.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:04 smithi204 ceph-mon[37076]: osdmap e66: 10 total, 9 up, 10 in 2024-05-20T17:55:05.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:04 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:05.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:04 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:05.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:04 smithi204 ceph-mon[37076]: from='osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755]' entity='osd.9' 2024-05-20T17:55:05.660 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-20T17:55:05.660 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-20T17:55:05.660 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-05-20T17:55:05.660 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-05-20T17:55:05.660 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115049 s, 911 MB/s 2024-05-20T17:55:05.660 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-05-20T17:55:05.995 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:05 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:55:05.995 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:05 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:55:05.995 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:05 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:05.995 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:05 smithi154 ceph-mon[36482]: osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755] boot 2024-05-20T17:55:05.995 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:05 smithi154 ceph-mon[36482]: osdmap e67: 10 total, 10 up, 10 in 2024-05-20T17:55:05.995 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:05 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:05.995 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:05 smithi154 ceph-mon[36482]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:06.120 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:05 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:55:06.120 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:05 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:55:06.120 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:05 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:06.120 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:05 smithi129 ceph-mon[32029]: osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755] boot 2024-05-20T17:55:06.120 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:05 smithi129 ceph-mon[32029]: osdmap e67: 10 total, 10 up, 10 in 2024-05-20T17:55:06.120 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:05 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:06.121 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:05 smithi129 ceph-mon[32029]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:06.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:05 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:55:06.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:05 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:55:06.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:05 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:06.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:05 smithi204 ceph-mon[37076]: osd.9 [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755] boot 2024-05-20T17:55:06.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:05 smithi204 ceph-mon[37076]: osdmap e67: 10 total, 10 up, 10 in 2024-05-20T17:55:06.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:05 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:55:06.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:05 smithi204 ceph-mon[37076]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-05-20T17:55:06.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi154:vg_nvme/lv_1 2024-05-20T17:55:06.578 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T17:55:07.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: osdmap e68: 10 total, 10 up, 10 in 2024-05-20T17:55:07.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi129 2024-05-20T17:55:07.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:07.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:07.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi129 to 4272M 2024-05-20T17:55:07.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:07.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:07.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:07.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:07 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:07.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: osdmap e68: 10 total, 10 up, 10 in 2024-05-20T17:55:07.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi129 2024-05-20T17:55:07.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:07.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:07.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi129 to 4272M 2024-05-20T17:55:07.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:07.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:07.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:07.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:07 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:07.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: osdmap e68: 10 total, 10 up, 10 in 2024-05-20T17:55:07.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi129 2024-05-20T17:55:07.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:07.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:07.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:07.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi129 to 4272M 2024-05-20T17:55:07.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:07.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:07.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:07.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:07 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:08.248 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.248+0000 7f00fad2c700 1 -- 172.21.15.154:0/3320158376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00f41031f0 msgr2=0x7f00f4103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:08.248 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.248+0000 7f00fad2c700 1 --2- 172.21.15.154:0/3320158376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00f41031f0 0x7f00f4103600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009990 tx=0x7f00e0009ca0 comp rx=0 tx=0).stop 2024-05-20T17:55:08.248 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.248+0000 7f00fad2c700 1 -- 172.21.15.154:0/3320158376 shutdown_connections 2024-05-20T17:55:08.248 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.248+0000 7f00fad2c700 1 --2- 172.21.15.154:0/3320158376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f00f4103b90 0x7f00f410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:08.248 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.248+0000 7f00fad2c700 1 --2- 172.21.15.154:0/3320158376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f41020c0 0x7f00f41024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:08.248 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.248+0000 7f00fad2c700 1 --2- 172.21.15.154:0/3320158376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00f41031f0 0x7f00f4103600 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:08.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.248+0000 7f00fad2c700 1 -- 172.21.15.154:0/3320158376 >> 172.21.15.154:0/3320158376 conn(0x7f00f40fd700 msgr2=0x7f00f40ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:55:08.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.248+0000 7f00fad2c700 1 -- 172.21.15.154:0/3320158376 shutdown_connections 2024-05-20T17:55:08.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.248+0000 7f00fad2c700 1 -- 172.21.15.154:0/3320158376 wait complete. 2024-05-20T17:55:08.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:08 smithi154 ceph-mon[36482]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:08.251 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.249+0000 7f00fad2c700 1 Processor -- start 2024-05-20T17:55:08.251 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.249+0000 7f00fad2c700 1 -- start start 2024-05-20T17:55:08.251 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.249+0000 7f00fad2c700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f00f41020c0 0x7f00f4199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:08.251 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.249+0000 7f00fad2c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00f41031f0 0x7f00f4199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:08.251 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.249+0000 7f00fad2c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f4103b90 0x7f00f41a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.249+0000 7f00fad2c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00f410dcb0 con 0x7f00f4103b90 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.249+0000 7f00fad2c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f00f410db30 con 0x7f00f41031f0 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.249+0000 7f00fad2c700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f00f410de30 con 0x7f00f41020c0 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f3fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00f41031f0 0x7f00f4199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f3fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00f41031f0 0x7f00f4199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:44616/0 (socket says 172.21.15.154:44616) 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f3fff700 1 -- 172.21.15.154:0/1986009274 learned_addr learned my addr 172.21.15.154:0/1986009274 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f8ac8700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f00f41020c0 0x7f00f4199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f92c9700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f4103b90 0x7f00f41a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f3fff700 1 -- 172.21.15.154:0/1986009274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f00f41020c0 msgr2=0x7f00f4199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f3fff700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f00f41020c0 0x7f00f4199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f3fff700 1 -- 172.21.15.154:0/1986009274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f4103b90 msgr2=0x7f00f41a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:08.252 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f3fff700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f4103b90 0x7f00f41a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:08.253 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f3fff700 1 -- 172.21.15.154:0/1986009274 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f41a13e0 con 0x7f00f41031f0 2024-05-20T17:55:08.253 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f3fff700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00f41031f0 0x7f00f4199dc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00e000b2e0 tx=0x7f00e000bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:55:08.253 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f1ffb700 1 -- 172.21.15.154:0/1986009274 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e0026070 con 0x7f00f41031f0 2024-05-20T17:55:08.253 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f1ffb700 1 -- 172.21.15.154:0/1986009274 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00e000f460 con 0x7f00f41031f0 2024-05-20T17:55:08.253 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00fad2c700 1 -- 172.21.15.154:0/1986009274 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f41a1670 con 0x7f00f41031f0 2024-05-20T17:55:08.253 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.250+0000 7f00f1ffb700 1 -- 172.21.15.154:0/1986009274 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e0021670 con 0x7f00f41031f0 2024-05-20T17:55:08.253 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.251+0000 7f00fad2c700 1 -- 172.21.15.154:0/1986009274 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f41a1ac0 con 0x7f00f41031f0 2024-05-20T17:55:08.253 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.252+0000 7f00fad2c700 1 -- 172.21.15.154:0/1986009274 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00f4050170 con 0x7f00f41031f0 2024-05-20T17:55:08.254 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.253+0000 7f00f1ffb700 1 -- 172.21.15.154:0/1986009274 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f00e0029070 con 0x7f00f41031f0 2024-05-20T17:55:08.254 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.254+0000 7f00f1ffb700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f00e40a2c20 0x7f00e40a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:08.254 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.254+0000 7f00f1ffb700 1 -- 172.21.15.154:0/1986009274 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f00e00cfbd0 con 0x7f00f41031f0 2024-05-20T17:55:08.254 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.254+0000 7f00f8ac8700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f00e40a2c20 0x7f00e40a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:08.254 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.254+0000 7f00f8ac8700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f00e40a2c20 0x7f00e40a5130 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f00f41030f0 tx=0x7f00e800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:55:08.256 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.257+0000 7f00f1ffb700 1 -- 172.21.15.154:0/1986009274 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f00e0013070 con 0x7f00f41031f0 2024-05-20T17:55:08.459 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:08.460+0000 7f00fad2c700 1 -- 172.21.15.154:0/1986009274 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f00f41a1de0 con 0x7f00e40a2c20 2024-05-20T17:55:08.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:08 smithi129 ceph-mon[32029]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:08.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:08 smithi204 ceph-mon[37076]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:09.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:09 smithi154 ceph-mon[36482]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:55:09.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:55:09.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:55:09.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:09.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:09 smithi129 ceph-mon[32029]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:55:09.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:55:09.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:55:09.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:09.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:09 smithi204 ceph-mon[37076]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:55:09.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:55:09.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:55:09.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:10.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:10 smithi154 ceph-mon[36482]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:10.645 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:10 smithi204 ceph-mon[37076]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:10.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:10 smithi129 ceph-mon[32029]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:11.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:11 smithi154 ceph-mon[36482]: from='client.? 172.21.15.154:0/639677111' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe45b366-5c36-454e-ae75-d6170e11aff1"}]: dispatch 2024-05-20T17:55:11.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:11 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe45b366-5c36-454e-ae75-d6170e11aff1"}]: dispatch 2024-05-20T17:55:11.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:11 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe45b366-5c36-454e-ae75-d6170e11aff1"}]': finished 2024-05-20T17:55:11.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:11 smithi154 ceph-mon[36482]: osdmap e69: 11 total, 10 up, 11 in 2024-05-20T17:55:11.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:11 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:11.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:11 smithi129 ceph-mon[32029]: from='client.? 172.21.15.154:0/639677111' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe45b366-5c36-454e-ae75-d6170e11aff1"}]: dispatch 2024-05-20T17:55:11.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:11 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe45b366-5c36-454e-ae75-d6170e11aff1"}]: dispatch 2024-05-20T17:55:11.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:11 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe45b366-5c36-454e-ae75-d6170e11aff1"}]': finished 2024-05-20T17:55:11.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:11 smithi129 ceph-mon[32029]: osdmap e69: 11 total, 10 up, 11 in 2024-05-20T17:55:11.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:11 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:11.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:11 smithi204 ceph-mon[37076]: from='client.? 172.21.15.154:0/639677111' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe45b366-5c36-454e-ae75-d6170e11aff1"}]: dispatch 2024-05-20T17:55:11.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:11 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe45b366-5c36-454e-ae75-d6170e11aff1"}]: dispatch 2024-05-20T17:55:11.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:11 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe45b366-5c36-454e-ae75-d6170e11aff1"}]': finished 2024-05-20T17:55:11.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:11 smithi204 ceph-mon[37076]: osdmap e69: 11 total, 10 up, 11 in 2024-05-20T17:55:11.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:11 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:12.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:12 smithi154 ceph-mon[36482]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:12.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:12 smithi154 ceph-mon[36482]: from='client.? 172.21.15.154:0/2426004433' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:55:12.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:12 smithi129 ceph-mon[32029]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:12.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:12 smithi129 ceph-mon[32029]: from='client.? 172.21.15.154:0/2426004433' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:55:12.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:12 smithi204 ceph-mon[37076]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:12.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:12 smithi204 ceph-mon[37076]: from='client.? 172.21.15.154:0/2426004433' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:55:14.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:13 smithi154 ceph-mon[36482]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:14.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:13 smithi129 ceph-mon[32029]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:14.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:13 smithi204 ceph-mon[37076]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:16.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:15 smithi154 ceph-mon[36482]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:16.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:15 smithi129 ceph-mon[32029]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:16.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:15 smithi204 ceph-mon[37076]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:17.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:17 smithi154 ceph-mon[36482]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:18.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:17 smithi129 ceph-mon[32029]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:18.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:17 smithi204 ceph-mon[37076]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:19.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:18 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-20T17:55:19.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:18 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:19.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:18 smithi154 ceph-mon[36482]: Deploying daemon osd.10 on smithi154 2024-05-20T17:55:19.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-20T17:55:19.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:19.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:18 smithi129 ceph-mon[32029]: Deploying daemon osd.10 on smithi154 2024-05-20T17:55:19.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:18 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-20T17:55:19.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:18 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:19.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:18 smithi204 ceph-mon[37076]: Deploying daemon osd.10 on smithi154 2024-05-20T17:55:19.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:19 smithi154 ceph-mon[36482]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:20.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:19 smithi129 ceph-mon[32029]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:20.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:19 smithi204 ceph-mon[37076]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:21.923 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:21 smithi154 ceph-mon[36482]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:22.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:21 smithi129 ceph-mon[32029]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:22.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:21 smithi204 ceph-mon[37076]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:23.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:23 smithi154 ceph-mon[36482]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:24.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:23 smithi129 ceph-mon[32029]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:24.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:23 smithi204 ceph-mon[37076]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:25.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:25 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:25.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:25 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:25.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:25 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:25.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:25 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:25.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:25 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:25.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:25 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:25.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:25 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:25.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:25 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:25.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:25 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:26.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:26 smithi154 ceph-mon[36482]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:26.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:26 smithi129 ceph-mon[32029]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:26.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:26 smithi204 ceph-mon[37076]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:28.575 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.576+0000 7f00f1ffb700 1 -- 172.21.15.154:0/1986009274 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f00f41a1de0 con 0x7f00e40a2c20 2024-05-20T17:55:28.575 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 10 on host 'smithi154' 2024-05-20T17:55:28.577 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 -- 172.21.15.154:0/1986009274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f00e40a2c20 msgr2=0x7f00e40a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:28.577 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f00e40a2c20 0x7f00e40a5130 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f00f41030f0 tx=0x7f00e800a380 comp rx=0 tx=0).stop 2024-05-20T17:55:28.577 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 -- 172.21.15.154:0/1986009274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00f41031f0 msgr2=0x7f00f4199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:28.578 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00f41031f0 0x7f00f4199dc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00e000b2e0 tx=0x7f00e000bbb0 comp rx=0 tx=0).stop 2024-05-20T17:55:28.578 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 -- 172.21.15.154:0/1986009274 shutdown_connections 2024-05-20T17:55:28.578 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f00e40a2c20 0x7f00e40a5130 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:28.578 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f4103b90 0x7f00f41a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:28.578 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f00f41020c0 0x7f00f4199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:28.578 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 --2- 172.21.15.154:0/1986009274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00f41031f0 0x7f00f4199dc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:28.578 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 -- 172.21.15.154:0/1986009274 >> 172.21.15.154:0/1986009274 conn(0x7f00f40fd700 msgr2=0x7f00f40fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:55:28.578 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 -- 172.21.15.154:0/1986009274 shutdown_connections 2024-05-20T17:55:28.578 INFO:teuthology.orchestra.run.smithi154.stderr:2024-05-20T17:55:28.578+0000 7f00fad2c700 1 -- 172.21.15.154:0/1986009274 wait complete. 2024-05-20T17:55:28.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:28 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:28.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:28 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:28.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:28 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:28.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:28 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:28.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:28 smithi154 ceph-mon[36482]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:28.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:28 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:28.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:28 smithi154 ceph-mon[36482]: from='osd.10 [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-20T17:55:28.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:28 smithi154 ceph-mon[36482]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-20T17:55:28.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:28 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:28.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:28 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:28.895 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:28 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:28 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:28 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:28 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:28 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:28 smithi204 ceph-mon[37076]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:28 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:28 smithi204 ceph-mon[37076]: from='osd.10 [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-20T17:55:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:28 smithi204 ceph-mon[37076]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-20T17:55:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:28 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:28 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:28.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:28 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:28 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:28 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:28 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:28 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:28 smithi129 ceph-mon[32029]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:28 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:29.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:28 smithi129 ceph-mon[32029]: from='osd.10 [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-20T17:55:29.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:28 smithi129 ceph-mon[32029]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-20T17:55:29.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:28 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:29.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:28 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:29.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:28 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:29.244 DEBUG:teuthology.orchestra.run.smithi154:osd.10> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.10.service 2024-05-20T17:55:29.246 INFO:tasks.cephadm:Deploying osd.11 on smithi204 with /dev/vg_nvme/lv_1... 2024-05-20T17:55:29.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-20T17:55:29.493 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:55:29.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:29 smithi204 ceph-mon[37076]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-20T17:55:29.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:29 smithi204 ceph-mon[37076]: from='osd.10 [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:55:29.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:29 smithi204 ceph-mon[37076]: osdmap e70: 11 total, 10 up, 11 in 2024-05-20T17:55:29.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:29 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:29.876 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:29 smithi204 ceph-mon[37076]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:55:29.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:29 smithi154 ceph-mon[36482]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-20T17:55:29.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:29 smithi154 ceph-mon[36482]: from='osd.10 [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:55:29.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:29 smithi154 ceph-mon[36482]: osdmap e70: 11 total, 10 up, 11 in 2024-05-20T17:55:29.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:29 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:29.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:29 smithi154 ceph-mon[36482]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:55:30.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:29 smithi129 ceph-mon[32029]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-20T17:55:30.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:29 smithi129 ceph-mon[32029]: from='osd.10 [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:55:30.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:29 smithi129 ceph-mon[32029]: osdmap e70: 11 total, 10 up, 11 in 2024-05-20T17:55:30.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:29 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:30.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:29 smithi129 ceph-mon[32029]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-05-20T17:55:30.394 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 17:55:30 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T17:55:30.141+0000 7fcf37bef700 -1 osd.10 0 waiting for initial osdmap 2024-05-20T17:55:30.394 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 17:55:30 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T17:55:30.156+0000 7fcf2f99c700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:55:30.968 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:30 smithi154 ceph-mon[36482]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:30.968 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:30 smithi154 ceph-mon[36482]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:55:30.968 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:30 smithi154 ceph-mon[36482]: osdmap e71: 11 total, 10 up, 11 in 2024-05-20T17:55:30.968 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:30.968 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:30.968 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:30 smithi154 ceph-mon[36482]: osdmap e72: 11 total, 10 up, 11 in 2024-05-20T17:55:30.968 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:30.968 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:30 smithi154 ceph-mon[36482]: from='osd.10 ' entity='osd.10' 2024-05-20T17:55:31.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:30 smithi204 ceph-mon[37076]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:31.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:30 smithi204 ceph-mon[37076]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:55:31.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:30 smithi204 ceph-mon[37076]: osdmap e71: 11 total, 10 up, 11 in 2024-05-20T17:55:31.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:31.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:31.033 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:30 smithi204 ceph-mon[37076]: osdmap e72: 11 total, 10 up, 11 in 2024-05-20T17:55:31.034 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:31.034 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:30 smithi204 ceph-mon[37076]: from='osd.10 ' entity='osd.10' 2024-05-20T17:55:31.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:30 smithi129 ceph-mon[32029]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:31.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:30 smithi129 ceph-mon[32029]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-05-20T17:55:31.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:30 smithi129 ceph-mon[32029]: osdmap e71: 11 total, 10 up, 11 in 2024-05-20T17:55:31.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:31.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:31.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:30 smithi129 ceph-mon[32029]: osdmap e72: 11 total, 10 up, 11 in 2024-05-20T17:55:31.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:31.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:30 smithi129 ceph-mon[32029]: from='osd.10 ' entity='osd.10' 2024-05-20T17:55:31.566 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-20T17:55:31.567 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-20T17:55:31.567 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-05-20T17:55:31.567 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-05-20T17:55:31.567 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0141259 s, 742 MB/s 2024-05-20T17:55:31.567 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-05-20T17:55:32.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:31 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:55:32.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:31 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:55:32.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:31 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:32.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:31 smithi154 ceph-mon[36482]: osd.10 [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880] boot 2024-05-20T17:55:32.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:31 smithi154 ceph-mon[36482]: osdmap e73: 11 total, 11 up, 11 in 2024-05-20T17:55:32.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:31 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:32.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:31 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:55:32.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:31 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:55:32.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:31 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:32.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:31 smithi129 ceph-mon[32029]: osd.10 [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880] boot 2024-05-20T17:55:32.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:31 smithi129 ceph-mon[32029]: osdmap e73: 11 total, 11 up, 11 in 2024-05-20T17:55:32.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:31 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:32.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:31 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:55:32.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:31 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:55:32.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:31 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:32.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:31 smithi204 ceph-mon[37076]: osd.10 [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880] boot 2024-05-20T17:55:32.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:31 smithi204 ceph-mon[37076]: osdmap e73: 11 total, 11 up, 11 in 2024-05-20T17:55:32.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:31 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:55:32.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_1 2024-05-20T17:55:32.499 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:55:32.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:32.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: osdmap e74: 11 total, 11 up, 11 in 2024-05-20T17:55:32.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:32.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:32.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:32.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:32.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:32.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:32.926 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:32.927 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:32.927 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:32.927 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: osdmap e74: 11 total, 11 up, 11 in 2024-05-20T17:55:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:33.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: osdmap e74: 11 total, 11 up, 11 in 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:33.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:33.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:33 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi154 2024-05-20T17:55:33.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:33 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi154 to 4272M 2024-05-20T17:55:33.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:33 smithi204 ceph-mon[37076]: osdmap e75: 11 total, 11 up, 11 in 2024-05-20T17:55:33.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:33 smithi204 ceph-mon[37076]: pgmap v238: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-05-20T17:55:34.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:33 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi154 2024-05-20T17:55:34.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:33 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi154 to 4272M 2024-05-20T17:55:34.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:33 smithi154 ceph-mon[36482]: osdmap e75: 11 total, 11 up, 11 in 2024-05-20T17:55:34.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:33 smithi154 ceph-mon[36482]: pgmap v238: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-05-20T17:55:34.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:33 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi154 2024-05-20T17:55:34.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:33 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi154 to 4272M 2024-05-20T17:55:34.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:33 smithi129 ceph-mon[32029]: osdmap e75: 11 total, 11 up, 11 in 2024-05-20T17:55:34.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:33 smithi129 ceph-mon[32029]: pgmap v238: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-05-20T17:55:34.216 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.216+0000 7f416d0c2700 1 -- 172.21.15.204:0/543023817 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41681031e0 msgr2=0x7f41681035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:34.216 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.216+0000 7f416d0c2700 1 --2- 172.21.15.204:0/543023817 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41681031e0 0x7f41681035f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f415c009990 tx=0x7f415c009ca0 comp rx=0 tx=0).stop 2024-05-20T17:55:34.216 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.216+0000 7f416d0c2700 1 -- 172.21.15.204:0/543023817 shutdown_connections 2024-05-20T17:55:34.216 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.216+0000 7f416d0c2700 1 --2- 172.21.15.204:0/543023817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41681020b0 0x7f41681024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:34.216 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.216+0000 7f416d0c2700 1 --2- 172.21.15.204:0/543023817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4168103b80 0x7f416810a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:34.216 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.216+0000 7f416d0c2700 1 --2- 172.21.15.204:0/543023817 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41681031e0 0x7f41681035f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:34.216 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.216+0000 7f416d0c2700 1 -- 172.21.15.204:0/543023817 >> 172.21.15.204:0/543023817 conn(0x7f41680fd6f0 msgr2=0x7f41680ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:55:34.216 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.216+0000 7f416d0c2700 1 -- 172.21.15.204:0/543023817 shutdown_connections 2024-05-20T17:55:34.216 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.216+0000 7f416d0c2700 1 -- 172.21.15.204:0/543023817 wait complete. 2024-05-20T17:55:34.218 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.217+0000 7f416d0c2700 1 Processor -- start 2024-05-20T17:55:34.218 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.217+0000 7f416d0c2700 1 -- start start 2024-05-20T17:55:34.218 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f416d0c2700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41681020b0 0x7f41681998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:34.218 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f416d0c2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41681031e0 0x7f4168199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:34.218 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f416d0c2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4168103b80 0x7f41681a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:34.218 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f416d0c2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f416810dc90 con 0x7f4168103b80 2024-05-20T17:55:34.218 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f416d0c2700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f416810db10 con 0x7f41681031e0 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f416d0c2700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f416810de10 con 0x7f41681020b0 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f4166d9d700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41681020b0 0x7f41681998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f4166d9d700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41681020b0 0x7f41681998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:38262/0 (socket says 172.21.15.204:38262) 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f4166d9d700 1 -- 172.21.15.204:0/3061957679 learned_addr learned my addr 172.21.15.204:0/3061957679 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f416659c700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41681031e0 0x7f4168199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f416759e700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4168103b80 0x7f41681a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f4166d9d700 1 -- 172.21.15.204:0/3061957679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41681031e0 msgr2=0x7f4168199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f4166d9d700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41681031e0 0x7f4168199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f4166d9d700 1 -- 172.21.15.204:0/3061957679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4168103b80 msgr2=0x7f41681a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f4166d9d700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4168103b80 0x7f41681a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f4166d9d700 1 -- 172.21.15.204:0/3061957679 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41681a1410 con 0x7f41681020b0 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f4166d9d700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41681020b0 0x7f41681998b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f415800e9f0 tx=0x7f415800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:55:34.219 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.218+0000 7f414ffff700 1 -- 172.21.15.204:0/3061957679 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415800cbd0 con 0x7f41681020b0 2024-05-20T17:55:34.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.219+0000 7f416d0c2700 1 -- 172.21.15.204:0/3061957679 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41681a1700 con 0x7f41681020b0 2024-05-20T17:55:34.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.219+0000 7f414ffff700 1 -- 172.21.15.204:0/3061957679 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4158004510 con 0x7f41681020b0 2024-05-20T17:55:34.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.219+0000 7f414ffff700 1 -- 172.21.15.204:0/3061957679 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4158010430 con 0x7f41681020b0 2024-05-20T17:55:34.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.219+0000 7f416d0c2700 1 -- 172.21.15.204:0/3061957679 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41681a1bf0 con 0x7f41681020b0 2024-05-20T17:55:34.220 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.219+0000 7f416d0c2700 1 -- 172.21.15.204:0/3061957679 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4168050170 con 0x7f41681020b0 2024-05-20T17:55:34.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.221+0000 7f414ffff700 1 -- 172.21.15.204:0/3061957679 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4158003ec0 con 0x7f41681020b0 2024-05-20T17:55:34.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.221+0000 7f414ffff700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41500a2c20 0x7f41500a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:34.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.221+0000 7f414ffff700 1 -- 172.21.15.204:0/3061957679 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7f41580ced80 con 0x7f41681020b0 2024-05-20T17:55:34.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.222+0000 7f416659c700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41500a2c20 0x7f41500a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:34.221 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.222+0000 7f416659c700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41500a2c20 0x7f41500a5130 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f415c00b2e0 tx=0x7f415c00b880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:55:34.224 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.225+0000 7f414ffff700 1 -- 172.21.15.204:0/3061957679 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f415809c330 con 0x7f41681020b0 2024-05-20T17:55:34.426 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:34.427+0000 7f416d0c2700 1 -- 172.21.15.204:0/3061957679 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f41681a1eb0 con 0x7f41500a2c20 2024-05-20T17:55:35.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:34 smithi154 ceph-mon[36482]: osdmap e76: 11 total, 11 up, 11 in 2024-05-20T17:55:35.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:34 smithi154 ceph-mon[36482]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:55:35.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:34 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:55:35.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:34 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:55:35.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:34 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:35.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:34 smithi129 ceph-mon[32029]: osdmap e76: 11 total, 11 up, 11 in 2024-05-20T17:55:35.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:34 smithi129 ceph-mon[32029]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:55:35.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:34 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:55:35.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:34 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:55:35.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:34 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:35.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:34 smithi204 ceph-mon[37076]: osdmap e76: 11 total, 11 up, 11 in 2024-05-20T17:55:35.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:34 smithi204 ceph-mon[37076]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:55:35.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:34 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-20T17:55:35.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:34 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-20T17:55:35.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:34 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:36.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:35 smithi204 ceph-mon[37076]: osdmap e77: 11 total, 11 up, 11 in 2024-05-20T17:55:36.122 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:35 smithi204 ceph-mon[37076]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:55:36.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:35 smithi154 ceph-mon[36482]: osdmap e77: 11 total, 11 up, 11 in 2024-05-20T17:55:36.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:35 smithi154 ceph-mon[36482]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:55:36.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:35 smithi129 ceph-mon[32029]: osdmap e77: 11 total, 11 up, 11 in 2024-05-20T17:55:36.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:35 smithi129 ceph-mon[32029]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-20T17:55:37.519 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:37 smithi204 ceph-mon[37076]: from='client.? 172.21.15.204:0/3842960314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc59d526-8616-4b3f-913a-c706b4d575c3"}]: dispatch 2024-05-20T17:55:37.519 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:37 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc59d526-8616-4b3f-913a-c706b4d575c3"}]: dispatch 2024-05-20T17:55:37.520 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:37 smithi204 ceph-mon[37076]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc59d526-8616-4b3f-913a-c706b4d575c3"}]': finished 2024-05-20T17:55:37.520 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:37 smithi204 ceph-mon[37076]: osdmap e78: 12 total, 11 up, 12 in 2024-05-20T17:55:37.520 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:37 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:37.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:37 smithi154 ceph-mon[36482]: from='client.? 172.21.15.204:0/3842960314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc59d526-8616-4b3f-913a-c706b4d575c3"}]: dispatch 2024-05-20T17:55:37.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:37 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc59d526-8616-4b3f-913a-c706b4d575c3"}]: dispatch 2024-05-20T17:55:37.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:37 smithi154 ceph-mon[36482]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc59d526-8616-4b3f-913a-c706b4d575c3"}]': finished 2024-05-20T17:55:37.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:37 smithi154 ceph-mon[36482]: osdmap e78: 12 total, 11 up, 12 in 2024-05-20T17:55:37.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:37 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:37.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:37 smithi129 ceph-mon[32029]: from='client.? 172.21.15.204:0/3842960314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc59d526-8616-4b3f-913a-c706b4d575c3"}]: dispatch 2024-05-20T17:55:37.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:37 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc59d526-8616-4b3f-913a-c706b4d575c3"}]: dispatch 2024-05-20T17:55:37.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:37 smithi129 ceph-mon[32029]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc59d526-8616-4b3f-913a-c706b4d575c3"}]': finished 2024-05-20T17:55:37.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:37 smithi129 ceph-mon[32029]: osdmap e78: 12 total, 11 up, 12 in 2024-05-20T17:55:37.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:37 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:38.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:38 smithi154 ceph-mon[36482]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-05-20T17:55:38.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:38 smithi154 ceph-mon[36482]: from='client.? 172.21.15.204:0/2618724729' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:55:38.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:38 smithi129 ceph-mon[32029]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-05-20T17:55:38.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:38 smithi129 ceph-mon[32029]: from='client.? 172.21.15.204:0/2618724729' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:55:38.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:38 smithi204 ceph-mon[37076]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-05-20T17:55:38.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:38 smithi204 ceph-mon[37076]: from='client.? 172.21.15.204:0/2618724729' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-20T17:55:40.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:39 smithi154 ceph-mon[36482]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:40.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:39 smithi129 ceph-mon[32029]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:40.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:39 smithi204 ceph-mon[37076]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:41.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:41 smithi204 ceph-mon[37076]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:42.101 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:41 smithi154 ceph-mon[36482]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:42.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:41 smithi129 ceph-mon[32029]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:44.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:43 smithi154 ceph-mon[36482]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:44.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:43 smithi129 ceph-mon[32029]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:44.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:43 smithi204 ceph-mon[37076]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:44.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-20T17:55:44.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:44 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:44.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:44 smithi204 ceph-mon[37076]: Deploying daemon osd.11 on smithi204 2024-05-20T17:55:45.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:44 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-20T17:55:45.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:44 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:45.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:44 smithi154 ceph-mon[36482]: Deploying daemon osd.11 on smithi204 2024-05-20T17:55:45.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-20T17:55:45.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:44 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:45.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:44 smithi129 ceph-mon[32029]: Deploying daemon osd.11 on smithi204 2024-05-20T17:55:45.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:45 smithi204 ceph-mon[37076]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:46.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:45 smithi154 ceph-mon[36482]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:46.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:45 smithi129 ceph-mon[32029]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:48.048 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:47 smithi204 ceph-mon[37076]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:48.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:47 smithi154 ceph-mon[36482]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:48.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:47 smithi129 ceph-mon[32029]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:50.041 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:49 smithi204 ceph-mon[37076]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:50.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:49 smithi154 ceph-mon[36482]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:50.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:49 smithi129 ceph-mon[32029]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:51.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:51 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:51.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:51 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:51.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:51 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:51.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:51 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:51.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:51 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:51.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:51 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:51.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:51 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:51.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:51 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:51.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:51 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:52.573 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:52 smithi204 ceph-mon[37076]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:52.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:52 smithi154 ceph-mon[36482]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:52.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:52 smithi129 ceph-mon[32029]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:54.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.296+0000 7f414ffff700 1 -- 172.21.15.204:0/3061957679 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f41681a1eb0 con 0x7f41500a2c20 2024-05-20T17:55:54.296 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 11 on host 'smithi204' 2024-05-20T17:55:54.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.298+0000 7f416d0c2700 1 -- 172.21.15.204:0/3061957679 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41500a2c20 msgr2=0x7f41500a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.298+0000 7f416d0c2700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41500a2c20 0x7f41500a5130 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f415c00b2e0 tx=0x7f415c00b880 comp rx=0 tx=0).stop 2024-05-20T17:55:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.298+0000 7f416d0c2700 1 -- 172.21.15.204:0/3061957679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41681020b0 msgr2=0x7f41681998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.298+0000 7f416d0c2700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41681020b0 0x7f41681998b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f415800e9f0 tx=0x7f415800ed00 comp rx=0 tx=0).stop 2024-05-20T17:55:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.299+0000 7f416d0c2700 1 -- 172.21.15.204:0/3061957679 shutdown_connections 2024-05-20T17:55:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.299+0000 7f416d0c2700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41500a2c20 0x7f41500a5130 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.299+0000 7f416d0c2700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4168103b80 0x7f41681a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.299+0000 7f416d0c2700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41681020b0 0x7f41681998b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.299+0000 7f416d0c2700 1 --2- 172.21.15.204:0/3061957679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41681031e0 0x7f4168199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.299+0000 7f416d0c2700 1 -- 172.21.15.204:0/3061957679 >> 172.21.15.204:0/3061957679 conn(0x7f41680fd6f0 msgr2=0x7f41680fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:55:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.299+0000 7f416d0c2700 1 -- 172.21.15.204:0/3061957679 shutdown_connections 2024-05-20T17:55:54.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:55:54.299+0000 7f416d0c2700 1 -- 172.21.15.204:0/3061957679 wait complete. 2024-05-20T17:55:54.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:54 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:54 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:54 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:54.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:54 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:54.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:54 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:54 smithi204 ceph-mon[37076]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:54.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:54 smithi204 ceph-mon[37076]: from='osd.11 [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-20T17:55:54.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:54 smithi204 ceph-mon[37076]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-20T17:55:54.610 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:54 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.611 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:54 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:54 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:54 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:54 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:54.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:54 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:54.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:54 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:54 smithi154 ceph-mon[36482]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:54.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:54 smithi154 ceph-mon[36482]: from='osd.11 [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-20T17:55:54.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:54 smithi154 ceph-mon[36482]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-20T17:55:54.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:54 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:54 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:54 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:54 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:54 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:54.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:54 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:54.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:54 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:54 smithi129 ceph-mon[32029]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:54.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:54 smithi129 ceph-mon[32029]: from='osd.11 [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-20T17:55:54.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:54 smithi129 ceph-mon[32029]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-20T17:55:54.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:54 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:54.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:54 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:55.021 DEBUG:teuthology.orchestra.run.smithi204:osd.11> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.11.service 2024-05-20T17:55:55.023 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-20T17:55:55.024 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd stat -f json 2024-05-20T17:55:55.271 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:55:55.582 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:55 smithi204 ceph-mon[37076]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-20T17:55:55.582 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:55 smithi204 ceph-mon[37076]: osdmap e79: 12 total, 11 up, 12 in 2024-05-20T17:55:55.582 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:55 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:55.583 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:55 smithi204 ceph-mon[37076]: from='osd.11 [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:55:55.583 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:55 smithi204 ceph-mon[37076]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:55:55.583 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:55 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:55.641 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:55 smithi129 ceph-mon[32029]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-20T17:55:55.641 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:55 smithi129 ceph-mon[32029]: osdmap e79: 12 total, 11 up, 12 in 2024-05-20T17:55:55.642 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:55 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:55.642 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:55 smithi129 ceph-mon[32029]: from='osd.11 [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:55:55.642 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:55 smithi129 ceph-mon[32029]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:55:55.642 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:55 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:55.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:55 smithi154 ceph-mon[36482]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-20T17:55:55.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:55 smithi154 ceph-mon[36482]: osdmap e79: 12 total, 11 up, 12 in 2024-05-20T17:55:55.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:55 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:55.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:55 smithi154 ceph-mon[36482]: from='osd.11 [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:55:55.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:55 smithi154 ceph-mon[36482]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-05-20T17:55:55.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:55 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:55:55.930 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 17:55:55 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T17:55:55.770+0000 7fee6e178700 -1 osd.11 0 waiting for initial osdmap 2024-05-20T17:55:55.930 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 17:55:55 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T17:55:55.784+0000 7fee65f25700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-20T17:55:56.507 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:56 smithi204 ceph-mon[37076]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:55:56.507 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:56 smithi204 ceph-mon[37076]: osdmap e80: 12 total, 11 up, 12 in 2024-05-20T17:55:56.507 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:56.507 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:56 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:56.507 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:56 smithi204 ceph-mon[37076]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:56.508 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:56 smithi204 ceph-mon[37076]: from='osd.11 ' entity='osd.11' 2024-05-20T17:55:56.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:56 smithi154 ceph-mon[36482]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:55:56.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:56 smithi154 ceph-mon[36482]: osdmap e80: 12 total, 11 up, 12 in 2024-05-20T17:55:56.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:56.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:56 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:56.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:56 smithi154 ceph-mon[36482]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:56.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:56 smithi154 ceph-mon[36482]: from='osd.11 ' entity='osd.11' 2024-05-20T17:55:56.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:56 smithi129 ceph-mon[32029]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-05-20T17:55:56.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:56 smithi129 ceph-mon[32029]: osdmap e80: 12 total, 11 up, 12 in 2024-05-20T17:55:56.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:56.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:56 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:56.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:56 smithi129 ceph-mon[32029]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-05-20T17:55:56.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:56 smithi129 ceph-mon[32029]: from='osd.11 ' entity='osd.11' 2024-05-20T17:55:57.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.048+0000 7f63671e0700 1 -- 172.21.15.129:0/1140855441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6360101b20 msgr2=0x7f636010b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:57.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.048+0000 7f63671e0700 1 --2- 172.21.15.129:0/1140855441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6360101b20 0x7f636010b740 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f635800b0a0 tx=0x7f635800b3b0 comp rx=0 tx=0).stop 2024-05-20T17:55:57.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.048+0000 7f63671e0700 1 -- 172.21.15.129:0/1140855441 shutdown_connections 2024-05-20T17:55:57.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.048+0000 7f63671e0700 1 --2- 172.21.15.129:0/1140855441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360104780 0x7f6360104b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:57.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.048+0000 7f63671e0700 1 --2- 172.21.15.129:0/1140855441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6360101b20 0x7f636010b740 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:57.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.048+0000 7f63671e0700 1 --2- 172.21.15.129:0/1140855441 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63601011d0 0x7f63601015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:57.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.048+0000 7f63671e0700 1 -- 172.21.15.129:0/1140855441 >> 172.21.15.129:0/1140855441 conn(0x7f63600fcea0 msgr2=0x7f63600ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:55:57.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.048+0000 7f63671e0700 1 -- 172.21.15.129:0/1140855441 shutdown_connections 2024-05-20T17:55:57.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.048+0000 7f63671e0700 1 -- 172.21.15.129:0/1140855441 wait complete. 2024-05-20T17:55:57.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.048+0000 7f63671e0700 1 Processor -- start 2024-05-20T17:55:57.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f63671e0700 1 -- start start 2024-05-20T17:55:57.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f63671e0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63601011d0 0x7f63601997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:57.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f63671e0700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6360101b20 0x7f6360199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:57.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f63671e0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360104780 0x7f63601a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:57.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f63671e0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f636010ed90 con 0x7f63601011d0 2024-05-20T17:55:57.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f63671e0700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f636010ec10 con 0x7f6360104780 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f63671e0700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f636010ef10 con 0x7f6360101b20 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f6364f7c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63601011d0 0x7f63601997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f6364f7c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63601011d0 0x7f63601997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43474/0 (socket says 172.21.15.129:43474) 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f6364f7c700 1 -- 172.21.15.129:0/3261401672 learned_addr learned my addr 172.21.15.129:0/3261401672 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f635ffff700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6360101b20 0x7f6360199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f636577d700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360104780 0x7f63601a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f6364f7c700 1 -- 172.21.15.129:0/3261401672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6360101b20 msgr2=0x7f6360199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f6364f7c700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6360101b20 0x7f6360199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f6364f7c700 1 -- 172.21.15.129:0/3261401672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360104780 msgr2=0x7f63601a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f6364f7c700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360104780 0x7f63601a0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f6364f7c700 1 -- 172.21.15.129:0/3261401672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63601a1330 con 0x7f63601011d0 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f636577d700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360104780 0x7f63601a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:55:57.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.049+0000 7f6364f7c700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63601011d0 0x7f63601997d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f635400c970 tx=0x7f635400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:55:57.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.050+0000 7f635dffb700 1 -- 172.21.15.129:0/3261401672 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635400ce60 con 0x7f63601011d0 2024-05-20T17:55:57.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.050+0000 7f635dffb700 1 -- 172.21.15.129:0/3261401672 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6354007de0 con 0x7f63601011d0 2024-05-20T17:55:57.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.050+0000 7f635dffb700 1 -- 172.21.15.129:0/3261401672 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354003d00 con 0x7f63601011d0 2024-05-20T17:55:57.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.050+0000 7f63671e0700 1 -- 172.21.15.129:0/3261401672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63601a1620 con 0x7f63601011d0 2024-05-20T17:55:57.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.050+0000 7f63671e0700 1 -- 172.21.15.129:0/3261401672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63601a1bb0 con 0x7f63601011d0 2024-05-20T17:55:57.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.051+0000 7f63671e0700 1 -- 172.21.15.129:0/3261401672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6360050170 con 0x7f63601011d0 2024-05-20T17:55:57.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.052+0000 7f635dffb700 1 -- 172.21.15.129:0/3261401672 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6354007900 con 0x7f63601011d0 2024-05-20T17:55:57.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.053+0000 7f635dffb700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f63500a2ca0 0x7f63500a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:55:57.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.053+0000 7f635ffff700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f63500a2ca0 0x7f63500a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:55:57.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.053+0000 7f635dffb700 1 -- 172.21.15.129:0/3261401672 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f63540ce6c0 con 0x7f63601011d0 2024-05-20T17:55:57.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.053+0000 7f635ffff700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f63500a2ca0 0x7f63500a51b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f636019a880 tx=0x7f634c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:55:57.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.058+0000 7f635dffb700 1 -- 172.21.15.129:0/3261401672 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6354014030 con 0x7f63601011d0 2024-05-20T17:55:57.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.254+0000 7f63671e0700 1 -- 172.21.15.129:0/3261401672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f636002dc60 con 0x7f63601011d0 2024-05-20T17:55:57.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.255+0000 7f635dffb700 1 -- 172.21.15.129:0/3261401672 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f635409bac0 con 0x7f63601011d0 2024-05-20T17:55:57.254 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.257+0000 7f63671e0700 1 -- 172.21.15.129:0/3261401672 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f63500a2ca0 msgr2=0x7f63500a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.258+0000 7f63671e0700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f63500a2ca0 0x7f63500a51b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f636019a880 tx=0x7f634c009260 comp rx=0 tx=0).stop 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.258+0000 7f63671e0700 1 -- 172.21.15.129:0/3261401672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63601011d0 msgr2=0x7f63601997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.258+0000 7f63671e0700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63601011d0 0x7f63601997d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f635400c970 tx=0x7f635400cc80 comp rx=0 tx=0).stop 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.258+0000 7f63671e0700 1 -- 172.21.15.129:0/3261401672 shutdown_connections 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.258+0000 7f63671e0700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f63500a2ca0 0x7f63500a51b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.258+0000 7f63671e0700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360104780 0x7f63601a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.258+0000 7f63671e0700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63601011d0 0x7f63601997d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.258+0000 7f63671e0700 1 --2- 172.21.15.129:0/3261401672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6360101b20 0x7f6360199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.258+0000 7f63671e0700 1 -- 172.21.15.129:0/3261401672 >> 172.21.15.129:0/3261401672 conn(0x7f63600fcea0 msgr2=0x7f6360107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.258+0000 7f63671e0700 1 -- 172.21.15.129:0/3261401672 shutdown_connections 2024-05-20T17:55:57.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:55:57.258+0000 7f63671e0700 1 -- 172.21.15.129:0/3261401672 wait complete. 2024-05-20T17:55:57.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:57 smithi154 ceph-mon[36482]: purged_snaps scrub starts 2024-05-20T17:55:57.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:57 smithi154 ceph-mon[36482]: purged_snaps scrub ok 2024-05-20T17:55:57.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:57 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:57.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:57 smithi154 ceph-mon[36482]: osd.11 [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241] boot 2024-05-20T17:55:57.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:57 smithi154 ceph-mon[36482]: osdmap e81: 12 total, 12 up, 12 in 2024-05-20T17:55:57.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:57 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:57.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:57 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3261401672' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-20T17:55:57.647 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:57 smithi129 ceph-mon[32029]: purged_snaps scrub starts 2024-05-20T17:55:57.647 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:57 smithi129 ceph-mon[32029]: purged_snaps scrub ok 2024-05-20T17:55:57.647 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:57 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:57.647 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:57 smithi129 ceph-mon[32029]: osd.11 [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241] boot 2024-05-20T17:55:57.647 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:57 smithi129 ceph-mon[32029]: osdmap e81: 12 total, 12 up, 12 in 2024-05-20T17:55:57.647 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:57 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:57.648 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:57 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3261401672' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-20T17:55:57.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:57 smithi204 ceph-mon[37076]: purged_snaps scrub starts 2024-05-20T17:55:57.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:57 smithi204 ceph-mon[37076]: purged_snaps scrub ok 2024-05-20T17:55:57.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:57 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:57.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:57 smithi204 ceph-mon[37076]: osd.11 [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241] boot 2024-05-20T17:55:57.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:57 smithi204 ceph-mon[37076]: osdmap e81: 12 total, 12 up, 12 in 2024-05-20T17:55:57.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:57 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:55:57.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:57 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3261401672' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-20T17:55:58.051 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1716227756,"num_in_osds":12,"osd_in_since":1716227737,"num_remapped_pgs":0} 2024-05-20T17:55:58.052 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd dump --format=json 2024-05-20T17:55:58.289 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:55:58.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:55:58.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: osdmap e82: 12 total, 12 up, 12 in 2024-05-20T17:55:58.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:58.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:58.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:58.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:58.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:58.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:58 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:58.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:55:58.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: osdmap e82: 12 total, 12 up, 12 in 2024-05-20T17:55:58.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:58.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:58.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:58.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:58.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:58.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:58 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:58.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:55:58.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: osdmap e82: 12 total, 12 up, 12 in 2024-05-20T17:55:58.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:58.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:58.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:55:58.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:58.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:55:58.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:55:58.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:58 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:55:59.599 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:59 smithi204 ceph-mon[37076]: Detected new or changed devices on smithi204 2024-05-20T17:55:59.600 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:55:59 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi204 to 4272M 2024-05-20T17:55:59.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:59 smithi154 ceph-mon[36482]: Detected new or changed devices on smithi204 2024-05-20T17:55:59.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:55:59 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi204 to 4272M 2024-05-20T17:55:59.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:59 smithi129 ceph-mon[32029]: Detected new or changed devices on smithi204 2024-05-20T17:55:59.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:55:59 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi204 to 4272M 2024-05-20T17:56:00.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.080+0000 7f420ea6a700 1 -- 172.21.15.129:0/1078081989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208107d80 msgr2=0x7f4208108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:00.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.080+0000 7f420ea6a700 1 --2- 172.21.15.129:0/1078081989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208107d80 0x7f4208108190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f41f800b0d0 tx=0x7f41f800b3e0 comp rx=0 tx=0).stop 2024-05-20T17:56:00.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.080+0000 7f420ea6a700 1 -- 172.21.15.129:0/1078081989 shutdown_connections 2024-05-20T17:56:00.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.080+0000 7f420ea6a700 1 --2- 172.21.15.129:0/1078081989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42081086d0 0x7f4208110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:00.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.080+0000 7f420ea6a700 1 --2- 172.21.15.129:0/1078081989 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4208103450 0x7f4208103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:00.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.080+0000 7f420ea6a700 1 --2- 172.21.15.129:0/1078081989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208107d80 0x7f4208108190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:00.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.080+0000 7f420ea6a700 1 -- 172.21.15.129:0/1078081989 >> 172.21.15.129:0/1078081989 conn(0x7f42080fed20 msgr2=0x7f4208101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:00.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.080+0000 7f420ea6a700 1 -- 172.21.15.129:0/1078081989 shutdown_connections 2024-05-20T17:56:00.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.080+0000 7f420ea6a700 1 -- 172.21.15.129:0/1078081989 wait complete. 2024-05-20T17:56:00.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.080+0000 7f420ea6a700 1 Processor -- start 2024-05-20T17:56:00.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.080+0000 7f420ea6a700 1 -- start start 2024-05-20T17:56:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420ea6a700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4208103450 0x7f4208199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420ea6a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4208107d80 0x7f420819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420ea6a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42081086d0 0x7f420819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420ea6a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4208113430 con 0x7f42081086d0 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420ea6a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f42081132b0 con 0x7f4208107d80 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420ea6a700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f42081135b0 con 0x7f4208103450 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420d007700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42081086d0 0x7f420819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420d007700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42081086d0 0x7f420819e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51884/0 (socket says 172.21.15.129:51884) 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420d007700 1 -- 172.21.15.129:0/2447694616 learned_addr learned my addr 172.21.15.129:0/2447694616 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f4207fff700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4208107d80 0x7f420819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420c806700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4208103450 0x7f4208199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420d007700 1 -- 172.21.15.129:0/2447694616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4208103450 msgr2=0x7f4208199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420d007700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4208103450 0x7f4208199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420d007700 1 -- 172.21.15.129:0/2447694616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4208107d80 msgr2=0x7f420819a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420d007700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4208107d80 0x7f420819a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420d007700 1 -- 172.21.15.129:0/2447694616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420819ec70 con 0x7f42081086d0 2024-05-20T17:56:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420c806700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4208103450 0x7f4208199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:56:00.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.081+0000 7f420d007700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42081086d0 0x7f420819e5d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f41f000d980 tx=0x7f41f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:00.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.082+0000 7f4205ffb700 1 -- 172.21.15.129:0/2447694616 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f0016070 con 0x7f42081086d0 2024-05-20T17:56:00.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.082+0000 7f420ea6a700 1 -- 172.21.15.129:0/2447694616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420819ef60 con 0x7f42081086d0 2024-05-20T17:56:00.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.082+0000 7f4205ffb700 1 -- 172.21.15.129:0/2447694616 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f0004d10 con 0x7f42081086d0 2024-05-20T17:56:00.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.082+0000 7f4205ffb700 1 -- 172.21.15.129:0/2447694616 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f00056c0 con 0x7f42081086d0 2024-05-20T17:56:00.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.082+0000 7f420ea6a700 1 -- 172.21.15.129:0/2447694616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42081a6880 con 0x7f42081086d0 2024-05-20T17:56:00.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.083+0000 7f420ea6a700 1 -- 172.21.15.129:0/2447694616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4208050170 con 0x7f42081086d0 2024-05-20T17:56:00.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.084+0000 7f4205ffb700 1 -- 172.21.15.129:0/2447694616 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f41f0004750 con 0x7f42081086d0 2024-05-20T17:56:00.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.084+0000 7f4205ffb700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41f40a2c20 0x7f41f40a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:00.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.085+0000 7f420c806700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41f40a2c20 0x7f41f40a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:00.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.085+0000 7f4205ffb700 1 -- 172.21.15.129:0/2447694616 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f41f00ce7b0 con 0x7f42081086d0 2024-05-20T17:56:00.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.085+0000 7f420c806700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41f40a2c20 0x7f41f40a5130 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4200011490 tx=0x7f4200009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:00.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.089+0000 7f4205ffb700 1 -- 172.21.15.129:0/2447694616 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f41f001f020 con 0x7f42081086d0 2024-05-20T17:56:00.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.285+0000 7f420ea6a700 1 -- 172.21.15.129:0/2447694616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f420802dc60 con 0x7f42081086d0 2024-05-20T17:56:00.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.286+0000 7f4205ffb700 1 -- 172.21.15.129:0/2447694616 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18477 (secure 0 0 0) 0x7f41f0022350 con 0x7f42081086d0 2024-05-20T17:56:00.285 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:00.286 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":82,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:55:57.775106+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:56:00.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.288+0000 7f420ea6a700 1 -- 172.21.15.129:0/2447694616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41f40a2c20 msgr2=0x7f41f40a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:00.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.288+0000 7f420ea6a700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41f40a2c20 0x7f41f40a5130 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4200011490 tx=0x7f4200009260 comp rx=0 tx=0).stop 2024-05-20T17:56:00.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.288+0000 7f420ea6a700 1 -- 172.21.15.129:0/2447694616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42081086d0 msgr2=0x7f420819e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:00.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.288+0000 7f420ea6a700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42081086d0 0x7f420819e5d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f41f000d980 tx=0x7f41f000dc90 comp rx=0 tx=0).stop 2024-05-20T17:56:00.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.288+0000 7f420ea6a700 1 -- 172.21.15.129:0/2447694616 shutdown_connections 2024-05-20T17:56:00.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.288+0000 7f420ea6a700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41f40a2c20 0x7f41f40a5130 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:00.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.288+0000 7f420ea6a700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42081086d0 0x7f420819e5d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:00.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.288+0000 7f420ea6a700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4208103450 0x7f4208199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:00.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.288+0000 7f420ea6a700 1 --2- 172.21.15.129:0/2447694616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4208107d80 0x7f420819a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:00.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.288+0000 7f420ea6a700 1 -- 172.21.15.129:0/2447694616 >> 172.21.15.129:0/2447694616 conn(0x7f42080fed20 msgr2=0x7f4208107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:00.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.289+0000 7f420ea6a700 1 -- 172.21.15.129:0/2447694616 shutdown_connections 2024-05-20T17:56:00.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:00.289+0000 7f420ea6a700 1 -- 172.21.15.129:0/2447694616 wait complete. 2024-05-20T17:56:00.611 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:00 smithi129 ceph-mon[32029]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:00.611 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:00 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2447694616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:00.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:00 smithi154 ceph-mon[36482]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:00.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:00 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2447694616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:00.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:00 smithi204 ceph-mon[37076]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:00.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:00 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2447694616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:00.913 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T17:56:00.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-20T17:56:01.154 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:56:01.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:01 smithi129 ceph-mon[32029]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:02.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:01 smithi154 ceph-mon[36482]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:02.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:01 smithi204 ceph-mon[37076]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:02.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.779+0000 7f697e7b8700 1 -- 172.21.15.129:0/2246044518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69781075e0 msgr2=0x7f697810ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:02.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.779+0000 7f697e7b8700 1 --2- 172.21.15.129:0/2246044518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69781075e0 0x7f697810ef40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f697000b0d0 tx=0x7f697000b3e0 comp rx=0 tx=0).stop 2024-05-20T17:56:02.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.779+0000 7f697e7b8700 1 -- 172.21.15.129:0/2246044518 shutdown_connections 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.779+0000 7f697e7b8700 1 --2- 172.21.15.129:0/2246044518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69781075e0 0x7f697810ef40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.779+0000 7f697e7b8700 1 --2- 172.21.15.129:0/2246044518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6978102360 0x7f6978102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.779+0000 7f697e7b8700 1 --2- 172.21.15.129:0/2246044518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6978106c90 0x7f69781070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.779+0000 7f697e7b8700 1 -- 172.21.15.129:0/2246044518 >> 172.21.15.129:0/2246044518 conn(0x7f69780fddb0 msgr2=0x7f69781001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.779+0000 7f697e7b8700 1 -- 172.21.15.129:0/2246044518 shutdown_connections 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.779+0000 7f697e7b8700 1 -- 172.21.15.129:0/2246044518 wait complete. 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.780+0000 7f697e7b8700 1 Processor -- start 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.780+0000 7f697e7b8700 1 -- start start 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.780+0000 7f697e7b8700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6978102360 0x7f6978199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.780+0000 7f697e7b8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6978106c90 0x7f697819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.780+0000 7f697e7b8700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69781075e0 0x7f697819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:02.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.780+0000 7f697e7b8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f69781123e0 con 0x7f6978106c90 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.780+0000 7f697e7b8700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6978112260 con 0x7f69781075e0 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.780+0000 7f697e7b8700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6978112560 con 0x7f6978102360 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f696ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6978106c90 0x7f697819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f696ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6978106c90 0x7f697819a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51912/0 (socket says 172.21.15.129:51912) 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f696ffff700 1 -- 172.21.15.129:0/234727691 learned_addr learned my addr 172.21.15.129:0/234727691 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f697cd55700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69781075e0 0x7f697819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f6977fff700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6978102360 0x7f6978199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f696ffff700 1 -- 172.21.15.129:0/234727691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6978102360 msgr2=0x7f6978199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f696ffff700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6978102360 0x7f6978199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f696ffff700 1 -- 172.21.15.129:0/234727691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69781075e0 msgr2=0x7f697819e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f696ffff700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69781075e0 0x7f697819e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f696ffff700 1 -- 172.21.15.129:0/234727691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697819ec40 con 0x7f6978106c90 2024-05-20T17:56:02.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f6977fff700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6978102360 0x7f6978199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:56:02.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f696ffff700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6978106c90 0x7f697819a360 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f696000c8f0 tx=0x7f696000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:02.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f6975ffb700 1 -- 172.21.15.129:0/234727691 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6960007940 con 0x7f6978106c90 2024-05-20T17:56:02.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f6975ffb700 1 -- 172.21.15.129:0/234727691 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f696000f450 con 0x7f6978106c90 2024-05-20T17:56:02.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f697e7b8700 1 -- 172.21.15.129:0/234727691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697819ef30 con 0x7f6978106c90 2024-05-20T17:56:02.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f6975ffb700 1 -- 172.21.15.129:0/234727691 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6960018660 con 0x7f6978106c90 2024-05-20T17:56:02.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.781+0000 7f697e7b8700 1 -- 172.21.15.129:0/234727691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69780fe1c0 con 0x7f6978106c90 2024-05-20T17:56:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.782+0000 7f697e7b8700 1 -- 172.21.15.129:0/234727691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6978103810 con 0x7f6978106c90 2024-05-20T17:56:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.784+0000 7f6975ffb700 1 -- 172.21.15.129:0/234727691 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f696000f600 con 0x7f6978106c90 2024-05-20T17:56:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.785+0000 7f6975ffb700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f69640a2c20 0x7f69640a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.785+0000 7f6977fff700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f69640a2c20 0x7f69640a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.785+0000 7f6975ffb700 1 -- 172.21.15.129:0/234727691 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f69600cebe0 con 0x7f6978106c90 2024-05-20T17:56:02.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.785+0000 7f6977fff700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f69640a2c20 0x7f69640a5130 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69680099d0 tx=0x7f6968008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:02.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.789+0000 7f6975ffb700 1 -- 172.21.15.129:0/234727691 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6960014030 con 0x7f6978106c90 2024-05-20T17:56:02.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.983+0000 7f697e7b8700 1 -- 172.21.15.129:0/234727691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f697819b0d0 con 0x7f6978106c90 2024-05-20T17:56:02.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.984+0000 7f6975ffb700 1 -- 172.21.15.129:0/234727691 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f696009bfd0 con 0x7f6978106c90 2024-05-20T17:56:02.983 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T17:56:02.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 -- 172.21.15.129:0/234727691 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f69640a2c20 msgr2=0x7f69640a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:02.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f69640a2c20 0x7f69640a5130 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69680099d0 tx=0x7f6968008040 comp rx=0 tx=0).stop 2024-05-20T17:56:02.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 -- 172.21.15.129:0/234727691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6978106c90 msgr2=0x7f697819a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:02.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6978106c90 0x7f697819a360 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f696000c8f0 tx=0x7f696000cc00 comp rx=0 tx=0).stop 2024-05-20T17:56:02.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 -- 172.21.15.129:0/234727691 shutdown_connections 2024-05-20T17:56:02.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f69640a2c20 0x7f69640a5130 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:02.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69781075e0 0x7f697819e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:02.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6978102360 0x7f6978199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:02.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 --2- 172.21.15.129:0/234727691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6978106c90 0x7f697819a360 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:02.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 -- 172.21.15.129:0/234727691 >> 172.21.15.129:0/234727691 conn(0x7f69780fddb0 msgr2=0x7f697806b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:02.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 -- 172.21.15.129:0/234727691 shutdown_connections 2024-05-20T17:56:02.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:02.986+0000 7f697e7b8700 1 -- 172.21.15.129:0/234727691 wait complete. 2024-05-20T17:56:03.359 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:03 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/234727691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:56:03.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:03 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/234727691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:56:03.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:03 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/234727691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:56:03.682 INFO:tasks.cephadm:Adding mds.a on smithi129 2024-05-20T17:56:03.682 INFO:tasks.cephadm:Adding mds.d on smithi129 2024-05-20T17:56:03.682 INFO:tasks.cephadm:Adding mds.g on smithi129 2024-05-20T17:56:03.683 INFO:tasks.cephadm:Adding mds.j on smithi129 2024-05-20T17:56:03.683 INFO:tasks.cephadm:Adding mds.b on smithi154 2024-05-20T17:56:03.683 INFO:tasks.cephadm:Adding mds.e on smithi154 2024-05-20T17:56:03.683 INFO:tasks.cephadm:Adding mds.h on smithi154 2024-05-20T17:56:03.683 INFO:tasks.cephadm:Adding mds.k on smithi154 2024-05-20T17:56:03.683 INFO:tasks.cephadm:Adding mds.c on smithi204 2024-05-20T17:56:03.683 INFO:tasks.cephadm:Adding mds.f on smithi204 2024-05-20T17:56:03.683 INFO:tasks.cephadm:Adding mds.i on smithi204 2024-05-20T17:56:03.683 INFO:tasks.cephadm:Adding mds.l on smithi204 2024-05-20T17:56:03.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph orch apply mds all '12;smithi129=a;smithi129=d;smithi129=g;smithi129=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l' 2024-05-20T17:56:03.936 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T17:56:04.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:04 smithi204 ceph-mon[37076]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:04.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:04 smithi154 ceph-mon[36482]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:04.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:04 smithi129 ceph-mon[32029]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:05.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.570+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3300831550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e01011d0 msgr2=0x7fc5e01015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:05.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.570+0000 7fc5e5be7700 1 --2- 172.21.15.204:0/3300831550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e01011d0 0x7fc5e01015e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009990 tx=0x7fc5c8009ca0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.570+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3300831550 shutdown_connections 2024-05-20T17:56:05.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.570+0000 7fc5e5be7700 1 --2- 172.21.15.204:0/3300831550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e0104780 0x7fc5e0104b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.570+0000 7fc5e5be7700 1 --2- 172.21.15.204:0/3300831550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e0101b20 0x7fc5e010b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.570+0000 7fc5e5be7700 1 --2- 172.21.15.204:0/3300831550 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e01011d0 0x7fc5e01015e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.569 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.570+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3300831550 >> 172.21.15.204:0/3300831550 conn(0x7fc5e00fcea0 msgr2=0x7fc5e00ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:05.570 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.570+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3300831550 shutdown_connections 2024-05-20T17:56:05.570 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.570+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3300831550 wait complete. 2024-05-20T17:56:05.571 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.571+0000 7fc5e5be7700 1 Processor -- start 2024-05-20T17:56:05.571 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.571+0000 7fc5e5be7700 1 -- start start 2024-05-20T17:56:05.571 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.571+0000 7fc5e5be7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e01011d0 0x7fc5e0199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:05.571 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.571+0000 7fc5e5be7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e0101b20 0x7fc5e0199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.571+0000 7fc5e5be7700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0104780 0x7fc5e01a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.571+0000 7fc5e5be7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e010ed90 con 0x7fc5e0101b20 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.571+0000 7fc5e5be7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e010ec10 con 0x7fc5e01011d0 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.571+0000 7fc5e5be7700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e010ef10 con 0x7fc5e0104780 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.571+0000 7fc5dffff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0104780 0x7fc5e01a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.571+0000 7fc5dffff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0104780 0x7fc5e01a0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:48246/0 (socket says 172.21.15.204:48246) 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.571+0000 7fc5dffff700 1 -- 172.21.15.204:0/3253869592 learned_addr learned my addr 172.21.15.204:0/3253869592 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5df7fe700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e01011d0 0x7fc5e0199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5deffd700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e0101b20 0x7fc5e0199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5dffff700 1 -- 172.21.15.204:0/3253869592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e01011d0 msgr2=0x7fc5e0199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5dffff700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e01011d0 0x7fc5e0199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5dffff700 1 -- 172.21.15.204:0/3253869592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e0101b20 msgr2=0x7fc5e0199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:05.572 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5dffff700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e0101b20 0x7fc5e0199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.573 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5dffff700 1 -- 172.21.15.204:0/3253869592 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e01a1370 con 0x7fc5e0104780 2024-05-20T17:56:05.573 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5dffff700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0104780 0x7fc5e01a0cf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8007ee0 tx=0x7fc5d800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:05.573 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5dcff9700 1 -- 172.21.15.204:0/3253869592 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d800dbd0 con 0x7fc5e0104780 2024-05-20T17:56:05.573 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5dcff9700 1 -- 172.21.15.204:0/3253869592 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5d800dd70 con 0x7fc5e0104780 2024-05-20T17:56:05.573 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3253869592 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e01a1660 con 0x7fc5e0104780 2024-05-20T17:56:05.573 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5dcff9700 1 -- 172.21.15.204:0/3253869592 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d80133f0 con 0x7fc5e0104780 2024-05-20T17:56:05.573 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.572+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3253869592 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e01a1b50 con 0x7fc5e0104780 2024-05-20T17:56:05.574 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.573+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3253869592 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5e0050170 con 0x7fc5e0104780 2024-05-20T17:56:05.574 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.574+0000 7fc5dcff9700 1 -- 172.21.15.204:0/3253869592 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc5d801a020 con 0x7fc5e0104780 2024-05-20T17:56:05.574 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.575+0000 7fc5dcff9700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5cc0a2c20 0x7fc5cc0a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:05.574 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.575+0000 7fc5dcff9700 1 -- 172.21.15.204:0/3253869592 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc5d80cea40 con 0x7fc5e0104780 2024-05-20T17:56:05.574 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.575+0000 7fc5df7fe700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5cc0a2c20 0x7fc5cc0a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:05.574 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.576+0000 7fc5df7fe700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5cc0a2c20 0x7fc5cc0a5130 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0005fd0 tx=0x7fc5d0005dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:05.577 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.579+0000 7fc5dcff9700 1 -- 172.21.15.204:0/3253869592 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc5d8010070 con 0x7fc5e0104780 2024-05-20T17:56:05.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.789+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3253869592 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi129=a;smithi129=d;smithi129=g;smithi129=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}) -- 0x7fc5e0062b80 con 0x7fc5cc0a2c20 2024-05-20T17:56:05.796 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.798+0000 7fc5dcff9700 1 -- 172.21.15.204:0/3253869592 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fc5e0062b80 con 0x7fc5cc0a2c20 2024-05-20T17:56:05.797 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mds.all update... 2024-05-20T17:56:05.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3253869592 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5cc0a2c20 msgr2=0x7fc5cc0a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:05.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5cc0a2c20 0x7fc5cc0a5130 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0005fd0 tx=0x7fc5d0005dd0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3253869592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0104780 msgr2=0x7fc5e01a0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:05.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0104780 0x7fc5e01a0cf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8007ee0 tx=0x7fc5d800d3b0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3253869592 shutdown_connections 2024-05-20T17:56:05.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5cc0a2c20 0x7fc5cc0a5130 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5e01011d0 0x7fc5e0199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc5e0104780 0x7fc5e01a0cf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 --2- 172.21.15.204:0/3253869592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e0101b20 0x7fc5e0199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:05.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3253869592 >> 172.21.15.204:0/3253869592 conn(0x7fc5e00fcea0 msgr2=0x7fc5e0107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:05.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3253869592 shutdown_connections 2024-05-20T17:56:05.800 INFO:teuthology.orchestra.run.smithi204.stderr:2024-05-20T17:56:05.800+0000 7fc5e5be7700 1 -- 172.21.15.204:0/3253869592 wait complete. 2024-05-20T17:56:05.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:05 smithi204 ceph-mon[37076]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:06.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:05 smithi154 ceph-mon[36482]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:06.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:05 smithi129 ceph-mon[32029]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:06.555 DEBUG:teuthology.orchestra.run.smithi129:mds.a> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.a.service 2024-05-20T17:56:06.558 DEBUG:teuthology.orchestra.run.smithi129:mds.d> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.d.service 2024-05-20T17:56:06.601 DEBUG:teuthology.orchestra.run.smithi129:mds.g> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.g.service 2024-05-20T17:56:06.646 DEBUG:teuthology.orchestra.run.smithi129:mds.j> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.j.service 2024-05-20T17:56:06.876 DEBUG:teuthology.orchestra.run.smithi154:mds.b> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.b.service 2024-05-20T17:56:06.879 DEBUG:teuthology.orchestra.run.smithi154:mds.e> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.e.service 2024-05-20T17:56:06.897 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:06 smithi129 ceph-mon[32029]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi129=a;smithi129=d;smithi129=g;smithi129=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:06.897 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:06 smithi129 ceph-mon[32029]: Saving service mds.all spec with placement smithi129=a;smithi129=d;smithi129=g;smithi129=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-05-20T17:56:06.897 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:06.897 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:56:06.897 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:06.897 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:56:06.898 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:06.898 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:06.898 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:06.898 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:06 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:06.898 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:06 smithi129 ceph-mon[32029]: Deploying daemon mds.l on smithi204 2024-05-20T17:56:06.924 DEBUG:teuthology.orchestra.run.smithi154:mds.h> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.h.service 2024-05-20T17:56:06.968 DEBUG:teuthology.orchestra.run.smithi154:mds.k> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.k.service 2024-05-20T17:56:07.012 DEBUG:teuthology.orchestra.run.smithi204:mds.c> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.c.service 2024-05-20T17:56:07.015 DEBUG:teuthology.orchestra.run.smithi204:mds.f> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.f.service 2024-05-20T17:56:07.059 DEBUG:teuthology.orchestra.run.smithi204:mds.i> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.i.service 2024-05-20T17:56:07.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:06 smithi204 ceph-mon[37076]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi129=a;smithi129=d;smithi129=g;smithi129=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:07.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:06 smithi204 ceph-mon[37076]: Saving service mds.all spec with placement smithi129=a;smithi129=d;smithi129=g;smithi129=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-05-20T17:56:07.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:07.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:56:07.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:07.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:56:07.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:07.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:07.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:07.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:06 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:07.076 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:06 smithi204 ceph-mon[37076]: Deploying daemon mds.l on smithi204 2024-05-20T17:56:07.077 DEBUG:teuthology.orchestra.run.smithi204:mds.l> sudo journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.l.service 2024-05-20T17:56:07.120 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-20T17:56:07.121 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:56:07.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:06 smithi154 ceph-mon[36482]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi129=a;smithi129=d;smithi129=g;smithi129=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:07.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:06 smithi154 ceph-mon[36482]: Saving service mds.all spec with placement smithi129=a;smithi129=d;smithi129=g;smithi129=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-05-20T17:56:07.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:07.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:56:07.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:07.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:56:07.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:07.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:07.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:07.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:06 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:07.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:06 smithi154 ceph-mon[36482]: Deploying daemon mds.l on smithi204 2024-05-20T17:56:07.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.268+0000 7fe27b6fe640 1 -- 172.21.15.129:0/2449131414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe274107080 msgr2=0x7fe2741094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.268+0000 7fe27b6fe640 1 --2- 172.21.15.129:0/2449131414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe274107080 0x7fe2741094a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe26c00adb0 tx=0x7fe26c02b7a0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.269+0000 7fe27b6fe640 1 -- 172.21.15.129:0/2449131414 shutdown_connections 2024-05-20T17:56:07.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.269+0000 7fe27b6fe640 1 --2- 172.21.15.129:0/2449131414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe274107080 0x7fe2741094a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.269+0000 7fe27b6fe640 1 --2- 172.21.15.129:0/2449131414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe274102380 0x7fe274102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.269+0000 7fe27b6fe640 1 --2- 172.21.15.129:0/2449131414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe274101140 0x7fe274101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.269+0000 7fe27b6fe640 1 -- 172.21.15.129:0/2449131414 >> 172.21.15.129:0/2449131414 conn(0x7fe2740fc310 msgr2=0x7fe2740fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:07.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.269+0000 7fe27b6fe640 1 -- 172.21.15.129:0/2449131414 shutdown_connections 2024-05-20T17:56:07.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.269+0000 7fe27b6fe640 1 -- 172.21.15.129:0/2449131414 wait complete. 2024-05-20T17:56:07.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.269+0000 7fe27b6fe640 1 Processor -- start 2024-05-20T17:56:07.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.269+0000 7fe27b6fe640 1 -- start start 2024-05-20T17:56:07.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe27b6fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe274101140 0x7fe274199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe27b6fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe274102380 0x7fe2741996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe27b6fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe274107080 0x7fe2741a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe27b6fe640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe27410c520 con 0x7fe274102380 2024-05-20T17:56:07.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe27b6fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe27410c3a0 con 0x7fe274107080 2024-05-20T17:56:07.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe27b6fe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe27410c6a0 con 0x7fe274101140 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe278c99640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe274102380 0x7fe2741996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe278c99640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe274102380 0x7fe2741996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51926/0 (socket says 172.21.15.129:51926) 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe278c99640 1 -- 172.21.15.129:0/3512639210 learned_addr learned my addr 172.21.15.129:0/3512639210 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe27949a640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe274101140 0x7fe274199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe279c9b640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe274107080 0x7fe2741a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe278c99640 1 -- 172.21.15.129:0/3512639210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe274101140 msgr2=0x7fe274199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe278c99640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe274101140 0x7fe274199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe278c99640 1 -- 172.21.15.129:0/3512639210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe274107080 msgr2=0x7fe2741a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe278c99640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe274107080 0x7fe2741a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe278c99640 1 -- 172.21.15.129:0/3512639210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2741a0cd0 con 0x7fe274102380 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe278c99640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe274102380 0x7fe2741996d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe26400e9f0 tx=0x7fe26400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe26a7fc640 1 -- 172.21.15.129:0/3512639210 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26400cd80 con 0x7fe274102380 2024-05-20T17:56:07.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe27b6fe640 1 -- 172.21.15.129:0/3512639210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2741a0fc0 con 0x7fe274102380 2024-05-20T17:56:07.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe26a7fc640 1 -- 172.21.15.129:0/3512639210 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe264005500 con 0x7fe274102380 2024-05-20T17:56:07.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe26a7fc640 1 -- 172.21.15.129:0/3512639210 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264010690 con 0x7fe274102380 2024-05-20T17:56:07.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.270+0000 7fe27b6fe640 1 -- 172.21.15.129:0/3512639210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2741a1430 con 0x7fe274102380 2024-05-20T17:56:07.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.271+0000 7fe27b6fe640 1 -- 172.21.15.129:0/3512639210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2741a16c0 con 0x7fe274102380 2024-05-20T17:56:07.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.273+0000 7fe26a7fc640 1 -- 172.21.15.129:0/3512639210 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fe2640046c0 con 0x7fe274102380 2024-05-20T17:56:07.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.273+0000 7fe26a7fc640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe2600a2b90 0x7fe2600a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.274+0000 7fe26a7fc640 1 -- 172.21.15.129:0/3512639210 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe2640cf260 con 0x7fe274102380 2024-05-20T17:56:07.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.274+0000 7fe27949a640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe2600a2b90 0x7fe2600a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.274+0000 7fe27949a640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe2600a2b90 0x7fe2600a50a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe274101d40 tx=0x7fe27000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:07.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.277+0000 7fe26a7fc640 1 -- 172.21.15.129:0/3512639210 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe264053850 con 0x7fe274102380 2024-05-20T17:56:07.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.473+0000 7fe27b6fe640 1 -- 172.21.15.129:0/3512639210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe2741a1890 con 0x7fe274102380 2024-05-20T17:56:07.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.474+0000 7fe26a7fc640 1 -- 172.21.15.129:0/3512639210 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18477 (secure 0 0 0) 0x7fe26409c660 con 0x7fe274102380 2024-05-20T17:56:07.473 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:07.474 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":82,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:55:57.775106+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:56:07.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 -- 172.21.15.129:0/3512639210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe2600a2b90 msgr2=0x7fe2600a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe2600a2b90 0x7fe2600a50a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe274101d40 tx=0x7fe27000b390 comp rx=0 tx=0).stop 2024-05-20T17:56:07.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 -- 172.21.15.129:0/3512639210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe274102380 msgr2=0x7fe2741996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe274102380 0x7fe2741996d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe26400e9f0 tx=0x7fe26400edf0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 -- 172.21.15.129:0/3512639210 shutdown_connections 2024-05-20T17:56:07.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe2600a2b90 0x7fe2600a50a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe274107080 0x7fe2741a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe274102380 0x7fe2741996d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 --2- 172.21.15.129:0/3512639210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe274101140 0x7fe274199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 -- 172.21.15.129:0/3512639210 >> 172.21.15.129:0/3512639210 conn(0x7fe2740fc310 msgr2=0x7fe274108d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:07.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 -- 172.21.15.129:0/3512639210 shutdown_connections 2024-05-20T17:56:07.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.476+0000 7fe27b6fe640 1 -- 172.21.15.129:0/3512639210 wait complete. 2024-05-20T17:56:07.487 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T17:56:07.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-20T17:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.601+0000 7f41b80e7640 1 -- 172.21.15.129:0/469049934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b0102bf0 msgr2=0x7f41b0103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.601+0000 7f41b80e7640 1 --2- 172.21.15.129:0/469049934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b0102bf0 0x7f41b0103000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009900 tx=0x7f41a002a410 comp rx=0 tx=0).stop 2024-05-20T17:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.601+0000 7f41b80e7640 1 -- 172.21.15.129:0/469049934 shutdown_connections 2024-05-20T17:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.601+0000 7f41b80e7640 1 --2- 172.21.15.129:0/469049934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41b01078f0 0x7f41b0109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.601+0000 7f41b80e7640 1 --2- 172.21.15.129:0/469049934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b0102bf0 0x7f41b0103000 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.601+0000 7f41b80e7640 1 --2- 172.21.15.129:0/469049934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41b01019b0 0x7f41b01026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.601+0000 7f41b80e7640 1 -- 172.21.15.129:0/469049934 >> 172.21.15.129:0/469049934 conn(0x7f41b00fcba0 msgr2=0x7f41b00ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b80e7640 1 -- 172.21.15.129:0/469049934 shutdown_connections 2024-05-20T17:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b80e7640 1 -- 172.21.15.129:0/469049934 wait complete. 2024-05-20T17:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b80e7640 1 Processor -- start 2024-05-20T17:56:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b80e7640 1 -- start start 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b80e7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41b01019b0 0x7f41b0199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b80e7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41b0102bf0 0x7f41b0199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b80e7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b01078f0 0x7f41b01a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b80e7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41b010e730 con 0x7f41b01078f0 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b80e7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41b010e5b0 con 0x7f41b01019b0 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b80e7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f41b010e8b0 con 0x7f41b0102bf0 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b6684640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b01078f0 0x7f41b01a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b6684640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b01078f0 0x7f41b01a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51936/0 (socket says 172.21.15.129:51936) 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b6684640 1 -- 172.21.15.129:0/1991049266 learned_addr learned my addr 172.21.15.129:0/1991049266 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.602+0000 7f41b5e83640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41b01019b0 0x7f41b0199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41b5682640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41b0102bf0 0x7f41b0199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41b6684640 1 -- 172.21.15.129:0/1991049266 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41b0102bf0 msgr2=0x7f41b0199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41b6684640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41b0102bf0 0x7f41b0199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41b6684640 1 -- 172.21.15.129:0/1991049266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41b01019b0 msgr2=0x7f41b0199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41b6684640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41b01019b0 0x7f41b0199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41b6684640 1 -- 172.21.15.129:0/1991049266 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b01a0e80 con 0x7f41b01078f0 2024-05-20T17:56:07.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41b6684640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b01078f0 0x7f41b01a0810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f41a800eaa0 tx=0x7f41a800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:07.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41a6ffd640 1 -- 172.21.15.129:0/1991049266 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a800cdb0 con 0x7f41b01078f0 2024-05-20T17:56:07.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41a6ffd640 1 -- 172.21.15.129:0/1991049266 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a8005020 con 0x7f41b01078f0 2024-05-20T17:56:07.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41b80e7640 1 -- 172.21.15.129:0/1991049266 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b01a1170 con 0x7f41b01078f0 2024-05-20T17:56:07.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41a6ffd640 1 -- 172.21.15.129:0/1991049266 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a80116b0 con 0x7f41b01078f0 2024-05-20T17:56:07.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.603+0000 7f41b80e7640 1 -- 172.21.15.129:0/1991049266 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b01a16c0 con 0x7f41b01078f0 2024-05-20T17:56:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.604+0000 7f41b80e7640 1 -- 172.21.15.129:0/1991049266 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b0106700 con 0x7f41b01078f0 2024-05-20T17:56:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.605+0000 7f41a6ffd640 1 -- 172.21.15.129:0/1991049266 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f41a8005560 con 0x7f41b01078f0 2024-05-20T17:56:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.606+0000 7f41a6ffd640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f419c0a2b90 0x7f419c0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.606+0000 7f41b5e83640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f419c0a2b90 0x7f419c0a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.606+0000 7f41a6ffd640 1 -- 172.21.15.129:0/1991049266 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f41a80cee00 con 0x7f41b01078f0 2024-05-20T17:56:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.606+0000 7f41b5e83640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f419c0a2b90 0x7f419c0a50a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f41b01025b0 tx=0x7f41ac00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:07.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.611+0000 7f41a6ffd640 1 -- 172.21.15.129:0/1991049266 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f41a8016030 con 0x7f41b01078f0 2024-05-20T17:56:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.788+0000 7f41b80e7640 1 -- 172.21.15.129:0/1991049266 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f41b0062750 con 0x7f41b01078f0 2024-05-20T17:56:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.788+0000 7f41a6ffd640 1 -- 172.21.15.129:0/1991049266 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f41a809bb10 con 0x7f41b01078f0 2024-05-20T17:56:07.788 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T17:56:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.790+0000 7f41b80e7640 1 -- 172.21.15.129:0/1991049266 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f419c0a2b90 msgr2=0x7f419c0a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.790+0000 7f41b80e7640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f419c0a2b90 0x7f419c0a50a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f41b01025b0 tx=0x7f41ac00b390 comp rx=0 tx=0).stop 2024-05-20T17:56:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.790+0000 7f41b80e7640 1 -- 172.21.15.129:0/1991049266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b01078f0 msgr2=0x7f41b01a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.790+0000 7f41b80e7640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b01078f0 0x7f41b01a0810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f41a800eaa0 tx=0x7f41a800ef60 comp rx=0 tx=0).stop 2024-05-20T17:56:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.790+0000 7f41b80e7640 1 -- 172.21.15.129:0/1991049266 shutdown_connections 2024-05-20T17:56:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.790+0000 7f41b80e7640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f419c0a2b90 0x7f419c0a50a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.790+0000 7f41b80e7640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b01078f0 0x7f41b01a0810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.790+0000 7f41b80e7640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41b0102bf0 0x7f41b0199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.790+0000 7f41b80e7640 1 --2- 172.21.15.129:0/1991049266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41b01019b0 0x7f41b0199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.790+0000 7f41b80e7640 1 -- 172.21.15.129:0/1991049266 >> 172.21.15.129:0/1991049266 conn(0x7f41b00fcba0 msgr2=0x7f41b01038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.791+0000 7f41b80e7640 1 -- 172.21.15.129:0/1991049266 shutdown_connections 2024-05-20T17:56:07.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.791+0000 7f41b80e7640 1 -- 172.21.15.129:0/1991049266 wait complete. 2024-05-20T17:56:07.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:07.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.913+0000 7f39f7545640 1 -- 172.21.15.129:0/69233481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f01078d0 msgr2=0x7f39f0109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.913+0000 7f39f7545640 1 --2- 172.21.15.129:0/69233481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f01078d0 0x7f39f0109cf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f39e8009ca0 tx=0x7f39e802a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.913+0000 7f39f7545640 1 -- 172.21.15.129:0/69233481 shutdown_connections 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.913+0000 7f39f7545640 1 --2- 172.21.15.129:0/69233481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f01078d0 0x7f39f0109cf0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.913+0000 7f39f7545640 1 --2- 172.21.15.129:0/69233481 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39f0102bd0 0x7f39f0102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.913+0000 7f39f7545640 1 --2- 172.21.15.129:0/69233481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0101990 0x7f39f0102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.913+0000 7f39f7545640 1 -- 172.21.15.129:0/69233481 >> 172.21.15.129:0/69233481 conn(0x7f39f00fcb80 msgr2=0x7f39f00ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.914+0000 7f39f7545640 1 -- 172.21.15.129:0/69233481 shutdown_connections 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.914+0000 7f39f7545640 1 -- 172.21.15.129:0/69233481 wait complete. 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.914+0000 7f39f7545640 1 Processor -- start 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.914+0000 7f39f7545640 1 -- start start 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.914+0000 7f39f7545640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0101990 0x7f39f0074810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.914+0000 7f39f7545640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f0102bd0 0x7f39f019e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.914+0000 7f39f7545640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39f01078d0 0x7f39f01a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.914+0000 7f39f7545640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f39f010d610 con 0x7f39f0102bd0 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.914+0000 7f39f7545640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39f010d490 con 0x7f39f0101990 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.914+0000 7f39f7545640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f39f010d790 con 0x7f39f01078d0 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.914+0000 7f39f4ae0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f0102bd0 0x7f39f019e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f4ae0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f0102bd0 0x7f39f019e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51958/0 (socket says 172.21.15.129:51958) 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f4ae0640 1 -- 172.21.15.129:0/4187714627 learned_addr learned my addr 172.21.15.129:0/4187714627 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f5ae2640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39f01078d0 0x7f39f01a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f52e1640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0101990 0x7f39f0074810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f4ae0640 1 -- 172.21.15.129:0/4187714627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39f01078d0 msgr2=0x7f39f01a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f4ae0640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39f01078d0 0x7f39f01a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f4ae0640 1 -- 172.21.15.129:0/4187714627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0101990 msgr2=0x7f39f0074810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f4ae0640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0101990 0x7f39f0074810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:07.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f4ae0640 1 -- 172.21.15.129:0/4187714627 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f01a0dd0 con 0x7f39f0102bd0 2024-05-20T17:56:07.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f52e1640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0101990 0x7f39f0074810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:56:07.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f4ae0640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f0102bd0 0x7f39f019e040 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f39e000d770 tx=0x7f39e000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:07.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39e67fc640 1 -- 172.21.15.129:0/4187714627 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e0017070 con 0x7f39f0102bd0 2024-05-20T17:56:07.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39e67fc640 1 -- 172.21.15.129:0/4187714627 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39e0005500 con 0x7f39f0102bd0 2024-05-20T17:56:07.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f7545640 1 -- 172.21.15.129:0/4187714627 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f01a10c0 con 0x7f39f0102bd0 2024-05-20T17:56:07.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39e67fc640 1 -- 172.21.15.129:0/4187714627 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e0004e00 con 0x7f39f0102bd0 2024-05-20T17:56:07.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.915+0000 7f39f7545640 1 -- 172.21.15.129:0/4187714627 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f01a15c0 con 0x7f39f0102bd0 2024-05-20T17:56:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.916+0000 7f39f7545640 1 -- 172.21.15.129:0/4187714627 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39f01066e0 con 0x7f39f0102bd0 2024-05-20T17:56:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.918+0000 7f39e67fc640 1 -- 172.21.15.129:0/4187714627 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f39e0004fa0 con 0x7f39f0102bd0 2024-05-20T17:56:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.919+0000 7f39e67fc640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39dc0a2b90 0x7f39dc0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.919+0000 7f39f52e1640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39dc0a2b90 0x7f39dc0a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.919+0000 7f39e67fc640 1 -- 172.21.15.129:0/4187714627 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f39e00ce9a0 con 0x7f39f0102bd0 2024-05-20T17:56:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.920+0000 7f39f52e1640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39dc0a2b90 0x7f39dc0a50a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f39ec005d00 tx=0x7f39ec005c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:07.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:07.923+0000 7f39e67fc640 1 -- 172.21.15.129:0/4187714627 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f39e009bda0 con 0x7f39f0102bd0 2024-05-20T17:56:07.985 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:07 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3512639210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:07.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:07 smithi204 ceph-mon[37076]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:07.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:07 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1991049266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:56:08.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.136+0000 7f39f7545640 1 -- 172.21.15.129:0/4187714627 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39f002d2a0 con 0x7f39f0102bd0 2024-05-20T17:56:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.136+0000 7f39e67fc640 1 -- 172.21.15.129:0/4187714627 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f39e009b6a0 con 0x7f39f0102bd0 2024-05-20T17:56:08.135 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:08.135 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-05-20T17:56:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 1 2024-05-20T17:56:08.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 -- 172.21.15.129:0/4187714627 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39dc0a2b90 msgr2=0x7f39dc0a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39dc0a2b90 0x7f39dc0a50a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f39ec005d00 tx=0x7f39ec005c90 comp rx=0 tx=0).stop 2024-05-20T17:56:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 -- 172.21.15.129:0/4187714627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f0102bd0 msgr2=0x7f39f019e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f0102bd0 0x7f39f019e040 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f39e000d770 tx=0x7f39e000dc30 comp rx=0 tx=0).stop 2024-05-20T17:56:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 -- 172.21.15.129:0/4187714627 shutdown_connections 2024-05-20T17:56:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39dc0a2b90 0x7f39dc0a50a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39f01078d0 0x7f39f01a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f0102bd0 0x7f39f019e040 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 --2- 172.21.15.129:0/4187714627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0101990 0x7f39f0074810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 -- 172.21.15.129:0/4187714627 >> 172.21.15.129:0/4187714627 conn(0x7f39f00fcb80 msgr2=0x7f39f01038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 -- 172.21.15.129:0/4187714627 shutdown_connections 2024-05-20T17:56:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:08.138+0000 7f39f7545640 1 -- 172.21.15.129:0/4187714627 wait complete. 2024-05-20T17:56:08.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:07 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3512639210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:08.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:07 smithi154 ceph-mon[36482]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:08.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:07 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1991049266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:56:08.148 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:07 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3512639210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:08.148 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:07 smithi129 ceph-mon[32029]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:08.148 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:07 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1991049266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:56:08.676 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 17:56:08 smithi204 systemd[1]: Starting Ceph mds.l for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:56:08.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:08 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/4187714627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:08.930 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 17:56:08 smithi204 podman[67125]: 2024-05-20 17:56:08.579878884 +0000 UTC m=+0.017201403 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:56:08.930 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 17:56:08 smithi204 podman[67125]: 2024-05-20 17:56:08.68990799 +0000 UTC m=+0.127230526 container create eab0016ca770245aea1614ff5aa25a79ba6b1a253daf5cebb3c6afd5ccdccfce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-l, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T17:56:08.930 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 17:56:08 smithi204 podman[67125]: 2024-05-20 17:56:08.885103222 +0000 UTC m=+0.322425724 container init eab0016ca770245aea1614ff5aa25a79ba6b1a253daf5cebb3c6afd5ccdccfce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux ) 2024-05-20T17:56:08.930 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 17:56:08 smithi204 podman[67125]: 2024-05-20 17:56:08.888302209 +0000 UTC m=+0.325624706 container start eab0016ca770245aea1614ff5aa25a79ba6b1a253daf5cebb3c6afd5ccdccfce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-l, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514) 2024-05-20T17:56:08.930 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 17:56:08 smithi204 ceph-mds[67144]: -- 172.21.15.204:0/966130360 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55644b6b9a40 con 0x55644b731180 2024-05-20T17:56:08.930 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 17:56:08 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-l[67140]: starting mds.l at 2024-05-20T17:56:09.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:08 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/4187714627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:09.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:08 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4187714627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:09.430 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 17:56:08 smithi204 bash[67125]: eab0016ca770245aea1614ff5aa25a79ba6b1a253daf5cebb3c6afd5ccdccfce 2024-05-20T17:56:09.430 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 17:56:08 smithi204 systemd[1]: Started Ceph mds.l for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:10.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:09 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.204:6832/3042064152,v1:172.21.15.204:6833/3042064152] up:boot 2024-05-20T17:56:10.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:09 smithi154 ceph-mon[36482]: fsmap 1 up:standby 2024-05-20T17:56:10.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-20T17:56:10.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:10.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:10.014 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:10.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:10.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:10.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:09 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:10.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:09 smithi154 ceph-mon[36482]: Deploying daemon mds.k on smithi154 2024-05-20T17:56:10.015 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:09 smithi154 ceph-mon[36482]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:10.150 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:10.178 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:09 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.204:6832/3042064152,v1:172.21.15.204:6833/3042064152] up:boot 2024-05-20T17:56:10.178 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:09 smithi129 ceph-mon[32029]: fsmap 1 up:standby 2024-05-20T17:56:10.178 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-20T17:56:10.179 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:10.179 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:10.179 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:10.179 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:10.179 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:10.179 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:09 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:10.179 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:09 smithi129 ceph-mon[32029]: Deploying daemon mds.k on smithi154 2024-05-20T17:56:10.179 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:09 smithi129 ceph-mon[32029]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:09 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.204:6832/3042064152,v1:172.21.15.204:6833/3042064152] up:boot 2024-05-20T17:56:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:09 smithi204 ceph-mon[37076]: fsmap 1 up:standby 2024-05-20T17:56:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-20T17:56:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:09 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:09 smithi204 ceph-mon[37076]: Deploying daemon mds.k on smithi154 2024-05-20T17:56:10.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:09 smithi204 ceph-mon[37076]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:10.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.274+0000 7fcd90f5e640 1 -- 172.21.15.129:0/3502767233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8c102bd0 msgr2=0x7fcd8c102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:10.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.274+0000 7fcd90f5e640 1 --2- 172.21.15.129:0/3502767233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8c102bd0 0x7fcd8c102fe0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009900 tx=0x7fcd7402a410 comp rx=0 tx=0).stop 2024-05-20T17:56:10.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.274+0000 7fcd90f5e640 1 -- 172.21.15.129:0/3502767233 shutdown_connections 2024-05-20T17:56:10.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.274+0000 7fcd90f5e640 1 --2- 172.21.15.129:0/3502767233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd8c1078d0 0x7fcd8c109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:10.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.274+0000 7fcd90f5e640 1 --2- 172.21.15.129:0/3502767233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8c102bd0 0x7fcd8c102fe0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:10.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.274+0000 7fcd90f5e640 1 --2- 172.21.15.129:0/3502767233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd8c101990 0x7fcd8c102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:10.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.274+0000 7fcd90f5e640 1 -- 172.21.15.129:0/3502767233 >> 172.21.15.129:0/3502767233 conn(0x7fcd8c0fcb80 msgr2=0x7fcd8c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:10.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.274+0000 7fcd90f5e640 1 -- 172.21.15.129:0/3502767233 shutdown_connections 2024-05-20T17:56:10.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.274+0000 7fcd90f5e640 1 -- 172.21.15.129:0/3502767233 wait complete. 2024-05-20T17:56:10.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.275+0000 7fcd90f5e640 1 Processor -- start 2024-05-20T17:56:10.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.275+0000 7fcd90f5e640 1 -- start start 2024-05-20T17:56:10.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd90f5e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8c101990 0x7fcd8c1995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:10.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd90f5e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd8c102bd0 0x7fcd8c199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:10.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd90f5e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd8c1078d0 0x7fcd8c1a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:10.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd90f5e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8c10d600 con 0x7fcd8c101990 2024-05-20T17:56:10.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd90f5e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8c10d480 con 0x7fcd8c1078d0 2024-05-20T17:56:10.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd90f5e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8c10d780 con 0x7fcd8c102bd0 2024-05-20T17:56:10.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd8a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8c101990 0x7fcd8c1995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:10.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd8a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8c101990 0x7fcd8c1995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60868/0 (socket says 172.21.15.129:60868) 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd8a59c640 1 -- 172.21.15.129:0/711010776 learned_addr learned my addr 172.21.15.129:0/711010776 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd8ad9d640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd8c1078d0 0x7fcd8c1a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd81d9b640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd8c102bd0 0x7fcd8c199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd8a59c640 1 -- 172.21.15.129:0/711010776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd8c102bd0 msgr2=0x7fcd8c199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd8a59c640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd8c102bd0 0x7fcd8c199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd8a59c640 1 -- 172.21.15.129:0/711010776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd8c1078d0 msgr2=0x7fcd8c1a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd8a59c640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd8c1078d0 0x7fcd8c1a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd8a59c640 1 -- 172.21.15.129:0/711010776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8c1a1100 con 0x7fcd8c101990 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd8ad9d640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd8c1078d0 0x7fcd8c1a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd81d9b640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd8c102bd0 0x7fcd8c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd8a59c640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8c101990 0x7fcd8c1995c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00c970 tx=0x7fcd7c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd83fff640 1 -- 172.21.15.129:0/711010776 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c004220 con 0x7fcd8c101990 2024-05-20T17:56:10.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd83fff640 1 -- 172.21.15.129:0/711010776 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd7c0043c0 con 0x7fcd8c101990 2024-05-20T17:56:10.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd90f5e640 1 -- 172.21.15.129:0/711010776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd8c1a13f0 con 0x7fcd8c101990 2024-05-20T17:56:10.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.276+0000 7fcd83fff640 1 -- 172.21.15.129:0/711010776 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c004a00 con 0x7fcd8c101990 2024-05-20T17:56:10.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.277+0000 7fcd90f5e640 1 -- 172.21.15.129:0/711010776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd8c1a18f0 con 0x7fcd8c101990 2024-05-20T17:56:10.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.278+0000 7fcd90f5e640 1 -- 172.21.15.129:0/711010776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd8c068b20 con 0x7fcd8c101990 2024-05-20T17:56:10.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.279+0000 7fcd83fff640 1 -- 172.21.15.129:0/711010776 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fcd7c0079b0 con 0x7fcd8c101990 2024-05-20T17:56:10.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.280+0000 7fcd83fff640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fcd780a2b90 0x7fcd780a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:10.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.280+0000 7fcd81d9b640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fcd780a2b90 0x7fcd780a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:10.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.280+0000 7fcd83fff640 1 -- 172.21.15.129:0/711010776 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fcd7c0cedb0 con 0x7fcd8c101990 2024-05-20T17:56:10.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.280+0000 7fcd81d9b640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fcd780a2b90 0x7fcd780a50a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcd74000c00 tx=0x7fcd740095f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:10.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.283+0000 7fcd83fff640 1 -- 172.21.15.129:0/711010776 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcd7c014030 con 0x7fcd8c101990 2024-05-20T17:56:10.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.497+0000 7fcd90f5e640 1 -- 172.21.15.129:0/711010776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd8c1a1bb0 con 0x7fcd8c101990 2024-05-20T17:56:10.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.497+0000 7fcd83fff640 1 -- 172.21.15.129:0/711010776 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fcd7c09c1b0 con 0x7fcd8c101990 2024-05-20T17:56:10.496 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:10.496 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-20T17:56:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 2 2024-05-20T17:56:10.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 -- 172.21.15.129:0/711010776 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fcd780a2b90 msgr2=0x7fcd780a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:10.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fcd780a2b90 0x7fcd780a50a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcd74000c00 tx=0x7fcd740095f0 comp rx=0 tx=0).stop 2024-05-20T17:56:10.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 -- 172.21.15.129:0/711010776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8c101990 msgr2=0x7fcd8c1995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:10.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8c101990 0x7fcd8c1995c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00c970 tx=0x7fcd7c00ce30 comp rx=0 tx=0).stop 2024-05-20T17:56:10.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 -- 172.21.15.129:0/711010776 shutdown_connections 2024-05-20T17:56:10.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fcd780a2b90 0x7fcd780a50a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:10.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd8c1078d0 0x7fcd8c1a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:10.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcd8c102bd0 0x7fcd8c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:10.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 --2- 172.21.15.129:0/711010776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8c101990 0x7fcd8c1995c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:10.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 -- 172.21.15.129:0/711010776 >> 172.21.15.129:0/711010776 conn(0x7fcd8c0fcb80 msgr2=0x7fcd8c108f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:10.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 -- 172.21.15.129:0/711010776 shutdown_connections 2024-05-20T17:56:10.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:10.499+0000 7fcd90f5e640 1 -- 172.21.15.129:0/711010776 wait complete. 2024-05-20T17:56:11.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:11 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/711010776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:11.394 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 17:56:11 smithi154 systemd[1]: Starting Ceph mds.k for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:56:11.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:11 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/711010776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:11.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:11 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/711010776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:11.791 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 17:56:11 smithi154 podman[65754]: 2024-05-20 17:56:11.371801282 +0000 UTC m=+0.017615905 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:56:11.791 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 17:56:11 smithi154 podman[65754]: 2024-05-20 17:56:11.548625872 +0000 UTC m=+0.194440514 container create 32548813b8d3335971d78b45035be0cc0c5b1136b24805f173988cf94603b609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True) 2024-05-20T17:56:11.791 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 17:56:11 smithi154 podman[65754]: 2024-05-20 17:56:11.792991007 +0000 UTC m=+0.438805629 container init 32548813b8d3335971d78b45035be0cc0c5b1136b24805f173988cf94603b609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-20T17:56:12.073 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 17:56:11 smithi154 podman[65754]: 2024-05-20 17:56:11.795690952 +0000 UTC m=+0.441505582 container start 32548813b8d3335971d78b45035be0cc0c5b1136b24805f173988cf94603b609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-k, ceph=True, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T17:56:12.074 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 17:56:11 smithi154 ceph-mds[65774]: -- 172.21.15.154:0/3736391650 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f30165a40 con 0x556f301dda80 2024-05-20T17:56:12.074 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 17:56:11 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-k[65770]: starting mds.k at 2024-05-20T17:56:12.074 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 17:56:11 smithi154 bash[65754]: 32548813b8d3335971d78b45035be0cc0c5b1136b24805f173988cf94603b609 2024-05-20T17:56:12.074 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 17:56:11 smithi154 systemd[1]: Started Ceph mds.k for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:12.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:12 smithi154 ceph-mon[36482]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:12.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:12.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:12.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:12.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:12.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:12.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:12 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:12.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:12 smithi129 ceph-mon[32029]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:12.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:12.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:12.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:12.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:12.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:12.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:12 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:12.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:12 smithi204 ceph-mon[37076]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:12.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:12.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:12.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:12.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:12.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:12.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:12 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:12.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:12.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.625+0000 7f4378964640 1 -- 172.21.15.129:0/1798987254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4374101970 msgr2=0x7f4374102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:12.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.625+0000 7f4378964640 1 --2- 172.21.15.129:0/1798987254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4374101970 0x7f4374102670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4364009900 tx=0x7f436402a410 comp rx=0 tx=0).stop 2024-05-20T17:56:12.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.625+0000 7f4378964640 1 -- 172.21.15.129:0/1798987254 shutdown_connections 2024-05-20T17:56:12.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.625+0000 7f4378964640 1 --2- 172.21.15.129:0/1798987254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43741078b0 0x7f4374109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:12.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.625+0000 7f4378964640 1 --2- 172.21.15.129:0/1798987254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4374102bb0 0x7f4374102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:12.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.625+0000 7f4378964640 1 --2- 172.21.15.129:0/1798987254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4374101970 0x7f4374102670 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:12.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.625+0000 7f4378964640 1 -- 172.21.15.129:0/1798987254 >> 172.21.15.129:0/1798987254 conn(0x7f43740fcb80 msgr2=0x7f43740ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:12.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.625+0000 7f4378964640 1 -- 172.21.15.129:0/1798987254 shutdown_connections 2024-05-20T17:56:12.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.626+0000 7f4378964640 1 -- 172.21.15.129:0/1798987254 wait complete. 2024-05-20T17:56:12.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.626+0000 7f4378964640 1 Processor -- start 2024-05-20T17:56:12.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.626+0000 7f4378964640 1 -- start start 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.626+0000 7f4378964640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4374101970 0x7f4374198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.626+0000 7f4378964640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4374102bb0 0x7f43741986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.626+0000 7f4378964640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43741078b0 0x7f437419f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.626+0000 7f4378964640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f437410d5a0 con 0x7f4374102bb0 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f4378964640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f437410d420 con 0x7f4374101970 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f4378964640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f437410d720 con 0x7f43741078b0 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4374102bb0 0x7f43741986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4374102bb0 0x7f43741986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60898/0 (socket says 172.21.15.129:60898) 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436bfff640 1 -- 172.21.15.129:0/273535421 learned_addr learned my addr 172.21.15.129:0/273535421 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f437259c640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4374101970 0x7f4374198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f4372d9d640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43741078b0 0x7f437419f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436bfff640 1 -- 172.21.15.129:0/273535421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43741078b0 msgr2=0x7f437419f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:12.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436bfff640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43741078b0 0x7f437419f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:12.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436bfff640 1 -- 172.21.15.129:0/273535421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4374101970 msgr2=0x7f4374198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:12.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436bfff640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4374101970 0x7f4374198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:12.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436bfff640 1 -- 172.21.15.129:0/273535421 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437419fc10 con 0x7f4374102bb0 2024-05-20T17:56:12.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436bfff640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4374102bb0 0x7f43741986c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f435c00d950 tx=0x7f435c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:12.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436b7fe640 1 -- 172.21.15.129:0/273535421 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c017070 con 0x7f4374102bb0 2024-05-20T17:56:12.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436b7fe640 1 -- 172.21.15.129:0/273535421 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435c009c30 con 0x7f4374102bb0 2024-05-20T17:56:12.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f4378964640 1 -- 172.21.15.129:0/273535421 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437419ff00 con 0x7f4374102bb0 2024-05-20T17:56:12.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f436b7fe640 1 -- 172.21.15.129:0/273535421 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c00f6b0 con 0x7f4374102bb0 2024-05-20T17:56:12.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.627+0000 7f4378964640 1 -- 172.21.15.129:0/273535421 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43741a03c0 con 0x7f4374102bb0 2024-05-20T17:56:12.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.628+0000 7f4378964640 1 -- 172.21.15.129:0/273535421 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43741066c0 con 0x7f4374102bb0 2024-05-20T17:56:12.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.629+0000 7f436b7fe640 1 -- 172.21.15.129:0/273535421 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f435c0109d0 con 0x7f4374102bb0 2024-05-20T17:56:12.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.630+0000 7f436b7fe640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f43600a2c10 0x7f43600a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:12.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.630+0000 7f437259c640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f43600a2c10 0x7f43600a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:12.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.630+0000 7f436b7fe640 1 -- 172.21.15.129:0/273535421 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f435c0ce770 con 0x7f4374102bb0 2024-05-20T17:56:12.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.631+0000 7f437259c640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f43600a2c10 0x7f43600a5120 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4364005ab0 tx=0x7f4364031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:12.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.634+0000 7f436b7fe640 1 -- 172.21.15.129:0/273535421 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f435c009de0 con 0x7f4374102bb0 2024-05-20T17:56:12.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.853+0000 7f4378964640 1 -- 172.21.15.129:0/273535421 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f437402d460 con 0x7f4374102bb0 2024-05-20T17:56:12.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.854+0000 7f436b7fe640 1 -- 172.21.15.129:0/273535421 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f435c09bb90 con 0x7f4374102bb0 2024-05-20T17:56:12.853 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:12.853 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-20T17:56:12.853 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 3 2024-05-20T17:56:12.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 -- 172.21.15.129:0/273535421 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f43600a2c10 msgr2=0x7f43600a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:12.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f43600a2c10 0x7f43600a5120 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4364005ab0 tx=0x7f4364031400 comp rx=0 tx=0).stop 2024-05-20T17:56:12.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 -- 172.21.15.129:0/273535421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4374102bb0 msgr2=0x7f43741986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:12.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4374102bb0 0x7f43741986c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f435c00d950 tx=0x7f435c00de10 comp rx=0 tx=0).stop 2024-05-20T17:56:12.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 -- 172.21.15.129:0/273535421 shutdown_connections 2024-05-20T17:56:12.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f43600a2c10 0x7f43600a5120 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:12.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f43741078b0 0x7f437419f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:12.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4374102bb0 0x7f43741986c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:12.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 --2- 172.21.15.129:0/273535421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4374101970 0x7f4374198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:12.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 -- 172.21.15.129:0/273535421 >> 172.21.15.129:0/273535421 conn(0x7f43740fcb80 msgr2=0x7f4374103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:12.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 -- 172.21.15.129:0/273535421 shutdown_connections 2024-05-20T17:56:12.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:12.856+0000 7f4378964640 1 -- 172.21.15.129:0/273535421 wait complete. 2024-05-20T17:56:13.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:13 smithi129 ceph-mon[32029]: Deploying daemon mds.j on smithi129 2024-05-20T17:56:13.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:13 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.154:6832/3269888601,v1:172.21.15.154:6833/3269888601] up:boot 2024-05-20T17:56:13.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:13 smithi129 ceph-mon[32029]: fsmap 2 up:standby 2024-05-20T17:56:13.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:13 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-20T17:56:13.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:13 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/273535421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:13.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:13 smithi154 ceph-mon[36482]: Deploying daemon mds.j on smithi129 2024-05-20T17:56:13.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:13 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.154:6832/3269888601,v1:172.21.15.154:6833/3269888601] up:boot 2024-05-20T17:56:13.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:13 smithi154 ceph-mon[36482]: fsmap 2 up:standby 2024-05-20T17:56:13.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:13 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-20T17:56:13.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:13 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/273535421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:13.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:13 smithi204 ceph-mon[37076]: Deploying daemon mds.j on smithi129 2024-05-20T17:56:13.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:13 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.154:6832/3269888601,v1:172.21.15.154:6833/3269888601] up:boot 2024-05-20T17:56:13.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:13 smithi204 ceph-mon[37076]: fsmap 2 up:standby 2024-05-20T17:56:13.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:13 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-20T17:56:13.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:13 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/273535421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:14.214 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:14 smithi154 ceph-mon[36482]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:14.349 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:14 smithi129 ceph-mon[32029]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:14.349 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 17:56:14 smithi129 systemd[1]: Starting Ceph mds.j for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:56:14.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:14 smithi204 ceph-mon[37076]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:14.625 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 17:56:14 smithi129 podman[70276]: 2024-05-20 17:56:14.253076063 +0000 UTC m=+0.016913716 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:56:14.625 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 17:56:14 smithi129 podman[70276]: 2024-05-20 17:56:14.361810184 +0000 UTC m=+0.125647879 container create 9f47465ebe1d0e8040c0c6b5509a5fa83d0b851ed5eb70199d35e8480f31f67e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-j, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T17:56:14.625 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 17:56:14 smithi129 podman[70276]: 2024-05-20 17:56:14.589705632 +0000 UTC m=+0.353543285 container init 9f47465ebe1d0e8040c0c6b5509a5fa83d0b851ed5eb70199d35e8480f31f67e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-j, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-20T17:56:14.625 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 17:56:14 smithi129 podman[70276]: 2024-05-20 17:56:14.592562665 +0000 UTC m=+0.356400319 container start 9f47465ebe1d0e8040c0c6b5509a5fa83d0b851ed5eb70199d35e8480f31f67e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-20T17:56:14.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:14.891 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 17:56:14 smithi129 ceph-mds[70295]: -- 172.21.15.129:0/832923384 <== mon.0 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ffdd683a40 con 0x55ffdd6fb600 2024-05-20T17:56:14.891 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 17:56:14 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-j[70291]: starting mds.j at 2024-05-20T17:56:14.891 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 17:56:14 smithi129 bash[70276]: 9f47465ebe1d0e8040c0c6b5509a5fa83d0b851ed5eb70199d35e8480f31f67e 2024-05-20T17:56:14.891 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 17:56:14 smithi129 systemd[1]: Started Ceph mds.j for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.985+0000 7f35fd1ea640 1 -- 172.21.15.129:0/2513945303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35f8103400 msgr2=0x7f35f8109b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.985+0000 7f35fd1ea640 1 --2- 172.21.15.129:0/2513945303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35f8103400 0x7f35f8109b50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009ca0 tx=0x7f35f002a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.985+0000 7f35fd1ea640 1 -- 172.21.15.129:0/2513945303 shutdown_connections 2024-05-20T17:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.985+0000 7f35fd1ea640 1 --2- 172.21.15.129:0/2513945303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f81028a0 0x7f35f8102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.985+0000 7f35fd1ea640 1 --2- 172.21.15.129:0/2513945303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35f8103400 0x7f35f8109b50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.985+0000 7f35fd1ea640 1 --2- 172.21.15.129:0/2513945303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35f8101ec0 0x7f35f81022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.985+0000 7f35fd1ea640 1 -- 172.21.15.129:0/2513945303 >> 172.21.15.129:0/2513945303 conn(0x7f35f80fc2e0 msgr2=0x7f35f80fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.985+0000 7f35fd1ea640 1 -- 172.21.15.129:0/2513945303 shutdown_connections 2024-05-20T17:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.985+0000 7f35fd1ea640 1 -- 172.21.15.129:0/2513945303 wait complete. 2024-05-20T17:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35fd1ea640 1 Processor -- start 2024-05-20T17:56:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35fd1ea640 1 -- start start 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35fd1ea640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35f8101ec0 0x7f35f8198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35fd1ea640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35f81028a0 0x7f35f8199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35fd1ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f819e2f0 0x7f35f81a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35fd1ea640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f35f810d620 con 0x7f35f8101ec0 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35fd1ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f35f810d4a0 con 0x7f35f819e2f0 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35fd1ea640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f35f810d7a0 con 0x7f35f81028a0 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35f6d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35f8101ec0 0x7f35f8198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35f6d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35f8101ec0 0x7f35f8198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60932/0 (socket says 172.21.15.129:60932) 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35f6d9d640 1 -- 172.21.15.129:0/1211868571 learned_addr learned my addr 172.21.15.129:0/1211868571 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.986+0000 7f35f759e640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f819e2f0 0x7f35f81a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35f659c640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35f81028a0 0x7f35f8199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35f6d9d640 1 -- 172.21.15.129:0/1211868571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35f81028a0 msgr2=0x7f35f8199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35f6d9d640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35f81028a0 0x7f35f8199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:14.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35f6d9d640 1 -- 172.21.15.129:0/1211868571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f819e2f0 msgr2=0x7f35f81a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:14.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35f6d9d640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f819e2f0 0x7f35f81a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:14.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35f6d9d640 1 -- 172.21.15.129:0/1211868571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f81a0d80 con 0x7f35f8101ec0 2024-05-20T17:56:14.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35f759e640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f819e2f0 0x7f35f81a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:56:14.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35f6d9d640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35f8101ec0 0x7f35f8198e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35e800c970 tx=0x7f35e800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:14.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35dffff640 1 -- 172.21.15.129:0/1211868571 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8004220 con 0x7f35f8101ec0 2024-05-20T17:56:14.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35fd1ea640 1 -- 172.21.15.129:0/1211868571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f81a1070 con 0x7f35f8101ec0 2024-05-20T17:56:14.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35dffff640 1 -- 172.21.15.129:0/1211868571 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35e80043c0 con 0x7f35f8101ec0 2024-05-20T17:56:14.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35dffff640 1 -- 172.21.15.129:0/1211868571 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8004a00 con 0x7f35f8101ec0 2024-05-20T17:56:14.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.987+0000 7f35fd1ea640 1 -- 172.21.15.129:0/1211868571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f81a15c0 con 0x7f35f8101ec0 2024-05-20T17:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.988+0000 7f35fd1ea640 1 -- 172.21.15.129:0/1211868571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35f81a1880 con 0x7f35f8101ec0 2024-05-20T17:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.989+0000 7f35dffff640 1 -- 172.21.15.129:0/1211868571 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f35e8004ba0 con 0x7f35f8101ec0 2024-05-20T17:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.990+0000 7f35dffff640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f35e00a2b90 0x7f35e00a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.990+0000 7f35dffff640 1 -- 172.21.15.129:0/1211868571 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f35e80ce9a0 con 0x7f35f8101ec0 2024-05-20T17:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.990+0000 7f35f659c640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f35e00a2b90 0x7f35e00a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:14.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.991+0000 7f35f659c640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f35e00a2b90 0x7f35e00a50a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f35f8199ed0 tx=0x7f35ec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:14.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:14.999+0000 7f35dffff640 1 -- 172.21.15.129:0/1211868571 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f35e8014030 con 0x7f35f8101ec0 2024-05-20T17:56:15.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.222+0000 7f35fd1ea640 1 -- 172.21.15.129:0/1211868571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35f81a1a50 con 0x7f35f8101ec0 2024-05-20T17:56:15.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.223+0000 7f35dffff640 1 -- 172.21.15.129:0/1211868571 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3052 (secure 0 0 0) 0x7f35e809bda0 con 0x7f35f8101ec0 2024-05-20T17:56:15.222 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:15.222 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-20T17:56:15.223 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 4 2024-05-20T17:56:15.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.225+0000 7f35fd1ea640 1 -- 172.21.15.129:0/1211868571 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f35e00a2b90 msgr2=0x7f35e00a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:15.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.225+0000 7f35fd1ea640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f35e00a2b90 0x7f35e00a50a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f35f8199ed0 tx=0x7f35ec005f90 comp rx=0 tx=0).stop 2024-05-20T17:56:15.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.225+0000 7f35fd1ea640 1 -- 172.21.15.129:0/1211868571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35f8101ec0 msgr2=0x7f35f8198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:15.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.225+0000 7f35fd1ea640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35f8101ec0 0x7f35f8198e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35e800c970 tx=0x7f35e800ce30 comp rx=0 tx=0).stop 2024-05-20T17:56:15.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.225+0000 7f35fd1ea640 1 -- 172.21.15.129:0/1211868571 shutdown_connections 2024-05-20T17:56:15.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.225+0000 7f35fd1ea640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f35e00a2b90 0x7f35e00a50a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:15.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.225+0000 7f35fd1ea640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35f819e2f0 0x7f35f81a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:15.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.225+0000 7f35fd1ea640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f35f81028a0 0x7f35f8199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:15.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.225+0000 7f35fd1ea640 1 --2- 172.21.15.129:0/1211868571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35f8101ec0 0x7f35f8198e30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:15.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.225+0000 7f35fd1ea640 1 -- 172.21.15.129:0/1211868571 >> 172.21.15.129:0/1211868571 conn(0x7f35f80fc2e0 msgr2=0x7f35f80fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:15.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.226+0000 7f35fd1ea640 1 -- 172.21.15.129:0/1211868571 shutdown_connections 2024-05-20T17:56:15.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:15.226+0000 7f35fd1ea640 1 -- 172.21.15.129:0/1211868571 wait complete. 2024-05-20T17:56:15.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: Deploying daemon mds.d on smithi129 2024-05-20T17:56:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.129:6834/2428051280,v1:172.21.15.129:6835/2428051280] up:boot 2024-05-20T17:56:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: fsmap 3 up:standby 2024-05-20T17:56:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-20T17:56:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1211868571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:15 smithi129 ceph-mon[32029]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: Deploying daemon mds.d on smithi129 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.129:6834/2428051280,v1:172.21.15.129:6835/2428051280] up:boot 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: fsmap 3 up:standby 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1211868571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:16.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:15 smithi154 ceph-mon[36482]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:16.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:16.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:16.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:16.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:16.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:16.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:16.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: Deploying daemon mds.d on smithi129 2024-05-20T17:56:16.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.129:6834/2428051280,v1:172.21.15.129:6835/2428051280] up:boot 2024-05-20T17:56:16.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: fsmap 3 up:standby 2024-05-20T17:56:16.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-20T17:56:16.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1211868571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:16.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:15 smithi204 ceph-mon[37076]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:17.159 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 17:56:16 smithi129 systemd[1]: Starting Ceph mds.d for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:56:17.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:17.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.355+0000 7efd8bfd5640 1 -- 172.21.15.129:0/1674902257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd84101990 msgr2=0x7efd84102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:17.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.355+0000 7efd8bfd5640 1 --2- 172.21.15.129:0/1674902257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd84101990 0x7efd84102690 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efd74009900 tx=0x7efd7402a410 comp rx=0 tx=0).stop 2024-05-20T17:56:17.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.355+0000 7efd8bfd5640 1 -- 172.21.15.129:0/1674902257 shutdown_connections 2024-05-20T17:56:17.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.355+0000 7efd8bfd5640 1 --2- 172.21.15.129:0/1674902257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd84102bd0 0x7efd84102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:17.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.355+0000 7efd8bfd5640 1 --2- 172.21.15.129:0/1674902257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd841078d0 0x7efd84109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:17.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.355+0000 7efd8bfd5640 1 --2- 172.21.15.129:0/1674902257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd84101990 0x7efd84102690 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:17.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.355+0000 7efd8bfd5640 1 -- 172.21.15.129:0/1674902257 >> 172.21.15.129:0/1674902257 conn(0x7efd840fcb80 msgr2=0x7efd840ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:17.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.355+0000 7efd8bfd5640 1 -- 172.21.15.129:0/1674902257 shutdown_connections 2024-05-20T17:56:17.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.355+0000 7efd8bfd5640 1 -- 172.21.15.129:0/1674902257 wait complete. 2024-05-20T17:56:17.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.356+0000 7efd8bfd5640 1 Processor -- start 2024-05-20T17:56:17.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.356+0000 7efd8bfd5640 1 -- start start 2024-05-20T17:56:17.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.356+0000 7efd8bfd5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd84101990 0x7efd84198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:17.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.356+0000 7efd8bfd5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd84102bd0 0x7efd84198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:17.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.356+0000 7efd8bfd5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd841078d0 0x7efd8419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:17.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.356+0000 7efd8bfd5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efd8410d5e0 con 0x7efd84102bd0 2024-05-20T17:56:17.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.356+0000 7efd8bfd5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd8410d460 con 0x7efd84101990 2024-05-20T17:56:17.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.356+0000 7efd8bfd5640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efd8410d760 con 0x7efd841078d0 2024-05-20T17:56:17.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.356+0000 7efd89570640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd84102bd0 0x7efd84198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.356+0000 7efd89570640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd84102bd0 0x7efd84198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60942/0 (socket says 172.21.15.129:60942) 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.356+0000 7efd89570640 1 -- 172.21.15.129:0/25858743 learned_addr learned my addr 172.21.15.129:0/25858743 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd8a572640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd841078d0 0x7efd8419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd89d71640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd84101990 0x7efd84198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd89570640 1 -- 172.21.15.129:0/25858743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd841078d0 msgr2=0x7efd8419f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd89570640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd841078d0 0x7efd8419f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd89570640 1 -- 172.21.15.129:0/25858743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd84101990 msgr2=0x7efd84198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd89570640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd84101990 0x7efd84198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd89570640 1 -- 172.21.15.129:0/25858743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8419fcb0 con 0x7efd84102bd0 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd89570640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd84102bd0 0x7efd84198760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efd8000c920 tx=0x7efd8000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd7affd640 1 -- 172.21.15.129:0/25858743 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd80007c70 con 0x7efd84102bd0 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd7affd640 1 -- 172.21.15.129:0/25858743 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd80007e10 con 0x7efd84102bd0 2024-05-20T17:56:17.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd7affd640 1 -- 172.21.15.129:0/25858743 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd80018890 con 0x7efd84102bd0 2024-05-20T17:56:17.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd8bfd5640 1 -- 172.21.15.129:0/25858743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd8419ffa0 con 0x7efd84102bd0 2024-05-20T17:56:17.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.357+0000 7efd8bfd5640 1 -- 172.21.15.129:0/25858743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd841a0460 con 0x7efd84102bd0 2024-05-20T17:56:17.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.358+0000 7efd8bfd5640 1 -- 172.21.15.129:0/25858743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd841066e0 con 0x7efd84102bd0 2024-05-20T17:56:17.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.359+0000 7efd7affd640 1 -- 172.21.15.129:0/25858743 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7efd80018a50 con 0x7efd84102bd0 2024-05-20T17:56:17.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.360+0000 7efd7affd640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd700a2c10 0x7efd700a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:17.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.360+0000 7efd89d71640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd700a2c10 0x7efd700a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:17.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.360+0000 7efd7affd640 1 -- 172.21.15.129:0/25858743 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7efd800cef90 con 0x7efd84102bd0 2024-05-20T17:56:17.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.361+0000 7efd89d71640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd700a2c10 0x7efd700a5120 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efd74031610 tx=0x7efd74000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:17.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.364+0000 7efd7affd640 1 -- 172.21.15.129:0/25858743 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7efd80014030 con 0x7efd84102bd0 2024-05-20T17:56:17.412 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 17:56:17 smithi129 podman[70618]: 2024-05-20 17:56:17.064438707 +0000 UTC m=+0.017778543 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:56:17.412 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 17:56:17 smithi129 podman[70618]: 2024-05-20 17:56:17.187473386 +0000 UTC m=+0.140813215 container create 9acc43aa3ab0895e27120ec5c40793d1288c981345e21884be0ac08b7b3ac2ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T17:56:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.612+0000 7efd8bfd5640 1 -- 172.21.15.129:0/25858743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd8402d2a0 con 0x7efd84102bd0 2024-05-20T17:56:17.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.612+0000 7efd7affd640 1 -- 172.21.15.129:0/25858743 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3052 (secure 0 0 0) 0x7efd8009c4a0 con 0x7efd84102bd0 2024-05-20T17:56:17.611 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:17.611 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-20T17:56:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 4 2024-05-20T17:56:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 -- 172.21.15.129:0/25858743 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd700a2c10 msgr2=0x7efd700a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd700a2c10 0x7efd700a5120 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efd74031610 tx=0x7efd74000bc0 comp rx=0 tx=0).stop 2024-05-20T17:56:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 -- 172.21.15.129:0/25858743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd84102bd0 msgr2=0x7efd84198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd84102bd0 0x7efd84198760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efd8000c920 tx=0x7efd8000cde0 comp rx=0 tx=0).stop 2024-05-20T17:56:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 -- 172.21.15.129:0/25858743 shutdown_connections 2024-05-20T17:56:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd700a2c10 0x7efd700a5120 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd841078d0 0x7efd8419f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:17.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd84102bd0 0x7efd84198760 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:17.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 --2- 172.21.15.129:0/25858743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd84101990 0x7efd84198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:17.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 -- 172.21.15.129:0/25858743 >> 172.21.15.129:0/25858743 conn(0x7efd840fcb80 msgr2=0x7efd841038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:17.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 -- 172.21.15.129:0/25858743 shutdown_connections 2024-05-20T17:56:17.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:17.614+0000 7efd8bfd5640 1 -- 172.21.15.129:0/25858743 wait complete. 2024-05-20T17:56:17.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:17 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/25858743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:17.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:17 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.129:6836/2359452744,v1:172.21.15.129:6837/2359452744] up:boot 2024-05-20T17:56:17.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:17 smithi129 ceph-mon[32029]: fsmap 4 up:standby 2024-05-20T17:56:17.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:17 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-20T17:56:17.913 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 17:56:17 smithi129 podman[70618]: 2024-05-20 17:56:17.564386836 +0000 UTC m=+0.517726674 container init 9acc43aa3ab0895e27120ec5c40793d1288c981345e21884be0ac08b7b3ac2ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-d, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-20T17:56:17.913 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 17:56:17 smithi129 podman[70618]: 2024-05-20 17:56:17.567586073 +0000 UTC m=+0.520925919 container start 9acc43aa3ab0895e27120ec5c40793d1288c981345e21884be0ac08b7b3ac2ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2) 2024-05-20T17:56:17.913 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 17:56:17 smithi129 ceph-mds[70676]: -- 172.21.15.129:0/2252298429 <== mon.0 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564f1cbd5a40 con 0x564f1cc4d600 2024-05-20T17:56:17.913 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 17:56:17 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-d[70672]: starting mds.d at 2024-05-20T17:56:17.913 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 17:56:17 smithi129 bash[70618]: 9acc43aa3ab0895e27120ec5c40793d1288c981345e21884be0ac08b7b3ac2ef 2024-05-20T17:56:17.913 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 17:56:17 smithi129 systemd[1]: Started Ceph mds.d for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:17.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:17 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/25858743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:17.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:17 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.129:6836/2359452744,v1:172.21.15.129:6837/2359452744] up:boot 2024-05-20T17:56:17.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:17 smithi204 ceph-mon[37076]: fsmap 4 up:standby 2024-05-20T17:56:17.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:17 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-20T17:56:18.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:17 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/25858743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:18.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:17 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.129:6836/2359452744,v1:172.21.15.129:6837/2359452744] up:boot 2024-05-20T17:56:18.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:17 smithi154 ceph-mon[36482]: fsmap 4 up:standby 2024-05-20T17:56:18.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:17 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-20T17:56:18.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:18 smithi129 ceph-mon[32029]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:18.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:18 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:19.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:18 smithi154 ceph-mon[36482]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:19.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:18 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:19.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:18 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:19.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:18 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:19.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:18 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:19.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:18 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:19.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:18 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:19.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:18 smithi204 ceph-mon[37076]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:19.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:18 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:19.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:18 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:19.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:18 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:19.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:18 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:19.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:18 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:19.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:18 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:19.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:19.709 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:19 smithi154 ceph-mon[36482]: Deploying daemon mds.e on smithi154 2024-05-20T17:56:19.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.742+0000 7eff3b8a2640 1 -- 172.21.15.129:0/3610188369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff341019c0 msgr2=0x7eff341026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:19.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.742+0000 7eff3b8a2640 1 --2- 172.21.15.129:0/3610188369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff341019c0 0x7eff341026c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7eff30009900 tx=0x7eff3002a410 comp rx=0 tx=0).stop 2024-05-20T17:56:19.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.742+0000 7eff3b8a2640 1 -- 172.21.15.129:0/3610188369 shutdown_connections 2024-05-20T17:56:19.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.742+0000 7eff3b8a2640 1 --2- 172.21.15.129:0/3610188369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff34102c00 0x7eff34103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:19.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.742+0000 7eff3b8a2640 1 --2- 172.21.15.129:0/3610188369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff34107900 0x7eff34109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:19.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.742+0000 7eff3b8a2640 1 --2- 172.21.15.129:0/3610188369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff341019c0 0x7eff341026c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:19.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.742+0000 7eff3b8a2640 1 -- 172.21.15.129:0/3610188369 >> 172.21.15.129:0/3610188369 conn(0x7eff340fcbb0 msgr2=0x7eff340ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:19.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.742+0000 7eff3b8a2640 1 -- 172.21.15.129:0/3610188369 shutdown_connections 2024-05-20T17:56:19.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.742+0000 7eff3b8a2640 1 -- 172.21.15.129:0/3610188369 wait complete. 2024-05-20T17:56:19.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.742+0000 7eff3b8a2640 1 Processor -- start 2024-05-20T17:56:19.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.742+0000 7eff3b8a2640 1 -- start start 2024-05-20T17:56:19.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3b8a2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341019c0 0x7eff3406acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:19.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3b8a2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff34102c00 0x7eff3419e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:19.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3b8a2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff34107900 0x7eff341a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:19.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3b8a2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7eff3410d610 con 0x7eff34107900 2024-05-20T17:56:19.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3b8a2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff3410d490 con 0x7eff341019c0 2024-05-20T17:56:19.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3b8a2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7eff3410d790 con 0x7eff34102c00 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff39e3f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff34107900 0x7eff341a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff39e3f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff34107900 0x7eff341a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51854/0 (socket says 172.21.15.129:51854) 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff39e3f640 1 -- 172.21.15.129:0/878163345 learned_addr learned my addr 172.21.15.129:0/878163345 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3963e640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341019c0 0x7eff3406acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff38e3d640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff34102c00 0x7eff3419e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3963e640 1 -- 172.21.15.129:0/878163345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff34102c00 msgr2=0x7eff3419e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3963e640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff34102c00 0x7eff3419e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3963e640 1 -- 172.21.15.129:0/878163345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff34107900 msgr2=0x7eff341a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3963e640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff34107900 0x7eff341a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3963e640 1 -- 172.21.15.129:0/878163345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff341a0e60 con 0x7eff341019c0 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff39e3f640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff34107900 0x7eff341a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.743+0000 7eff3963e640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341019c0 0x7eff3406acb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7eff3002a930 tx=0x7eff300042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:19.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.744+0000 7eff267fc640 1 -- 172.21.15.129:0/878163345 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3003f030 con 0x7eff341019c0 2024-05-20T17:56:19.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.744+0000 7eff3b8a2640 1 -- 172.21.15.129:0/878163345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff341a10f0 con 0x7eff341019c0 2024-05-20T17:56:19.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.744+0000 7eff267fc640 1 -- 172.21.15.129:0/878163345 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff30031a00 con 0x7eff341019c0 2024-05-20T17:56:19.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.744+0000 7eff267fc640 1 -- 172.21.15.129:0/878163345 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30035070 con 0x7eff341019c0 2024-05-20T17:56:19.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.744+0000 7eff3b8a2640 1 -- 172.21.15.129:0/878163345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff341a15b0 con 0x7eff341019c0 2024-05-20T17:56:19.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.745+0000 7eff3b8a2640 1 -- 172.21.15.129:0/878163345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff34106710 con 0x7eff341019c0 2024-05-20T17:56:19.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.746+0000 7eff267fc640 1 -- 172.21.15.129:0/878163345 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7eff30042030 con 0x7eff341019c0 2024-05-20T17:56:19.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.747+0000 7eff267fc640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff200a2b90 0x7eff200a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:19.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.747+0000 7eff38e3d640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff200a2b90 0x7eff200a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:19.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.747+0000 7eff267fc640 1 -- 172.21.15.129:0/878163345 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7eff300ef1e0 con 0x7eff341019c0 2024-05-20T17:56:19.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.747+0000 7eff38e3d640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff200a2b90 0x7eff200a50a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7eff3406a650 tx=0x7eff28005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:19.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.750+0000 7eff267fc640 1 -- 172.21.15.129:0/878163345 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7eff300bc5e0 con 0x7eff341019c0 2024-05-20T17:56:19.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:19 smithi129 ceph-mon[32029]: Deploying daemon mds.e on smithi154 2024-05-20T17:56:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.967+0000 7eff3b8a2640 1 -- 172.21.15.129:0/878163345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff3402d2a0 con 0x7eff341019c0 2024-05-20T17:56:19.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.968+0000 7eff267fc640 1 -- 172.21.15.129:0/878163345 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3880 (secure 0 0 0) 0x7eff30005dc0 con 0x7eff341019c0 2024-05-20T17:56:19.967 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:19.967 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-20T17:56:19.968 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 5 2024-05-20T17:56:19.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 -- 172.21.15.129:0/878163345 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff200a2b90 msgr2=0x7eff200a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:19.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff200a2b90 0x7eff200a50a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7eff3406a650 tx=0x7eff28005f90 comp rx=0 tx=0).stop 2024-05-20T17:56:19.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 -- 172.21.15.129:0/878163345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341019c0 msgr2=0x7eff3406acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:19.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341019c0 0x7eff3406acb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7eff3002a930 tx=0x7eff300042f0 comp rx=0 tx=0).stop 2024-05-20T17:56:19.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 -- 172.21.15.129:0/878163345 shutdown_connections 2024-05-20T17:56:19.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7eff200a2b90 0x7eff200a50a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:19.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff34107900 0x7eff341a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:19.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff34102c00 0x7eff3419e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:19.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 --2- 172.21.15.129:0/878163345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff341019c0 0x7eff3406acb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:19.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 -- 172.21.15.129:0/878163345 >> 172.21.15.129:0/878163345 conn(0x7eff340fcbb0 msgr2=0x7eff341038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:19.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 -- 172.21.15.129:0/878163345 shutdown_connections 2024-05-20T17:56:19.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:19.970+0000 7eff3b8a2640 1 -- 172.21.15.129:0/878163345 wait complete. 2024-05-20T17:56:20.051 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 17:56:20 smithi154 systemd[1]: Starting Ceph mds.e for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:56:20.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:19 smithi204 ceph-mon[37076]: Deploying daemon mds.e on smithi154 2024-05-20T17:56:20.394 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 17:56:20 smithi154 podman[66067]: 2024-05-20 17:56:20.171829289 +0000 UTC m=+0.018140810 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:56:20.394 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 17:56:20 smithi154 podman[66067]: 2024-05-20 17:56:20.278938169 +0000 UTC m=+0.125249718 container create 9cd4a95b1ef887af37c8bc5afc3e9d61c62718e6f543151a1bdc8fd244e36e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c) 2024-05-20T17:56:20.710 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 17:56:20 smithi154 podman[66067]: 2024-05-20 17:56:20.523434479 +0000 UTC m=+0.369745983 container init 9cd4a95b1ef887af37c8bc5afc3e9d61c62718e6f543151a1bdc8fd244e36e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-e, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-20T17:56:20.710 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 17:56:20 smithi154 podman[66067]: 2024-05-20 17:56:20.525976146 +0000 UTC m=+0.372287654 container start 9cd4a95b1ef887af37c8bc5afc3e9d61c62718e6f543151a1bdc8fd244e36e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-e, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-20T17:56:20.710 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 17:56:20 smithi154 ceph-mds[66085]: -- 172.21.15.154:0/2774785289 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a7d8523a40 con 0x55a7d859b180 2024-05-20T17:56:20.710 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 17:56:20 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-e[66081]: starting mds.e at 2024-05-20T17:56:20.710 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 17:56:20 smithi154 bash[66067]: 9cd4a95b1ef887af37c8bc5afc3e9d61c62718e6f543151a1bdc8fd244e36e6b 2024-05-20T17:56:20.710 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 17:56:20 smithi154 systemd[1]: Started Ceph mds.e for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:21.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:20 smithi154 ceph-mon[36482]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:21.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:20 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/878163345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:21.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:20 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:21.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:20 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:21.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:20 smithi129 ceph-mon[32029]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:21.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:20 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/878163345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:21.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:20 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:21.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:20 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:21.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:20 smithi204 ceph-mon[37076]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:21.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:20 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/878163345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:21.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:20 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:21.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:20 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:21.981 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:22.008 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:21 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.154:6834/3512970065,v1:172.21.15.154:6835/3512970065] up:boot 2024-05-20T17:56:22.008 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:21 smithi129 ceph-mon[32029]: fsmap 5 up:standby 2024-05-20T17:56:22.008 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-20T17:56:22.008 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:22.008 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:22.008 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:22.008 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:21 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:22.008 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:21 smithi129 ceph-mon[32029]: Deploying daemon mds.f on smithi204 2024-05-20T17:56:22.009 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:21 smithi129 ceph-mon[32029]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.107+0000 7f3849644640 1 -- 172.21.15.129:0/1624887921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38441078b0 msgr2=0x7f3844109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.107+0000 7f3849644640 1 --2- 172.21.15.129:0/1624887921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38441078b0 0x7f3844109cd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f383c009ca0 tx=0x7f383c02a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.107+0000 7f3849644640 1 -- 172.21.15.129:0/1624887921 shutdown_connections 2024-05-20T17:56:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.107+0000 7f3849644640 1 --2- 172.21.15.129:0/1624887921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3844102bb0 0x7f3844102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.107+0000 7f3849644640 1 --2- 172.21.15.129:0/1624887921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38441078b0 0x7f3844109cd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.107+0000 7f3849644640 1 --2- 172.21.15.129:0/1624887921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3844101970 0x7f3844102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.107+0000 7f3849644640 1 -- 172.21.15.129:0/1624887921 >> 172.21.15.129:0/1624887921 conn(0x7f38440fcb80 msgr2=0x7f38440ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:22.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.107+0000 7f3849644640 1 -- 172.21.15.129:0/1624887921 shutdown_connections 2024-05-20T17:56:22.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.108+0000 7f3849644640 1 -- 172.21.15.129:0/1624887921 wait complete. 2024-05-20T17:56:22.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.108+0000 7f3849644640 1 Processor -- start 2024-05-20T17:56:22.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.108+0000 7f3849644640 1 -- start start 2024-05-20T17:56:22.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.108+0000 7f3849644640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3844101970 0x7f384419b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:22.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f3849644640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3844102bb0 0x7f384419b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:22.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f3849644640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38441078b0 0x7f38441a2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:22.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f3849644640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f384410c020 con 0x7f38441078b0 2024-05-20T17:56:22.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f3849644640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f384410bea0 con 0x7f3844102bb0 2024-05-20T17:56:22.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f3849644640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f384410c1a0 con 0x7f3844101970 2024-05-20T17:56:22.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f38437fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38441078b0 0x7f38441a2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:22.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f3842ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3844101970 0x7f384419b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:22.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f38437fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38441078b0 0x7f38441a2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51876/0 (socket says 172.21.15.129:51876) 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f38437fe640 1 -- 172.21.15.129:0/2246702567 learned_addr learned my addr 172.21.15.129:0/2246702567 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f38427fc640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3844102bb0 0x7f384419b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f38437fe640 1 -- 172.21.15.129:0/2246702567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3844101970 msgr2=0x7f384419b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f38437fe640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3844101970 0x7f384419b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f38437fe640 1 -- 172.21.15.129:0/2246702567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3844102bb0 msgr2=0x7f384419b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f38437fe640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3844102bb0 0x7f384419b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f38437fe640 1 -- 172.21.15.129:0/2246702567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38441a2fd0 con 0x7f38441078b0 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f3842ffd640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3844101970 0x7f384419b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f38437fe640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38441078b0 0x7f38441a2960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f383c005cc0 tx=0x7f383c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f382bfff640 1 -- 172.21.15.129:0/2246702567 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c003b70 con 0x7f38441078b0 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f382bfff640 1 -- 172.21.15.129:0/2246702567 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f383c003d10 con 0x7f38441078b0 2024-05-20T17:56:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f3849644640 1 -- 172.21.15.129:0/2246702567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38441a3260 con 0x7f38441078b0 2024-05-20T17:56:22.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f382bfff640 1 -- 172.21.15.129:0/2246702567 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c02fbd0 con 0x7f38441078b0 2024-05-20T17:56:22.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.109+0000 7f3849644640 1 -- 172.21.15.129:0/2246702567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38441a37b0 con 0x7f38441078b0 2024-05-20T17:56:22.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.110+0000 7f3849644640 1 -- 172.21.15.129:0/2246702567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3844105fb0 con 0x7f38441078b0 2024-05-20T17:56:22.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.112+0000 7f382bfff640 1 -- 172.21.15.129:0/2246702567 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f383c02fd70 con 0x7f38441078b0 2024-05-20T17:56:22.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.112+0000 7f382bfff640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f382c0a2b90 0x7f382c0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:22.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.112+0000 7f3842ffd640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f382c0a2b90 0x7f382c0a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:22.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.113+0000 7f382bfff640 1 -- 172.21.15.129:0/2246702567 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f383c0f0050 con 0x7f38441078b0 2024-05-20T17:56:22.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.113+0000 7f3842ffd640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f382c0a2b90 0x7f382c0a50a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3844102570 tx=0x7f3834005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.116+0000 7f382bfff640 1 -- 172.21.15.129:0/2246702567 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f383c0bd450 con 0x7f38441078b0 2024-05-20T17:56:22.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:21 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.154:6834/3512970065,v1:172.21.15.154:6835/3512970065] up:boot 2024-05-20T17:56:22.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:21 smithi154 ceph-mon[36482]: fsmap 5 up:standby 2024-05-20T17:56:22.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-20T17:56:22.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:22.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:22.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:22.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:21 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:22.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:21 smithi154 ceph-mon[36482]: Deploying daemon mds.f on smithi204 2024-05-20T17:56:22.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:21 smithi154 ceph-mon[36482]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:22.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:21 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.154:6834/3512970065,v1:172.21.15.154:6835/3512970065] up:boot 2024-05-20T17:56:22.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:21 smithi204 ceph-mon[37076]: fsmap 5 up:standby 2024-05-20T17:56:22.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-20T17:56:22.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:22.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:22.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:22.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:21 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:22.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:21 smithi204 ceph-mon[37076]: Deploying daemon mds.f on smithi204 2024-05-20T17:56:22.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:21 smithi204 ceph-mon[37076]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:22.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.329+0000 7f3849644640 1 -- 172.21.15.129:0/2246702567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f384402d2a0 con 0x7f38441078b0 2024-05-20T17:56:22.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.329+0000 7f382bfff640 1 -- 172.21.15.129:0/2246702567 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4708 (secure 0 0 0) 0x7f383c0bcd50 con 0x7f38441078b0 2024-05-20T17:56:22.330 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:22.330 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-20T17:56:22.331 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 6 2024-05-20T17:56:22.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 -- 172.21.15.129:0/2246702567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f382c0a2b90 msgr2=0x7f382c0a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:22.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f382c0a2b90 0x7f382c0a50a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3844102570 tx=0x7f3834005f90 comp rx=0 tx=0).stop 2024-05-20T17:56:22.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 -- 172.21.15.129:0/2246702567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38441078b0 msgr2=0x7f38441a2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:22.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38441078b0 0x7f38441a2960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f383c005cc0 tx=0x7f383c004ab0 comp rx=0 tx=0).stop 2024-05-20T17:56:22.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 -- 172.21.15.129:0/2246702567 shutdown_connections 2024-05-20T17:56:22.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f382c0a2b90 0x7f382c0a50a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:22.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3844102bb0 0x7f384419b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:22.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38441078b0 0x7f38441a2960 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:22.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 --2- 172.21.15.129:0/2246702567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3844101970 0x7f384419b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:22.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 -- 172.21.15.129:0/2246702567 >> 172.21.15.129:0/2246702567 conn(0x7f38440fcb80 msgr2=0x7f38440fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:22.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 -- 172.21.15.129:0/2246702567 shutdown_connections 2024-05-20T17:56:22.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:22.332+0000 7f3849644640 1 -- 172.21.15.129:0/2246702567 wait complete. 2024-05-20T17:56:22.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:22 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2246702567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:22.930 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 17:56:22 smithi204 systemd[1]: Starting Ceph mds.f for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:56:23.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:22 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2246702567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:23.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:22 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2246702567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:23.316 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 17:56:23 smithi204 podman[67429]: 2024-05-20 17:56:22.964059684 +0000 UTC m=+0.017130943 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:56:23.316 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 17:56:23 smithi204 podman[67429]: 2024-05-20 17:56:23.081449882 +0000 UTC m=+0.134521128 container create 756cb0b9c6660f7338ccb14ea57796b55c7e2d977cd5e2fcbc462abc9a79d9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-20T17:56:23.680 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 17:56:23 smithi204 podman[67429]: 2024-05-20 17:56:23.318073766 +0000 UTC m=+0.371145020 container init 756cb0b9c6660f7338ccb14ea57796b55c7e2d977cd5e2fcbc462abc9a79d9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-f, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-20T17:56:23.680 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 17:56:23 smithi204 podman[67429]: 2024-05-20 17:56:23.320890185 +0000 UTC m=+0.373961428 container start 756cb0b9c6660f7338ccb14ea57796b55c7e2d977cd5e2fcbc462abc9a79d9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-f, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-20T17:56:23.680 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 17:56:23 smithi204 ceph-mds[67448]: -- 172.21.15.204:0/3075737515 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ff6b799a40 con 0x55ff6b811a80 2024-05-20T17:56:23.680 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 17:56:23 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-f[67444]: starting mds.f at 2024-05-20T17:56:23.680 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 17:56:23 smithi204 bash[67429]: 756cb0b9c6660f7338ccb14ea57796b55c7e2d977cd5e2fcbc462abc9a79d9af 2024-05-20T17:56:23.680 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 17:56:23 smithi204 systemd[1]: Started Ceph mds.f for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:24.343 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:24.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.471+0000 7f06288e2640 1 -- 172.21.15.129:0/4017985657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06241028c0 msgr2=0x7f0624102d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:24.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.471+0000 7f06288e2640 1 --2- 172.21.15.129:0/4017985657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06241028c0 0x7f0624102d70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0618009900 tx=0x7f061802a410 comp rx=0 tx=0).stop 2024-05-20T17:56:24.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.471+0000 7f06288e2640 1 -- 172.21.15.129:0/4017985657 shutdown_connections 2024-05-20T17:56:24.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.471+0000 7f06288e2640 1 --2- 172.21.15.129:0/4017985657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0624103400 0x7f0624109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:24.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.471+0000 7f06288e2640 1 --2- 172.21.15.129:0/4017985657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06241028c0 0x7f0624102d70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:24.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.471+0000 7f06288e2640 1 --2- 172.21.15.129:0/4017985657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0624101ee0 0x7f06241022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:24.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.471+0000 7f06288e2640 1 -- 172.21.15.129:0/4017985657 >> 172.21.15.129:0/4017985657 conn(0x7f06240fc2e0 msgr2=0x7f06240fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:24.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.472+0000 7f06288e2640 1 -- 172.21.15.129:0/4017985657 shutdown_connections 2024-05-20T17:56:24.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.472+0000 7f06288e2640 1 -- 172.21.15.129:0/4017985657 wait complete. 2024-05-20T17:56:24.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.472+0000 7f06288e2640 1 Processor -- start 2024-05-20T17:56:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.472+0000 7f06288e2640 1 -- start start 2024-05-20T17:56:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f06288e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0624101ee0 0x7f06241981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f06288e2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06241028c0 0x7f0624198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f06288e2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0624103400 0x7f062419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f06288e2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f062410d620 con 0x7f0624101ee0 2024-05-20T17:56:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f06288e2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f062410d4a0 con 0x7f0624103400 2024-05-20T17:56:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f06288e2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f062410d7a0 con 0x7f06241028c0 2024-05-20T17:56:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f062259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0624101ee0 0x7f06241981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f062259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0624101ee0 0x7f06241981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51894/0 (socket says 172.21.15.129:51894) 2024-05-20T17:56:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f062259c640 1 -- 172.21.15.129:0/1333262483 learned_addr learned my addr 172.21.15.129:0/1333262483 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f0621d9b640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06241028c0 0x7f0624198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f0622d9d640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0624103400 0x7f062419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f062259c640 1 -- 172.21.15.129:0/1333262483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06241028c0 msgr2=0x7f0624198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f062259c640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06241028c0 0x7f0624198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f062259c640 1 -- 172.21.15.129:0/1333262483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0624103400 msgr2=0x7f062419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f062259c640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0624103400 0x7f062419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f062259c640 1 -- 172.21.15.129:0/1333262483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062419fd00 con 0x7f0624101ee0 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f062259c640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0624101ee0 0x7f06241981c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f061400e9c0 tx=0x7f061400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f06137fe640 1 -- 172.21.15.129:0/1333262483 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061400ccc0 con 0x7f0624101ee0 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f06137fe640 1 -- 172.21.15.129:0/1333262483 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0614005500 con 0x7f0624101ee0 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f06137fe640 1 -- 172.21.15.129:0/1333262483 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614010690 con 0x7f0624101ee0 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f06288e2640 1 -- 172.21.15.129:0/1333262483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f062419fff0 con 0x7f0624101ee0 2024-05-20T17:56:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.473+0000 7f06288e2640 1 -- 172.21.15.129:0/1333262483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06241a0400 con 0x7f0624101ee0 2024-05-20T17:56:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.475+0000 7f06288e2640 1 -- 172.21.15.129:0/1333262483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0624106880 con 0x7f0624101ee0 2024-05-20T17:56:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.476+0000 7f06137fe640 1 -- 172.21.15.129:0/1333262483 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0614004e80 con 0x7f0624101ee0 2024-05-20T17:56:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.476+0000 7f06137fe640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f060c0a2b90 0x7f060c0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.476+0000 7f0621d9b640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f060c0a2b90 0x7f060c0a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:24.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.477+0000 7f06137fe640 1 -- 172.21.15.129:0/1333262483 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f06140cef30 con 0x7f0624101ee0 2024-05-20T17:56:24.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.477+0000 7f0621d9b640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f060c0a2b90 0x7f060c0a50a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0618009230 tx=0x7f0618031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:24.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.481+0000 7f06137fe640 1 -- 172.21.15.129:0/1333262483 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f061409c330 con 0x7f0624101ee0 2024-05-20T17:56:24.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:24.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:24.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:24.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:24.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:24.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:24.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:24 smithi129 ceph-mon[32029]: Deploying daemon mds.c on smithi204 2024-05-20T17:56:24.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:24 smithi129 ceph-mon[32029]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:24.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:24 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.204:6834/4037294668,v1:172.21.15.204:6835/4037294668] up:boot 2024-05-20T17:56:24.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:24 smithi129 ceph-mon[32029]: fsmap 6 up:standby 2024-05-20T17:56:24.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:24 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-20T17:56:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:24 smithi204 ceph-mon[37076]: Deploying daemon mds.c on smithi204 2024-05-20T17:56:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:24 smithi204 ceph-mon[37076]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:24 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.204:6834/4037294668,v1:172.21.15.204:6835/4037294668] up:boot 2024-05-20T17:56:24.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:24 smithi204 ceph-mon[37076]: fsmap 6 up:standby 2024-05-20T17:56:24.681 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:24 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-20T17:56:24.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.694+0000 7f06288e2640 1 -- 172.21.15.129:0/1333262483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f062402d470 con 0x7f0624101ee0 2024-05-20T17:56:24.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.695+0000 7f06137fe640 1 -- 172.21.15.129:0/1333262483 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5536 (secure 0 0 0) 0x7f061409bc30 con 0x7f0624101ee0 2024-05-20T17:56:24.694 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:24.694 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-20T17:56:24.694 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 7 2024-05-20T17:56:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 -- 172.21.15.129:0/1333262483 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f060c0a2b90 msgr2=0x7f060c0a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f060c0a2b90 0x7f060c0a50a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0618009230 tx=0x7f0618031400 comp rx=0 tx=0).stop 2024-05-20T17:56:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 -- 172.21.15.129:0/1333262483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0624101ee0 msgr2=0x7f06241981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0624101ee0 0x7f06241981c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f061400e9c0 tx=0x7f061400ee80 comp rx=0 tx=0).stop 2024-05-20T17:56:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 -- 172.21.15.129:0/1333262483 shutdown_connections 2024-05-20T17:56:24.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f060c0a2b90 0x7f060c0a50a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:24.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0624103400 0x7f062419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:24.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06241028c0 0x7f0624198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:24.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 --2- 172.21.15.129:0/1333262483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0624101ee0 0x7f06241981c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:24.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 -- 172.21.15.129:0/1333262483 >> 172.21.15.129:0/1333262483 conn(0x7f06240fc2e0 msgr2=0x7f0624103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:24.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 -- 172.21.15.129:0/1333262483 shutdown_connections 2024-05-20T17:56:24.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:24.697+0000 7f06288e2640 1 -- 172.21.15.129:0/1333262483 wait complete. 2024-05-20T17:56:24.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:24.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:24.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:24.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:24.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:24.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:24.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:24 smithi154 ceph-mon[36482]: Deploying daemon mds.c on smithi204 2024-05-20T17:56:24.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:24 smithi154 ceph-mon[36482]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:24.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:24 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.204:6834/4037294668,v1:172.21.15.204:6835/4037294668] up:boot 2024-05-20T17:56:24.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:24 smithi154 ceph-mon[36482]: fsmap 6 up:standby 2024-05-20T17:56:24.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:24 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-20T17:56:25.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:25 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1333262483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:25.680 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 17:56:25 smithi204 systemd[1]: Starting Ceph mds.c for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:56:25.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:25 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1333262483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:25.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:25 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1333262483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:26.031 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 17:56:25 smithi204 podman[67733]: 2024-05-20 17:56:25.680492019 +0000 UTC m=+0.017026533 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:56:26.031 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 17:56:25 smithi204 podman[67733]: 2024-05-20 17:56:25.814285322 +0000 UTC m=+0.150819838 container create 3938857b9ea6d929322ba977e094c8a10325b3942297a0c01902d40043797f29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514) 2024-05-20T17:56:26.031 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 17:56:26 smithi204 podman[67733]: 2024-05-20 17:56:26.03347499 +0000 UTC m=+0.370009504 container init 3938857b9ea6d929322ba977e094c8a10325b3942297a0c01902d40043797f29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c) 2024-05-20T17:56:26.429 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 17:56:26 smithi204 podman[67733]: 2024-05-20 17:56:26.036088819 +0000 UTC m=+0.372623333 container start 3938857b9ea6d929322ba977e094c8a10325b3942297a0c01902d40043797f29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T17:56:26.430 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 17:56:26 smithi204 ceph-mds[67753]: -- 172.21.15.204:0/985745772 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562288327a40 con 0x56228839f180 2024-05-20T17:56:26.430 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 17:56:26 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-c[67749]: starting mds.c at 2024-05-20T17:56:26.430 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 17:56:26 smithi204 bash[67733]: 3938857b9ea6d929322ba977e094c8a10325b3942297a0c01902d40043797f29 2024-05-20T17:56:26.430 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 17:56:26 smithi204 systemd[1]: Started Ceph mds.c for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:26.708 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:26.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.835+0000 7fd22a4ab640 1 -- 172.21.15.129:0/2232504800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd224102bf0 msgr2=0x7fd224103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:26.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.835+0000 7fd22a4ab640 1 --2- 172.21.15.129:0/2232504800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd224102bf0 0x7fd224103000 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009900 tx=0x7fd20c02a410 comp rx=0 tx=0).stop 2024-05-20T17:56:26.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.835+0000 7fd22a4ab640 1 -- 172.21.15.129:0/2232504800 shutdown_connections 2024-05-20T17:56:26.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.835+0000 7fd22a4ab640 1 --2- 172.21.15.129:0/2232504800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2241078f0 0x7fd224109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:26.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.835+0000 7fd22a4ab640 1 --2- 172.21.15.129:0/2232504800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd224102bf0 0x7fd224103000 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.835+0000 7fd22a4ab640 1 --2- 172.21.15.129:0/2232504800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2241019b0 0x7fd2241026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.835+0000 7fd22a4ab640 1 -- 172.21.15.129:0/2232504800 >> 172.21.15.129:0/2232504800 conn(0x7fd2240fcba0 msgr2=0x7fd2240ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.835+0000 7fd22a4ab640 1 -- 172.21.15.129:0/2232504800 shutdown_connections 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.835+0000 7fd22a4ab640 1 -- 172.21.15.129:0/2232504800 wait complete. 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd22a4ab640 1 Processor -- start 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd22a4ab640 1 -- start start 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd22a4ab640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2241019b0 0x7fd2241981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd22a4ab640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd224102bf0 0x7fd224198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd22a4ab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2241078f0 0x7fd22419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd22a4ab640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd22410d650 con 0x7fd224102bf0 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd22a4ab640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd22410d4d0 con 0x7fd2241078f0 2024-05-20T17:56:26.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd22a4ab640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd22410d7d0 con 0x7fd2241019b0 2024-05-20T17:56:26.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd2237fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd224102bf0 0x7fd224198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:26.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd2237fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd224102bf0 0x7fd224198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51914/0 (socket says 172.21.15.129:51914) 2024-05-20T17:56:26.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd2237fe640 1 -- 172.21.15.129:0/1323436330 learned_addr learned my addr 172.21.15.129:0/1323436330 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:26.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.836+0000 7fd228a48640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2241078f0 0x7fd22419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:26.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd223fff640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2241019b0 0x7fd2241981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:26.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd2237fe640 1 -- 172.21.15.129:0/1323436330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2241019b0 msgr2=0x7fd2241981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:26.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd2237fe640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2241019b0 0x7fd2241981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:26.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd2237fe640 1 -- 172.21.15.129:0/1323436330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2241078f0 msgr2=0x7fd22419f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:26.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd2237fe640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2241078f0 0x7fd22419f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:26.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd2237fe640 1 -- 172.21.15.129:0/1323436330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd224073e80 con 0x7fd224102bf0 2024-05-20T17:56:26.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd2237fe640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd224102bf0 0x7fd224198700 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd20c031440 tx=0x7fd20c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:26.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd2217fa640 1 -- 172.21.15.129:0/1323436330 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c03f030 con 0x7fd224102bf0 2024-05-20T17:56:26.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd2217fa640 1 -- 172.21.15.129:0/1323436330 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd20c003ce0 con 0x7fd224102bf0 2024-05-20T17:56:26.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd22a4ab640 1 -- 172.21.15.129:0/1323436330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd224074110 con 0x7fd224102bf0 2024-05-20T17:56:26.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd2217fa640 1 -- 172.21.15.129:0/1323436330 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c035070 con 0x7fd224102bf0 2024-05-20T17:56:26.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.837+0000 7fd22a4ab640 1 -- 172.21.15.129:0/1323436330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd224074600 con 0x7fd224102bf0 2024-05-20T17:56:26.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.838+0000 7fd22a4ab640 1 -- 172.21.15.129:0/1323436330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd224106700 con 0x7fd224102bf0 2024-05-20T17:56:26.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.840+0000 7fd2217fa640 1 -- 172.21.15.129:0/1323436330 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd20c042030 con 0x7fd224102bf0 2024-05-20T17:56:26.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.841+0000 7fd2217fa640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2100a2b90 0x7fd2100a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:26.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.841+0000 7fd2217fa640 1 -- 172.21.15.129:0/1323436330 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd20c0ef580 con 0x7fd224102bf0 2024-05-20T17:56:26.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.841+0000 7fd223fff640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2100a2b90 0x7fd2100a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:26.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.841+0000 7fd223fff640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2100a2b90 0x7fd2100a50a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd214005d00 tx=0x7fd214005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:26.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:26.845+0000 7fd2217fa640 1 -- 172.21.15.129:0/1323436330 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd20c0bc980 con 0x7fd224102bf0 2024-05-20T17:56:26.888 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:26 smithi154 ceph-mon[36482]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:26.888 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:26 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:26.888 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:26 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:26.888 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:26 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:26.888 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:26 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:26.888 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:26 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:26.888 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:26 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:26.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:26 smithi129 ceph-mon[32029]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:26.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:26.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:26.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:26.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:26.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:26.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:26 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:26.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:26 smithi204 ceph-mon[37076]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:26.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:26 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:26.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:26 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:26.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:26 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:26.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:26 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:26.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:26 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:26.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:26 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:27.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.057+0000 7fd22a4ab640 1 -- 172.21.15.129:0/1323436330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd22402d2a0 con 0x7fd224102bf0 2024-05-20T17:56:27.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.057+0000 7fd2217fa640 1 -- 172.21.15.129:0/1323436330 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6361 (secure 0 0 0) 0x7fd20c0bc280 con 0x7fd224102bf0 2024-05-20T17:56:27.055 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:27.056 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-20T17:56:27.056 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 8 2024-05-20T17:56:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.059+0000 7fd22a4ab640 1 -- 172.21.15.129:0/1323436330 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2100a2b90 msgr2=0x7fd2100a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.059+0000 7fd22a4ab640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2100a2b90 0x7fd2100a50a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd214005d00 tx=0x7fd214005c50 comp rx=0 tx=0).stop 2024-05-20T17:56:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.060+0000 7fd22a4ab640 1 -- 172.21.15.129:0/1323436330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd224102bf0 msgr2=0x7fd224198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.060+0000 7fd22a4ab640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd224102bf0 0x7fd224198700 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd20c031440 tx=0x7fd20c032ce0 comp rx=0 tx=0).stop 2024-05-20T17:56:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.060+0000 7fd22a4ab640 1 -- 172.21.15.129:0/1323436330 shutdown_connections 2024-05-20T17:56:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.060+0000 7fd22a4ab640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2100a2b90 0x7fd2100a50a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.060+0000 7fd22a4ab640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2241078f0 0x7fd22419f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.060+0000 7fd22a4ab640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd224102bf0 0x7fd224198700 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.060+0000 7fd22a4ab640 1 --2- 172.21.15.129:0/1323436330 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd2241019b0 0x7fd2241981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.060+0000 7fd22a4ab640 1 -- 172.21.15.129:0/1323436330 >> 172.21.15.129:0/1323436330 conn(0x7fd2240fcba0 msgr2=0x7fd2241038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.060+0000 7fd22a4ab640 1 -- 172.21.15.129:0/1323436330 shutdown_connections 2024-05-20T17:56:27.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:27.060+0000 7fd22a4ab640 1 -- 172.21.15.129:0/1323436330 wait complete. 2024-05-20T17:56:27.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:27 smithi154 ceph-mon[36482]: Deploying daemon mds.h on smithi154 2024-05-20T17:56:27.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:27 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.204:6836/259458619,v1:172.21.15.204:6837/259458619] up:boot 2024-05-20T17:56:27.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:27 smithi154 ceph-mon[36482]: fsmap 7 up:standby 2024-05-20T17:56:27.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:27 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-20T17:56:27.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:27 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1323436330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:27.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:27 smithi129 ceph-mon[32029]: Deploying daemon mds.h on smithi154 2024-05-20T17:56:27.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:27 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.204:6836/259458619,v1:172.21.15.204:6837/259458619] up:boot 2024-05-20T17:56:27.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:27 smithi129 ceph-mon[32029]: fsmap 7 up:standby 2024-05-20T17:56:27.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:27 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-20T17:56:27.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:27 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1323436330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:27.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:27 smithi204 ceph-mon[37076]: Deploying daemon mds.h on smithi154 2024-05-20T17:56:27.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:27 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.204:6836/259458619,v1:172.21.15.204:6837/259458619] up:boot 2024-05-20T17:56:27.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:27 smithi204 ceph-mon[37076]: fsmap 7 up:standby 2024-05-20T17:56:27.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:27 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-20T17:56:27.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:27 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1323436330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:28.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:28 smithi154 ceph-mon[36482]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:28.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:28 smithi129 ceph-mon[32029]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:28.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:28 smithi204 ceph-mon[37076]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:29.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:29.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.198+0000 7fc565785640 1 -- 172.21.15.129:0/1616747863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc560101970 msgr2=0x7fc560102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:29.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.198+0000 7fc565785640 1 --2- 172.21.15.129:0/1616747863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc560101970 0x7fc560102670 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc550009ca0 tx=0x7fc55002a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:29.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.198+0000 7fc565785640 1 -- 172.21.15.129:0/1616747863 shutdown_connections 2024-05-20T17:56:29.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.198+0000 7fc565785640 1 --2- 172.21.15.129:0/1616747863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5601078b0 0x7fc560109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:29.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.198+0000 7fc565785640 1 --2- 172.21.15.129:0/1616747863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc560102bb0 0x7fc560102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:29.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.198+0000 7fc565785640 1 --2- 172.21.15.129:0/1616747863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc560101970 0x7fc560102670 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:29.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.198+0000 7fc565785640 1 -- 172.21.15.129:0/1616747863 >> 172.21.15.129:0/1616747863 conn(0x7fc5600fcb80 msgr2=0x7fc5600ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:29.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.198+0000 7fc565785640 1 -- 172.21.15.129:0/1616747863 shutdown_connections 2024-05-20T17:56:29.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.198+0000 7fc565785640 1 -- 172.21.15.129:0/1616747863 wait complete. 2024-05-20T17:56:29.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.199+0000 7fc565785640 1 Processor -- start 2024-05-20T17:56:29.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.199+0000 7fc565785640 1 -- start start 2024-05-20T17:56:29.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.199+0000 7fc565785640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc560101970 0x7fc5601981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:29.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.199+0000 7fc565785640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc560102bb0 0x7fc560198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.199+0000 7fc565785640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5601078b0 0x7fc56019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.199+0000 7fc565785640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010d5c0 con 0x7fc560101970 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.199+0000 7fc565785640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010d440 con 0x7fc5601078b0 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.199+0000 7fc565785640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010d740 con 0x7fc560102bb0 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc560101970 0x7fc5601981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc560101970 0x7fc5601981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51934/0 (socket says 172.21.15.129:51934) 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55effd640 1 -- 172.21.15.129:0/1280258044 learned_addr learned my addr 172.21.15.129:0/1280258044 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55e7fc640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc560102bb0 0x7fc560198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55f7fe640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5601078b0 0x7fc56019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55effd640 1 -- 172.21.15.129:0/1280258044 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc560102bb0 msgr2=0x7fc560198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55effd640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc560102bb0 0x7fc560198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55effd640 1 -- 172.21.15.129:0/1280258044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5601078b0 msgr2=0x7fc56019f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55effd640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5601078b0 0x7fc56019f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:29.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55effd640 1 -- 172.21.15.129:0/1280258044 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56019fc50 con 0x7fc560101970 2024-05-20T17:56:29.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55f7fe640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5601078b0 0x7fc56019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:56:29.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc55effd640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc560101970 0x7fc5601981c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc550000f80 tx=0x7fc550031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:29.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc547fff640 1 -- 172.21.15.129:0/1280258044 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55002faa0 con 0x7fc560101970 2024-05-20T17:56:29.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc547fff640 1 -- 172.21.15.129:0/1280258044 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc55002fc40 con 0x7fc560101970 2024-05-20T17:56:29.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc565785640 1 -- 172.21.15.129:0/1280258044 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56019fee0 con 0x7fc560101970 2024-05-20T17:56:29.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc547fff640 1 -- 172.21.15.129:0/1280258044 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55003a7f0 con 0x7fc560101970 2024-05-20T17:56:29.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.200+0000 7fc565785640 1 -- 172.21.15.129:0/1280258044 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5601a0340 con 0x7fc560101970 2024-05-20T17:56:29.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.202+0000 7fc565785640 1 -- 172.21.15.129:0/1280258044 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5601066c0 con 0x7fc560101970 2024-05-20T17:56:29.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.203+0000 7fc547fff640 1 -- 172.21.15.129:0/1280258044 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc550030bc0 con 0x7fc560101970 2024-05-20T17:56:29.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.204+0000 7fc547fff640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5480a2b90 0x7fc5480a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:29.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.204+0000 7fc55e7fc640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5480a2b90 0x7fc5480a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:29.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.204+0000 7fc547fff640 1 -- 172.21.15.129:0/1280258044 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc5500f0320 con 0x7fc560101970 2024-05-20T17:56:29.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.204+0000 7fc55e7fc640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5480a2b90 0x7fc5480a50a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc560199260 tx=0x7fc554005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.207+0000 7fc547fff640 1 -- 172.21.15.129:0/1280258044 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc5500bd720 con 0x7fc560101970 2024-05-20T17:56:29.393 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:56:29 smithi154 systemd[1]: Started Ceph mds.h for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.424+0000 7fc565785640 1 -- 172.21.15.129:0/1280258044 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc56002d3c0 con 0x7fc560101970 2024-05-20T17:56:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.425+0000 7fc547fff640 1 -- 172.21.15.129:0/1280258044 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6361 (secure 0 0 0) 0x7fc5500bd020 con 0x7fc560101970 2024-05-20T17:56:29.424 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:29.424 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-20T17:56:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 8 2024-05-20T17:56:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 -- 172.21.15.129:0/1280258044 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5480a2b90 msgr2=0x7fc5480a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5480a2b90 0x7fc5480a50a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc560199260 tx=0x7fc554005f90 comp rx=0 tx=0).stop 2024-05-20T17:56:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 -- 172.21.15.129:0/1280258044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc560101970 msgr2=0x7fc5601981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc560101970 0x7fc5601981c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc550000f80 tx=0x7fc550031ee0 comp rx=0 tx=0).stop 2024-05-20T17:56:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 -- 172.21.15.129:0/1280258044 shutdown_connections 2024-05-20T17:56:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc5480a2b90 0x7fc5480a50a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5601078b0 0x7fc56019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc560102bb0 0x7fc560198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 --2- 172.21.15.129:0/1280258044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc560101970 0x7fc5601981c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 -- 172.21.15.129:0/1280258044 >> 172.21.15.129:0/1280258044 conn(0x7fc5600fcb80 msgr2=0x7fc560103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 -- 172.21.15.129:0/1280258044 shutdown_connections 2024-05-20T17:56:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:29.427+0000 7fc565785640 1 -- 172.21.15.129:0/1280258044 wait complete. 2024-05-20T17:56:30.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:30.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:30.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:30.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:30.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:30.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:30.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: Deploying daemon mds.i on smithi204 2024-05-20T17:56:30.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1280258044' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:30.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.154:6836/1478533749,v1:172.21.15.154:6837/1478533749] up:boot 2024-05-20T17:56:30.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: fsmap 8 up:standby 2024-05-20T17:56:30.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T17:56:30.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:30 smithi154 ceph-mon[36482]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:30.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:30.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:30.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:30.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:30.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:30.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:30.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: Deploying daemon mds.i on smithi204 2024-05-20T17:56:30.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1280258044' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:30.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.154:6836/1478533749,v1:172.21.15.154:6837/1478533749] up:boot 2024-05-20T17:56:30.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: fsmap 8 up:standby 2024-05-20T17:56:30.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T17:56:30.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:30 smithi129 ceph-mon[32029]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:30.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:30.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:30.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:30.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:30.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:30.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:30.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: Deploying daemon mds.i on smithi204 2024-05-20T17:56:30.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1280258044' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:30.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.154:6836/1478533749,v1:172.21.15.154:6837/1478533749] up:boot 2024-05-20T17:56:30.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: fsmap 8 up:standby 2024-05-20T17:56:30.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T17:56:30.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:30 smithi204 ceph-mon[37076]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:31.430 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 17:56:31 smithi204 systemd[1]: Starting Ceph mds.i for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:56:31.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:31.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.564+0000 7fbe082d7640 1 -- 172.21.15.129:0/1653366237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe00102ab0 msgr2=0x7fbe001037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:31.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.564+0000 7fbe082d7640 1 --2- 172.21.15.129:0/1653366237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe00102ab0 0x7fbe001037b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009900 tx=0x7fbdfc02a410 comp rx=0 tx=0).stop 2024-05-20T17:56:31.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.564+0000 7fbe082d7640 1 -- 172.21.15.129:0/1653366237 shutdown_connections 2024-05-20T17:56:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.564+0000 7fbe082d7640 1 --2- 172.21.15.129:0/1653366237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe00103cf0 0x7fbe00104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.564+0000 7fbe082d7640 1 --2- 172.21.15.129:0/1653366237 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe001089f0 0x7fbe0010ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.564+0000 7fbe082d7640 1 --2- 172.21.15.129:0/1653366237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe00102ab0 0x7fbe001037b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.564+0000 7fbe082d7640 1 -- 172.21.15.129:0/1653366237 >> 172.21.15.129:0/1653366237 conn(0x7fbe000fdcc0 msgr2=0x7fbe00100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.564+0000 7fbe082d7640 1 -- 172.21.15.129:0/1653366237 shutdown_connections 2024-05-20T17:56:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.564+0000 7fbe082d7640 1 -- 172.21.15.129:0/1653366237 wait complete. 2024-05-20T17:56:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.564+0000 7fbe082d7640 1 Processor -- start 2024-05-20T17:56:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.564+0000 7fbe082d7640 1 -- start start 2024-05-20T17:56:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe082d7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe00102ab0 0x7fbe001992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe082d7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe00103cf0 0x7fbe00199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe082d7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe001089f0 0x7fbe001a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe082d7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0010e700 con 0x7fbe00103cf0 2024-05-20T17:56:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe082d7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0010e580 con 0x7fbe001089f0 2024-05-20T17:56:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe082d7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0010e880 con 0x7fbe00102ab0 2024-05-20T17:56:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe05872640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe00103cf0 0x7fbe00199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe05872640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe00103cf0 0x7fbe00199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39940/0 (socket says 172.21.15.129:39940) 2024-05-20T17:56:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe05872640 1 -- 172.21.15.129:0/1410046854 learned_addr learned my addr 172.21.15.129:0/1410046854 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe06874640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe001089f0 0x7fbe001a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe06073640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe00102ab0 0x7fbe001992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe05872640 1 -- 172.21.15.129:0/1410046854 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe00102ab0 msgr2=0x7fbe001992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe05872640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe00102ab0 0x7fbe001992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe05872640 1 -- 172.21.15.129:0/1410046854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe001089f0 msgr2=0x7fbe001a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe05872640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe001089f0 0x7fbe001a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe05872640 1 -- 172.21.15.129:0/1410046854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe001a0e10 con 0x7fbe00103cf0 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe05872640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe00103cf0 0x7fbe00199810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000e9f0 tx=0x7fbdf000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbdf77fe640 1 -- 172.21.15.129:0/1410046854 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf000cd80 con 0x7fbe00103cf0 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbdf77fe640 1 -- 172.21.15.129:0/1410046854 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdf0010910 con 0x7fbe00103cf0 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbdf77fe640 1 -- 172.21.15.129:0/1410046854 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0018990 con 0x7fbe00103cf0 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.565+0000 7fbe082d7640 1 -- 172.21.15.129:0/1410046854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe001a1100 con 0x7fbe00103cf0 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.566+0000 7fbe082d7640 1 -- 172.21.15.129:0/1410046854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe001a16a0 con 0x7fbe00103cf0 2024-05-20T17:56:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.566+0000 7fbe082d7640 1 -- 172.21.15.129:0/1410046854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe00107800 con 0x7fbe00103cf0 2024-05-20T17:56:31.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.568+0000 7fbdf77fe640 1 -- 172.21.15.129:0/1410046854 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbdf0010ac0 con 0x7fbe00103cf0 2024-05-20T17:56:31.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.569+0000 7fbdf77fe640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbdec0a2c10 0x7fbdec0a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:31.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.569+0000 7fbe06073640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbdec0a2c10 0x7fbdec0a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:31.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.569+0000 7fbdf77fe640 1 -- 172.21.15.129:0/1410046854 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbdf00cf320 con 0x7fbe00103cf0 2024-05-20T17:56:31.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.569+0000 7fbe06073640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbdec0a2c10 0x7fbdec0a5120 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009230 tx=0x7fbdfc031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.573+0000 7fbdf77fe640 1 -- 172.21.15.129:0/1410046854 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbdf009c700 con 0x7fbe00103cf0 2024-05-20T17:56:31.766 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 17:56:31 smithi204 podman[68046]: 2024-05-20 17:56:31.419754853 +0000 UTC m=+0.017361415 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:56:31.766 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 17:56:31 smithi204 podman[68046]: 2024-05-20 17:56:31.545263259 +0000 UTC m=+0.142869845 container create ab802f7dcb52ddf9da091ffff945a0e4c44a3a4c7a60abe4a635b16d9c9ea829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-20T17:56:31.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.792+0000 7fbe082d7640 1 -- 172.21.15.129:0/1410046854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe0002d2a0 con 0x7fbe00103cf0 2024-05-20T17:56:31.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.793+0000 7fbdf77fe640 1 -- 172.21.15.129:0/1410046854 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7189 (secure 0 0 0) 0x7fbdf00bc0c0 con 0x7fbe00103cf0 2024-05-20T17:56:31.792 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:31.792 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-20T17:56:31.792 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 9 2024-05-20T17:56:31.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 -- 172.21.15.129:0/1410046854 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbdec0a2c10 msgr2=0x7fbdec0a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:31.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbdec0a2c10 0x7fbdec0a5120 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009230 tx=0x7fbdfc031400 comp rx=0 tx=0).stop 2024-05-20T17:56:31.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 -- 172.21.15.129:0/1410046854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe00103cf0 msgr2=0x7fbe00199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:31.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe00103cf0 0x7fbe00199810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000e9f0 tx=0x7fbdf000eeb0 comp rx=0 tx=0).stop 2024-05-20T17:56:31.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 -- 172.21.15.129:0/1410046854 shutdown_connections 2024-05-20T17:56:31.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbdec0a2c10 0x7fbdec0a5120 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:31.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe001089f0 0x7fbe001a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:31.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe00103cf0 0x7fbe00199810 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:31.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 --2- 172.21.15.129:0/1410046854 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe00102ab0 0x7fbe001992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:31.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 -- 172.21.15.129:0/1410046854 >> 172.21.15.129:0/1410046854 conn(0x7fbe000fdcc0 msgr2=0x7fbe001049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:31.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 -- 172.21.15.129:0/1410046854 shutdown_connections 2024-05-20T17:56:31.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:31.795+0000 7fbe082d7640 1 -- 172.21.15.129:0/1410046854 wait complete. 2024-05-20T17:56:31.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:31 smithi154 ceph-mon[36482]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:32.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:31 smithi129 ceph-mon[32029]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:32.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:31 smithi204 ceph-mon[37076]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:32.180 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 17:56:31 smithi204 podman[68046]: 2024-05-20 17:56:31.803491536 +0000 UTC m=+0.401098078 container init ab802f7dcb52ddf9da091ffff945a0e4c44a3a4c7a60abe4a635b16d9c9ea829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-20T17:56:32.180 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 17:56:31 smithi204 podman[68046]: 2024-05-20 17:56:31.806704171 +0000 UTC m=+0.404310727 container start ab802f7dcb52ddf9da091ffff945a0e4c44a3a4c7a60abe4a635b16d9c9ea829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-20T17:56:32.180 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 17:56:31 smithi204 ceph-mds[68065]: -- 172.21.15.204:0/1724259649 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558ad72a3a40 con 0x558ad731ba80 2024-05-20T17:56:32.180 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 17:56:31 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: starting mds.i at 2024-05-20T17:56:32.180 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 17:56:31 smithi204 bash[68046]: ab802f7dcb52ddf9da091ffff945a0e4c44a3a4c7a60abe4a635b16d9c9ea829 2024-05-20T17:56:32.180 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 17:56:31 smithi204 systemd[1]: Started Ceph mds.i for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:32.909 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:32 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1410046854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:32.909 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:32.909 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:32.909 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:32.909 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:32.909 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:32.909 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:32 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:32.909 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:32 smithi129 ceph-mon[32029]: Deploying daemon mds.g on smithi129 2024-05-20T17:56:33.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:32 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1410046854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:32 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:33.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:32 smithi154 ceph-mon[36482]: Deploying daemon mds.g on smithi129 2024-05-20T17:56:33.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:32 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1410046854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:33.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:33.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:33.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:33.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:33.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:33.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:32 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:33.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:32 smithi204 ceph-mon[37076]: Deploying daemon mds.g on smithi129 2024-05-20T17:56:33.807 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:33.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:33 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.204:6838/2880425505,v1:172.21.15.204:6839/2880425505] up:boot 2024-05-20T17:56:33.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:33 smithi129 ceph-mon[32029]: fsmap 9 up:standby 2024-05-20T17:56:33.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:33 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T17:56:33.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:33 smithi129 ceph-mon[32029]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.028+0000 7fa749957640 1 -- 172.21.15.129:0/913398196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa744108720 msgr2=0x7fa74410ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.028+0000 7fa749957640 1 --2- 172.21.15.129:0/913398196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa744108720 0x7fa74410ac30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009ca0 tx=0x7fa73c02a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.028+0000 7fa749957640 1 -- 172.21.15.129:0/913398196 shutdown_connections 2024-05-20T17:56:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.028+0000 7fa749957640 1 --2- 172.21.15.129:0/913398196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744105bb0 0x7fa744108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.028+0000 7fa749957640 1 --2- 172.21.15.129:0/913398196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa744108720 0x7fa74410ac30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.028+0000 7fa749957640 1 --2- 172.21.15.129:0/913398196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa744100dd0 0x7fa7441011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:34.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.028+0000 7fa749957640 1 -- 172.21.15.129:0/913398196 >> 172.21.15.129:0/913398196 conn(0x7fa7440fc2e0 msgr2=0x7fa7440fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:34.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.028+0000 7fa749957640 1 -- 172.21.15.129:0/913398196 shutdown_connections 2024-05-20T17:56:34.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.028+0000 7fa749957640 1 -- 172.21.15.129:0/913398196 wait complete. 2024-05-20T17:56:34.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa749957640 1 Processor -- start 2024-05-20T17:56:34.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa749957640 1 -- start start 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa749957640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa744100dd0 0x7fa744199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa749957640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa744105bb0 0x7fa744199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa749957640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744108720 0x7fa7441a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa749957640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa74410e700 con 0x7fa744100dd0 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa749957640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa74410e580 con 0x7fa744108720 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa749957640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa74410e880 con 0x7fa744105bb0 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa742ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa744100dd0 0x7fa744199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa742ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa744100dd0 0x7fa744199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39962/0 (socket says 172.21.15.129:39962) 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa742ffd640 1 -- 172.21.15.129:0/3441272968 learned_addr learned my addr 172.21.15.129:0/3441272968 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.029+0000 7fa7427fc640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa744105bb0 0x7fa744199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.030+0000 7fa742ffd640 1 -- 172.21.15.129:0/3441272968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa744105bb0 msgr2=0x7fa744199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.030+0000 7fa742ffd640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa744105bb0 0x7fa744199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.030+0000 7fa742ffd640 1 -- 172.21.15.129:0/3441272968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744108720 msgr2=0x7fa7441a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:34.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.030+0000 7fa742ffd640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744108720 0x7fa7441a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:34.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.030+0000 7fa742ffd640 1 -- 172.21.15.129:0/3441272968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7441a0d80 con 0x7fa744100dd0 2024-05-20T17:56:34.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.030+0000 7fa742ffd640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa744100dd0 0x7fa744199240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa73800d740 tx=0x7fa73800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:34.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.030+0000 7fa748955640 1 -- 172.21.15.129:0/3441272968 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738017070 con 0x7fa744100dd0 2024-05-20T17:56:34.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.030+0000 7fa748955640 1 -- 172.21.15.129:0/3441272968 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa73800de50 con 0x7fa744100dd0 2024-05-20T17:56:34.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.030+0000 7fa749957640 1 -- 172.21.15.129:0/3441272968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7441a1070 con 0x7fa744100dd0 2024-05-20T17:56:34.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.030+0000 7fa748955640 1 -- 172.21.15.129:0/3441272968 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738004e00 con 0x7fa744100dd0 2024-05-20T17:56:34.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.030+0000 7fa749957640 1 -- 172.21.15.129:0/3441272968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7441a1570 con 0x7fa744100dd0 2024-05-20T17:56:34.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.031+0000 7fa749957640 1 -- 172.21.15.129:0/3441272968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7440fde00 con 0x7fa744100dd0 2024-05-20T17:56:34.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.033+0000 7fa748955640 1 -- 172.21.15.129:0/3441272968 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa738004fa0 con 0x7fa744100dd0 2024-05-20T17:56:34.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.033+0000 7fa748955640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa72c0a2b90 0x7fa72c0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:34.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.033+0000 7fa7427fc640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa72c0a2b90 0x7fa72c0a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:34.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.033+0000 7fa748955640 1 -- 172.21.15.129:0/3441272968 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa7380cea30 con 0x7fa744100dd0 2024-05-20T17:56:34.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.034+0000 7fa7427fc640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa72c0a2b90 0x7fa72c0a50a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa734005fd0 tx=0x7fa734005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:34.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.037+0000 7fa748955640 1 -- 172.21.15.129:0/3441272968 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa73809be30 con 0x7fa744100dd0 2024-05-20T17:56:34.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:33 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.204:6838/2880425505,v1:172.21.15.204:6839/2880425505] up:boot 2024-05-20T17:56:34.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:33 smithi154 ceph-mon[36482]: fsmap 9 up:standby 2024-05-20T17:56:34.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:33 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T17:56:34.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:33 smithi154 ceph-mon[36482]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:34.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:33 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.204:6838/2880425505,v1:172.21.15.204:6839/2880425505] up:boot 2024-05-20T17:56:34.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:33 smithi204 ceph-mon[37076]: fsmap 9 up:standby 2024-05-20T17:56:34.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:33 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T17:56:34.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:33 smithi204 ceph-mon[37076]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:34.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.262+0000 7fa749957640 1 -- 172.21.15.129:0/3441272968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa74402d3c0 con 0x7fa744100dd0 2024-05-20T17:56:34.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.264+0000 7fa748955640 1 -- 172.21.15.129:0/3441272968 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8019 (secure 0 0 0) 0x7fa73809b730 con 0x7fa744100dd0 2024-05-20T17:56:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:34.263 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-20T17:56:34.263 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 10 2024-05-20T17:56:34.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 -- 172.21.15.129:0/3441272968 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa72c0a2b90 msgr2=0x7fa72c0a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:34.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa72c0a2b90 0x7fa72c0a50a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa734005fd0 tx=0x7fa734005850 comp rx=0 tx=0).stop 2024-05-20T17:56:34.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 -- 172.21.15.129:0/3441272968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa744100dd0 msgr2=0x7fa744199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:34.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa744100dd0 0x7fa744199240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa73800d740 tx=0x7fa73800dc00 comp rx=0 tx=0).stop 2024-05-20T17:56:34.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 -- 172.21.15.129:0/3441272968 shutdown_connections 2024-05-20T17:56:34.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa72c0a2b90 0x7fa72c0a50a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:34.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744108720 0x7fa7441a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:34.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa744105bb0 0x7fa744199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:34.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 --2- 172.21.15.129:0/3441272968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa744100dd0 0x7fa744199240 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:34.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 -- 172.21.15.129:0/3441272968 >> 172.21.15.129:0/3441272968 conn(0x7fa7440fc2e0 msgr2=0x7fa744106b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:34.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 -- 172.21.15.129:0/3441272968 shutdown_connections 2024-05-20T17:56:34.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:34.266+0000 7fa749957640 1 -- 172.21.15.129:0/3441272968 wait complete. 2024-05-20T17:56:34.277 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 17:56:34 smithi129 systemd[1]: Starting Ceph mds.g for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:56:34.528 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 17:56:34 smithi129 podman[71261]: 2024-05-20 17:56:34.281967959 +0000 UTC m=+0.019291794 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:56:34.528 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 17:56:34 smithi129 podman[71261]: 2024-05-20 17:56:34.403094363 +0000 UTC m=+0.140418205 container create ebdb95f64391801f02a02f3058e98fc409644346dd130852f8f208f28fa9656e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T17:56:34.785 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 17:56:34 smithi129 podman[71261]: 2024-05-20 17:56:34.672814915 +0000 UTC m=+0.410138745 container init ebdb95f64391801f02a02f3058e98fc409644346dd130852f8f208f28fa9656e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-g, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=) 2024-05-20T17:56:34.785 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 17:56:34 smithi129 podman[71261]: 2024-05-20 17:56:34.675749018 +0000 UTC m=+0.413072848 container start ebdb95f64391801f02a02f3058e98fc409644346dd130852f8f208f28fa9656e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-20T17:56:34.785 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 17:56:34 smithi129 ceph-mds[71282]: -- 172.21.15.129:0/227588538 <== mon.0 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc9aed1a40 con 0x55dc9af49180 2024-05-20T17:56:34.785 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 17:56:34 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-g[71278]: starting mds.g at 2024-05-20T17:56:34.785 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 17:56:34 smithi129 bash[71261]: ebdb95f64391801f02a02f3058e98fc409644346dd130852f8f208f28fa9656e 2024-05-20T17:56:34.785 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 17:56:34 smithi129 systemd[1]: Started Ceph mds.g for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:35.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:34 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3441272968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:35.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:34 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3441272968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:35.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:34 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3441272968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:36.012 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:35 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.129:6838/2206181560,v1:172.21.15.129:6839/2206181560] up:boot 2024-05-20T17:56:36.012 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:35 smithi129 ceph-mon[32029]: fsmap 10 up:standby 2024-05-20T17:56:36.013 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-20T17:56:36.013 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:36.013 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:36.013 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:36.013 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:36.013 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:36.013 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:35 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:36.013 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:35 smithi129 ceph-mon[32029]: Deploying daemon mds.a on smithi129 2024-05-20T17:56:36.013 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:35 smithi129 ceph-mon[32029]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:36.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:35 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.129:6838/2206181560,v1:172.21.15.129:6839/2206181560] up:boot 2024-05-20T17:56:36.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:35 smithi154 ceph-mon[36482]: fsmap 10 up:standby 2024-05-20T17:56:36.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-20T17:56:36.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:36.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:36.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:36.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:36.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:36.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:35 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:36.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:35 smithi154 ceph-mon[36482]: Deploying daemon mds.a on smithi129 2024-05-20T17:56:36.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:35 smithi154 ceph-mon[36482]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:36.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:35 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.129:6838/2206181560,v1:172.21.15.129:6839/2206181560] up:boot 2024-05-20T17:56:36.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:35 smithi204 ceph-mon[37076]: fsmap 10 up:standby 2024-05-20T17:56:36.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-20T17:56:36.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:36.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:36.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:36.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:36.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:36.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:35 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:36.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:35 smithi204 ceph-mon[37076]: Deploying daemon mds.a on smithi129 2024-05-20T17:56:36.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:35 smithi204 ceph-mon[37076]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:36.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:36.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.408+0000 7f7e2953e640 1 -- 172.21.15.129:0/868055140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e24102a80 msgr2=0x7f7e24103780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:36.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.408+0000 7f7e2953e640 1 --2- 172.21.15.129:0/868055140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e24102a80 0x7f7e24103780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009900 tx=0x7f7e1402a410 comp rx=0 tx=0).stop 2024-05-20T17:56:36.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.409+0000 7f7e2953e640 1 -- 172.21.15.129:0/868055140 shutdown_connections 2024-05-20T17:56:36.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.409+0000 7f7e2953e640 1 --2- 172.21.15.129:0/868055140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e241089c0 0x7f7e2410ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:36.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.409+0000 7f7e2953e640 1 --2- 172.21.15.129:0/868055140 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e24103cc0 0x7f7e241040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:36.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.409+0000 7f7e2953e640 1 --2- 172.21.15.129:0/868055140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e24102a80 0x7f7e24103780 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:36.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.409+0000 7f7e2953e640 1 -- 172.21.15.129:0/868055140 >> 172.21.15.129:0/868055140 conn(0x7f7e240fdc70 msgr2=0x7f7e24100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:36.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.409+0000 7f7e2953e640 1 -- 172.21.15.129:0/868055140 shutdown_connections 2024-05-20T17:56:36.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.409+0000 7f7e2953e640 1 -- 172.21.15.129:0/868055140 wait complete. 2024-05-20T17:56:36.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.410+0000 7f7e2953e640 1 Processor -- start 2024-05-20T17:56:36.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.411+0000 7f7e2953e640 1 -- start start 2024-05-20T17:56:36.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.411+0000 7f7e2953e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e24102a80 0x7f7e241990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:36.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.411+0000 7f7e2953e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e24103cc0 0x7f7e24199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:36.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.411+0000 7f7e2953e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e241089c0 0x7f7e241a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:36.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.411+0000 7f7e2953e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2410e720 con 0x7f7e24103cc0 2024-05-20T17:56:36.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.412+0000 7f7e2953e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2410e5a0 con 0x7f7e24102a80 2024-05-20T17:56:36.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.412+0000 7f7e237fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e241089c0 0x7f7e241a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:36.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.412+0000 7f7e237fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e241089c0 0x7f7e241a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:37936/0 (socket says 172.21.15.129:37936) 2024-05-20T17:56:36.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.412+0000 7f7e237fe640 1 -- 172.21.15.129:0/4122922698 learned_addr learned my addr 172.21.15.129:0/4122922698 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:36.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.412+0000 7f7e2953e640 1 -- 172.21.15.129:0/4122922698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2410e8a0 con 0x7f7e241089c0 2024-05-20T17:56:36.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.414+0000 7f7e227fc640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e24103cc0 0x7f7e24199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:36.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.414+0000 7f7e237fe640 1 -- 172.21.15.129:0/4122922698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e24102a80 msgr2=0x7f7e241990d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:56:36.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.414+0000 7f7e237fe640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e24102a80 0x7f7e241990d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:36.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.414+0000 7f7e237fe640 1 -- 172.21.15.129:0/4122922698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e24103cc0 msgr2=0x7f7e24199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:36.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.414+0000 7f7e237fe640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e24103cc0 0x7f7e24199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:36.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.414+0000 7f7e237fe640 1 -- 172.21.15.129:0/4122922698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e241a0c10 con 0x7f7e241089c0 2024-05-20T17:56:36.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.415+0000 7f7e227fc640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e24103cc0 0x7f7e24199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:56:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.418+0000 7f7e237fe640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e241089c0 0x7f7e241a05a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00eaa0 tx=0x7f7e1c00ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.418+0000 7f7e0bfff640 1 -- 172.21.15.129:0/4122922698 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c00cdb0 con 0x7f7e241089c0 2024-05-20T17:56:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.418+0000 7f7e2953e640 1 -- 172.21.15.129:0/4122922698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e241a0f00 con 0x7f7e241089c0 2024-05-20T17:56:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.419+0000 7f7e2953e640 1 -- 172.21.15.129:0/4122922698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e241a1360 con 0x7f7e241089c0 2024-05-20T17:56:36.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.419+0000 7f7e0bfff640 1 -- 172.21.15.129:0/4122922698 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e1c005020 con 0x7f7e241089c0 2024-05-20T17:56:36.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.419+0000 7f7e0bfff640 1 -- 172.21.15.129:0/4122922698 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c0116b0 con 0x7f7e241089c0 2024-05-20T17:56:36.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.421+0000 7f7e2953e640 1 -- 172.21.15.129:0/4122922698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e10005290 con 0x7f7e241089c0 2024-05-20T17:56:36.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.424+0000 7f7e0bfff640 1 -- 172.21.15.129:0/4122922698 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7e1c004a40 con 0x7f7e241089c0 2024-05-20T17:56:36.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.424+0000 7f7e0bfff640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7e0c0a2c10 0x7f7e0c0a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:36.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.424+0000 7f7e0bfff640 1 -- 172.21.15.129:0/4122922698 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7e1c0cedf0 con 0x7f7e241089c0 2024-05-20T17:56:36.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.425+0000 7f7e22ffd640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7e0c0a2c10 0x7f7e0c0a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.425+0000 7f7e22ffd640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7e0c0a2c10 0x7f7e0c0a5120 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009230 tx=0x7f7e14000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:36.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.429+0000 7f7e0bfff640 1 -- 172.21.15.129:0/4122922698 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7e1c016030 con 0x7f7e241089c0 2024-05-20T17:56:36.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.666+0000 7f7e2953e640 1 -- 172.21.15.129:0/4122922698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e10005850 con 0x7f7e241089c0 2024-05-20T17:56:36.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.667+0000 7f7e0bfff640 1 -- 172.21.15.129:0/4122922698 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8848 (secure 0 0 0) 0x7f7e1c0ce020 con 0x7f7e241089c0 2024-05-20T17:56:36.668 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:36.668 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-20T17:56:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 11 2024-05-20T17:56:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 -- 172.21.15.129:0/4122922698 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7e0c0a2c10 msgr2=0x7f7e0c0a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:36.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7e0c0a2c10 0x7f7e0c0a5120 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009230 tx=0x7f7e14000bc0 comp rx=0 tx=0).stop 2024-05-20T17:56:36.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 -- 172.21.15.129:0/4122922698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e241089c0 msgr2=0x7f7e241a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:36.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e241089c0 0x7f7e241a05a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00eaa0 tx=0x7f7e1c00ef60 comp rx=0 tx=0).stop 2024-05-20T17:56:36.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 -- 172.21.15.129:0/4122922698 shutdown_connections 2024-05-20T17:56:36.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7e0c0a2c10 0x7f7e0c0a5120 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e241089c0 0x7f7e241a05a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e24103cc0 0x7f7e24199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 --2- 172.21.15.129:0/4122922698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e24102a80 0x7f7e241990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 -- 172.21.15.129:0/4122922698 >> 172.21.15.129:0/4122922698 conn(0x7f7e240fdc70 msgr2=0x7f7e24100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 -- 172.21.15.129:0/4122922698 shutdown_connections 2024-05-20T17:56:36.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:36.670+0000 7f7e2953e640 1 -- 172.21.15.129:0/4122922698 wait complete. 2024-05-20T17:56:36.846 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 17:56:36 smithi129 systemd[1]: Starting Ceph mds.a for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:56:37.124 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:36 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4122922698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:37.124 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 17:56:36 smithi129 podman[71614]: 2024-05-20 17:56:36.895150251 +0000 UTC m=+0.017240849 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:56:37.124 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 17:56:37 smithi129 podman[71614]: 2024-05-20 17:56:37.018777529 +0000 UTC m=+0.140868118 container create 9e62bdb20c4361caec038a550b211c705c20265a496cbafef53d4bd7ccea1ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_BRANCH=HEAD) 2024-05-20T17:56:37.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:36 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/4122922698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:37.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:36 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/4122922698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:37.412 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 17:56:37 smithi129 podman[71614]: 2024-05-20 17:56:37.271860057 +0000 UTC m=+0.393950662 container init 9e62bdb20c4361caec038a550b211c705c20265a496cbafef53d4bd7ccea1ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-20T17:56:37.412 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 17:56:37 smithi129 podman[71614]: 2024-05-20 17:56:37.275555799 +0000 UTC m=+0.397646397 container start 9e62bdb20c4361caec038a550b211c705c20265a496cbafef53d4bd7ccea1ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-a, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-20T17:56:37.413 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 17:56:37 smithi129 ceph-mds[71635]: -- 172.21.15.129:0/845624506 <== mon.0 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556063fb3a40 con 0x55606402ba80 2024-05-20T17:56:37.413 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 17:56:37 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-a[71631]: starting mds.a at 2024-05-20T17:56:37.413 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 17:56:37 smithi129 bash[71614]: 9e62bdb20c4361caec038a550b211c705c20265a496cbafef53d4bd7ccea1ee1 2024-05-20T17:56:37.413 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 17:56:37 smithi129 systemd[1]: Started Ceph mds.a for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:38.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:38.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:38.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:38.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:38.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:38.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:38.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:38 smithi154 ceph-mon[36482]: Deploying daemon mds.b on smithi154 2024-05-20T17:56:38.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:38 smithi154 ceph-mon[36482]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:38.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:38 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.129:6840/1623696468,v1:172.21.15.129:6841/1623696468] up:boot 2024-05-20T17:56:38.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:38 smithi154 ceph-mon[36482]: fsmap 11 up:standby 2024-05-20T17:56:38.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:38 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-20T17:56:38.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:38.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:38.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:38.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:38.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:38.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:38.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:38 smithi129 ceph-mon[32029]: Deploying daemon mds.b on smithi154 2024-05-20T17:56:38.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:38 smithi129 ceph-mon[32029]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:38.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:38 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.129:6840/1623696468,v1:172.21.15.129:6841/1623696468] up:boot 2024-05-20T17:56:38.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:38 smithi129 ceph-mon[32029]: fsmap 11 up:standby 2024-05-20T17:56:38.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:38 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-20T17:56:38.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:38.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:38.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:38.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-20T17:56:38.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-20T17:56:38.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:38.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:38 smithi204 ceph-mon[37076]: Deploying daemon mds.b on smithi154 2024-05-20T17:56:38.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:38 smithi204 ceph-mon[37076]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:38.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:38 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.129:6840/1623696468,v1:172.21.15.129:6841/1623696468] up:boot 2024-05-20T17:56:38.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:38 smithi204 ceph-mon[37076]: fsmap 11 up:standby 2024-05-20T17:56:38.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:38 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-20T17:56:38.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:38.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.811+0000 7f2c9e27a640 1 -- 172.21.15.129:0/4035867814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c98102900 msgr2=0x7f2c98102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:38.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.811+0000 7f2c9e27a640 1 --2- 172.21.15.129:0/4035867814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c98102900 0x7f2c98102d90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c80009900 tx=0x7f2c8002a410 comp rx=0 tx=0).stop 2024-05-20T17:56:38.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.811+0000 7f2c9e27a640 1 -- 172.21.15.129:0/4035867814 shutdown_connections 2024-05-20T17:56:38.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.811+0000 7f2c9e27a640 1 --2- 172.21.15.129:0/4035867814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c98103420 0x7f2c98109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:38.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.811+0000 7f2c9e27a640 1 --2- 172.21.15.129:0/4035867814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c98102900 0x7f2c98102d90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.811+0000 7f2c9e27a640 1 --2- 172.21.15.129:0/4035867814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c98101f20 0x7f2c98102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.811+0000 7f2c9e27a640 1 -- 172.21.15.129:0/4035867814 >> 172.21.15.129:0/4035867814 conn(0x7f2c980fc320 msgr2=0x7f2c980fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.811+0000 7f2c9e27a640 1 -- 172.21.15.129:0/4035867814 shutdown_connections 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.811+0000 7f2c9e27a640 1 -- 172.21.15.129:0/4035867814 wait complete. 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c9e27a640 1 Processor -- start 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c9e27a640 1 -- start start 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c9e27a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c98101f20 0x7f2c98199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c9e27a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c98102900 0x7f2c981997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c9e27a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c98103420 0x7f2c981a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c9e27a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9810d640 con 0x7f2c98102900 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c9e27a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9810d4c0 con 0x7f2c98103420 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c9e27a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9810d7c0 con 0x7f2c98101f20 2024-05-20T17:56:38.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c8ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c98102900 0x7f2c981997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c8ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c98102900 0x7f2c981997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40054/0 (socket says 172.21.15.129:40054) 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c8ffff640 1 -- 172.21.15.129:0/1813226275 learned_addr learned my addr 172.21.15.129:0/1813226275 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c9c817640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c98103420 0x7f2c981a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c97fff640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c98101f20 0x7f2c98199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.812+0000 7f2c8ffff640 1 -- 172.21.15.129:0/1813226275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c98101f20 msgr2=0x7f2c98199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.813+0000 7f2c8ffff640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c98101f20 0x7f2c98199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.813+0000 7f2c8ffff640 1 -- 172.21.15.129:0/1813226275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c98103420 msgr2=0x7f2c981a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.813+0000 7f2c8ffff640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c98103420 0x7f2c981a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.813+0000 7f2c8ffff640 1 -- 172.21.15.129:0/1813226275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c981a0db0 con 0x7f2c98102900 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.813+0000 7f2c8ffff640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c98102900 0x7f2c981997b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2c80006010 tx=0x7f2c800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.813+0000 7f2c95ffb640 1 -- 172.21.15.129:0/1813226275 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8003f030 con 0x7f2c98102900 2024-05-20T17:56:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.813+0000 7f2c95ffb640 1 -- 172.21.15.129:0/1813226275 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c80003ce0 con 0x7f2c98102900 2024-05-20T17:56:38.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.813+0000 7f2c9e27a640 1 -- 172.21.15.129:0/1813226275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c981a1040 con 0x7f2c98102900 2024-05-20T17:56:38.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.813+0000 7f2c95ffb640 1 -- 172.21.15.129:0/1813226275 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c80035070 con 0x7f2c98102900 2024-05-20T17:56:38.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.813+0000 7f2c9e27a640 1 -- 172.21.15.129:0/1813226275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c981a1580 con 0x7f2c98102900 2024-05-20T17:56:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.814+0000 7f2c9e27a640 1 -- 172.21.15.129:0/1813226275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c981a17e0 con 0x7f2c98102900 2024-05-20T17:56:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.816+0000 7f2c95ffb640 1 -- 172.21.15.129:0/1813226275 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f2c80042030 con 0x7f2c98102900 2024-05-20T17:56:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.817+0000 7f2c95ffb640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2c840a2c10 0x7f2c840a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.817+0000 7f2c97fff640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2c840a2c10 0x7f2c840a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.817+0000 7f2c95ffb640 1 -- 172.21.15.129:0/1813226275 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2c800ef670 con 0x7f2c98102900 2024-05-20T17:56:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.817+0000 7f2c97fff640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2c840a2c10 0x7f2c840a5120 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2c9806a2b0 tx=0x7f2c88005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:38.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:38.820+0000 7f2c95ffb640 1 -- 172.21.15.129:0/1813226275 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2c800bca70 con 0x7f2c98102900 2024-05-20T17:56:39.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.036+0000 7f2c9e27a640 1 -- 172.21.15.129:0/1813226275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c981a19b0 con 0x7f2c98102900 2024-05-20T17:56:39.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.036+0000 7f2c95ffb640 1 -- 172.21.15.129:0/1813226275 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9677 (secure 0 0 0) 0x7f2c800bc370 con 0x7f2c98102900 2024-05-20T17:56:39.035 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:39.036 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-20T17:56:39.036 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 12 2024-05-20T17:56:39.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.038+0000 7f2c9e27a640 1 -- 172.21.15.129:0/1813226275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2c840a2c10 msgr2=0x7f2c840a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:39.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.038+0000 7f2c9e27a640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2c840a2c10 0x7f2c840a5120 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2c9806a2b0 tx=0x7f2c88005f90 comp rx=0 tx=0).stop 2024-05-20T17:56:39.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.038+0000 7f2c9e27a640 1 -- 172.21.15.129:0/1813226275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c98102900 msgr2=0x7f2c981997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:39.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.038+0000 7f2c9e27a640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c98102900 0x7f2c981997b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2c80006010 tx=0x7f2c800042f0 comp rx=0 tx=0).stop 2024-05-20T17:56:39.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.038+0000 7f2c9e27a640 1 -- 172.21.15.129:0/1813226275 shutdown_connections 2024-05-20T17:56:39.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.038+0000 7f2c9e27a640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2c840a2c10 0x7f2c840a5120 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:39.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.038+0000 7f2c9e27a640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c98103420 0x7f2c981a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:39.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.038+0000 7f2c9e27a640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2c98102900 0x7f2c981997b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:39.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.038+0000 7f2c9e27a640 1 --2- 172.21.15.129:0/1813226275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c98101f20 0x7f2c98199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:39.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.038+0000 7f2c9e27a640 1 -- 172.21.15.129:0/1813226275 >> 172.21.15.129:0/1813226275 conn(0x7f2c980fc320 msgr2=0x7f2c98108c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:39.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.039+0000 7f2c9e27a640 1 -- 172.21.15.129:0/1813226275 shutdown_connections 2024-05-20T17:56:39.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:39.039+0000 7f2c9e27a640 1 -- 172.21.15.129:0/1813226275 wait complete. 2024-05-20T17:56:39.864 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:39 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1813226275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:39.865 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 17:56:39 smithi154 systemd[1]: Starting Ceph mds.b for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:56:39.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:39 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1813226275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:39.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:39 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1813226275' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:40.116 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 17:56:39 smithi154 podman[66677]: 2024-05-20 17:56:39.768955045 +0000 UTC m=+0.016878895 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:56:40.116 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 17:56:39 smithi154 podman[66677]: 2024-05-20 17:56:39.873834953 +0000 UTC m=+0.121758792 container create 7b06ee3220d54cf332ab1907d16b6c8ce4329782bef29749f5a49c3f2e654797 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, ceph=True) 2024-05-20T17:56:40.116 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 17:56:40 smithi154 podman[66677]: 2024-05-20 17:56:40.118436709 +0000 UTC m=+0.366360543 container init 7b06ee3220d54cf332ab1907d16b6c8ce4329782bef29749f5a49c3f2e654797 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-20T17:56:40.393 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 17:56:40 smithi154 podman[66677]: 2024-05-20 17:56:40.121085244 +0000 UTC m=+0.369009078 container start 7b06ee3220d54cf332ab1907d16b6c8ce4329782bef29749f5a49c3f2e654797 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T17:56:40.393 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 17:56:40 smithi154 ceph-mds[66696]: -- 172.21.15.154:0/3030271207 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56069de57a40 con 0x56069decfa80 2024-05-20T17:56:40.393 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 17:56:40 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b[66692]: starting mds.b at 2024-05-20T17:56:40.393 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 17:56:40 smithi154 bash[66677]: 7b06ee3220d54cf332ab1907d16b6c8ce4329782bef29749f5a49c3f2e654797 2024-05-20T17:56:40.394 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 17:56:40 smithi154 systemd[1]: Started Ceph mds.b for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:56:40.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:40 smithi154 ceph-mon[36482]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:40.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:40 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:40 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:40 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:40 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:40 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:40 smithi129 ceph-mon[32029]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:40.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:40 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:40 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:40 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:40 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:40 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:40 smithi204 ceph-mon[37076]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:40.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:40 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:40 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:40 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:40 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:40.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:40 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:41.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:41.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.169+0000 7f7b55bff640 1 -- 172.21.15.129:0/3791969820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b501006e0 msgr2=0x7f7b50100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.169+0000 7f7b55bff640 1 --2- 172.21.15.129:0/3791969820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b501006e0 0x7f7b50100b50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7b44009900 tx=0x7f7b4402a410 comp rx=0 tx=0).stop 2024-05-20T17:56:41.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.169+0000 7f7b55bff640 1 -- 172.21.15.129:0/3791969820 shutdown_connections 2024-05-20T17:56:41.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.169+0000 7f7b55bff640 1 --2- 172.21.15.129:0/3791969820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b501011e0 0x7f7b5010ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.169+0000 7f7b55bff640 1 --2- 172.21.15.129:0/3791969820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b501006e0 0x7f7b50100b50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.169+0000 7f7b55bff640 1 --2- 172.21.15.129:0/3791969820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b50104030 0x7f7b50104440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.169+0000 7f7b55bff640 1 -- 172.21.15.129:0/3791969820 >> 172.21.15.129:0/3791969820 conn(0x7f7b500fc2e0 msgr2=0x7f7b500fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:41.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.169+0000 7f7b55bff640 1 -- 172.21.15.129:0/3791969820 shutdown_connections 2024-05-20T17:56:41.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.169+0000 7f7b55bff640 1 -- 172.21.15.129:0/3791969820 wait complete. 2024-05-20T17:56:41.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b55bff640 1 Processor -- start 2024-05-20T17:56:41.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b55bff640 1 -- start start 2024-05-20T17:56:41.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b55bff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b501006e0 0x7f7b501992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b55bff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b501011e0 0x7f7b501997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b55bff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b50104030 0x7f7b501a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b55bff640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5010e6f0 con 0x7f7b501011e0 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b55bff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5010e570 con 0x7f7b501006e0 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b55bff640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5010e870 con 0x7f7b50104030 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b4effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b501011e0 0x7f7b501997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b4effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b501011e0 0x7f7b501997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53500/0 (socket says 172.21.15.129:53500) 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b4effd640 1 -- 172.21.15.129:0/1133180495 learned_addr learned my addr 172.21.15.129:0/1133180495 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b4f7fe640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b501006e0 0x7f7b501992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.170+0000 7f7b4ffff640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b50104030 0x7f7b501a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.171+0000 7f7b4effd640 1 -- 172.21.15.129:0/1133180495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b50104030 msgr2=0x7f7b501a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.171+0000 7f7b4effd640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b50104030 0x7f7b501a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.171+0000 7f7b4effd640 1 -- 172.21.15.129:0/1133180495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b501006e0 msgr2=0x7f7b501992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.171+0000 7f7b4effd640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b501006e0 0x7f7b501992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.171+0000 7f7b4effd640 1 -- 172.21.15.129:0/1133180495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b501a0de0 con 0x7f7b501011e0 2024-05-20T17:56:41.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.171+0000 7f7b4effd640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b501011e0 0x7f7b501997e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7b44006010 tx=0x7f7b44004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:41.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.171+0000 7f7b4cff9640 1 -- 172.21.15.129:0/1133180495 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4403f030 con 0x7f7b501011e0 2024-05-20T17:56:41.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.171+0000 7f7b55bff640 1 -- 172.21.15.129:0/1133180495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b501a1070 con 0x7f7b501011e0 2024-05-20T17:56:41.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.171+0000 7f7b4cff9640 1 -- 172.21.15.129:0/1133180495 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b44003ce0 con 0x7f7b501011e0 2024-05-20T17:56:41.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.171+0000 7f7b4cff9640 1 -- 172.21.15.129:0/1133180495 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b44035070 con 0x7f7b501011e0 2024-05-20T17:56:41.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.171+0000 7f7b55bff640 1 -- 172.21.15.129:0/1133180495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b501a1560 con 0x7f7b501011e0 2024-05-20T17:56:41.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.172+0000 7f7b55bff640 1 -- 172.21.15.129:0/1133180495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b501079c0 con 0x7f7b501011e0 2024-05-20T17:56:41.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.174+0000 7f7b4cff9640 1 -- 172.21.15.129:0/1133180495 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7b44042030 con 0x7f7b501011e0 2024-05-20T17:56:41.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.174+0000 7f7b4cff9640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7b380a2b90 0x7f7b380a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.174+0000 7f7b4f7fe640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7b380a2b90 0x7f7b380a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.174+0000 7f7b4cff9640 1 -- 172.21.15.129:0/1133180495 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7b440ef700 con 0x7f7b501011e0 2024-05-20T17:56:41.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.175+0000 7f7b4f7fe640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7b380a2b90 0x7f7b380a50a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7b40005fd0 tx=0x7f7b4000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:41.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.178+0000 7f7b4cff9640 1 -- 172.21.15.129:0/1133180495 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7b440bc4b0 con 0x7f7b501011e0 2024-05-20T17:56:41.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.399+0000 7f7b55bff640 1 -- 172.21.15.129:0/1133180495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b5002d2a0 con 0x7f7b501011e0 2024-05-20T17:56:41.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.399+0000 7f7b4cff9640 1 -- 172.21.15.129:0/1133180495 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10503 (secure 0 0 0) 0x7f7b440c07b0 con 0x7f7b501011e0 2024-05-20T17:56:41.398 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:41.399 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24560,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-20T17:56:41.399 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 13 2024-05-20T17:56:41.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 -- 172.21.15.129:0/1133180495 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7b380a2b90 msgr2=0x7f7b380a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7b380a2b90 0x7f7b380a50a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7b40005fd0 tx=0x7f7b4000b410 comp rx=0 tx=0).stop 2024-05-20T17:56:41.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 -- 172.21.15.129:0/1133180495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b501011e0 msgr2=0x7f7b501997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b501011e0 0x7f7b501997e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7b44006010 tx=0x7f7b44004360 comp rx=0 tx=0).stop 2024-05-20T17:56:41.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 -- 172.21.15.129:0/1133180495 shutdown_connections 2024-05-20T17:56:41.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f7b380a2b90 0x7f7b380a50a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b50104030 0x7f7b501a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b501011e0 0x7f7b501997e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 --2- 172.21.15.129:0/1133180495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b501006e0 0x7f7b501992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 -- 172.21.15.129:0/1133180495 >> 172.21.15.129:0/1133180495 conn(0x7f7b500fc2e0 msgr2=0x7f7b50104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:41.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 -- 172.21.15.129:0/1133180495 shutdown_connections 2024-05-20T17:56:41.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.402+0000 7f7b55bff640 1 -- 172.21.15.129:0/1133180495 wait complete. 2024-05-20T17:56:41.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:56:41.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.529+0000 7fbf4ac84640 1 -- 172.21.15.129:0/182686073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf44102350 msgr2=0x7fbf44102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.529+0000 7fbf4ac84640 1 --2- 172.21.15.129:0/182686073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf44102350 0x7fbf44102760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009900 tx=0x7fbf3002a410 comp rx=0 tx=0).stop 2024-05-20T17:56:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.529+0000 7fbf4ac84640 1 -- 172.21.15.129:0/182686073 shutdown_connections 2024-05-20T17:56:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.529+0000 7fbf4ac84640 1 --2- 172.21.15.129:0/182686073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf44107050 0x7fbf44109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.529+0000 7fbf4ac84640 1 --2- 172.21.15.129:0/182686073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf44102350 0x7fbf44102760 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.529+0000 7fbf4ac84640 1 --2- 172.21.15.129:0/182686073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf44101110 0x7fbf44101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.529+0000 7fbf4ac84640 1 -- 172.21.15.129:0/182686073 >> 172.21.15.129:0/182686073 conn(0x7fbf440fc2e0 msgr2=0x7fbf440fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.530+0000 7fbf4ac84640 1 -- 172.21.15.129:0/182686073 shutdown_connections 2024-05-20T17:56:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.530+0000 7fbf4ac84640 1 -- 172.21.15.129:0/182686073 wait complete. 2024-05-20T17:56:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.530+0000 7fbf4ac84640 1 Processor -- start 2024-05-20T17:56:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.530+0000 7fbf4ac84640 1 -- start start 2024-05-20T17:56:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.530+0000 7fbf4ac84640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf44101110 0x7fbf4410a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.530+0000 7fbf4ac84640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf44102350 0x7fbf4419e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.530+0000 7fbf4ac84640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf44107050 0x7fbf441a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.530+0000 7fbf4ac84640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4410d610 con 0x7fbf44107050 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.530+0000 7fbf4ac84640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4410d490 con 0x7fbf44101110 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.530+0000 7fbf4ac84640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4410d790 con 0x7fbf44102350 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.530+0000 7fbf49221640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf44107050 0x7fbf441a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf43fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf44102350 0x7fbf4419e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf49221640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf44107050 0x7fbf441a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53520/0 (socket says 172.21.15.129:53520) 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf49221640 1 -- 172.21.15.129:0/3255753645 learned_addr learned my addr 172.21.15.129:0/3255753645 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf48a20640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf44101110 0x7fbf4410a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf49221640 1 -- 172.21.15.129:0/3255753645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf44102350 msgr2=0x7fbf4419e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf49221640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf44102350 0x7fbf4419e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf49221640 1 -- 172.21.15.129:0/3255753645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf44101110 msgr2=0x7fbf4410a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf49221640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf44101110 0x7fbf4410a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf49221640 1 -- 172.21.15.129:0/3255753645 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf441a0da0 con 0x7fbf44107050 2024-05-20T17:56:41.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf43fff640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf44102350 0x7fbf4419e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:56:41.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf49221640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf44107050 0x7fbf441a0580 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009fd0 tx=0x7fbf3c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:41.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf41ffb640 1 -- 172.21.15.129:0/3255753645 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c017070 con 0x7fbf44107050 2024-05-20T17:56:41.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf41ffb640 1 -- 172.21.15.129:0/3255753645 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf3c003e50 con 0x7fbf44107050 2024-05-20T17:56:41.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf4ac84640 1 -- 172.21.15.129:0/3255753645 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf441a1090 con 0x7fbf44107050 2024-05-20T17:56:41.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf41ffb640 1 -- 172.21.15.129:0/3255753645 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c0106c0 con 0x7fbf44107050 2024-05-20T17:56:41.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.531+0000 7fbf4ac84640 1 -- 172.21.15.129:0/3255753645 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf441a1590 con 0x7fbf44107050 2024-05-20T17:56:41.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.533+0000 7fbf4ac84640 1 -- 172.21.15.129:0/3255753645 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf44105e60 con 0x7fbf44107050 2024-05-20T17:56:41.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.534+0000 7fbf41ffb640 1 -- 172.21.15.129:0/3255753645 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbf3c003970 con 0x7fbf44107050 2024-05-20T17:56:41.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.534+0000 7fbf41ffb640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbf340a2b90 0x7fbf340a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.534+0000 7fbf48a20640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbf340a2b90 0x7fbf340a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.534+0000 7fbf41ffb640 1 -- 172.21.15.129:0/3255753645 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbf3c0ce8b0 con 0x7fbf44107050 2024-05-20T17:56:41.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.535+0000 7fbf48a20640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbf340a2b90 0x7fbf340a50a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbf44101d10 tx=0x7fbf38005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:41.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.539+0000 7fbf41ffb640 1 -- 172.21.15.129:0/3255753645 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbf3c09b660 con 0x7fbf44107050 2024-05-20T17:56:41.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:41 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] up:boot 2024-05-20T17:56:41.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:41 smithi129 ceph-mon[32029]: fsmap 12 up:standby 2024-05-20T17:56:41.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:41 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-20T17:56:41.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:41 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:56:41.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:41 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1133180495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:41.666 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:41 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] up:boot 2024-05-20T17:56:41.666 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:41 smithi154 ceph-mon[36482]: fsmap 12 up:standby 2024-05-20T17:56:41.666 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:41 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-20T17:56:41.666 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:41 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:56:41.667 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:41 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1133180495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:41.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:41 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] up:boot 2024-05-20T17:56:41.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:41 smithi204 ceph-mon[37076]: fsmap 12 up:standby 2024-05-20T17:56:41.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:41 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-20T17:56:41.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:41 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:56:41.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:41 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1133180495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:41.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.723+0000 7fbf4ac84640 1 -- 172.21.15.129:0/3255753645 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf4402d4c0 con 0x7fbf44107050 2024-05-20T17:56:41.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.724+0000 7fbf41ffb640 1 -- 172.21.15.129:0/3255753645 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18477 (secure 0 0 0) 0x7fbf3c09f960 con 0x7fbf44107050 2024-05-20T17:56:41.723 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:41.723 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":82,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:55:57.775106+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:56:41.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 -- 172.21.15.129:0/3255753645 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbf340a2b90 msgr2=0x7fbf340a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbf340a2b90 0x7fbf340a50a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbf44101d10 tx=0x7fbf38005f90 comp rx=0 tx=0).stop 2024-05-20T17:56:41.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 -- 172.21.15.129:0/3255753645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf44107050 msgr2=0x7fbf441a0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf44107050 0x7fbf441a0580 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009fd0 tx=0x7fbf3c00b260 comp rx=0 tx=0).stop 2024-05-20T17:56:41.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 -- 172.21.15.129:0/3255753645 shutdown_connections 2024-05-20T17:56:41.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbf340a2b90 0x7fbf340a50a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf44107050 0x7fbf441a0580 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf44102350 0x7fbf4419e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 --2- 172.21.15.129:0/3255753645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf44101110 0x7fbf4410a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 -- 172.21.15.129:0/3255753645 >> 172.21.15.129:0/3255753645 conn(0x7fbf440fc2e0 msgr2=0x7fbf44103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:41.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 -- 172.21.15.129:0/3255753645 shutdown_connections 2024-05-20T17:56:41.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.726+0000 7fbf4ac84640 1 -- 172.21.15.129:0/3255753645 wait complete. 2024-05-20T17:56:41.737 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T17:56:41.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-20T17:56:41.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.858+0000 7f73cb557640 1 -- 172.21.15.129:0/2132570747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c4101120 msgr2=0x7f73c4101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.858+0000 7f73cb557640 1 --2- 172.21.15.129:0/2132570747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c4101120 0x7f73c4101e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009900 tx=0x7f73c002a410 comp rx=0 tx=0).stop 2024-05-20T17:56:41.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.858+0000 7f73cb557640 1 -- 172.21.15.129:0/2132570747 shutdown_connections 2024-05-20T17:56:41.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.858+0000 7f73cb557640 1 --2- 172.21.15.129:0/2132570747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73c4107060 0x7f73c4109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.858+0000 7f73cb557640 1 --2- 172.21.15.129:0/2132570747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73c4102360 0x7f73c4102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.858+0000 7f73cb557640 1 --2- 172.21.15.129:0/2132570747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c4101120 0x7f73c4101e20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.858+0000 7f73cb557640 1 -- 172.21.15.129:0/2132570747 >> 172.21.15.129:0/2132570747 conn(0x7f73c40fc2f0 msgr2=0x7f73c40fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:41.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.858+0000 7f73cb557640 1 -- 172.21.15.129:0/2132570747 shutdown_connections 2024-05-20T17:56:41.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.858+0000 7f73cb557640 1 -- 172.21.15.129:0/2132570747 wait complete. 2024-05-20T17:56:41.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.858+0000 7f73cb557640 1 Processor -- start 2024-05-20T17:56:41.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.858+0000 7f73cb557640 1 -- start start 2024-05-20T17:56:41.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73cb557640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c4101120 0x7f73c41992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73cb557640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73c4102360 0x7f73c4199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73cb557640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73c4107060 0x7f73c41a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73cb557640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f73c410c500 con 0x7f73c4101120 2024-05-20T17:56:41.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73cb557640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73c410c380 con 0x7f73c4102360 2024-05-20T17:56:41.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73cb557640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f73c410c680 con 0x7f73c4107060 2024-05-20T17:56:41.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73c92f3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c4101120 0x7f73c41992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73c92f3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c4101120 0x7f73c41992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53542/0 (socket says 172.21.15.129:53542) 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73c92f3640 1 -- 172.21.15.129:0/1744107159 learned_addr learned my addr 172.21.15.129:0/1744107159 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73c8af2640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73c4102360 0x7f73c4199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73c9af4640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73c4107060 0x7f73c41a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73c92f3640 1 -- 172.21.15.129:0/1744107159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73c4107060 msgr2=0x7f73c41a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73c92f3640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73c4107060 0x7f73c41a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73c92f3640 1 -- 172.21.15.129:0/1744107159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73c4102360 msgr2=0x7f73c4199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73c92f3640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73c4102360 0x7f73c4199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73c92f3640 1 -- 172.21.15.129:0/1744107159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c41a0e30 con 0x7f73c4101120 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73c92f3640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c4101120 0x7f73c41992f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f73c0005ab0 tx=0x7f73c0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73ba7fc640 1 -- 172.21.15.129:0/1744107159 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c003f030 con 0x7f73c4101120 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73ba7fc640 1 -- 172.21.15.129:0/1744107159 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73c0003ad0 con 0x7f73c4101120 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73cb557640 1 -- 172.21.15.129:0/1744107159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c41a10c0 con 0x7f73c4101120 2024-05-20T17:56:41.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.859+0000 7f73ba7fc640 1 -- 172.21.15.129:0/1744107159 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0035070 con 0x7f73c4101120 2024-05-20T17:56:41.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.860+0000 7f73cb557640 1 -- 172.21.15.129:0/1744107159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c41a1600 con 0x7f73c4101120 2024-05-20T17:56:41.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.861+0000 7f73cb557640 1 -- 172.21.15.129:0/1744107159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73c4105e70 con 0x7f73c4101120 2024-05-20T17:56:41.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.862+0000 7f73ba7fc640 1 -- 172.21.15.129:0/1744107159 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f73c0042030 con 0x7f73c4101120 2024-05-20T17:56:41.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.863+0000 7f73ba7fc640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f73b00a2c10 0x7f73b00a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:41.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.863+0000 7f73ba7fc640 1 -- 172.21.15.129:0/1744107159 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f73c00ef180 con 0x7f73c4101120 2024-05-20T17:56:41.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.863+0000 7f73c8af2640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f73b00a2c10 0x7f73b00a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:41.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.863+0000 7f73c8af2640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f73b00a2c10 0x7f73b00a5120 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f73c419a390 tx=0x7f73b4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:41.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:41.868+0000 7f73ba7fc640 1 -- 172.21.15.129:0/1744107159 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f73c00bc580 con 0x7f73c4101120 2024-05-20T17:56:42.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.073+0000 7f73cb557640 1 -- 172.21.15.129:0/1744107159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f73c402d2a0 con 0x7f73c4101120 2024-05-20T17:56:42.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.074+0000 7f73ba7fc640 1 -- 172.21.15.129:0/1744107159 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f73c00bbe80 con 0x7f73c4101120 2024-05-20T17:56:42.072 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T17:56:42.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 -- 172.21.15.129:0/1744107159 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f73b00a2c10 msgr2=0x7f73b00a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:42.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f73b00a2c10 0x7f73b00a5120 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f73c419a390 tx=0x7f73b4005f90 comp rx=0 tx=0).stop 2024-05-20T17:56:42.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 -- 172.21.15.129:0/1744107159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c4101120 msgr2=0x7f73c41992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:42.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c4101120 0x7f73c41992f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f73c0005ab0 tx=0x7f73c0004270 comp rx=0 tx=0).stop 2024-05-20T17:56:42.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 -- 172.21.15.129:0/1744107159 shutdown_connections 2024-05-20T17:56:42.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f73b00a2c10 0x7f73b00a5120 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73c4102360 0x7f73c4199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73c4107060 0x7f73c41a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 --2- 172.21.15.129:0/1744107159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73c4101120 0x7f73c41992f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 -- 172.21.15.129:0/1744107159 >> 172.21.15.129:0/1744107159 conn(0x7f73c40fc2f0 msgr2=0x7f73c4103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:42.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 -- 172.21.15.129:0/1744107159 shutdown_connections 2024-05-20T17:56:42.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.076+0000 7f73cb557640 1 -- 172.21.15.129:0/1744107159 wait complete. 2024-05-20T17:56:42.087 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-20T17:56:42.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.214+0000 7faaae7aa640 1 -- 172.21.15.129:0/3785044702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa8105ee0 msgr2=0x7faaa8108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:42.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.214+0000 7faaae7aa640 1 --2- 172.21.15.129:0/3785044702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa8105ee0 0x7faaa8108300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009900 tx=0x7faa9c02a410 comp rx=0 tx=0).stop 2024-05-20T17:56:42.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.214+0000 7faaae7aa640 1 -- 172.21.15.129:0/3785044702 shutdown_connections 2024-05-20T17:56:42.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.214+0000 7faaae7aa640 1 --2- 172.21.15.129:0/3785044702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faaa8108840 0x7faaa810ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.214+0000 7faaae7aa640 1 --2- 172.21.15.129:0/3785044702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa8105ee0 0x7faaa8108300 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.214+0000 7faaae7aa640 1 --2- 172.21.15.129:0/3785044702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faaa8103580 0x7faaa81059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.214+0000 7faaae7aa640 1 -- 172.21.15.129:0/3785044702 >> 172.21.15.129:0/3785044702 conn(0x7faaa80fcb80 msgr2=0x7faaa80ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:42.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.214+0000 7faaae7aa640 1 -- 172.21.15.129:0/3785044702 shutdown_connections 2024-05-20T17:56:42.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.214+0000 7faaae7aa640 1 -- 172.21.15.129:0/3785044702 wait complete. 2024-05-20T17:56:42.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.215+0000 7faaae7aa640 1 Processor -- start 2024-05-20T17:56:42.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.215+0000 7faaae7aa640 1 -- start start 2024-05-20T17:56:42.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.215+0000 7faaae7aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa8103580 0x7faaa819f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:42.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.215+0000 7faaae7aa640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faaa8105ee0 0x7faaa819fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:42.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.215+0000 7faaae7aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faaa8108840 0x7faaa81a6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:42.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.215+0000 7faaae7aa640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faaa8114d40 con 0x7faaa8103580 2024-05-20T17:56:42.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.215+0000 7faaae7aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faaa8114bc0 con 0x7faaa8108840 2024-05-20T17:56:42.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.215+0000 7faaae7aa640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faaa8114ec0 con 0x7faaa8105ee0 2024-05-20T17:56:42.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa8103580 0x7faaa819f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:42.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa8103580 0x7faaa819f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53558/0 (socket says 172.21.15.129:53558) 2024-05-20T17:56:42.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa77fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faaa8105ee0 0x7faaa819fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:42.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa7fff640 1 -- 172.21.15.129:0/3242806964 learned_addr learned my addr 172.21.15.129:0/3242806964 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:42.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaacd47640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faaa8108840 0x7faaa81a6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:42.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa7fff640 1 -- 172.21.15.129:0/3242806964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faaa8105ee0 msgr2=0x7faaa819fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:42.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa7fff640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faaa8105ee0 0x7faaa819fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa7fff640 1 -- 172.21.15.129:0/3242806964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faaa8108840 msgr2=0x7faaa81a6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:42.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa7fff640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faaa8108840 0x7faaa81a6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa7fff640 1 -- 172.21.15.129:0/3242806964 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa81a7450 con 0x7faaa8103580 2024-05-20T17:56:42.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa77fe640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faaa8105ee0 0x7faaa819fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:56:42.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa7fff640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa8103580 0x7faaa819f910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7faa9800d740 tx=0x7faa9800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:42.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa57fa640 1 -- 172.21.15.129:0/3242806964 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa98017070 con 0x7faaa8103580 2024-05-20T17:56:42.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaae7aa640 1 -- 172.21.15.129:0/3242806964 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaa81a7740 con 0x7faaa8103580 2024-05-20T17:56:42.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa57fa640 1 -- 172.21.15.129:0/3242806964 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa9800de50 con 0x7faaa8103580 2024-05-20T17:56:42.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaa57fa640 1 -- 172.21.15.129:0/3242806964 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa98004e00 con 0x7faaa8103580 2024-05-20T17:56:42.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.216+0000 7faaae7aa640 1 -- 172.21.15.129:0/3242806964 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaa81a7c90 con 0x7faaa8103580 2024-05-20T17:56:42.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.217+0000 7faaae7aa640 1 -- 172.21.15.129:0/3242806964 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaa80fe6a0 con 0x7faaa8103580 2024-05-20T17:56:42.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.219+0000 7faaa57fa640 1 -- 172.21.15.129:0/3242806964 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7faa98004fa0 con 0x7faaa8103580 2024-05-20T17:56:42.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.219+0000 7faaa57fa640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faa900a2b80 0x7faa900a5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:42.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.220+0000 7faaa77fe640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faa900a2b80 0x7faa900a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:42.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.220+0000 7faaa57fa640 1 -- 172.21.15.129:0/3242806964 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7faa980ce8d0 con 0x7faaa8103580 2024-05-20T17:56:42.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.220+0000 7faaa77fe640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faa900a2b80 0x7faa900a5090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009230 tx=0x7faa9c0315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:42.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.223+0000 7faaa57fa640 1 -- 172.21.15.129:0/3242806964 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7faa9809b680 con 0x7faaa8103580 2024-05-20T17:56:42.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.407+0000 7faaae7aa640 1 -- 172.21.15.129:0/3242806964 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7faaa802d470 con 0x7faaa8103580 2024-05-20T17:56:42.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.407+0000 7faaa57fa640 1 -- 172.21.15.129:0/3242806964 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7faa9809f980 con 0x7faaa8103580 2024-05-20T17:56:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:42.407 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-05-20T17:56:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T17:56:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-05-20T17:56:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-05-20T17:56:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T17:56:42.411 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-05-20T17:56:42.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.409+0000 7faaae7aa640 1 -- 172.21.15.129:0/3242806964 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faa900a2b80 msgr2=0x7faa900a5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:42.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.409+0000 7faaae7aa640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faa900a2b80 0x7faa900a5090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009230 tx=0x7faa9c0315b0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.409+0000 7faaae7aa640 1 -- 172.21.15.129:0/3242806964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa8103580 msgr2=0x7faaa819f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:42.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.409+0000 7faaae7aa640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa8103580 0x7faaa819f910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7faa9800d740 tx=0x7faa9800dc00 comp rx=0 tx=0).stop 2024-05-20T17:56:42.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.409+0000 7faaae7aa640 1 -- 172.21.15.129:0/3242806964 shutdown_connections 2024-05-20T17:56:42.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.409+0000 7faaae7aa640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faa900a2b80 0x7faa900a5090 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.409+0000 7faaae7aa640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faaa8108840 0x7faaa81a6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.409+0000 7faaae7aa640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faaa8105ee0 0x7faaa819fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.409+0000 7faaae7aa640 1 --2- 172.21.15.129:0/3242806964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faaa8103580 0x7faaa819f910 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.410+0000 7faaae7aa640 1 -- 172.21.15.129:0/3242806964 >> 172.21.15.129:0/3242806964 conn(0x7faaa80fcb80 msgr2=0x7faaa8104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:42.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.410+0000 7faaae7aa640 1 -- 172.21.15.129:0/3242806964 shutdown_connections 2024-05-20T17:56:42.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.410+0000 7faaae7aa640 1 -- 172.21.15.129:0/3242806964 wait complete. 2024-05-20T17:56:42.420 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-20T17:56:42.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-20T17:56:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.537+0000 7fd79674c640 1 -- 172.21.15.129:0/2400411934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7901089e0 msgr2=0x7fd79010ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.537+0000 7fd79674c640 1 --2- 172.21.15.129:0/2400411934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7901089e0 0x7fd79010ae00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd788009ca0 tx=0x7fd78802a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.537+0000 7fd79674c640 1 -- 172.21.15.129:0/2400411934 shutdown_connections 2024-05-20T17:56:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.537+0000 7fd79674c640 1 --2- 172.21.15.129:0/2400411934 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7901089e0 0x7fd79010ae00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.537+0000 7fd79674c640 1 --2- 172.21.15.129:0/2400411934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd790103ce0 0x7fd7901040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.537+0000 7fd79674c640 1 --2- 172.21.15.129:0/2400411934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd790102aa0 0x7fd7901037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.537+0000 7fd79674c640 1 -- 172.21.15.129:0/2400411934 >> 172.21.15.129:0/2400411934 conn(0x7fd7900fdc90 msgr2=0x7fd790100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.537+0000 7fd79674c640 1 -- 172.21.15.129:0/2400411934 shutdown_connections 2024-05-20T17:56:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.537+0000 7fd79674c640 1 -- 172.21.15.129:0/2400411934 wait complete. 2024-05-20T17:56:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd79674c640 1 Processor -- start 2024-05-20T17:56:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd79674c640 1 -- start start 2024-05-20T17:56:42.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd79674c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd790102aa0 0x7fd790199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:42.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd79674c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd790103ce0 0x7fd7901997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd79674c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7901089e0 0x7fd7901a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd79674c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd79010e770 con 0x7fd7901089e0 2024-05-20T17:56:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd79674c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd79010e5f0 con 0x7fd790103ce0 2024-05-20T17:56:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd79674c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd79010e8f0 con 0x7fd790102aa0 2024-05-20T17:56:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd794ce9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7901089e0 0x7fd7901a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd794ce9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7901089e0 0x7fd7901a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53570/0 (socket says 172.21.15.129:53570) 2024-05-20T17:56:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd794ce9640 1 -- 172.21.15.129:0/2748053638 learned_addr learned my addr 172.21.15.129:0/2748053638 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd78ffff640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd790102aa0 0x7fd790199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd78f7fe640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd790103ce0 0x7fd7901997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd794ce9640 1 -- 172.21.15.129:0/2748053638 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd790102aa0 msgr2=0x7fd790199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd794ce9640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd790102aa0 0x7fd790199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd794ce9640 1 -- 172.21.15.129:0/2748053638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd790103ce0 msgr2=0x7fd7901997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd794ce9640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd790103ce0 0x7fd7901997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.538+0000 7fd794ce9640 1 -- 172.21.15.129:0/2748053638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7901a0dd0 con 0x7fd7901089e0 2024-05-20T17:56:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.539+0000 7fd794ce9640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7901089e0 0x7fd7901a0760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd7880319f0 tx=0x7fd78802f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.539+0000 7fd78d7fa640 1 -- 172.21.15.129:0/2748053638 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd788003bf0 con 0x7fd7901089e0 2024-05-20T17:56:42.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.539+0000 7fd78d7fa640 1 -- 172.21.15.129:0/2748053638 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd788003d90 con 0x7fd7901089e0 2024-05-20T17:56:42.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.539+0000 7fd78d7fa640 1 -- 172.21.15.129:0/2748053638 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78802fd00 con 0x7fd7901089e0 2024-05-20T17:56:42.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.539+0000 7fd79674c640 1 -- 172.21.15.129:0/2748053638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7901a1060 con 0x7fd7901089e0 2024-05-20T17:56:42.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.539+0000 7fd79674c640 1 -- 172.21.15.129:0/2748053638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7901a1600 con 0x7fd7901089e0 2024-05-20T17:56:42.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.540+0000 7fd79674c640 1 -- 172.21.15.129:0/2748053638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7901077f0 con 0x7fd7901089e0 2024-05-20T17:56:42.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.541+0000 7fd78d7fa640 1 -- 172.21.15.129:0/2748053638 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd788004040 con 0x7fd7901089e0 2024-05-20T17:56:42.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.542+0000 7fd78d7fa640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd77c0a2c10 0x7fd77c0a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:42.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.542+0000 7fd78ffff640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd77c0a2c10 0x7fd77c0a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:42.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.542+0000 7fd78d7fa640 1 -- 172.21.15.129:0/2748053638 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd7880f0300 con 0x7fd7901089e0 2024-05-20T17:56:42.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.542+0000 7fd78ffff640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd77c0a2c10 0x7fd77c0a5120 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd7901036a0 tx=0x7fd78000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:42.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.546+0000 7fd78d7fa640 1 -- 172.21.15.129:0/2748053638 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd7880bd700 con 0x7fd7901089e0 2024-05-20T17:56:42.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:42.729+0000 7fd79674c640 1 -- 172.21.15.129:0/2748053638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd79002d2a0 con 0x7fd7901089e0 2024-05-20T17:56:42.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:42 smithi154 ceph-mon[36482]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:42.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:42 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3255753645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:42.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:42 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1744107159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:56:42.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:42 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.894 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:42 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3242806964' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-20T17:56:42.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:42 smithi129 ceph-mon[32029]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:42.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:42 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3255753645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:42.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:42 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1744107159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:56:42.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:42 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:42 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3242806964' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-20T17:56:42.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:42 smithi204 ceph-mon[37076]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:42 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3255753645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:42 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1744107159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:56:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:42 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:42.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:42 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3242806964' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-20T17:56:43.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.695+0000 7fd78d7fa640 1 -- 172.21.15.129:0/2748053638 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7fd7880dd0c0 con 0x7fd7901089e0 2024-05-20T17:56:43.695 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-05-20T17:56:43.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.698+0000 7fd79674c640 1 -- 172.21.15.129:0/2748053638 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd77c0a2c10 msgr2=0x7fd77c0a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.698+0000 7fd79674c640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd77c0a2c10 0x7fd77c0a5120 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd7901036a0 tx=0x7fd78000a2d0 comp rx=0 tx=0).stop 2024-05-20T17:56:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.698+0000 7fd79674c640 1 -- 172.21.15.129:0/2748053638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7901089e0 msgr2=0x7fd7901a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.698+0000 7fd79674c640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7901089e0 0x7fd7901a0760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd7880319f0 tx=0x7fd78802f380 comp rx=0 tx=0).stop 2024-05-20T17:56:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.699+0000 7fd79674c640 1 -- 172.21.15.129:0/2748053638 shutdown_connections 2024-05-20T17:56:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.699+0000 7fd79674c640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd77c0a2c10 0x7fd77c0a5120 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.699+0000 7fd79674c640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd790103ce0 0x7fd7901997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.699+0000 7fd79674c640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7901089e0 0x7fd7901a0760 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.699+0000 7fd79674c640 1 --2- 172.21.15.129:0/2748053638 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd790102aa0 0x7fd790199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.699+0000 7fd79674c640 1 -- 172.21.15.129:0/2748053638 >> 172.21.15.129:0/2748053638 conn(0x7fd7900fdc90 msgr2=0x7fd7901049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.699+0000 7fd79674c640 1 -- 172.21.15.129:0/2748053638 shutdown_connections 2024-05-20T17:56:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.699+0000 7fd79674c640 1 -- 172.21.15.129:0/2748053638 wait complete. 2024-05-20T17:56:43.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-20T17:56:43.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.839+0000 7f569ee6c640 1 -- 172.21.15.129:0/3993485793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5698103400 msgr2=0x7f5698109b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:43.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.839+0000 7f569ee6c640 1 --2- 172.21.15.129:0/3993485793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5698103400 0x7f5698109b40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5690009ca0 tx=0x7f569002a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:43.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.839+0000 7f569ee6c640 1 -- 172.21.15.129:0/3993485793 shutdown_connections 2024-05-20T17:56:43.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.839+0000 7f569ee6c640 1 --2- 172.21.15.129:0/3993485793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56981028c0 0x7f5698102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:43.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.839+0000 7f569ee6c640 1 --2- 172.21.15.129:0/3993485793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5698103400 0x7f5698109b40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:43.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.839+0000 7f569ee6c640 1 --2- 172.21.15.129:0/3993485793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5698101ee0 0x7f56981022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:43.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.839+0000 7f569ee6c640 1 -- 172.21.15.129:0/3993485793 >> 172.21.15.129:0/3993485793 conn(0x7f56980fc2e0 msgr2=0x7f56980fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:43.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.839+0000 7f569ee6c640 1 -- 172.21.15.129:0/3993485793 shutdown_connections 2024-05-20T17:56:43.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.839+0000 7f569ee6c640 1 -- 172.21.15.129:0/3993485793 wait complete. 2024-05-20T17:56:43.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569ee6c640 1 Processor -- start 2024-05-20T17:56:43.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569ee6c640 1 -- start start 2024-05-20T17:56:43.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569ee6c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5698101ee0 0x7f5698101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:43.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569ee6c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56981028c0 0x7f569819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:43.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569ee6c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5698103400 0x7f56981a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:43.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569ee6c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f569810d620 con 0x7f5698101ee0 2024-05-20T17:56:43.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569ee6c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f569810d4a0 con 0x7f5698103400 2024-05-20T17:56:43.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569ee6c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f569810d7a0 con 0x7f56981028c0 2024-05-20T17:56:43.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569cc08640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5698101ee0 0x7f5698101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:43.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569cc08640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5698101ee0 0x7f5698101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53580/0 (socket says 172.21.15.129:53580) 2024-05-20T17:56:43.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569cc08640 1 -- 172.21.15.129:0/2076558567 learned_addr learned my addr 172.21.15.129:0/2076558567 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f5697fff640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56981028c0 0x7f569819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569d409640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5698103400 0x7f56981a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569cc08640 1 -- 172.21.15.129:0/2076558567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56981028c0 msgr2=0x7f569819dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569cc08640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56981028c0 0x7f569819dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569cc08640 1 -- 172.21.15.129:0/2076558567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5698103400 msgr2=0x7f56981a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569cc08640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5698103400 0x7f56981a0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569cc08640 1 -- 172.21.15.129:0/2076558567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56981a0d50 con 0x7f5698101ee0 2024-05-20T17:56:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.840+0000 7f569d409640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5698103400 0x7f56981a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:56:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.841+0000 7f569cc08640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5698101ee0 0x7f5698101180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f568c00c970 tx=0x7f568c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.841+0000 7f5695ffb640 1 -- 172.21.15.129:0/2076558567 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c004220 con 0x7f5698101ee0 2024-05-20T17:56:43.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.841+0000 7f569ee6c640 1 -- 172.21.15.129:0/2076558567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56981a0fe0 con 0x7f5698101ee0 2024-05-20T17:56:43.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.841+0000 7f5695ffb640 1 -- 172.21.15.129:0/2076558567 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f568c0043c0 con 0x7f5698101ee0 2024-05-20T17:56:43.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.841+0000 7f5695ffb640 1 -- 172.21.15.129:0/2076558567 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c004a00 con 0x7f5698101ee0 2024-05-20T17:56:43.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.841+0000 7f569ee6c640 1 -- 172.21.15.129:0/2076558567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56981a14a0 con 0x7f5698101ee0 2024-05-20T17:56:43.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.842+0000 7f569ee6c640 1 -- 172.21.15.129:0/2076558567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5698108890 con 0x7f5698101ee0 2024-05-20T17:56:43.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.843+0000 7f5695ffb640 1 -- 172.21.15.129:0/2076558567 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f568c0079b0 con 0x7f5698101ee0 2024-05-20T17:56:43.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.844+0000 7f5695ffb640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f56880a2b90 0x7f56880a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:43.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.844+0000 7f5697fff640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f56880a2b90 0x7f56880a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:43.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.844+0000 7f5695ffb640 1 -- 172.21.15.129:0/2076558567 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7f568c0cee60 con 0x7f5698101ee0 2024-05-20T17:56:43.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.845+0000 7f5697fff640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f56880a2b90 0x7f56880a50a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5684005fd0 tx=0x7f5684005e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:43.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:43.848+0000 7f5695ffb640 1 -- 172.21.15.129:0/2076558567 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f568c014030 con 0x7f5698101ee0 2024-05-20T17:56:43.963 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:43 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:43.964 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:43 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:43.964 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:43 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:56:43.964 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:43 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2748053638' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-20T17:56:43.964 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:43 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:43.964 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:43 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:56:43.989 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:43 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:43.990 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:43 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:43.990 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:43 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:56:43.990 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:43 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2748053638' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-20T17:56:43.990 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:43 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:43.990 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:43 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:56:43.994 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:43 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:43.994 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:43 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:43.994 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:43 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:56:43.994 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:43 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2748053638' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-20T17:56:43.995 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:43 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:43.995 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:43 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:56:44.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.032+0000 7f569ee6c640 1 -- 172.21.15.129:0/2076558567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f569802d2a0 con 0x7f5698101ee0 2024-05-20T17:56:44.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.698+0000 7f5695ffb640 1 -- 172.21.15.129:0/2076558567 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) ==== 170+0+0 (secure 0 0 0) 0x7f568c09c0f0 con 0x7f5698101ee0 2024-05-20T17:56:44.697 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-05-20T17:56:44.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 -- 172.21.15.129:0/2076558567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f56880a2b90 msgr2=0x7f56880a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:44.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f56880a2b90 0x7f56880a50a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5684005fd0 tx=0x7f5684005e40 comp rx=0 tx=0).stop 2024-05-20T17:56:44.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 -- 172.21.15.129:0/2076558567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5698101ee0 msgr2=0x7f5698101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:44.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5698101ee0 0x7f5698101180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f568c00c970 tx=0x7f568c00ce30 comp rx=0 tx=0).stop 2024-05-20T17:56:44.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 -- 172.21.15.129:0/2076558567 shutdown_connections 2024-05-20T17:56:44.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f56880a2b90 0x7f56880a50a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:44.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5698103400 0x7f56981a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:44.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f56981028c0 0x7f569819dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:44.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 --2- 172.21.15.129:0/2076558567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5698101ee0 0x7f5698101180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:44.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 -- 172.21.15.129:0/2076558567 >> 172.21.15.129:0/2076558567 conn(0x7f56980fc2e0 msgr2=0x7f5698105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:44.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 -- 172.21.15.129:0/2076558567 shutdown_connections 2024-05-20T17:56:44.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.700+0000 7f569ee6c640 1 -- 172.21.15.129:0/2076558567 wait complete. 2024-05-20T17:56:44.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-20T17:56:44.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.858+0000 7f4201196640 1 -- 172.21.15.129:0/787847526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41fc108730 msgr2=0x7f41fc10ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.858+0000 7f4201196640 1 --2- 172.21.15.129:0/787847526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41fc108730 0x7f41fc10ac40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009ca0 tx=0x7f41f402a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.858+0000 7f4201196640 1 -- 172.21.15.129:0/787847526 shutdown_connections 2024-05-20T17:56:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.858+0000 7f4201196640 1 --2- 172.21.15.129:0/787847526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41fc105bc0 0x7f41fc1080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.858+0000 7f4201196640 1 --2- 172.21.15.129:0/787847526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41fc108730 0x7f41fc10ac40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.858+0000 7f4201196640 1 --2- 172.21.15.129:0/787847526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41fc100de0 0x7f41fc1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.858+0000 7f4201196640 1 -- 172.21.15.129:0/787847526 >> 172.21.15.129:0/787847526 conn(0x7f41fc0fc2f0 msgr2=0x7f41fc0fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.858+0000 7f4201196640 1 -- 172.21.15.129:0/787847526 shutdown_connections 2024-05-20T17:56:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.858+0000 7f4201196640 1 -- 172.21.15.129:0/787847526 wait complete. 2024-05-20T17:56:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.859+0000 7f4201196640 1 Processor -- start 2024-05-20T17:56:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.859+0000 7f4201196640 1 -- start start 2024-05-20T17:56:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f4201196640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41fc100de0 0x7f41fc199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f4201196640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41fc105bc0 0x7f41fc199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f4201196640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41fc108730 0x7f41fc1a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f4201196640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc10e740 con 0x7f41fc105bc0 2024-05-20T17:56:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f4201196640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc10e5c0 con 0x7f41fc100de0 2024-05-20T17:56:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f4201196640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc10e8c0 con 0x7f41fc108730 2024-05-20T17:56:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fa59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41fc105bc0 0x7f41fc199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fa59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41fc105bc0 0x7f41fc199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53592/0 (socket says 172.21.15.129:53592) 2024-05-20T17:56:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fa59c640 1 -- 172.21.15.129:0/3946168182 learned_addr learned my addr 172.21.15.129:0/3946168182 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fad9d640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41fc100de0 0x7f41fc199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fb59e640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41fc108730 0x7f41fc1a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fa59c640 1 -- 172.21.15.129:0/3946168182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41fc108730 msgr2=0x7f41fc1a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fa59c640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41fc108730 0x7f41fc1a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fa59c640 1 -- 172.21.15.129:0/3946168182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41fc100de0 msgr2=0x7f41fc199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fa59c640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41fc100de0 0x7f41fc199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fa59c640 1 -- 172.21.15.129:0/3946168182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41fc1a0d50 con 0x7f41fc105bc0 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fb59e640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41fc108730 0x7f41fc1a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.860+0000 7f41fad9d640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41fc100de0 0x7f41fc199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.861+0000 7f41fa59c640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41fc105bc0 0x7f41fc199750 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f41f000e9f0 tx=0x7f41f000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.861+0000 7f41e3fff640 1 -- 172.21.15.129:0/3946168182 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f000cd80 con 0x7f41fc105bc0 2024-05-20T17:56:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.861+0000 7f41e3fff640 1 -- 172.21.15.129:0/3946168182 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f0010910 con 0x7f41fc105bc0 2024-05-20T17:56:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.861+0000 7f4201196640 1 -- 172.21.15.129:0/3946168182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41fc1a0fe0 con 0x7f41fc105bc0 2024-05-20T17:56:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.861+0000 7f41e3fff640 1 -- 172.21.15.129:0/3946168182 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f0018990 con 0x7f41fc105bc0 2024-05-20T17:56:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.861+0000 7f4201196640 1 -- 172.21.15.129:0/3946168182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41fc1a1520 con 0x7f41fc105bc0 2024-05-20T17:56:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.862+0000 7f4201196640 1 -- 172.21.15.129:0/3946168182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41fc0fde10 con 0x7f41fc105bc0 2024-05-20T17:56:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.864+0000 7f41e3fff640 1 -- 172.21.15.129:0/3946168182 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f41f0010ac0 con 0x7f41fc105bc0 2024-05-20T17:56:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.865+0000 7f41e3fff640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41e40a2c10 0x7f41e40a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.865+0000 7f41fad9d640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41e40a2c10 0x7f41e40a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.865+0000 7f41e3fff640 1 -- 172.21.15.129:0/3946168182 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f41f00cf520 con 0x7f41fc105bc0 2024-05-20T17:56:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.865+0000 7f41fad9d640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41e40a2c10 0x7f41e40a5120 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f41ec0095f0 tx=0x7f41ec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:44.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:44.869+0000 7f41e3fff640 1 -- 172.21.15.129:0/3946168182 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f41f009bfe0 con 0x7f41fc105bc0 2024-05-20T17:56:45.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.094+0000 7f4201196640 1 -- 172.21.15.129:0/3946168182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f41fc02d2a0 con 0x7f41fc105bc0 2024-05-20T17:56:45.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:44 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2748053638' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-20T17:56:45.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:44 smithi154 ceph-mon[36482]: osdmap e83: 12 total, 12 up, 12 in 2024-05-20T17:56:45.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:44 smithi154 ceph-mon[36482]: pgmap v281: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:45.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:44 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2076558567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-20T17:56:45.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:44 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2748053638' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-20T17:56:45.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:44 smithi129 ceph-mon[32029]: osdmap e83: 12 total, 12 up, 12 in 2024-05-20T17:56:45.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:44 smithi129 ceph-mon[32029]: pgmap v281: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:45.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:44 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2076558567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-20T17:56:45.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:44 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2748053638' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-20T17:56:45.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:44 smithi204 ceph-mon[37076]: osdmap e83: 12 total, 12 up, 12 in 2024-05-20T17:56:45.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:44 smithi204 ceph-mon[37076]: pgmap v281: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:45.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:44 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2076558567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-20T17:56:45.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.957+0000 7f41e3fff640 1 -- 172.21.15.129:0/3946168182 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f41f00a02e0 con 0x7f41fc105bc0 2024-05-20T17:56:45.956 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 2 and data pool 3 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 -- 172.21.15.129:0/3946168182 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41e40a2c10 msgr2=0x7f41e40a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41e40a2c10 0x7f41e40a5120 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f41ec0095f0 tx=0x7f41ec005f90 comp rx=0 tx=0).stop 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 -- 172.21.15.129:0/3946168182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41fc105bc0 msgr2=0x7f41fc199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41fc105bc0 0x7f41fc199750 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f41f000e9f0 tx=0x7f41f000eeb0 comp rx=0 tx=0).stop 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 -- 172.21.15.129:0/3946168182 shutdown_connections 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f41e40a2c10 0x7f41e40a5120 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f41fc108730 0x7f41fc1a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41fc105bc0 0x7f41fc199750 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 --2- 172.21.15.129:0/3946168182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41fc100de0 0x7f41fc199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 -- 172.21.15.129:0/3946168182 >> 172.21.15.129:0/3946168182 conn(0x7f41fc0fc2f0 msgr2=0x7f41fc106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 -- 172.21.15.129:0/3946168182 shutdown_connections 2024-05-20T17:56:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:45.960+0000 7f4201196640 1 -- 172.21.15.129:0/3946168182 wait complete. 2024-05-20T17:56:45.975 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:56:45.997 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-20T17:56:45.997 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2076558567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-20T17:56:45.997 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: osdmap e84: 12 total, 12 up, 12 in 2024-05-20T17:56:45.997 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:45.997 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:45.997 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:45.997 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:45.997 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:45.997 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:45.997 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:45.998 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:56:45.998 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:45.998 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3946168182' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-20T17:56:45.998 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-20T17:56:45.998 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:45 smithi129 ceph-mon[32029]: pgmap v283: 97 pgs: 6 creating+peering, 2 active+clean, 89 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:46.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.089+0000 7f2900075640 1 -- 172.21.15.129:0/1777868213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f8103cd0 msgr2=0x7f28f81040e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.089+0000 7f2900075640 1 --2- 172.21.15.129:0/1777868213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f8103cd0 0x7f28f81040e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009900 tx=0x7f28e802a410 comp rx=0 tx=0).stop 2024-05-20T17:56:46.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.089+0000 7f2900075640 1 -- 172.21.15.129:0/1777868213 shutdown_connections 2024-05-20T17:56:46.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.089+0000 7f2900075640 1 --2- 172.21.15.129:0/1777868213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f81089d0 0x7f28f810adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.089+0000 7f2900075640 1 --2- 172.21.15.129:0/1777868213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f8103cd0 0x7f28f81040e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.089+0000 7f2900075640 1 --2- 172.21.15.129:0/1777868213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f8102a90 0x7f28f8103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.089+0000 7f2900075640 1 -- 172.21.15.129:0/1777868213 >> 172.21.15.129:0/1777868213 conn(0x7f28f80fdc80 msgr2=0x7f28f8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.089+0000 7f2900075640 1 -- 172.21.15.129:0/1777868213 shutdown_connections 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.089+0000 7f2900075640 1 -- 172.21.15.129:0/1777868213 wait complete. 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f2900075640 1 Processor -- start 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f2900075640 1 -- start start 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f2900075640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f8102a90 0x7f28f81992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f2900075640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f8103cd0 0x7f28f81997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f2900075640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f81089d0 0x7f28f81a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f2900075640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f28f810e700 con 0x7f28f81089d0 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f2900075640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28f810e580 con 0x7f28f8103cd0 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f2900075640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f28f810e880 con 0x7f28f8102a90 2024-05-20T17:56:46.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fe612640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f81089d0 0x7f28f81a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fe612640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f81089d0 0x7f28f81a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53612/0 (socket says 172.21.15.129:53612) 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fe612640 1 -- 172.21.15.129:0/1855809660 learned_addr learned my addr 172.21.15.129:0/1855809660 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fde11640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f8102a90 0x7f28f81992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fd610640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f8103cd0 0x7f28f81997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fe612640 1 -- 172.21.15.129:0/1855809660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f8102a90 msgr2=0x7f28f81992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fe612640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f8102a90 0x7f28f81992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fe612640 1 -- 172.21.15.129:0/1855809660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f8103cd0 msgr2=0x7f28f81997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fe612640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f8103cd0 0x7f28f81997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fe612640 1 -- 172.21.15.129:0/1855809660 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f81a0de0 con 0x7f28f81089d0 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fd610640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f8103cd0 0x7f28f81997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28fe612640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f81089d0 0x7f28f81a0770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f28f000eaa0 tx=0x7f28f000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.090+0000 7f28eeffd640 1 -- 172.21.15.129:0/1855809660 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f000cdb0 con 0x7f28f81089d0 2024-05-20T17:56:46.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.091+0000 7f28eeffd640 1 -- 172.21.15.129:0/1855809660 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f0005020 con 0x7f28f81089d0 2024-05-20T17:56:46.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.091+0000 7f2900075640 1 -- 172.21.15.129:0/1855809660 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f81a10d0 con 0x7f28f81089d0 2024-05-20T17:56:46.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.091+0000 7f28eeffd640 1 -- 172.21.15.129:0/1855809660 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f00116b0 con 0x7f28f81089d0 2024-05-20T17:56:46.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.091+0000 7f2900075640 1 -- 172.21.15.129:0/1855809660 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f81a15d0 con 0x7f28f81089d0 2024-05-20T17:56:46.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.091+0000 7f2900075640 1 -- 172.21.15.129:0/1855809660 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f81077e0 con 0x7f28f81089d0 2024-05-20T17:56:46.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.094+0000 7f28eeffd640 1 -- 172.21.15.129:0/1855809660 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f28f0005560 con 0x7f28f81089d0 2024-05-20T17:56:46.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.094+0000 7f28eeffd640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f28e40a2b90 0x7f28e40a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.094+0000 7f28fde11640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f28e40a2b90 0x7f28e40a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.094+0000 7f28eeffd640 1 -- 172.21.15.129:0/1855809660 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f28f00cf5e0 con 0x7f28f81089d0 2024-05-20T17:56:46.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.095+0000 7f28fde11640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f28e40a2b90 0x7f28e40a50a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f28f4005fd0 tx=0x7f28f4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:46.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.098+0000 7f28eeffd640 1 -- 172.21.15.129:0/1855809660 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f28f0016030 con 0x7f28f81089d0 2024-05-20T17:56:46.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-20T17:56:46.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2076558567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-20T17:56:46.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: osdmap e84: 12 total, 12 up, 12 in 2024-05-20T17:56:46.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:46.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:56:46.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3946168182' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-20T17:56:46.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-20T17:56:46.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:45 smithi154 ceph-mon[36482]: pgmap v283: 97 pgs: 6 creating+peering, 2 active+clean, 89 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-20T17:56:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2076558567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-20T17:56:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: osdmap e84: 12 total, 12 up, 12 in 2024-05-20T17:56:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:56:46.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:56:46.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:56:46.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3946168182' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-20T17:56:46.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-20T17:56:46.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:45 smithi204 ceph-mon[37076]: pgmap v283: 97 pgs: 6 creating+peering, 2 active+clean, 89 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:56:46.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.276+0000 7f2900075640 1 -- 172.21.15.129:0/1855809660 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28f8062750 con 0x7f28f81089d0 2024-05-20T17:56:46.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.278+0000 7f28eeffd640 1 -- 172.21.15.129:0/1855809660 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22253 (secure 0 0 0) 0x7f28f009c690 con 0x7f28f81089d0 2024-05-20T17:56:46.277 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:46.277 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":85,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:56:45.095753+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:56:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 -- 172.21.15.129:0/1855809660 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f28e40a2b90 msgr2=0x7f28e40a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f28e40a2b90 0x7f28e40a50a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f28f4005fd0 tx=0x7f28f4005850 comp rx=0 tx=0).stop 2024-05-20T17:56:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 -- 172.21.15.129:0/1855809660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f81089d0 msgr2=0x7f28f81a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f81089d0 0x7f28f81a0770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f28f000eaa0 tx=0x7f28f000ef60 comp rx=0 tx=0).stop 2024-05-20T17:56:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 -- 172.21.15.129:0/1855809660 shutdown_connections 2024-05-20T17:56:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f28e40a2b90 0x7f28e40a50a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f8103cd0 0x7f28f81997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f81089d0 0x7f28f81a0770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 --2- 172.21.15.129:0/1855809660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f8102a90 0x7f28f81992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 -- 172.21.15.129:0/1855809660 >> 172.21.15.129:0/1855809660 conn(0x7f28f80fdc80 msgr2=0x7f28f81049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:46.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 -- 172.21.15.129:0/1855809660 shutdown_connections 2024-05-20T17:56:46.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.280+0000 7f2900075640 1 -- 172.21.15.129:0/1855809660 wait complete. 2024-05-20T17:56:46.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:56:46.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.443+0000 7f6252075640 1 -- 172.21.15.129:0/1568710149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624c102aa0 msgr2=0x7f624c1037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.443+0000 7f6252075640 1 --2- 172.21.15.129:0/1568710149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624c102aa0 0x7f624c1037a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f623c009900 tx=0x7f623c02a410 comp rx=0 tx=0).stop 2024-05-20T17:56:46.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.443+0000 7f6252075640 1 -- 172.21.15.129:0/1568710149 shutdown_connections 2024-05-20T17:56:46.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.443+0000 7f6252075640 1 --2- 172.21.15.129:0/1568710149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f624c1089e0 0x7f624c10ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.443+0000 7f6252075640 1 --2- 172.21.15.129:0/1568710149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f624c103ce0 0x7f624c1040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.443+0000 7f6252075640 1 --2- 172.21.15.129:0/1568710149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624c102aa0 0x7f624c1037a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.443+0000 7f6252075640 1 -- 172.21.15.129:0/1568710149 >> 172.21.15.129:0/1568710149 conn(0x7f624c0fdcf0 msgr2=0x7f624c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.443+0000 7f6252075640 1 -- 172.21.15.129:0/1568710149 shutdown_connections 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.443+0000 7f6252075640 1 -- 172.21.15.129:0/1568710149 wait complete. 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.443+0000 7f6252075640 1 Processor -- start 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.443+0000 7f6252075640 1 -- start start 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f6252075640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624c102aa0 0x7f624c199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f6252075640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f624c103ce0 0x7f624c199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f6252075640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f624c1089e0 0x7f624c1a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f6252075640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f624c10e740 con 0x7f624c102aa0 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f6252075640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f624c10e5c0 con 0x7f624c103ce0 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f6252075640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f624c10e8c0 con 0x7f624c1089e0 2024-05-20T17:56:46.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f624b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624c102aa0 0x7f624c199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f624b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624c102aa0 0x7f624c199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53630/0 (socket says 172.21.15.129:53630) 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f624b7fe640 1 -- 172.21.15.129:0/1643502912 learned_addr learned my addr 172.21.15.129:0/1643502912 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f624bfff640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f624c1089e0 0x7f624c1a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f624b7fe640 1 -- 172.21.15.129:0/1643502912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f624c1089e0 msgr2=0x7f624c1a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f6243fff640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f624c103ce0 0x7f624c199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f624b7fe640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f624c1089e0 0x7f624c1a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f624b7fe640 1 -- 172.21.15.129:0/1643502912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f624c103ce0 msgr2=0x7f624c199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f624b7fe640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f624c103ce0 0x7f624c199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f624b7fe640 1 -- 172.21.15.129:0/1643502912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f624c1a0de0 con 0x7f624c102aa0 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f624b7fe640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624c102aa0 0x7f624c199310 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f623c031440 tx=0x7f623c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f62497fa640 1 -- 172.21.15.129:0/1643502912 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c03f030 con 0x7f624c102aa0 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f62497fa640 1 -- 172.21.15.129:0/1643502912 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f623c003ce0 con 0x7f624c102aa0 2024-05-20T17:56:46.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f6252075640 1 -- 172.21.15.129:0/1643502912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f624c1a1070 con 0x7f624c102aa0 2024-05-20T17:56:46.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.444+0000 7f62497fa640 1 -- 172.21.15.129:0/1643502912 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c035070 con 0x7f624c102aa0 2024-05-20T17:56:46.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.445+0000 7f6252075640 1 -- 172.21.15.129:0/1643502912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f624c1a1560 con 0x7f624c102aa0 2024-05-20T17:56:46.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.445+0000 7f6252075640 1 -- 172.21.15.129:0/1643502912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f624c1077f0 con 0x7f624c102aa0 2024-05-20T17:56:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.447+0000 7f62497fa640 1 -- 172.21.15.129:0/1643502912 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f623c042030 con 0x7f624c102aa0 2024-05-20T17:56:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.448+0000 7f62497fa640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62380a2b90 0x7f62380a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.448+0000 7f6243fff640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62380a2b90 0x7f62380a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.448+0000 7f62497fa640 1 -- 172.21.15.129:0/1643502912 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f623c0ef940 con 0x7f624c102aa0 2024-05-20T17:56:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.448+0000 7f6243fff640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62380a2b90 0x7f62380a50a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6234005fd0 tx=0x7f6234005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:46.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.451+0000 7f62497fa640 1 -- 172.21.15.129:0/1643502912 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f623c0bc9d0 con 0x7f624c102aa0 2024-05-20T17:56:46.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.629+0000 7f6252075640 1 -- 172.21.15.129:0/1643502912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f624c062750 con 0x7f624c102aa0 2024-05-20T17:56:46.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.631+0000 7f62497fa640 1 -- 172.21.15.129:0/1643502912 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22253 (secure 0 0 0) 0x7f623c0bc2d0 con 0x7f624c102aa0 2024-05-20T17:56:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:46.630 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":85,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:56:45.095753+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:56:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 -- 172.21.15.129:0/1643502912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62380a2b90 msgr2=0x7f62380a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62380a2b90 0x7f62380a50a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6234005fd0 tx=0x7f6234005850 comp rx=0 tx=0).stop 2024-05-20T17:56:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 -- 172.21.15.129:0/1643502912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624c102aa0 msgr2=0x7f624c199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624c102aa0 0x7f624c199310 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f623c031440 tx=0x7f623c032ce0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 -- 172.21.15.129:0/1643502912 shutdown_connections 2024-05-20T17:56:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f62380a2b90 0x7f62380a50a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f624c103ce0 0x7f624c199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f624c1089e0 0x7f624c1a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 --2- 172.21.15.129:0/1643502912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f624c102aa0 0x7f624c199310 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 -- 172.21.15.129:0/1643502912 >> 172.21.15.129:0/1643502912 conn(0x7f624c0fdcf0 msgr2=0x7f624c1049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 -- 172.21.15.129:0/1643502912 shutdown_connections 2024-05-20T17:56:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.633+0000 7f6252075640 1 -- 172.21.15.129:0/1643502912 wait complete. 2024-05-20T17:56:46.644 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-20T17:56:46.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.761+0000 7f395da67640 1 -- 172.21.15.129:0/3119609165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39581078e0 msgr2=0x7f3958109d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.761+0000 7f395da67640 1 --2- 172.21.15.129:0/3119609165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39581078e0 0x7f3958109d00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3950009ca0 tx=0x7f395002a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.761+0000 7f395da67640 1 -- 172.21.15.129:0/3119609165 shutdown_connections 2024-05-20T17:56:46.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.761+0000 7f395da67640 1 --2- 172.21.15.129:0/3119609165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39581078e0 0x7f3958109d00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.761+0000 7f395da67640 1 --2- 172.21.15.129:0/3119609165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3958102be0 0x7f3958102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.761+0000 7f395da67640 1 --2- 172.21.15.129:0/3119609165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39581019a0 0x7f39581026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.761+0000 7f395da67640 1 -- 172.21.15.129:0/3119609165 >> 172.21.15.129:0/3119609165 conn(0x7f39580fcb70 msgr2=0x7f39580ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:46.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.761+0000 7f395da67640 1 -- 172.21.15.129:0/3119609165 shutdown_connections 2024-05-20T17:56:46.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.761+0000 7f395da67640 1 -- 172.21.15.129:0/3119609165 wait complete. 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.761+0000 7f395da67640 1 Processor -- start 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f395da67640 1 -- start start 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f395da67640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39581019a0 0x7f3958074810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f395da67640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3958102be0 0x7f395819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f395da67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39581078e0 0x7f39581a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f395da67640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f395810d620 con 0x7f39581078e0 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f395da67640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f395810d4a0 con 0x7f39581019a0 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f395da67640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f395810d7a0 con 0x7f3958102be0 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f3957fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39581078e0 0x7f39581a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f3957fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39581078e0 0x7f39581a0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53652/0 (socket says 172.21.15.129:53652) 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f3957fff640 1 -- 172.21.15.129:0/254387024 learned_addr learned my addr 172.21.15.129:0/254387024 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f394ffff640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3958102be0 0x7f395819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f39577fe640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39581019a0 0x7f3958074810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f3957fff640 1 -- 172.21.15.129:0/254387024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3958102be0 msgr2=0x7f395819dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f3957fff640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3958102be0 0x7f395819dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f3957fff640 1 -- 172.21.15.129:0/254387024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39581019a0 msgr2=0x7f3958074810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:46.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f3957fff640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39581019a0 0x7f3958074810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:46.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f3957fff640 1 -- 172.21.15.129:0/254387024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39581a0d50 con 0x7f39581078e0 2024-05-20T17:56:46.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f3957fff640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39581078e0 0x7f39581a0530 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3950005cc0 tx=0x7f3950004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:46.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.762+0000 7f39557fa640 1 -- 172.21.15.129:0/254387024 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3950003b70 con 0x7f39581078e0 2024-05-20T17:56:46.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.763+0000 7f39557fa640 1 -- 172.21.15.129:0/254387024 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3950003d10 con 0x7f39581078e0 2024-05-20T17:56:46.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.763+0000 7f395da67640 1 -- 172.21.15.129:0/254387024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39581a0fe0 con 0x7f39581078e0 2024-05-20T17:56:46.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.763+0000 7f39557fa640 1 -- 172.21.15.129:0/254387024 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395002fbd0 con 0x7f39581078e0 2024-05-20T17:56:46.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.763+0000 7f395da67640 1 -- 172.21.15.129:0/254387024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39581a1440 con 0x7f39581078e0 2024-05-20T17:56:46.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.763+0000 7f395da67640 1 -- 172.21.15.129:0/254387024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3958069110 con 0x7f39581078e0 2024-05-20T17:56:46.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.765+0000 7f39557fa640 1 -- 172.21.15.129:0/254387024 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f395002fd70 con 0x7f39581078e0 2024-05-20T17:56:46.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.766+0000 7f39557fa640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39440a2c10 0x7f39440a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:46.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.766+0000 7f39557fa640 1 -- 172.21.15.129:0/254387024 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f39500f0410 con 0x7f39581078e0 2024-05-20T17:56:46.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.766+0000 7f39577fe640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39440a2c10 0x7f39440a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:46.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.766+0000 7f39577fe640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39440a2c10 0x7f39440a5120 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f39581025a0 tx=0x7f3948005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:46.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.769+0000 7f39557fa640 1 -- 172.21.15.129:0/254387024 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f39500bd420 con 0x7f39581078e0 2024-05-20T17:56:46.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:46 smithi129 ceph-mon[32029]: osdmap e85: 12 total, 12 up, 12 in 2024-05-20T17:56:46.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:46 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3946168182' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-20T17:56:46.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:46 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-20T17:56:46.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:46 smithi129 ceph-mon[32029]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-20T17:56:46.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:46 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1855809660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:46.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:46 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1643502912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:46.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:46.986+0000 7f395da67640 1 -- 172.21.15.129:0/254387024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3958106740 con 0x7f39581078e0 2024-05-20T17:56:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:46 smithi154 ceph-mon[36482]: osdmap e85: 12 total, 12 up, 12 in 2024-05-20T17:56:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:46 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3946168182' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-20T17:56:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:46 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-20T17:56:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:46 smithi154 ceph-mon[36482]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-20T17:56:47.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:46 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1855809660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:47.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:46 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1643502912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:47.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:46 smithi204 ceph-mon[37076]: osdmap e85: 12 total, 12 up, 12 in 2024-05-20T17:56:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:46 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3946168182' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-20T17:56:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:46 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-20T17:56:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:46 smithi204 ceph-mon[37076]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-20T17:56:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:46 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1855809660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:46 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1643502912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:47.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.975+0000 7f39557fa640 1 -- 172.21.15.129:0/254387024 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f3958106740 con 0x7f39581078e0 2024-05-20T17:56:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 -- 172.21.15.129:0/254387024 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39440a2c10 msgr2=0x7f39440a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39440a2c10 0x7f39440a5120 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f39581025a0 tx=0x7f3948005f90 comp rx=0 tx=0).stop 2024-05-20T17:56:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 -- 172.21.15.129:0/254387024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39581078e0 msgr2=0x7f39581a0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39581078e0 0x7f39581a0530 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3950005cc0 tx=0x7f3950004ab0 comp rx=0 tx=0).stop 2024-05-20T17:56:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 -- 172.21.15.129:0/254387024 shutdown_connections 2024-05-20T17:56:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39440a2c10 0x7f39440a5120 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39581078e0 0x7f39581a0530 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3958102be0 0x7f395819dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 --2- 172.21.15.129:0/254387024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39581019a0 0x7f3958074810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 -- 172.21.15.129:0/254387024 >> 172.21.15.129:0/254387024 conn(0x7f39580fcb70 msgr2=0x7f39581038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 -- 172.21.15.129:0/254387024 shutdown_connections 2024-05-20T17:56:47.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:47.978+0000 7f395da67640 1 -- 172.21.15.129:0/254387024 wait complete. 2024-05-20T17:56:47.987 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:47 smithi129 ceph-mon[32029]: osdmap e86: 12 total, 12 up, 12 in 2024-05-20T17:56:47.987 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:47 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] up:active 2024-05-20T17:56:47.987 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:47 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:47.987 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:47 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:47.987 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:47 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/254387024' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-20T17:56:47.987 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:47 smithi129 ceph-mon[32029]: pgmap v286: 97 pgs: 6 creating+peering, 39 active+clean, 52 unknown; 578 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-05-20T17:56:47.988 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-20T17:56:47.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-05-20T17:56:48.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.106+0000 7f3e46232640 1 -- 172.21.15.129:0/2117910597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e401078e0 msgr2=0x7f3e40109d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.106+0000 7f3e46232640 1 --2- 172.21.15.129:0/2117910597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e401078e0 0x7f3e40109d00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009ca0 tx=0x7f3e3802a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.106+0000 7f3e46232640 1 -- 172.21.15.129:0/2117910597 shutdown_connections 2024-05-20T17:56:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.106+0000 7f3e46232640 1 --2- 172.21.15.129:0/2117910597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e401078e0 0x7f3e40109d00 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.106+0000 7f3e46232640 1 --2- 172.21.15.129:0/2117910597 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e40102be0 0x7f3e40102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.106+0000 7f3e46232640 1 --2- 172.21.15.129:0/2117910597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e401019a0 0x7f3e401026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:48.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.106+0000 7f3e46232640 1 -- 172.21.15.129:0/2117910597 >> 172.21.15.129:0/2117910597 conn(0x7f3e400fcb90 msgr2=0x7f3e400ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:48.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.106+0000 7f3e46232640 1 -- 172.21.15.129:0/2117910597 shutdown_connections 2024-05-20T17:56:48.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.106+0000 7f3e46232640 1 -- 172.21.15.129:0/2117910597 wait complete. 2024-05-20T17:56:48.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e46232640 1 Processor -- start 2024-05-20T17:56:48.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e46232640 1 -- start start 2024-05-20T17:56:48.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e46232640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e401019a0 0x7f3e40198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:48.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e46232640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e40102be0 0x7f3e401986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:48.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e46232640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e401078e0 0x7f3e4019f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:48.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e46232640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4010d600 con 0x7f3e40102be0 2024-05-20T17:56:48.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e46232640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4010d480 con 0x7f3e401019a0 2024-05-20T17:56:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e46232640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4010d780 con 0x7f3e401078e0 2024-05-20T17:56:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e3effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e40102be0 0x7f3e401986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e3effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e40102be0 0x7f3e401986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53666/0 (socket says 172.21.15.129:53666) 2024-05-20T17:56:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e3effd640 1 -- 172.21.15.129:0/1058340595 learned_addr learned my addr 172.21.15.129:0/1058340595 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.107+0000 7f3e3ffff640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e401078e0 0x7f3e4019f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3f7fe640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e401019a0 0x7f3e40198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3effd640 1 -- 172.21.15.129:0/1058340595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e401078e0 msgr2=0x7f3e4019f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3effd640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e401078e0 0x7f3e4019f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3effd640 1 -- 172.21.15.129:0/1058340595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e401019a0 msgr2=0x7f3e40198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3effd640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e401019a0 0x7f3e40198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3effd640 1 -- 172.21.15.129:0/1058340595 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4019fcb0 con 0x7f3e40102be0 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3ffff640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e401078e0 0x7f3e4019f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3f7fe640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e401019a0 0x7f3e40198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3effd640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e40102be0 0x7f3e401986b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3e3400c890 tx=0x7f3e3400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3cff9640 1 -- 172.21.15.129:0/1058340595 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e34004220 con 0x7f3e40102be0 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3cff9640 1 -- 172.21.15.129:0/1058340595 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e340043c0 con 0x7f3e40102be0 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e3cff9640 1 -- 172.21.15.129:0/1058340595 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e340049e0 con 0x7f3e40102be0 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e46232640 1 -- 172.21.15.129:0/1058340595 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4019ffa0 con 0x7f3e40102be0 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.108+0000 7f3e46232640 1 -- 172.21.15.129:0/1058340595 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e401a0460 con 0x7f3e40102be0 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.109+0000 7f3e46232640 1 -- 172.21.15.129:0/1058340595 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e401066f0 con 0x7f3e40102be0 2024-05-20T17:56:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.110+0000 7f3e3cff9640 1 -- 172.21.15.129:0/1058340595 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3e340079b0 con 0x7f3e40102be0 2024-05-20T17:56:48.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.111+0000 7f3e3cff9640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3e280a2c10 0x7f3e280a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:48.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.111+0000 7f3e3cff9640 1 -- 172.21.15.129:0/1058340595 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f3e340cefc0 con 0x7f3e40102be0 2024-05-20T17:56:48.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.111+0000 7f3e3f7fe640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3e280a2c10 0x7f3e280a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:48.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.111+0000 7f3e3f7fe640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3e280a2c10 0x7f3e280a5120 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3e401025a0 tx=0x7f3e30005e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:48.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.116+0000 7f3e3cff9640 1 -- 172.21.15.129:0/1058340595 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3e34014030 con 0x7f3e40102be0 2024-05-20T17:56:48.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:47 smithi154 ceph-mon[36482]: osdmap e86: 12 total, 12 up, 12 in 2024-05-20T17:56:48.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:47 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] up:active 2024-05-20T17:56:48.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:47 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:48.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:47 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:48.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:47 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/254387024' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-20T17:56:48.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:47 smithi154 ceph-mon[36482]: pgmap v286: 97 pgs: 6 creating+peering, 39 active+clean, 52 unknown; 578 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-05-20T17:56:48.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:47 smithi204 ceph-mon[37076]: osdmap e86: 12 total, 12 up, 12 in 2024-05-20T17:56:48.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:47 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] up:active 2024-05-20T17:56:48.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:47 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:48.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:47 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:47 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/254387024' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-20T17:56:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:47 smithi204 ceph-mon[37076]: pgmap v286: 97 pgs: 6 creating+peering, 39 active+clean, 52 unknown; 578 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-05-20T17:56:48.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.351+0000 7f3e46232640 1 -- 172.21.15.129:0/1058340595 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f3e4002d2a0 con 0x7f3e40102be0 2024-05-20T17:56:48.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.977+0000 7f3e3cff9640 1 -- 172.21.15.129:0/1058340595 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f3e3409c070 con 0x7f3e40102be0 2024-05-20T17:56:48.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.979+0000 7f3e46232640 1 -- 172.21.15.129:0/1058340595 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3e280a2c10 msgr2=0x7f3e280a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.979+0000 7f3e46232640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3e280a2c10 0x7f3e280a5120 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3e401025a0 tx=0x7f3e30005e70 comp rx=0 tx=0).stop 2024-05-20T17:56:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.979+0000 7f3e46232640 1 -- 172.21.15.129:0/1058340595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e40102be0 msgr2=0x7f3e401986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.979+0000 7f3e46232640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e40102be0 0x7f3e401986b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3e3400c890 tx=0x7f3e3400cd50 comp rx=0 tx=0).stop 2024-05-20T17:56:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.979+0000 7f3e46232640 1 -- 172.21.15.129:0/1058340595 shutdown_connections 2024-05-20T17:56:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.979+0000 7f3e46232640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3e280a2c10 0x7f3e280a5120 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.979+0000 7f3e46232640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3e401078e0 0x7f3e4019f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.979+0000 7f3e46232640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e40102be0 0x7f3e401986b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.979+0000 7f3e46232640 1 --2- 172.21.15.129:0/1058340595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e401019a0 0x7f3e40198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.979+0000 7f3e46232640 1 -- 172.21.15.129:0/1058340595 >> 172.21.15.129:0/1058340595 conn(0x7f3e400fcb90 msgr2=0x7f3e401038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.980+0000 7f3e46232640 1 -- 172.21.15.129:0/1058340595 shutdown_connections 2024-05-20T17:56:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:48.980+0000 7f3e46232640 1 -- 172.21.15.129:0/1058340595 wait complete. 2024-05-20T17:56:48.991 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:48 smithi129 ceph-mon[32029]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-20T17:56:48.991 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:48 smithi129 ceph-mon[32029]: Cluster is now healthy 2024-05-20T17:56:48.992 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:48 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/254387024' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-20T17:56:48.992 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:48 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:48.992 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:48 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1058340595' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-20T17:56:48.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-05-20T17:56:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.113+0000 7f2687dcc640 1 -- 172.21.15.129:0/1812683627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26801019b0 msgr2=0x7f26801026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.113+0000 7f2687dcc640 1 --2- 172.21.15.129:0/1812683627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26801019b0 0x7f26801026b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f267c009900 tx=0x7f267c02a410 comp rx=0 tx=0).stop 2024-05-20T17:56:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.113+0000 7f2687dcc640 1 -- 172.21.15.129:0/1812683627 shutdown_connections 2024-05-20T17:56:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.113+0000 7f2687dcc640 1 --2- 172.21.15.129:0/1812683627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2680102bf0 0x7f2680103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.113+0000 7f2687dcc640 1 --2- 172.21.15.129:0/1812683627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26801078f0 0x7f2680109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.113+0000 7f2687dcc640 1 --2- 172.21.15.129:0/1812683627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26801019b0 0x7f26801026b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.113+0000 7f2687dcc640 1 -- 172.21.15.129:0/1812683627 >> 172.21.15.129:0/1812683627 conn(0x7f26800fcba0 msgr2=0x7f26800ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.113+0000 7f2687dcc640 1 -- 172.21.15.129:0/1812683627 shutdown_connections 2024-05-20T17:56:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.113+0000 7f2687dcc640 1 -- 172.21.15.129:0/1812683627 wait complete. 2024-05-20T17:56:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2687dcc640 1 Processor -- start 2024-05-20T17:56:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2687dcc640 1 -- start start 2024-05-20T17:56:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2687dcc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26801019b0 0x7f26801992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2687dcc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2680102bf0 0x7f26801997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2687dcc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26801078f0 0x7f26801a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2687dcc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f268010e730 con 0x7f2680102bf0 2024-05-20T17:56:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2687dcc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f268010e5b0 con 0x7f26801078f0 2024-05-20T17:56:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2687dcc640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f268010e8b0 con 0x7f26801019b0 2024-05-20T17:56:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2685367640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2680102bf0 0x7f26801997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2685367640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2680102bf0 0x7f26801997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53686/0 (socket says 172.21.15.129:53686) 2024-05-20T17:56:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2685367640 1 -- 172.21.15.129:0/3866157065 learned_addr learned my addr 172.21.15.129:0/3866157065 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2685b68640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26801019b0 0x7f26801992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.114+0000 7f2686369640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26801078f0 0x7f26801a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2685367640 1 -- 172.21.15.129:0/3866157065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26801019b0 msgr2=0x7f26801992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2685367640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26801019b0 0x7f26801992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2685367640 1 -- 172.21.15.129:0/3866157065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26801078f0 msgr2=0x7f26801a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2685367640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26801078f0 0x7f26801a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2685367640 1 -- 172.21.15.129:0/3866157065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26801a0df0 con 0x7f2680102bf0 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2685b68640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26801019b0 0x7f26801992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2685367640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2680102bf0 0x7f26801997f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f267000d950 tx=0x7f267000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2676ffd640 1 -- 172.21.15.129:0/3866157065 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670017070 con 0x7f2680102bf0 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2676ffd640 1 -- 172.21.15.129:0/3866157065 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2670009c30 con 0x7f2680102bf0 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2687dcc640 1 -- 172.21.15.129:0/3866157065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26801a10e0 con 0x7f2680102bf0 2024-05-20T17:56:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2676ffd640 1 -- 172.21.15.129:0/3866157065 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267000f6b0 con 0x7f2680102bf0 2024-05-20T17:56:49.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.115+0000 7f2687dcc640 1 -- 172.21.15.129:0/3866157065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26801a1630 con 0x7f2680102bf0 2024-05-20T17:56:49.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.116+0000 7f2687dcc640 1 -- 172.21.15.129:0/3866157065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2680106700 con 0x7f2680102bf0 2024-05-20T17:56:49.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.117+0000 7f2676ffd640 1 -- 172.21.15.129:0/3866157065 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f26700109d0 con 0x7f2680102bf0 2024-05-20T17:56:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.118+0000 7f2676ffd640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f266c0a2b90 0x7f266c0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.118+0000 7f2685b68640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f266c0a2b90 0x7f266c0a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.118+0000 7f2676ffd640 1 -- 172.21.15.129:0/3866157065 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f26700cebe0 con 0x7f2680102bf0 2024-05-20T17:56:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.119+0000 7f2685b68640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f266c0a2b90 0x7f266c0a50a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f267c009230 tx=0x7f267c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:49.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.122+0000 7f2676ffd640 1 -- 172.21.15.129:0/3866157065 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2670009de0 con 0x7f2680102bf0 2024-05-20T17:56:49.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:48 smithi154 ceph-mon[36482]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-20T17:56:49.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:48 smithi154 ceph-mon[36482]: Cluster is now healthy 2024-05-20T17:56:49.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:48 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/254387024' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-20T17:56:49.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:48 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:49.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:48 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1058340595' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-20T17:56:49.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:48 smithi204 ceph-mon[37076]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-20T17:56:49.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:48 smithi204 ceph-mon[37076]: Cluster is now healthy 2024-05-20T17:56:49.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:48 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/254387024' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-20T17:56:49.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:48 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:49.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:48 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1058340595' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-20T17:56:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.334+0000 7f2687dcc640 1 -- 172.21.15.129:0/3866157065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f268002d2a0 con 0x7f2680102bf0 2024-05-20T17:56:49.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.991+0000 7f2676ffd640 1 -- 172.21.15.129:0/3866157065 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7f267009bc90 con 0x7f2680102bf0 2024-05-20T17:56:49.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 -- 172.21.15.129:0/3866157065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f266c0a2b90 msgr2=0x7f266c0a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:49.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f266c0a2b90 0x7f266c0a50a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f267c009230 tx=0x7f267c000bc0 comp rx=0 tx=0).stop 2024-05-20T17:56:49.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 -- 172.21.15.129:0/3866157065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2680102bf0 msgr2=0x7f26801997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:49.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2680102bf0 0x7f26801997f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f267000d950 tx=0x7f267000de10 comp rx=0 tx=0).stop 2024-05-20T17:56:49.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 -- 172.21.15.129:0/3866157065 shutdown_connections 2024-05-20T17:56:49.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f266c0a2b90 0x7f266c0a50a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:49.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26801078f0 0x7f26801a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:49.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2680102bf0 0x7f26801997f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:49.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 --2- 172.21.15.129:0/3866157065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f26801019b0 0x7f26801992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:49.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 -- 172.21.15.129:0/3866157065 >> 172.21.15.129:0/3866157065 conn(0x7f26800fcba0 msgr2=0x7f26801038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:49.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 -- 172.21.15.129:0/3866157065 shutdown_connections 2024-05-20T17:56:49.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:49.994+0000 7f2687dcc640 1 -- 172.21.15.129:0/3866157065 wait complete. 2024-05-20T17:56:50.004 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-05-20T17:56:50.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-05-20T17:56:50.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.121+0000 7f428bd77640 1 -- 172.21.15.129:0/423740412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4284102ad0 msgr2=0x7f4284102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.121+0000 7f428bd77640 1 --2- 172.21.15.129:0/423740412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4284102ad0 0x7f4284102fa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4274009900 tx=0x7f427402a410 comp rx=0 tx=0).stop 2024-05-20T17:56:50.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.121+0000 7f428bd77640 1 -- 172.21.15.129:0/423740412 shutdown_connections 2024-05-20T17:56:50.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.121+0000 7f428bd77640 1 --2- 172.21.15.129:0/423740412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4284103630 0x7f4284109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.121+0000 7f428bd77640 1 --2- 172.21.15.129:0/423740412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4284102ad0 0x7f4284102fa0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.121+0000 7f428bd77640 1 --2- 172.21.15.129:0/423740412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4284101f70 0x7f4284102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.121+0000 7f428bd77640 1 -- 172.21.15.129:0/423740412 >> 172.21.15.129:0/423740412 conn(0x7f42840fc2f0 msgr2=0x7f42840fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:50.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.121+0000 7f428bd77640 1 -- 172.21.15.129:0/423740412 shutdown_connections 2024-05-20T17:56:50.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.121+0000 7f428bd77640 1 -- 172.21.15.129:0/423740412 wait complete. 2024-05-20T17:56:50.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.121+0000 7f428bd77640 1 Processor -- start 2024-05-20T17:56:50.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.121+0000 7f428bd77640 1 -- start start 2024-05-20T17:56:50.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f428bd77640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4284101f70 0x7f42841990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f428bd77640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4284102ad0 0x7f4284199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f428bd77640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f428419e580 0x7f42841a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f428bd77640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f428410e9b0 con 0x7f4284102ad0 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f428bd77640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f428410e830 con 0x7f428419e580 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f428bd77640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f428410eb30 con 0x7f4284101f70 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f4289312640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4284102ad0 0x7f4284199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f4289312640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4284102ad0 0x7f4284199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46570/0 (socket says 172.21.15.129:46570) 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f4289312640 1 -- 172.21.15.129:0/2153247905 learned_addr learned my addr 172.21.15.129:0/2153247905 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f4289b13640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4284101f70 0x7f42841990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f428a314640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f428419e580 0x7f42841a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f4289312640 1 -- 172.21.15.129:0/2153247905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4284101f70 msgr2=0x7f42841990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f4289312640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4284101f70 0x7f42841990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f4289312640 1 -- 172.21.15.129:0/2153247905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f428419e580 msgr2=0x7f42841a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f4289312640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f428419e580 0x7f42841a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f4289312640 1 -- 172.21.15.129:0/2153247905 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42841a1010 con 0x7f4284102ad0 2024-05-20T17:56:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f4289312640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4284102ad0 0x7f4284199600 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4274031440 tx=0x7f4274032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f427affd640 1 -- 172.21.15.129:0/2153247905 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427403f030 con 0x7f4284102ad0 2024-05-20T17:56:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f427affd640 1 -- 172.21.15.129:0/2153247905 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4274003ce0 con 0x7f4284102ad0 2024-05-20T17:56:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f428bd77640 1 -- 172.21.15.129:0/2153247905 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42841a12a0 con 0x7f4284102ad0 2024-05-20T17:56:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.122+0000 7f427affd640 1 -- 172.21.15.129:0/2153247905 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274035070 con 0x7f4284102ad0 2024-05-20T17:56:50.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.123+0000 7f428bd77640 1 -- 172.21.15.129:0/2153247905 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42841a1790 con 0x7f4284102ad0 2024-05-20T17:56:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.123+0000 7f428bd77640 1 -- 172.21.15.129:0/2153247905 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4284068c20 con 0x7f4284102ad0 2024-05-20T17:56:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.125+0000 7f427affd640 1 -- 172.21.15.129:0/2153247905 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4274042030 con 0x7f4284102ad0 2024-05-20T17:56:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.126+0000 7f427affd640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f42700a2b90 0x7f42700a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.126+0000 7f4289b13640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f42700a2b90 0x7f42700a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.126+0000 7f427affd640 1 -- 172.21.15.129:0/2153247905 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f42740ef8f0 con 0x7f4284102ad0 2024-05-20T17:56:50.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.126+0000 7f4289b13640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f42700a2b90 0x7f42700a50a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f42841029d0 tx=0x7f428000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:50.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.130+0000 7f427affd640 1 -- 172.21.15.129:0/2153247905 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f42740bc980 con 0x7f4284102ad0 2024-05-20T17:56:50.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.330+0000 7f428bd77640 1 -- 172.21.15.129:0/2153247905 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f428402d560 con 0x7f42700a2b90 2024-05-20T17:56:50.335 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:49 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1058340595' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-20T17:56:50.335 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:49 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:50.335 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:49 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T17:56:50.336 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:49 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3866157065' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-20T17:56:50.336 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:49 smithi129 ceph-mon[32029]: pgmap v287: 97 pgs: 6 creating+peering, 91 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-05-20T17:56:50.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.351+0000 7f427affd640 1 -- 172.21.15.129:0/2153247905 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f428402d560 con 0x7f42700a2b90 2024-05-20T17:56:50.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 -- 172.21.15.129:0/2153247905 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f42700a2b90 msgr2=0x7f42700a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f42700a2b90 0x7f42700a50a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f42841029d0 tx=0x7f428000b390 comp rx=0 tx=0).stop 2024-05-20T17:56:50.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 -- 172.21.15.129:0/2153247905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4284102ad0 msgr2=0x7f4284199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4284102ad0 0x7f4284199600 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4274031440 tx=0x7f4274032ce0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 -- 172.21.15.129:0/2153247905 shutdown_connections 2024-05-20T17:56:50.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f42700a2b90 0x7f42700a50a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f428419e580 0x7f42841a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4284102ad0 0x7f4284199600 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 --2- 172.21.15.129:0/2153247905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4284101f70 0x7f42841990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 -- 172.21.15.129:0/2153247905 >> 172.21.15.129:0/2153247905 conn(0x7f42840fc2f0 msgr2=0x7f42840fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:50.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 -- 172.21.15.129:0/2153247905 shutdown_connections 2024-05-20T17:56:50.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.353+0000 7f428bd77640 1 -- 172.21.15.129:0/2153247905 wait complete. 2024-05-20T17:56:50.363 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-20T17:56:50.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:49 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1058340595' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-20T17:56:50.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:49 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:50.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:49 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T17:56:50.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:49 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3866157065' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-20T17:56:50.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:49 smithi154 ceph-mon[36482]: pgmap v287: 97 pgs: 6 creating+peering, 91 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-05-20T17:56:50.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:49 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1058340595' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-20T17:56:50.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:49 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:56:50.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:49 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T17:56:50.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:49 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3866157065' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-20T17:56:50.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:49 smithi204 ceph-mon[37076]: pgmap v287: 97 pgs: 6 creating+peering, 91 active+clean; 578 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-05-20T17:56:50.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.482+0000 7fe640492640 1 -- 172.21.15.129:0/3879955388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe638107080 msgr2=0x7fe6381094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.482+0000 7fe640492640 1 --2- 172.21.15.129:0/3879955388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe638107080 0x7fe6381094a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe630009ca0 tx=0x7fe63002a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.482+0000 7fe640492640 1 -- 172.21.15.129:0/3879955388 shutdown_connections 2024-05-20T17:56:50.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.482+0000 7fe640492640 1 --2- 172.21.15.129:0/3879955388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe638107080 0x7fe6381094a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.482+0000 7fe640492640 1 --2- 172.21.15.129:0/3879955388 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe638102380 0x7fe638102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.482+0000 7fe640492640 1 --2- 172.21.15.129:0/3879955388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe638101140 0x7fe638101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.482+0000 7fe640492640 1 -- 172.21.15.129:0/3879955388 >> 172.21.15.129:0/3879955388 conn(0x7fe6380fc330 msgr2=0x7fe6380fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:50.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.482+0000 7fe640492640 1 -- 172.21.15.129:0/3879955388 shutdown_connections 2024-05-20T17:56:50.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.482+0000 7fe640492640 1 -- 172.21.15.129:0/3879955388 wait complete. 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe640492640 1 Processor -- start 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe640492640 1 -- start start 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe640492640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe638101140 0x7fe63810a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe640492640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe638102380 0x7fe63819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe640492640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe638107080 0x7fe6381a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe640492640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe63810d640 con 0x7fe638102380 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe640492640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe63810d4c0 con 0x7fe638101140 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe640492640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe63810d7c0 con 0x7fe638107080 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe63da2d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe638102380 0x7fe63819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe63da2d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe638102380 0x7fe63819dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46618/0 (socket says 172.21.15.129:46618) 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe63da2d640 1 -- 172.21.15.129:0/1541926177 learned_addr learned my addr 172.21.15.129:0/1541926177 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe63e22e640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe638101140 0x7fe63810a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:50.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe63da2d640 1 -- 172.21.15.129:0/1541926177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe638107080 msgr2=0x7fe6381a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe63da2d640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe638107080 0x7fe6381a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe63da2d640 1 -- 172.21.15.129:0/1541926177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe638101140 msgr2=0x7fe63810a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe63da2d640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe638101140 0x7fe63810a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.483+0000 7fe63da2d640 1 -- 172.21.15.129:0/1541926177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6381a0d50 con 0x7fe638102380 2024-05-20T17:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.484+0000 7fe63da2d640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe638102380 0x7fe63819dfc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe62800c970 tx=0x7fe62800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.484+0000 7fe62f7fe640 1 -- 172.21.15.129:0/1541926177 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe628007bb0 con 0x7fe638102380 2024-05-20T17:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.484+0000 7fe62f7fe640 1 -- 172.21.15.129:0/1541926177 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe628007d50 con 0x7fe638102380 2024-05-20T17:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.484+0000 7fe640492640 1 -- 172.21.15.129:0/1541926177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6381a0fe0 con 0x7fe638102380 2024-05-20T17:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.484+0000 7fe62f7fe640 1 -- 172.21.15.129:0/1541926177 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62800f450 con 0x7fe638102380 2024-05-20T17:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.484+0000 7fe640492640 1 -- 172.21.15.129:0/1541926177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6381a14b0 con 0x7fe638102380 2024-05-20T17:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.485+0000 7fe640492640 1 -- 172.21.15.129:0/1541926177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe638105e90 con 0x7fe638102380 2024-05-20T17:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.486+0000 7fe62f7fe640 1 -- 172.21.15.129:0/1541926177 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fe628004fe0 con 0x7fe638102380 2024-05-20T17:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.487+0000 7fe62f7fe640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe6240a2b90 0x7fe6240a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.487+0000 7fe63e22e640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe6240a2b90 0x7fe6240a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.487+0000 7fe62f7fe640 1 -- 172.21.15.129:0/1541926177 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fe6280cf120 con 0x7fe638102380 2024-05-20T17:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.487+0000 7fe63e22e640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe6240a2b90 0x7fe6240a50a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe634005db0 tx=0x7fe634005d40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:50.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.491+0000 7fe62f7fe640 1 -- 172.21.15.129:0/1541926177 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe628014030 con 0x7fe638102380 2024-05-20T17:56:50.662 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:56:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:56:50.336+0000 7f0c68b63700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:56:50.663 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:56:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:56:50.336+0000 7f0c68b63700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:56:50.663 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:56:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:56:50.336+0000 7f0c68b63700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:56:50.663 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:56:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:56:50.336+0000 7f0c68b63700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:56:50.663 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:56:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:56:50.336+0000 7f0c68b63700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:56:50.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.695+0000 7fe640492640 1 -- 172.21.15.129:0/1541926177 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe638061f90 con 0x7fe6240a2b90 2024-05-20T17:56:50.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.724+0000 7fe62f7fe640 1 -- 172.21.15.129:0/1541926177 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe638061f90 con 0x7fe6240a2b90 2024-05-20T17:56:50.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 -- 172.21.15.129:0/1541926177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe6240a2b90 msgr2=0x7fe6240a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe6240a2b90 0x7fe6240a50a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe634005db0 tx=0x7fe634005d40 comp rx=0 tx=0).stop 2024-05-20T17:56:50.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 -- 172.21.15.129:0/1541926177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe638102380 msgr2=0x7fe63819dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe638102380 0x7fe63819dfc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe62800c970 tx=0x7fe62800ce30 comp rx=0 tx=0).stop 2024-05-20T17:56:50.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 -- 172.21.15.129:0/1541926177 shutdown_connections 2024-05-20T17:56:50.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fe6240a2b90 0x7fe6240a50a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe638107080 0x7fe6381a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe638102380 0x7fe63819dfc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 --2- 172.21.15.129:0/1541926177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe638101140 0x7fe63810a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 -- 172.21.15.129:0/1541926177 >> 172.21.15.129:0/1541926177 conn(0x7fe6380fc330 msgr2=0x7fe638103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:50.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 -- 172.21.15.129:0/1541926177 shutdown_connections 2024-05-20T17:56:50.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.726+0000 7fe640492640 1 -- 172.21.15.129:0/1541926177 wait complete. 2024-05-20T17:56:50.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-20T17:56:50.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.854+0000 7f913553e640 1 -- 172.21.15.129:0/1733346364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9130101200 msgr2=0x7f913010ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.854+0000 7f913553e640 1 --2- 172.21.15.129:0/1733346364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9130101200 0x7f913010ac60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9128009ca0 tx=0x7f912802a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.854+0000 7f913553e640 1 -- 172.21.15.129:0/1733346364 shutdown_connections 2024-05-20T17:56:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.854+0000 7f913553e640 1 --2- 172.21.15.129:0/1733346364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9130101200 0x7f913010ac60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.854+0000 7f913553e640 1 --2- 172.21.15.129:0/1733346364 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9130100700 0x7f9130100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.854+0000 7f913553e640 1 --2- 172.21.15.129:0/1733346364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9130104050 0x7f9130104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.854+0000 7f913553e640 1 -- 172.21.15.129:0/1733346364 >> 172.21.15.129:0/1733346364 conn(0x7f91300fc300 msgr2=0x7f91300fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.854+0000 7f913553e640 1 -- 172.21.15.129:0/1733346364 shutdown_connections 2024-05-20T17:56:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.854+0000 7f913553e640 1 -- 172.21.15.129:0/1733346364 wait complete. 2024-05-20T17:56:50.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f913553e640 1 Processor -- start 2024-05-20T17:56:50.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f913553e640 1 -- start start 2024-05-20T17:56:50.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f913553e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9130100700 0x7f91301991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f913553e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9130101200 0x7f9130199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f913553e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9130104050 0x7f91301a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f913553e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f913010e710 con 0x7f9130100700 2024-05-20T17:56:50.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f913553e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f913010e590 con 0x7f9130104050 2024-05-20T17:56:50.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f913553e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f913010e890 con 0x7f9130101200 2024-05-20T17:56:50.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f912effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9130100700 0x7f91301991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:50.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f912effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9130100700 0x7f91301991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46646/0 (socket says 172.21.15.129:46646) 2024-05-20T17:56:50.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f912effd640 1 -- 172.21.15.129:0/670395896 learned_addr learned my addr 172.21.15.129:0/670395896 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:50.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.855+0000 7f912e7fc640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9130101200 0x7f9130199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f912f7fe640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9130104050 0x7f91301a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f912effd640 1 -- 172.21.15.129:0/670395896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9130101200 msgr2=0x7f9130199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f912effd640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9130101200 0x7f9130199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f912effd640 1 -- 172.21.15.129:0/670395896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9130104050 msgr2=0x7f91301a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f912effd640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9130104050 0x7f91301a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f912effd640 1 -- 172.21.15.129:0/670395896 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91301a0d30 con 0x7f9130100700 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f912f7fe640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9130104050 0x7f91301a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f912effd640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9130100700 0x7f91301991f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f912000d740 tx=0x7f912000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f9117fff640 1 -- 172.21.15.129:0/670395896 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120017070 con 0x7f9130100700 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f913553e640 1 -- 172.21.15.129:0/670395896 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91301a0fc0 con 0x7f9130100700 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f9117fff640 1 -- 172.21.15.129:0/670395896 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f912000de50 con 0x7f9130100700 2024-05-20T17:56:50.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f9117fff640 1 -- 172.21.15.129:0/670395896 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120003e00 con 0x7f9130100700 2024-05-20T17:56:50.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.856+0000 7f913553e640 1 -- 172.21.15.129:0/670395896 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91301a14d0 con 0x7f9130100700 2024-05-20T17:56:50.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.857+0000 7f913553e640 1 -- 172.21.15.129:0/670395896 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9130069110 con 0x7f9130100700 2024-05-20T17:56:50.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.858+0000 7f9117fff640 1 -- 172.21.15.129:0/670395896 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9120003fa0 con 0x7f9130100700 2024-05-20T17:56:50.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.859+0000 7f9117fff640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f91180a2b90 0x7f91180a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:50.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.859+0000 7f912e7fc640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f91180a2b90 0x7f91180a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:50.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.859+0000 7f9117fff640 1 -- 172.21.15.129:0/670395896 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f91200ce8b0 con 0x7f9130100700 2024-05-20T17:56:50.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.860+0000 7f912e7fc640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f91180a2b90 0x7f91180a50a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f91240058c0 tx=0x7f912400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:50.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:50.864+0000 7f9117fff640 1 -- 172.21.15.129:0/670395896 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f912009b960 con 0x7f9130100700 2024-05-20T17:56:50.893 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 17:56:50 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b[66692]: 2024-05-20T17:56:50.459+0000 7f511f1d3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-20T17:56:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.075+0000 7f913553e640 1 -- 172.21.15.129:0/670395896 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f9130062020 con 0x7f91180a2b90 2024-05-20T17:56:51.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.080+0000 7f9117fff640 1 -- 172.21.15.129:0/670395896 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f9130062020 con 0x7f91180a2b90 2024-05-20T17:56:51.079 INFO:teuthology.orchestra.run.smithi129.stdout:/volumes/qa/sv_0/168e5759-0b03-4628-beb5-607ce0af66ac 2024-05-20T17:56:51.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 -- 172.21.15.129:0/670395896 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f91180a2b90 msgr2=0x7f91180a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f91180a2b90 0x7f91180a50a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f91240058c0 tx=0x7f912400a2d0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 -- 172.21.15.129:0/670395896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9130100700 msgr2=0x7f91301991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9130100700 0x7f91301991f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f912000d740 tx=0x7f912000dc00 comp rx=0 tx=0).stop 2024-05-20T17:56:51.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 -- 172.21.15.129:0/670395896 shutdown_connections 2024-05-20T17:56:51.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f91180a2b90 0x7f91180a50a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9130104050 0x7f91301a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9130101200 0x7f9130199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 --2- 172.21.15.129:0/670395896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9130100700 0x7f91301991f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 -- 172.21.15.129:0/670395896 >> 172.21.15.129:0/670395896 conn(0x7f91300fc300 msgr2=0x7f9130106bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:51.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 -- 172.21.15.129:0/670395896 shutdown_connections 2024-05-20T17:56:51.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.082+0000 7f913553e640 1 -- 172.21.15.129:0/670395896 wait complete. 2024-05-20T17:56:51.092 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-20T17:56:51.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.210+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1454752336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee0103ce0 msgr2=0x7f9ee01040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.210+0000 7f9ee6fce640 1 --2- 172.21.15.129:0/1454752336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee0103ce0 0x7f9ee01040f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009900 tx=0x7f9ecc02a410 comp rx=0 tx=0).stop 2024-05-20T17:56:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.210+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1454752336 shutdown_connections 2024-05-20T17:56:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.210+0000 7f9ee6fce640 1 --2- 172.21.15.129:0/1454752336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ee01089e0 0x7f9ee010ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.210+0000 7f9ee6fce640 1 --2- 172.21.15.129:0/1454752336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee0103ce0 0x7f9ee01040f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.210+0000 7f9ee6fce640 1 --2- 172.21.15.129:0/1454752336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0102aa0 0x7f9ee01037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.210+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1454752336 >> 172.21.15.129:0/1454752336 conn(0x7f9ee00fdc90 msgr2=0x7f9ee0100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.210+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1454752336 shutdown_connections 2024-05-20T17:56:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.210+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1454752336 wait complete. 2024-05-20T17:56:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9ee6fce640 1 Processor -- start 2024-05-20T17:56:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9ee6fce640 1 -- start start 2024-05-20T17:56:51.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9ee6fce640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0102aa0 0x7f9ee0199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9ee6fce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee0103ce0 0x7f9ee01997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9ee6fce640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ee01089e0 0x7f9ee01a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9ee6fce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee010e720 con 0x7f9ee0103ce0 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9ee6fce640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee010e5a0 con 0x7f9ee0102aa0 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9ee6fce640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee010e8a0 con 0x7f9ee01089e0 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9edffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee0103ce0 0x7f9ee01997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9edffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee0103ce0 0x7f9ee01997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46664/0 (socket says 172.21.15.129:46664) 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9edffff640 1 -- 172.21.15.129:0/1721534380 learned_addr learned my addr 172.21.15.129:0/1721534380 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9ee4d6a640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0102aa0 0x7f9ee0199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9edffff640 1 -- 172.21.15.129:0/1721534380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ee01089e0 msgr2=0x7f9ee01a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9edffff640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ee01089e0 0x7f9ee01a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9edffff640 1 -- 172.21.15.129:0/1721534380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0102aa0 msgr2=0x7f9ee0199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9edffff640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0102aa0 0x7f9ee0199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9edffff640 1 -- 172.21.15.129:0/1721534380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee01a0dd0 con 0x7f9ee0103ce0 2024-05-20T17:56:51.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9edffff640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee0103ce0 0x7f9ee01997d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc006010 tx=0x7f9ecc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:51.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9eddffb640 1 -- 172.21.15.129:0/1721534380 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc03f030 con 0x7f9ee0103ce0 2024-05-20T17:56:51.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9eddffb640 1 -- 172.21.15.129:0/1721534380 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ecc003ce0 con 0x7f9ee0103ce0 2024-05-20T17:56:51.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.211+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1721534380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee01a1060 con 0x7f9ee0103ce0 2024-05-20T17:56:51.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.212+0000 7f9eddffb640 1 -- 172.21.15.129:0/1721534380 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc035070 con 0x7f9ee0103ce0 2024-05-20T17:56:51.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.212+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1721534380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee01a1550 con 0x7f9ee0103ce0 2024-05-20T17:56:51.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.213+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1721534380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee01077f0 con 0x7f9ee0103ce0 2024-05-20T17:56:51.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.214+0000 7f9eddffb640 1 -- 172.21.15.129:0/1721534380 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9ecc042030 con 0x7f9ee0103ce0 2024-05-20T17:56:51.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.214+0000 7f9eddffb640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9ed00a2b90 0x7f9ed00a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.215+0000 7f9ee4d6a640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9ed00a2b90 0x7f9ed00a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:51.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.215+0000 7f9eddffb640 1 -- 172.21.15.129:0/1721534380 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f9ecc0efbb0 con 0x7f9ee0103ce0 2024-05-20T17:56:51.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.215+0000 7f9ee4d6a640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9ed00a2b90 0x7f9ed00a50a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4005fd0 tx=0x7f9ed4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:51.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.218+0000 7f9eddffb640 1 -- 172.21.15.129:0/1721534380 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9ecc0bcc60 con 0x7f9ee0103ce0 2024-05-20T17:56:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:50 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3866157065' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-20T17:56:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:50 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T17:56:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:50 smithi154 ceph-mon[36482]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:50 smithi154 ceph-mon[36482]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:50 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3866157065' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-20T17:56:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:50 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T17:56:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:50 smithi129 ceph-mon[32029]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:50 smithi129 ceph-mon[32029]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.423+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1721534380 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9ee0061f90 con 0x7f9ed00a2b90 2024-05-20T17:56:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:50 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3866157065' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-20T17:56:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:50 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T17:56:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:50 smithi204 ceph-mon[37076]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:50 smithi204 ceph-mon[37076]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:51.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.452+0000 7f9eddffb640 1 -- 172.21.15.129:0/1721534380 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9ee0061f90 con 0x7f9ed00a2b90 2024-05-20T17:56:51.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1721534380 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9ed00a2b90 msgr2=0x7f9ed00a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9ed00a2b90 0x7f9ed00a50a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4005fd0 tx=0x7f9ed4005850 comp rx=0 tx=0).stop 2024-05-20T17:56:51.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1721534380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee0103ce0 msgr2=0x7f9ee01997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee0103ce0 0x7f9ee01997d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc006010 tx=0x7f9ecc004360 comp rx=0 tx=0).stop 2024-05-20T17:56:51.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1721534380 shutdown_connections 2024-05-20T17:56:51.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9ed00a2b90 0x7f9ed00a50a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ee01089e0 0x7f9ee01a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee0103ce0 0x7f9ee01997d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 --2- 172.21.15.129:0/1721534380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0102aa0 0x7f9ee0199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1721534380 >> 172.21.15.129:0/1721534380 conn(0x7f9ee00fdc90 msgr2=0x7f9ee01049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:51.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1721534380 shutdown_connections 2024-05-20T17:56:51.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.454+0000 7f9ee6fce640 1 -- 172.21.15.129:0/1721534380 wait complete. 2024-05-20T17:56:51.464 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-20T17:56:51.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.582+0000 7f9c525ce640 1 -- 172.21.15.129:0/372619791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c4c1078e0 msgr2=0x7f9c4c109d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.582+0000 7f9c525ce640 1 --2- 172.21.15.129:0/372619791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c4c1078e0 0x7f9c4c109d00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009ca0 tx=0x7f9c4402a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.582+0000 7f9c525ce640 1 -- 172.21.15.129:0/372619791 shutdown_connections 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.582+0000 7f9c525ce640 1 --2- 172.21.15.129:0/372619791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c4c1078e0 0x7f9c4c109d00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.582+0000 7f9c525ce640 1 --2- 172.21.15.129:0/372619791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c4c102be0 0x7f9c4c102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.582+0000 7f9c525ce640 1 --2- 172.21.15.129:0/372619791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c4c1019a0 0x7f9c4c1026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.582+0000 7f9c525ce640 1 -- 172.21.15.129:0/372619791 >> 172.21.15.129:0/372619791 conn(0x7f9c4c0fcbb0 msgr2=0x7f9c4c0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.582+0000 7f9c525ce640 1 -- 172.21.15.129:0/372619791 shutdown_connections 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.582+0000 7f9c525ce640 1 -- 172.21.15.129:0/372619791 wait complete. 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.582+0000 7f9c525ce640 1 Processor -- start 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.582+0000 7f9c525ce640 1 -- start start 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c525ce640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c4c1019a0 0x7f9c4c199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c525ce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c4c102be0 0x7f9c4c199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c525ce640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c4c1078e0 0x7f9c4c1a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c525ce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c10e780 con 0x7f9c4c102be0 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c525ce640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c10e600 con 0x7f9c4c1078e0 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c525ce640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c10e900 con 0x7f9c4c1019a0 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c4b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c4c102be0 0x7f9c4c199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c4b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c4c102be0 0x7f9c4c199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46670/0 (socket says 172.21.15.129:46670) 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c4b7fe640 1 -- 172.21.15.129:0/980915412 learned_addr learned my addr 172.21.15.129:0/980915412 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c50b6b640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c4c1078e0 0x7f9c4c1a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c4bfff640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c4c1019a0 0x7f9c4c199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c4b7fe640 1 -- 172.21.15.129:0/980915412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c4c1019a0 msgr2=0x7f9c4c199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c4b7fe640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c4c1019a0 0x7f9c4c199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c4b7fe640 1 -- 172.21.15.129:0/980915412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c4c1078e0 msgr2=0x7f9c4c1a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c4b7fe640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c4c1078e0 0x7f9c4c1a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c4b7fe640 1 -- 172.21.15.129:0/980915412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4c1a0d90 con 0x7f9c4c102be0 2024-05-20T17:56:51.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c50b6b640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c4c1078e0 0x7f9c4c1a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:56:51.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c4b7fe640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c4c102be0 0x7f9c4c199790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000d770 tx=0x7f9c4000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:51.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.583+0000 7f9c497fa640 1 -- 172.21.15.129:0/980915412 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c40017070 con 0x7f9c4c102be0 2024-05-20T17:56:51.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.584+0000 7f9c497fa640 1 -- 172.21.15.129:0/980915412 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c40005500 con 0x7f9c4c102be0 2024-05-20T17:56:51.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.584+0000 7f9c525ce640 1 -- 172.21.15.129:0/980915412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c4c1a1080 con 0x7f9c4c102be0 2024-05-20T17:56:51.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.584+0000 7f9c497fa640 1 -- 172.21.15.129:0/980915412 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c40004e00 con 0x7f9c4c102be0 2024-05-20T17:56:51.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.584+0000 7f9c525ce640 1 -- 172.21.15.129:0/980915412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c4c1a1580 con 0x7f9c4c102be0 2024-05-20T17:56:51.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.585+0000 7f9c525ce640 1 -- 172.21.15.129:0/980915412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c4c1066f0 con 0x7f9c4c102be0 2024-05-20T17:56:51.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.586+0000 7f9c497fa640 1 -- 172.21.15.129:0/980915412 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9c40004fa0 con 0x7f9c4c102be0 2024-05-20T17:56:51.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.587+0000 7f9c497fa640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9c340a2b90 0x7f9c340a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.587+0000 7f9c497fa640 1 -- 172.21.15.129:0/980915412 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f9c400cee50 con 0x7f9c4c102be0 2024-05-20T17:56:51.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.587+0000 7f9c4bfff640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9c340a2b90 0x7f9c340a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:51.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.587+0000 7f9c4bfff640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9c340a2b90 0x7f9c340a50a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c005fd0 tx=0x7f9c3c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:51.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.590+0000 7f9c497fa640 1 -- 172.21.15.129:0/980915412 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9c4009bf00 con 0x7f9c4c102be0 2024-05-20T17:56:51.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.796+0000 7f9c525ce640 1 -- 172.21.15.129:0/980915412 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f9c4c062020 con 0x7f9c340a2b90 2024-05-20T17:56:51.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.801+0000 7f9c497fa640 1 -- 172.21.15.129:0/980915412 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f9c4c062020 con 0x7f9c340a2b90 2024-05-20T17:56:51.800 INFO:teuthology.orchestra.run.smithi129.stdout:/volumes/qa/sv_1/dab013c0-2723-47a5-b465-c3539a792f5b 2024-05-20T17:56:51.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 -- 172.21.15.129:0/980915412 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9c340a2b90 msgr2=0x7f9c340a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9c340a2b90 0x7f9c340a50a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c005fd0 tx=0x7f9c3c005850 comp rx=0 tx=0).stop 2024-05-20T17:56:51.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 -- 172.21.15.129:0/980915412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c4c102be0 msgr2=0x7f9c4c199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c4c102be0 0x7f9c4c199790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000d770 tx=0x7f9c4000dc30 comp rx=0 tx=0).stop 2024-05-20T17:56:51.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 -- 172.21.15.129:0/980915412 shutdown_connections 2024-05-20T17:56:51.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9c340a2b90 0x7f9c340a50a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c4c1078e0 0x7f9c4c1a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c4c102be0 0x7f9c4c199790 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 --2- 172.21.15.129:0/980915412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9c4c1019a0 0x7f9c4c199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 -- 172.21.15.129:0/980915412 >> 172.21.15.129:0/980915412 conn(0x7f9c4c0fcbb0 msgr2=0x7f9c4c1038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:51.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 -- 172.21.15.129:0/980915412 shutdown_connections 2024-05-20T17:56:51.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.803+0000 7f9c525ce640 1 -- 172.21.15.129:0/980915412 wait complete. 2024-05-20T17:56:51.813 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:56:51.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.925+0000 7f0425738640 1 -- 172.21.15.129:0/2973422293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0420101110 msgr2=0x7f0420101e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.925+0000 7f0425738640 1 --2- 172.21.15.129:0/2973422293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0420101110 0x7f0420101e10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0410009ca0 tx=0x7f041002a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.925+0000 7f0425738640 1 -- 172.21.15.129:0/2973422293 shutdown_connections 2024-05-20T17:56:51.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.925+0000 7f0425738640 1 --2- 172.21.15.129:0/2973422293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0420102350 0x7f0420102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.925+0000 7f0425738640 1 --2- 172.21.15.129:0/2973422293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0420107050 0x7f0420109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.925+0000 7f0425738640 1 --2- 172.21.15.129:0/2973422293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0420101110 0x7f0420101e10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.925+0000 7f0425738640 1 -- 172.21.15.129:0/2973422293 >> 172.21.15.129:0/2973422293 conn(0x7f04200fc2e0 msgr2=0x7f04200fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.925+0000 7f0425738640 1 -- 172.21.15.129:0/2973422293 shutdown_connections 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.925+0000 7f0425738640 1 -- 172.21.15.129:0/2973422293 wait complete. 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.925+0000 7f0425738640 1 Processor -- start 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f0425738640 1 -- start start 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f0425738640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0420101110 0x7f0420108f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f0425738640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0420102350 0x7f042019e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f0425738640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0420107050 0x7f04201a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f0425738640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f042010cd80 con 0x7f0420107050 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f0425738640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f042010cc00 con 0x7f0420102350 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f0425738640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f042010cf00 con 0x7f0420101110 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f041f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0420107050 0x7f04201a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:51.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f041f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0420107050 0x7f04201a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46688/0 (socket says 172.21.15.129:46688) 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f041f7fe640 1 -- 172.21.15.129:0/2949988027 learned_addr learned my addr 172.21.15.129:0/2949988027 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f041e7fc640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0420102350 0x7f042019e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f041effd640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0420101110 0x7f0420108f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f041f7fe640 1 -- 172.21.15.129:0/2949988027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0420101110 msgr2=0x7f0420108f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f041f7fe640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0420101110 0x7f0420108f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f041f7fe640 1 -- 172.21.15.129:0/2949988027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0420102350 msgr2=0x7f042019e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f041f7fe640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0420102350 0x7f042019e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f041f7fe640 1 -- 172.21.15.129:0/2949988027 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04201a0e10 con 0x7f0420107050 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f041f7fe640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0420107050 0x7f04201a05f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f041800d750 tx=0x7f041800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.926+0000 7f0407fff640 1 -- 172.21.15.129:0/2949988027 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0418017070 con 0x7f0420107050 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.927+0000 7f0407fff640 1 -- 172.21.15.129:0/2949988027 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0418005500 con 0x7f0420107050 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.927+0000 7f0425738640 1 -- 172.21.15.129:0/2949988027 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04201a1100 con 0x7f0420107050 2024-05-20T17:56:51.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.927+0000 7f0407fff640 1 -- 172.21.15.129:0/2949988027 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0418004d90 con 0x7f0420107050 2024-05-20T17:56:51.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.927+0000 7f0425738640 1 -- 172.21.15.129:0/2949988027 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04201a1650 con 0x7f0420107050 2024-05-20T17:56:51.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.928+0000 7f0425738640 1 -- 172.21.15.129:0/2949988027 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0420106750 con 0x7f0420107050 2024-05-20T17:56:51.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.929+0000 7f0407fff640 1 -- 172.21.15.129:0/2949988027 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0418004f30 con 0x7f0420107050 2024-05-20T17:56:51.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.930+0000 7f0407fff640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04080a2b90 0x7f04080a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:51.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.930+0000 7f041effd640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04080a2b90 0x7f04080a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:51.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.930+0000 7f0407fff640 1 -- 172.21.15.129:0/2949988027 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f04180ce8c0 con 0x7f0420107050 2024-05-20T17:56:51.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.930+0000 7f041effd640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04080a2b90 0x7f04080a50a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0410006010 tx=0x7f0410000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:51.934+0000 7f0407fff640 1 -- 172.21.15.129:0/2949988027 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f041809b970 con 0x7f0420107050 2024-05-20T17:56:52.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.017+0000 7f0407fff640 1 -- 172.21.15.129:0/2949988027 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f04180978e0 con 0x7f0420107050 2024-05-20T17:56:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.160+0000 7f0425738640 1 -- 172.21.15.129:0/2949988027 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f042002d2a0 con 0x7f0420107050 2024-05-20T17:56:52.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.161+0000 7f0407fff640 1 -- 172.21.15.129:0/2949988027 <== mon.0 v2:172.21.15.129:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11829 (secure 0 0 0) 0x7f041800de60 con 0x7f0420107050 2024-05-20T17:56:52.159 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:52.160 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:56:49.336282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T17:56:52.160 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 20 2024-05-20T17:56:52.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 -- 172.21.15.129:0/2949988027 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04080a2b90 msgr2=0x7f04080a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04080a2b90 0x7f04080a50a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0410006010 tx=0x7f0410000f40 comp rx=0 tx=0).stop 2024-05-20T17:56:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 -- 172.21.15.129:0/2949988027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0420107050 msgr2=0x7f04201a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0420107050 0x7f04201a05f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f041800d750 tx=0x7f041800dc10 comp rx=0 tx=0).stop 2024-05-20T17:56:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 -- 172.21.15.129:0/2949988027 shutdown_connections 2024-05-20T17:56:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04080a2b90 0x7f04080a50a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0420102350 0x7f042019e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0420107050 0x7f04201a05f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 --2- 172.21.15.129:0/2949988027 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0420101110 0x7f0420108f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 -- 172.21.15.129:0/2949988027 >> 172.21.15.129:0/2949988027 conn(0x7f04200fc2e0 msgr2=0x7f04200fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 -- 172.21.15.129:0/2949988027 shutdown_connections 2024-05-20T17:56:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.163+0000 7f0425738640 1 -- 172.21.15.129:0/2949988027 wait complete. 2024-05-20T17:56:52.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:52 smithi129 ceph-mon[32029]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:52.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:52 smithi129 ceph-mon[32029]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:52.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:52 smithi129 ceph-mon[32029]: pgmap v288: 97 pgs: 97 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 146 B/s rd, 2.0 KiB/s wr, 6 op/s 2024-05-20T17:56:52.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:56:52.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.285+0000 7fae089b5640 1 -- 172.21.15.129:0/95536901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae04108710 msgr2=0x7fae0410ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.285+0000 7fae089b5640 1 --2- 172.21.15.129:0/95536901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae04108710 0x7fae0410ac20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009ca0 tx=0x7fadfc02a8c0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.285+0000 7fae089b5640 1 -- 172.21.15.129:0/95536901 shutdown_connections 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.285+0000 7fae089b5640 1 --2- 172.21.15.129:0/95536901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae04108710 0x7fae0410ac20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.285+0000 7fae089b5640 1 --2- 172.21.15.129:0/95536901 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae041012f0 0x7fae04108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.285+0000 7fae089b5640 1 --2- 172.21.15.129:0/95536901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae04100df0 0x7fae04105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.285+0000 7fae089b5640 1 -- 172.21.15.129:0/95536901 >> 172.21.15.129:0/95536901 conn(0x7fae040fc2e0 msgr2=0x7fae040fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.285+0000 7fae089b5640 1 -- 172.21.15.129:0/95536901 shutdown_connections 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.285+0000 7fae089b5640 1 -- 172.21.15.129:0/95536901 wait complete. 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.286+0000 7fae089b5640 1 Processor -- start 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.286+0000 7fae089b5640 1 -- start start 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.286+0000 7fae089b5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae04100df0 0x7fae041991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.286+0000 7fae089b5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae041012f0 0x7fae04199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.286+0000 7fae089b5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae04108710 0x7fae041a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.286+0000 7fae089b5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fae0410e6d0 con 0x7fae04108710 2024-05-20T17:56:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.286+0000 7fae089b5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae0410e550 con 0x7fae04100df0 2024-05-20T17:56:52.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.286+0000 7fae089b5640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fae0410e850 con 0x7fae041012f0 2024-05-20T17:56:52.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae02d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae04108710 0x7fae041a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae02d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae04108710 0x7fae041a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46698/0 (socket says 172.21.15.129:46698) 2024-05-20T17:56:52.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae02d9d640 1 -- 172.21.15.129:0/2375148432 learned_addr learned my addr 172.21.15.129:0/2375148432 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:52.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fadfbfff640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae041012f0 0x7fae04199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae0259c640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae04100df0 0x7fae041991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae02d9d640 1 -- 172.21.15.129:0/2375148432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae041012f0 msgr2=0x7fae04199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae02d9d640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae041012f0 0x7fae04199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae02d9d640 1 -- 172.21.15.129:0/2375148432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae04100df0 msgr2=0x7fae041991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae02d9d640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae04100df0 0x7fae041991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae02d9d640 1 -- 172.21.15.129:0/2375148432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae041a0c70 con 0x7fae04108710 2024-05-20T17:56:52.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae02d9d640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae04108710 0x7fae041a06b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fadfc005cc0 tx=0x7fadfc009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:52.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fadfb7fe640 1 -- 172.21.15.129:0/2375148432 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc003c30 con 0x7fae04108710 2024-05-20T17:56:52.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fadfb7fe640 1 -- 172.21.15.129:0/2375148432 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadfc003dd0 con 0x7fae04108710 2024-05-20T17:56:52.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae089b5640 1 -- 172.21.15.129:0/2375148432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae041a0f00 con 0x7fae04108710 2024-05-20T17:56:52.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fadfb7fe640 1 -- 172.21.15.129:0/2375148432 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadfc02fc50 con 0x7fae04108710 2024-05-20T17:56:52.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.287+0000 7fae089b5640 1 -- 172.21.15.129:0/2375148432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae041a1360 con 0x7fae04108710 2024-05-20T17:56:52.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.288+0000 7fae089b5640 1 -- 172.21.15.129:0/2375148432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae040682c0 con 0x7fae04108710 2024-05-20T17:56:52.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.290+0000 7fadfb7fe640 1 -- 172.21.15.129:0/2375148432 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fadfc004050 con 0x7fae04108710 2024-05-20T17:56:52.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.290+0000 7fadfb7fe640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fadf00a2c20 0x7fadf00a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.290+0000 7fae0259c640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fadf00a2c20 0x7fadf00a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.290+0000 7fadfb7fe640 1 -- 172.21.15.129:0/2375148432 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fadfc0f05b0 con 0x7fae04108710 2024-05-20T17:56:52.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.291+0000 7fae0259c640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fadf00a2c20 0x7fadf00a5130 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fadf4005e10 tx=0x7fadf4005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:52.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.294+0000 7fadfb7fe640 1 -- 172.21.15.129:0/2375148432 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fadfc0bd670 con 0x7fae04108710 2024-05-20T17:56:52.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:52 smithi154 ceph-mon[36482]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:52.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:52 smithi154 ceph-mon[36482]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:52.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:52 smithi154 ceph-mon[36482]: pgmap v288: 97 pgs: 97 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 146 B/s rd, 2.0 KiB/s wr, 6 op/s 2024-05-20T17:56:52.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:52 smithi204 ceph-mon[37076]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:52.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:52 smithi204 ceph-mon[37076]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:52.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:52 smithi204 ceph-mon[37076]: pgmap v288: 97 pgs: 97 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 146 B/s rd, 2.0 KiB/s wr, 6 op/s 2024-05-20T17:56:52.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.470+0000 7fae089b5640 1 -- 172.21.15.129:0/2375148432 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae0402d370 con 0x7fae04108710 2024-05-20T17:56:52.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.472+0000 7fadfb7fe640 1 -- 172.21.15.129:0/2375148432 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22240 (secure 0 0 0) 0x7fadfc0bcf70 con 0x7fae04108710 2024-05-20T17:56:52.471 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:52.471 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":86,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:56:46.728647+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 -- 172.21.15.129:0/2375148432 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fadf00a2c20 msgr2=0x7fadf00a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fadf00a2c20 0x7fadf00a5130 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fadf4005e10 tx=0x7fadf4005da0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 -- 172.21.15.129:0/2375148432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae04108710 msgr2=0x7fae041a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae04108710 0x7fae041a06b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fadfc005cc0 tx=0x7fadfc009890 comp rx=0 tx=0).stop 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 -- 172.21.15.129:0/2375148432 shutdown_connections 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fadf00a2c20 0x7fadf00a5130 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae04108710 0x7fae041a06b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae041012f0 0x7fae04199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 --2- 172.21.15.129:0/2375148432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae04100df0 0x7fae041991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 -- 172.21.15.129:0/2375148432 >> 172.21.15.129:0/2375148432 conn(0x7fae040fc2e0 msgr2=0x7fae04104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 -- 172.21.15.129:0/2375148432 shutdown_connections 2024-05-20T17:56:52.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.474+0000 7fae089b5640 1 -- 172.21.15.129:0/2375148432 wait complete. 2024-05-20T17:56:52.485 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-20T17:56:52.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-20T17:56:52.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.598+0000 7f5ee38de640 1 -- 172.21.15.129:0/492176130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc103cf0 msgr2=0x7f5edc104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.598+0000 7f5ee38de640 1 --2- 172.21.15.129:0/492176130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc103cf0 0x7f5edc104100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8009900 tx=0x7f5ed802a410 comp rx=0 tx=0).stop 2024-05-20T17:56:52.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.598+0000 7f5ee38de640 1 -- 172.21.15.129:0/492176130 shutdown_connections 2024-05-20T17:56:52.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.598+0000 7f5ee38de640 1 --2- 172.21.15.129:0/492176130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5edc1089f0 0x7f5edc10ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.598+0000 7f5ee38de640 1 --2- 172.21.15.129:0/492176130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc103cf0 0x7f5edc104100 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.598+0000 7f5ee38de640 1 --2- 172.21.15.129:0/492176130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5edc102ab0 0x7f5edc1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.598+0000 7f5ee38de640 1 -- 172.21.15.129:0/492176130 >> 172.21.15.129:0/492176130 conn(0x7f5edc0fdcc0 msgr2=0x7f5edc100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:52.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.598+0000 7f5ee38de640 1 -- 172.21.15.129:0/492176130 shutdown_connections 2024-05-20T17:56:52.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.598+0000 7f5ee38de640 1 -- 172.21.15.129:0/492176130 wait complete. 2024-05-20T17:56:52.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee38de640 1 Processor -- start 2024-05-20T17:56:52.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee38de640 1 -- start start 2024-05-20T17:56:52.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee38de640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5edc102ab0 0x7f5edc199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee38de640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5edc103cf0 0x7f5edc1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee38de640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc1089f0 0x7f5edc1a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee38de640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc10e780 con 0x7f5edc1089f0 2024-05-20T17:56:52.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee38de640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc10e600 con 0x7f5edc102ab0 2024-05-20T17:56:52.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee38de640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc10e900 con 0x7f5edc103cf0 2024-05-20T17:56:52.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee1e7b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc1089f0 0x7f5edc1a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee1e7b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc1089f0 0x7f5edc1a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46716/0 (socket says 172.21.15.129:46716) 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee1e7b640 1 -- 172.21.15.129:0/1940700811 learned_addr learned my addr 172.21.15.129:0/1940700811 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee0e79640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5edc103cf0 0x7f5edc1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee167a640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5edc102ab0 0x7f5edc199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee1e7b640 1 -- 172.21.15.129:0/1940700811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5edc103cf0 msgr2=0x7f5edc1997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee1e7b640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5edc103cf0 0x7f5edc1997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee1e7b640 1 -- 172.21.15.129:0/1940700811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5edc102ab0 msgr2=0x7f5edc199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee1e7b640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5edc102ab0 0x7f5edc199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.599+0000 7f5ee1e7b640 1 -- 172.21.15.129:0/1940700811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5edc1a0dd0 con 0x7f5edc1089f0 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.600+0000 7f5ee1e7b640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc1089f0 0x7f5edc1a0760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5ed400eb10 tx=0x7f5ed400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.600+0000 7f5ed27fc640 1 -- 172.21.15.129:0/1940700811 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed4004b20 con 0x7f5edc1089f0 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.600+0000 7f5ed27fc640 1 -- 172.21.15.129:0/1940700811 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ed4004cc0 con 0x7f5edc1089f0 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.600+0000 7f5ee38de640 1 -- 172.21.15.129:0/1940700811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5edc1a10c0 con 0x7f5edc1089f0 2024-05-20T17:56:52.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.600+0000 7f5ed27fc640 1 -- 172.21.15.129:0/1940700811 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed4010760 con 0x7f5edc1089f0 2024-05-20T17:56:52.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.600+0000 7f5ee38de640 1 -- 172.21.15.129:0/1940700811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5edc1a1610 con 0x7f5edc1089f0 2024-05-20T17:56:52.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.601+0000 7f5ee38de640 1 -- 172.21.15.129:0/1940700811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5edc107800 con 0x7f5edc1089f0 2024-05-20T17:56:52.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.603+0000 7f5ed27fc640 1 -- 172.21.15.129:0/1940700811 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f5ed4005000 con 0x7f5edc1089f0 2024-05-20T17:56:52.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.603+0000 7f5ed27fc640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5ec80a2c20 0x7f5ec80a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.603+0000 7f5ee167a640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5ec80a2c20 0x7f5ec80a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.603+0000 7f5ed27fc640 1 -- 172.21.15.129:0/1940700811 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f5ed40cfb90 con 0x7f5edc1089f0 2024-05-20T17:56:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.604+0000 7f5ee167a640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5ec80a2c20 0x7f5ec80a5130 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc005fd0 tx=0x7f5ecc005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:52.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.607+0000 7f5ed27fc640 1 -- 172.21.15.129:0/1940700811 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5ed409cb40 con 0x7f5edc1089f0 2024-05-20T17:56:52.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.783+0000 7f5ee38de640 1 -- 172.21.15.129:0/1940700811 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5edc061f90 con 0x7f5ec80a2c20 2024-05-20T17:56:52.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.791+0000 7f5ed27fc640 1 -- 172.21.15.129:0/1940700811 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258857 (secure 0 0 0) 0x7f5edc061f90 con 0x7f5ec80a2c20 2024-05-20T17:56:52.793 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:52.794 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-05-20T17:56:52.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 -- 172.21.15.129:0/1940700811 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5ec80a2c20 msgr2=0x7f5ec80a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5ec80a2c20 0x7f5ec80a5130 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc005fd0 tx=0x7f5ecc005850 comp rx=0 tx=0).stop 2024-05-20T17:56:52.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 -- 172.21.15.129:0/1940700811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc1089f0 msgr2=0x7f5edc1a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc1089f0 0x7f5edc1a0760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5ed400eb10 tx=0x7f5ed400c2d0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 -- 172.21.15.129:0/1940700811 shutdown_connections 2024-05-20T17:56:52.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5ec80a2c20 0x7f5ec80a5130 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc1089f0 0x7f5edc1a0760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5edc103cf0 0x7f5edc1997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 --2- 172.21.15.129:0/1940700811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5edc102ab0 0x7f5edc199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 -- 172.21.15.129:0/1940700811 >> 172.21.15.129:0/1940700811 conn(0x7f5edc0fdcc0 msgr2=0x7f5edc1049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:52.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 -- 172.21.15.129:0/1940700811 shutdown_connections 2024-05-20T17:56:52.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.794+0000 7f5ee38de640 1 -- 172.21.15.129:0/1940700811 wait complete. 2024-05-20T17:56:52.804 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-05-20T17:56:51.706836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":137,"num_read_kb":120,"num_write":226,"num_write_kb":2952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176,"ondisk_log_size":176,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":246,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364428,"kb_used_data":11040,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124577396,"statfs":{"total":1151940427776,"available":1151567253504,"internally_reserved":0,"allocated":11304960,"data_stored":7086166,"data_compressed":106947,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.003656"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.754034+0000","last_change":"2024-05-20T17:56:45.746793+0000","last_active":"2024-05-20T17:56:45.754034+0000","last_peered":"2024-05-20T17:56:45.754034+0000","last_clean":"2024-05-20T17:56:45.754034+0000","last_became_active":"2024-05-20T17:56:45.746244+0000","last_became_peered":"2024-05-20T17:56:45.746244+0000","last_unstale":"2024-05-20T17:56:45.754034+0000","last_undegraded":"2024-05-20T17:56:45.754034+0000","last_fullsized":"2024-05-20T17:56:45.754034+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:20:36.166355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618845+0000","last_change":"2024-05-20T17:56:45.744538+0000","last_active":"2024-05-20T17:56:48.618845+0000","last_peered":"2024-05-20T17:56:48.618845+0000","last_clean":"2024-05-20T17:56:48.618845+0000","last_became_active":"2024-05-20T17:56:45.744213+0000","last_became_peered":"2024-05-20T17:56:45.744213+0000","last_unstale":"2024-05-20T17:56:48.618845+0000","last_undegraded":"2024-05-20T17:56:48.618845+0000","last_fullsized":"2024-05-20T17:56:48.618845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:19:26.163003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618644+0000","last_change":"2024-05-20T17:56:45.749725+0000","last_active":"2024-05-20T17:56:48.618644+0000","last_peered":"2024-05-20T17:56:48.618644+0000","last_clean":"2024-05-20T17:56:48.618644+0000","last_became_active":"2024-05-20T17:56:45.749392+0000","last_became_peered":"2024-05-20T17:56:45.749392+0000","last_unstale":"2024-05-20T17:56:48.618644+0000","last_undegraded":"2024-05-20T17:56:48.618644+0000","last_fullsized":"2024-05-20T17:56:48.618644+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:43:33.184192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738902+0000","last_change":"2024-05-20T17:56:45.755191+0000","last_active":"2024-05-20T17:56:46.738902+0000","last_peered":"2024-05-20T17:56:46.738902+0000","last_clean":"2024-05-20T17:56:46.738902+0000","last_became_active":"2024-05-20T17:56:45.753524+0000","last_became_peered":"2024-05-20T17:56:45.753524+0000","last_unstale":"2024-05-20T17:56:46.738902+0000","last_undegraded":"2024-05-20T17:56:46.738902+0000","last_fullsized":"2024-05-20T17:56:46.738902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:11:23.099919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888391+0000","last_change":"2024-05-20T17:56:45.756291+0000","last_active":"2024-05-20T17:56:46.888391+0000","last_peered":"2024-05-20T17:56:46.888391+0000","last_clean":"2024-05-20T17:56:46.888391+0000","last_became_active":"2024-05-20T17:56:45.755910+0000","last_became_peered":"2024-05-20T17:56:45.755910+0000","last_unstale":"2024-05-20T17:56:46.888391+0000","last_undegraded":"2024-05-20T17:56:46.888391+0000","last_fullsized":"2024-05-20T17:56:46.888391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:37:36.941012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.750755+0000","last_change":"2024-05-20T17:56:45.748756+0000","last_active":"2024-05-20T17:56:45.750755+0000","last_peered":"2024-05-20T17:56:45.750755+0000","last_clean":"2024-05-20T17:56:45.750755+0000","last_became_active":"2024-05-20T17:56:45.748406+0000","last_became_peered":"2024-05-20T17:56:45.748406+0000","last_unstale":"2024-05-20T17:56:45.750755+0000","last_undegraded":"2024-05-20T17:56:45.750755+0000","last_fullsized":"2024-05-20T17:56:45.750755+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:11:37.959445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585243+0000","last_change":"2024-05-20T17:56:45.761108+0000","last_active":"2024-05-20T17:56:47.585243+0000","last_peered":"2024-05-20T17:56:47.585243+0000","last_clean":"2024-05-20T17:56:47.585243+0000","last_became_active":"2024-05-20T17:56:45.760322+0000","last_became_peered":"2024-05-20T17:56:45.760322+0000","last_unstale":"2024-05-20T17:56:47.585243+0000","last_undegraded":"2024-05-20T17:56:47.585243+0000","last_fullsized":"2024-05-20T17:56:47.585243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:44:01.754913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737037+0000","last_change":"2024-05-20T17:56:45.757387+0000","last_active":"2024-05-20T17:56:46.737037+0000","last_peered":"2024-05-20T17:56:46.737037+0000","last_clean":"2024-05-20T17:56:46.737037+0000","last_became_active":"2024-05-20T17:56:45.756374+0000","last_became_peered":"2024-05-20T17:56:45.756374+0000","last_unstale":"2024-05-20T17:56:46.737037+0000","last_undegraded":"2024-05-20T17:56:46.737037+0000","last_fullsized":"2024-05-20T17:56:46.737037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:53:52.751019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.918434+0000","last_change":"2024-05-20T17:56:45.751933+0000","last_active":"2024-05-20T17:56:46.918434+0000","last_peered":"2024-05-20T17:56:46.918434+0000","last_clean":"2024-05-20T17:56:46.918434+0000","last_became_active":"2024-05-20T17:56:45.751414+0000","last_became_peered":"2024-05-20T17:56:45.751414+0000","last_unstale":"2024-05-20T17:56:46.918434+0000","last_undegraded":"2024-05-20T17:56:46.918434+0000","last_fullsized":"2024-05-20T17:56:46.918434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:56:52.974830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.750734+0000","last_change":"2024-05-20T17:56:45.749511+0000","last_active":"2024-05-20T17:56:45.750734+0000","last_peered":"2024-05-20T17:56:45.750734+0000","last_clean":"2024-05-20T17:56:45.750734+0000","last_became_active":"2024-05-20T17:56:45.749211+0000","last_became_peered":"2024-05-20T17:56:45.749211+0000","last_unstale":"2024-05-20T17:56:45.750734+0000","last_undegraded":"2024-05-20T17:56:45.750734+0000","last_fullsized":"2024-05-20T17:56:45.750734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:22:35.456052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585399+0000","last_change":"2024-05-20T17:56:45.753743+0000","last_active":"2024-05-20T17:56:47.585399+0000","last_peered":"2024-05-20T17:56:47.585399+0000","last_clean":"2024-05-20T17:56:47.585399+0000","last_became_active":"2024-05-20T17:56:45.753378+0000","last_became_peered":"2024-05-20T17:56:45.753378+0000","last_unstale":"2024-05-20T17:56:47.585399+0000","last_undegraded":"2024-05-20T17:56:47.585399+0000","last_fullsized":"2024-05-20T17:56:47.585399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:54.563895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738825+0000","last_change":"2024-05-20T17:56:45.754890+0000","last_active":"2024-05-20T17:56:46.738825+0000","last_peered":"2024-05-20T17:56:46.738825+0000","last_clean":"2024-05-20T17:56:46.738825+0000","last_became_active":"2024-05-20T17:56:45.754333+0000","last_became_peered":"2024-05-20T17:56:45.754333+0000","last_unstale":"2024-05-20T17:56:46.738825+0000","last_undegraded":"2024-05-20T17:56:46.738825+0000","last_fullsized":"2024-05-20T17:56:46.738825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:51.025516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130635+0000","last_change":"2024-05-20T17:56:45.745662+0000","last_active":"2024-05-20T17:56:48.130635+0000","last_peered":"2024-05-20T17:56:48.130635+0000","last_clean":"2024-05-20T17:56:48.130635+0000","last_became_active":"2024-05-20T17:56:45.745281+0000","last_became_peered":"2024-05-20T17:56:45.745281+0000","last_unstale":"2024-05-20T17:56:48.130635+0000","last_undegraded":"2024-05-20T17:56:48.130635+0000","last_fullsized":"2024-05-20T17:56:48.130635+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:38:57.057301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618507+0000","last_change":"2024-05-20T17:56:45.749164+0000","last_active":"2024-05-20T17:56:48.618507+0000","last_peered":"2024-05-20T17:56:48.618507+0000","last_clean":"2024-05-20T17:56:48.618507+0000","last_became_active":"2024-05-20T17:56:45.748822+0000","last_became_peered":"2024-05-20T17:56:45.748822+0000","last_unstale":"2024-05-20T17:56:48.618507+0000","last_undegraded":"2024-05-20T17:56:48.618507+0000","last_fullsized":"2024-05-20T17:56:48.618507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:45:47.487116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618461+0000","last_change":"2024-05-20T17:56:45.751185+0000","last_active":"2024-05-20T17:56:48.618461+0000","last_peered":"2024-05-20T17:56:48.618461+0000","last_clean":"2024-05-20T17:56:48.618461+0000","last_became_active":"2024-05-20T17:56:45.750851+0000","last_became_peered":"2024-05-20T17:56:45.750851+0000","last_unstale":"2024-05-20T17:56:48.618461+0000","last_undegraded":"2024-05-20T17:56:48.618461+0000","last_fullsized":"2024-05-20T17:56:48.618461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:31:07.792717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.917837+0000","last_change":"2024-05-20T17:56:45.748101+0000","last_active":"2024-05-20T17:56:46.917837+0000","last_peered":"2024-05-20T17:56:46.917837+0000","last_clean":"2024-05-20T17:56:46.917837+0000","last_became_active":"2024-05-20T17:56:45.747754+0000","last_became_peered":"2024-05-20T17:56:45.747754+0000","last_unstale":"2024-05-20T17:56:46.917837+0000","last_undegraded":"2024-05-20T17:56:46.917837+0000","last_fullsized":"2024-05-20T17:56:46.917837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:24:54.702601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.754059+0000","last_change":"2024-05-20T17:56:45.747497+0000","last_active":"2024-05-20T17:56:45.754059+0000","last_peered":"2024-05-20T17:56:45.754059+0000","last_clean":"2024-05-20T17:56:45.754059+0000","last_became_active":"2024-05-20T17:56:45.747139+0000","last_became_peered":"2024-05-20T17:56:45.747139+0000","last_unstale":"2024-05-20T17:56:45.754059+0000","last_undegraded":"2024-05-20T17:56:45.754059+0000","last_fullsized":"2024-05-20T17:56:45.754059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:41:17.652537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.963938+0000","last_change":"2024-05-20T17:56:45.746194+0000","last_active":"2024-05-20T17:56:47.963938+0000","last_peered":"2024-05-20T17:56:47.963938+0000","last_clean":"2024-05-20T17:56:47.963938+0000","last_became_active":"2024-05-20T17:56:45.745809+0000","last_became_peered":"2024-05-20T17:56:45.745809+0000","last_unstale":"2024-05-20T17:56:47.963938+0000","last_undegraded":"2024-05-20T17:56:47.963938+0000","last_fullsized":"2024-05-20T17:56:47.963938+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:26:29.133790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518565+0000","last_change":"2024-05-20T17:56:45.749918+0000","last_active":"2024-05-20T17:56:47.518565+0000","last_peered":"2024-05-20T17:56:47.518565+0000","last_clean":"2024-05-20T17:56:47.518565+0000","last_became_active":"2024-05-20T17:56:45.749560+0000","last_became_peered":"2024-05-20T17:56:45.749560+0000","last_unstale":"2024-05-20T17:56:47.518565+0000","last_undegraded":"2024-05-20T17:56:47.518565+0000","last_fullsized":"2024-05-20T17:56:47.518565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:33:12.737206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.964018+0000","last_change":"2024-05-20T17:56:44.718812+0000","last_active":"2024-05-20T17:56:47.964018+0000","last_peered":"2024-05-20T17:56:47.964018+0000","last_clean":"2024-05-20T17:56:47.964018+0000","last_became_active":"2024-05-20T17:56:44.718496+0000","last_became_peered":"2024-05-20T17:56:44.718496+0000","last_unstale":"2024-05-20T17:56:47.964018+0000","last_undegraded":"2024-05-20T17:56:47.964018+0000","last_fullsized":"2024-05-20T17:56:47.964018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:40:50.059357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737470+0000","last_change":"2024-05-20T17:56:45.758330+0000","last_active":"2024-05-20T17:56:46.737470+0000","last_peered":"2024-05-20T17:56:46.737470+0000","last_clean":"2024-05-20T17:56:46.737470+0000","last_became_active":"2024-05-20T17:56:45.755873+0000","last_became_peered":"2024-05-20T17:56:45.755873+0000","last_unstale":"2024-05-20T17:56:46.737470+0000","last_undegraded":"2024-05-20T17:56:46.737470+0000","last_fullsized":"2024-05-20T17:56:46.737470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:53:31.653497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737626+0000","last_change":"2024-05-20T17:56:44.728508+0000","last_active":"2024-05-20T17:56:46.737626+0000","last_peered":"2024-05-20T17:56:46.737626+0000","last_clean":"2024-05-20T17:56:46.737626+0000","last_became_active":"2024-05-20T17:56:44.727875+0000","last_became_peered":"2024-05-20T17:56:44.727875+0000","last_unstale":"2024-05-20T17:56:46.737626+0000","last_undegraded":"2024-05-20T17:56:46.737626+0000","last_fullsized":"2024-05-20T17:56:46.737626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:12:14.349039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618930+0000","last_change":"2024-05-20T17:56:45.750000+0000","last_active":"2024-05-20T17:56:48.618930+0000","last_peered":"2024-05-20T17:56:48.618930+0000","last_clean":"2024-05-20T17:56:48.618930+0000","last_became_active":"2024-05-20T17:56:45.749644+0000","last_became_peered":"2024-05-20T17:56:45.749644+0000","last_unstale":"2024-05-20T17:56:48.618930+0000","last_undegraded":"2024-05-20T17:56:48.618930+0000","last_fullsized":"2024-05-20T17:56:48.618930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:32:46.085081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888739+0000","last_change":"2024-05-20T17:56:44.722692+0000","last_active":"2024-05-20T17:56:46.888739+0000","last_peered":"2024-05-20T17:56:46.888739+0000","last_clean":"2024-05-20T17:56:46.888739+0000","last_became_active":"2024-05-20T17:56:44.722170+0000","last_became_peered":"2024-05-20T17:56:44.722170+0000","last_unstale":"2024-05-20T17:56:46.888739+0000","last_undegraded":"2024-05-20T17:56:46.888739+0000","last_fullsized":"2024-05-20T17:56:46.888739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:38:03.679562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.918091+0000","last_change":"2024-05-20T17:56:45.751666+0000","last_active":"2024-05-20T17:56:46.918091+0000","last_peered":"2024-05-20T17:56:46.918091+0000","last_clean":"2024-05-20T17:56:46.918091+0000","last_became_active":"2024-05-20T17:56:45.751291+0000","last_became_peered":"2024-05-20T17:56:45.751291+0000","last_unstale":"2024-05-20T17:56:46.918091+0000","last_undegraded":"2024-05-20T17:56:46.918091+0000","last_fullsized":"2024-05-20T17:56:46.918091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:29:51.517363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888590+0000","last_change":"2024-05-20T17:56:44.721121+0000","last_active":"2024-05-20T17:56:46.888590+0000","last_peered":"2024-05-20T17:56:46.888590+0000","last_clean":"2024-05-20T17:56:46.888590+0000","last_became_active":"2024-05-20T17:56:44.720693+0000","last_became_peered":"2024-05-20T17:56:44.720693+0000","last_unstale":"2024-05-20T17:56:46.888590+0000","last_undegraded":"2024-05-20T17:56:46.888590+0000","last_fullsized":"2024-05-20T17:56:46.888590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:44.223828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738200+0000","last_change":"2024-05-20T17:56:45.745925+0000","last_active":"2024-05-20T17:56:46.738200+0000","last_peered":"2024-05-20T17:56:46.738200+0000","last_clean":"2024-05-20T17:56:46.738200+0000","last_became_active":"2024-05-20T17:56:45.745303+0000","last_became_peered":"2024-05-20T17:56:45.745303+0000","last_unstale":"2024-05-20T17:56:46.738200+0000","last_undegraded":"2024-05-20T17:56:46.738200+0000","last_fullsized":"2024-05-20T17:56:46.738200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:41:00.230681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737929+0000","last_change":"2024-05-20T17:56:44.719048+0000","last_active":"2024-05-20T17:56:46.737929+0000","last_peered":"2024-05-20T17:56:46.737929+0000","last_clean":"2024-05-20T17:56:46.737929+0000","last_became_active":"2024-05-20T17:56:44.718397+0000","last_became_peered":"2024-05-20T17:56:44.718397+0000","last_unstale":"2024-05-20T17:56:46.737929+0000","last_undegraded":"2024-05-20T17:56:46.737929+0000","last_fullsized":"2024-05-20T17:56:46.737929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:11:53.898226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888498+0000","last_change":"2024-05-20T17:56:45.752395+0000","last_active":"2024-05-20T17:56:46.888498+0000","last_peered":"2024-05-20T17:56:46.888498+0000","last_clean":"2024-05-20T17:56:46.888498+0000","last_became_active":"2024-05-20T17:56:45.751940+0000","last_became_peered":"2024-05-20T17:56:45.751940+0000","last_unstale":"2024-05-20T17:56:46.888498+0000","last_undegraded":"2024-05-20T17:56:46.888498+0000","last_fullsized":"2024-05-20T17:56:46.888498+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:33:57.780042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.968759+0000","last_change":"2024-05-20T17:56:44.714712+0000","last_active":"2024-05-20T17:56:45.968759+0000","last_peered":"2024-05-20T17:56:45.968759+0000","last_clean":"2024-05-20T17:56:45.968759+0000","last_became_active":"2024-05-20T17:56:44.714396+0000","last_became_peered":"2024-05-20T17:56:44.714396+0000","last_unstale":"2024-05-20T17:56:45.968759+0000","last_undegraded":"2024-05-20T17:56:45.968759+0000","last_fullsized":"2024-05-20T17:56:45.968759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:21:04.831289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738281+0000","last_change":"2024-05-20T17:56:45.758160+0000","last_active":"2024-05-20T17:56:46.738281+0000","last_peered":"2024-05-20T17:56:46.738281+0000","last_clean":"2024-05-20T17:56:46.738281+0000","last_became_active":"2024-05-20T17:56:45.757405+0000","last_became_peered":"2024-05-20T17:56:45.757405+0000","last_unstale":"2024-05-20T17:56:46.738281+0000","last_undegraded":"2024-05-20T17:56:46.738281+0000","last_fullsized":"2024-05-20T17:56:46.738281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:17.625246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130033+0000","last_change":"2024-05-20T17:56:44.716342+0000","last_active":"2024-05-20T17:56:48.130033+0000","last_peered":"2024-05-20T17:56:48.130033+0000","last_clean":"2024-05-20T17:56:48.130033+0000","last_became_active":"2024-05-20T17:56:44.716010+0000","last_became_peered":"2024-05-20T17:56:44.716010+0000","last_unstale":"2024-05-20T17:56:48.130033+0000","last_undegraded":"2024-05-20T17:56:48.130033+0000","last_fullsized":"2024-05-20T17:56:48.130033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:00:25.073312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737356+0000","last_change":"2024-05-20T17:56:45.755288+0000","last_active":"2024-05-20T17:56:46.737356+0000","last_peered":"2024-05-20T17:56:46.737356+0000","last_clean":"2024-05-20T17:56:46.737356+0000","last_became_active":"2024-05-20T17:56:45.754952+0000","last_became_peered":"2024-05-20T17:56:45.754952+0000","last_unstale":"2024-05-20T17:56:46.737356+0000","last_undegraded":"2024-05-20T17:56:46.737356+0000","last_fullsized":"2024-05-20T17:56:46.737356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:44:27.631722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737462+0000","last_change":"2024-05-20T17:56:44.717887+0000","last_active":"2024-05-20T17:56:46.737462+0000","last_peered":"2024-05-20T17:56:46.737462+0000","last_clean":"2024-05-20T17:56:46.737462+0000","last_became_active":"2024-05-20T17:56:44.717581+0000","last_became_peered":"2024-05-20T17:56:44.717581+0000","last_unstale":"2024-05-20T17:56:46.737462+0000","last_undegraded":"2024-05-20T17:56:46.737462+0000","last_fullsized":"2024-05-20T17:56:46.737462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:42:54.703112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.964000+0000","last_change":"2024-05-20T17:56:45.748448+0000","last_active":"2024-05-20T17:56:47.964000+0000","last_peered":"2024-05-20T17:56:47.964000+0000","last_clean":"2024-05-20T17:56:47.964000+0000","last_became_active":"2024-05-20T17:56:45.748016+0000","last_became_peered":"2024-05-20T17:56:45.748016+0000","last_unstale":"2024-05-20T17:56:47.964000+0000","last_undegraded":"2024-05-20T17:56:47.964000+0000","last_fullsized":"2024-05-20T17:56:47.964000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:31:39.350477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.737414+0000","last_change":"2024-05-20T17:56:44.709483+0000","last_active":"2024-05-20T17:56:45.737414+0000","last_peered":"2024-05-20T17:56:45.737414+0000","last_clean":"2024-05-20T17:56:45.737414+0000","last_became_active":"2024-05-20T17:56:44.708649+0000","last_became_peered":"2024-05-20T17:56:44.708649+0000","last_unstale":"2024-05-20T17:56:45.737414+0000","last_undegraded":"2024-05-20T17:56:45.737414+0000","last_fullsized":"2024-05-20T17:56:45.737414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:31:46.580256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.918300+0000","last_change":"2024-05-20T17:56:45.752279+0000","last_active":"2024-05-20T17:56:46.918300+0000","last_peered":"2024-05-20T17:56:46.918300+0000","last_clean":"2024-05-20T17:56:46.918300+0000","last_became_active":"2024-05-20T17:56:45.751790+0000","last_became_peered":"2024-05-20T17:56:45.751790+0000","last_unstale":"2024-05-20T17:56:46.918300+0000","last_undegraded":"2024-05-20T17:56:46.918300+0000","last_fullsized":"2024-05-20T17:56:46.918300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:06:38.113559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585434+0000","last_change":"2024-05-20T17:56:44.726040+0000","last_active":"2024-05-20T17:56:47.585434+0000","last_peered":"2024-05-20T17:56:47.585434+0000","last_clean":"2024-05-20T17:56:47.585434+0000","last_became_active":"2024-05-20T17:56:44.725714+0000","last_became_peered":"2024-05-20T17:56:44.725714+0000","last_unstale":"2024-05-20T17:56:47.585434+0000","last_undegraded":"2024-05-20T17:56:47.585434+0000","last_fullsized":"2024-05-20T17:56:47.585434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:22:59.081600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618909+0000","last_change":"2024-05-20T17:56:45.756464+0000","last_active":"2024-05-20T17:56:48.618909+0000","last_peered":"2024-05-20T17:56:48.618909+0000","last_clean":"2024-05-20T17:56:48.618909+0000","last_became_active":"2024-05-20T17:56:45.755922+0000","last_became_peered":"2024-05-20T17:56:45.755922+0000","last_unstale":"2024-05-20T17:56:48.618909+0000","last_undegraded":"2024-05-20T17:56:48.618909+0000","last_fullsized":"2024-05-20T17:56:48.618909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:01:53.689939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738135+0000","last_change":"2024-05-20T17:56:44.709548+0000","last_active":"2024-05-20T17:56:46.738135+0000","last_peered":"2024-05-20T17:56:46.738135+0000","last_clean":"2024-05-20T17:56:46.738135+0000","last_became_active":"2024-05-20T17:56:44.709209+0000","last_became_peered":"2024-05-20T17:56:44.709209+0000","last_unstale":"2024-05-20T17:56:46.738135+0000","last_undegraded":"2024-05-20T17:56:46.738135+0000","last_fullsized":"2024-05-20T17:56:46.738135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:28:13.118336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.918263+0000","last_change":"2024-05-20T17:56:45.755141+0000","last_active":"2024-05-20T17:56:46.918263+0000","last_peered":"2024-05-20T17:56:46.918263+0000","last_clean":"2024-05-20T17:56:46.918263+0000","last_became_active":"2024-05-20T17:56:45.754786+0000","last_became_peered":"2024-05-20T17:56:45.754786+0000","last_unstale":"2024-05-20T17:56:46.918263+0000","last_undegraded":"2024-05-20T17:56:46.918263+0000","last_fullsized":"2024-05-20T17:56:46.918263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:45:21.044602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'6","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.972399+0000","last_change":"2024-05-20T17:56:44.713578+0000","last_active":"2024-05-20T17:56:45.972399+0000","last_peered":"2024-05-20T17:56:45.972399+0000","last_clean":"2024-05-20T17:56:45.972399+0000","last_became_active":"2024-05-20T17:56:44.713203+0000","last_became_peered":"2024-05-20T17:56:44.713203+0000","last_unstale":"2024-05-20T17:56:45.972399+0000","last_undegraded":"2024-05-20T17:56:45.972399+0000","last_fullsized":"2024-05-20T17:56:45.972399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:36:54.386036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888624+0000","last_change":"2024-05-20T17:56:45.752023+0000","last_active":"2024-05-20T17:56:46.888624+0000","last_peered":"2024-05-20T17:56:46.888624+0000","last_clean":"2024-05-20T17:56:46.888624+0000","last_became_active":"2024-05-20T17:56:45.751672+0000","last_became_peered":"2024-05-20T17:56:45.751672+0000","last_unstale":"2024-05-20T17:56:46.888624+0000","last_undegraded":"2024-05-20T17:56:46.888624+0000","last_fullsized":"2024-05-20T17:56:46.888624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:23:49.127016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518208+0000","last_change":"2024-05-20T17:56:44.717336+0000","last_active":"2024-05-20T17:56:47.518208+0000","last_peered":"2024-05-20T17:56:47.518208+0000","last_clean":"2024-05-20T17:56:47.518208+0000","last_became_active":"2024-05-20T17:56:44.717016+0000","last_became_peered":"2024-05-20T17:56:44.717016+0000","last_unstale":"2024-05-20T17:56:47.518208+0000","last_undegraded":"2024-05-20T17:56:47.518208+0000","last_fullsized":"2024-05-20T17:56:47.518208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:50:46.433461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130207+0000","last_change":"2024-05-20T17:56:45.752411+0000","last_active":"2024-05-20T17:56:48.130207+0000","last_peered":"2024-05-20T17:56:48.130207+0000","last_clean":"2024-05-20T17:56:48.130207+0000","last_became_active":"2024-05-20T17:56:45.752078+0000","last_became_peered":"2024-05-20T17:56:45.752078+0000","last_unstale":"2024-05-20T17:56:48.130207+0000","last_undegraded":"2024-05-20T17:56:48.130207+0000","last_fullsized":"2024-05-20T17:56:48.130207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:22:15.560666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737897+0000","last_change":"2024-05-20T17:56:44.723262+0000","last_active":"2024-05-20T17:56:46.737897+0000","last_peered":"2024-05-20T17:56:46.737897+0000","last_clean":"2024-05-20T17:56:46.737897+0000","last_became_active":"2024-05-20T17:56:44.722865+0000","last_became_peered":"2024-05-20T17:56:44.722865+0000","last_unstale":"2024-05-20T17:56:46.737897+0000","last_undegraded":"2024-05-20T17:56:46.737897+0000","last_fullsized":"2024-05-20T17:56:46.737897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:15:02.820718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585443+0000","last_change":"2024-05-20T17:56:45.751126+0000","last_active":"2024-05-20T17:56:47.585443+0000","last_peered":"2024-05-20T17:56:47.585443+0000","last_clean":"2024-05-20T17:56:47.585443+0000","last_became_active":"2024-05-20T17:56:45.750042+0000","last_became_peered":"2024-05-20T17:56:45.750042+0000","last_unstale":"2024-05-20T17:56:47.585443+0000","last_undegraded":"2024-05-20T17:56:47.585443+0000","last_fullsized":"2024-05-20T17:56:47.585443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:48:35.371790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738600+0000","last_change":"2024-05-20T17:56:44.718396+0000","last_active":"2024-05-20T17:56:46.738600+0000","last_peered":"2024-05-20T17:56:46.738600+0000","last_clean":"2024-05-20T17:56:46.738600+0000","last_became_active":"2024-05-20T17:56:44.718031+0000","last_became_peered":"2024-05-20T17:56:44.718031+0000","last_unstale":"2024-05-20T17:56:46.738600+0000","last_undegraded":"2024-05-20T17:56:46.738600+0000","last_fullsized":"2024-05-20T17:56:46.738600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:42:40.955520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.584993+0000","last_change":"2024-05-20T17:56:45.759302+0000","last_active":"2024-05-20T17:56:47.584993+0000","last_peered":"2024-05-20T17:56:47.584993+0000","last_clean":"2024-05-20T17:56:47.584993+0000","last_became_active":"2024-05-20T17:56:45.758889+0000","last_became_peered":"2024-05-20T17:56:45.758889+0000","last_unstale":"2024-05-20T17:56:47.584993+0000","last_undegraded":"2024-05-20T17:56:47.584993+0000","last_fullsized":"2024-05-20T17:56:47.584993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:40.685471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888157+0000","last_change":"2024-05-20T17:56:44.726502+0000","last_active":"2024-05-20T17:56:46.888157+0000","last_peered":"2024-05-20T17:56:46.888157+0000","last_clean":"2024-05-20T17:56:46.888157+0000","last_became_active":"2024-05-20T17:56:44.726168+0000","last_became_peered":"2024-05-20T17:56:44.726168+0000","last_unstale":"2024-05-20T17:56:46.888157+0000","last_undegraded":"2024-05-20T17:56:46.888157+0000","last_fullsized":"2024-05-20T17:56:46.888157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:45:14.523968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737716+0000","last_change":"2024-05-20T17:56:45.759355+0000","last_active":"2024-05-20T17:56:46.737716+0000","last_peered":"2024-05-20T17:56:46.737716+0000","last_clean":"2024-05-20T17:56:46.737716+0000","last_became_active":"2024-05-20T17:56:45.758920+0000","last_became_peered":"2024-05-20T17:56:45.758920+0000","last_unstale":"2024-05-20T17:56:46.737716+0000","last_undegraded":"2024-05-20T17:56:46.737716+0000","last_fullsized":"2024-05-20T17:56:46.737716+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:46:02.372250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888717+0000","last_change":"2024-05-20T17:56:44.718507+0000","last_active":"2024-05-20T17:56:46.888717+0000","last_peered":"2024-05-20T17:56:46.888717+0000","last_clean":"2024-05-20T17:56:46.888717+0000","last_became_active":"2024-05-20T17:56:44.718188+0000","last_became_peered":"2024-05-20T17:56:44.718188+0000","last_unstale":"2024-05-20T17:56:46.888717+0000","last_undegraded":"2024-05-20T17:56:46.888717+0000","last_fullsized":"2024-05-20T17:56:46.888717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:43:59.792942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.754182+0000","last_change":"2024-05-20T17:56:45.751361+0000","last_active":"2024-05-20T17:56:45.754182+0000","last_peered":"2024-05-20T17:56:45.754182+0000","last_clean":"2024-05-20T17:56:45.754182+0000","last_became_active":"2024-05-20T17:56:45.750032+0000","last_became_peered":"2024-05-20T17:56:45.750032+0000","last_unstale":"2024-05-20T17:56:45.754182+0000","last_undegraded":"2024-05-20T17:56:45.754182+0000","last_fullsized":"2024-05-20T17:56:45.754182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:24:43.801133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.963937+0000","last_change":"2024-05-20T17:56:44.714404+0000","last_active":"2024-05-20T17:56:47.963937+0000","last_peered":"2024-05-20T17:56:47.963937+0000","last_clean":"2024-05-20T17:56:47.963937+0000","last_became_active":"2024-05-20T17:56:44.714072+0000","last_became_peered":"2024-05-20T17:56:44.714072+0000","last_unstale":"2024-05-20T17:56:47.963937+0000","last_undegraded":"2024-05-20T17:56:47.963937+0000","last_fullsized":"2024-05-20T17:56:47.963937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:39:53.983513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.750849+0000","last_change":"2024-05-20T17:56:45.749785+0000","last_active":"2024-05-20T17:56:45.750849+0000","last_peered":"2024-05-20T17:56:45.750849+0000","last_clean":"2024-05-20T17:56:45.750849+0000","last_became_active":"2024-05-20T17:56:45.749457+0000","last_became_peered":"2024-05-20T17:56:45.749457+0000","last_unstale":"2024-05-20T17:56:45.750849+0000","last_undegraded":"2024-05-20T17:56:45.750849+0000","last_fullsized":"2024-05-20T17:56:45.750849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:00:16.303032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585282+0000","last_change":"2024-05-20T17:56:44.722787+0000","last_active":"2024-05-20T17:56:47.585282+0000","last_peered":"2024-05-20T17:56:47.585282+0000","last_clean":"2024-05-20T17:56:47.585282+0000","last_became_active":"2024-05-20T17:56:44.722214+0000","last_became_peered":"2024-05-20T17:56:44.722214+0000","last_unstale":"2024-05-20T17:56:47.585282+0000","last_undegraded":"2024-05-20T17:56:47.585282+0000","last_fullsized":"2024-05-20T17:56:47.585282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:40:16.705718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.758042+0000","last_change":"2024-05-20T17:56:45.757515+0000","last_active":"2024-05-20T17:56:45.758042+0000","last_peered":"2024-05-20T17:56:45.758042+0000","last_clean":"2024-05-20T17:56:45.758042+0000","last_became_active":"2024-05-20T17:56:45.757174+0000","last_became_peered":"2024-05-20T17:56:45.757174+0000","last_unstale":"2024-05-20T17:56:45.758042+0000","last_undegraded":"2024-05-20T17:56:45.758042+0000","last_fullsized":"2024-05-20T17:56:45.758042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:36:59.941562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737726+0000","last_change":"2024-05-20T17:56:45.759018+0000","last_active":"2024-05-20T17:56:46.737726+0000","last_peered":"2024-05-20T17:56:46.737726+0000","last_clean":"2024-05-20T17:56:46.737726+0000","last_became_active":"2024-05-20T17:56:45.757901+0000","last_became_peered":"2024-05-20T17:56:45.757901+0000","last_unstale":"2024-05-20T17:56:46.737726+0000","last_undegraded":"2024-05-20T17:56:46.737726+0000","last_fullsized":"2024-05-20T17:56:46.737726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:25:13.554683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.129927+0000","last_change":"2024-05-20T17:56:44.724642+0000","last_active":"2024-05-20T17:56:48.129927+0000","last_peered":"2024-05-20T17:56:48.129927+0000","last_clean":"2024-05-20T17:56:48.129927+0000","last_became_active":"2024-05-20T17:56:44.724311+0000","last_became_peered":"2024-05-20T17:56:44.724311+0000","last_unstale":"2024-05-20T17:56:48.129927+0000","last_undegraded":"2024-05-20T17:56:48.129927+0000","last_fullsized":"2024-05-20T17:56:48.129927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:47:35.336236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737319+0000","last_change":"2024-05-20T17:56:45.758466+0000","last_active":"2024-05-20T17:56:46.737319+0000","last_peered":"2024-05-20T17:56:46.737319+0000","last_clean":"2024-05-20T17:56:46.737319+0000","last_became_active":"2024-05-20T17:56:45.758126+0000","last_became_peered":"2024-05-20T17:56:45.758126+0000","last_unstale":"2024-05-20T17:56:46.737319+0000","last_undegraded":"2024-05-20T17:56:46.737319+0000","last_fullsized":"2024-05-20T17:56:46.737319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:36.830828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738744+0000","last_change":"2024-05-20T17:56:44.711173+0000","last_active":"2024-05-20T17:56:46.738744+0000","last_peered":"2024-05-20T17:56:46.738744+0000","last_clean":"2024-05-20T17:56:46.738744+0000","last_became_active":"2024-05-20T17:56:44.710778+0000","last_became_peered":"2024-05-20T17:56:44.710778+0000","last_unstale":"2024-05-20T17:56:46.738744+0000","last_undegraded":"2024-05-20T17:56:46.738744+0000","last_fullsized":"2024-05-20T17:56:46.738744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:20:14.150346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.584782+0000","last_change":"2024-05-20T17:55:34.871483+0000","last_active":"2024-05-20T17:56:47.584782+0000","last_peered":"2024-05-20T17:56:47.584782+0000","last_clean":"2024-05-20T17:56:47.584782+0000","last_became_active":"2024-05-20T17:55:34.871028+0000","last_became_peered":"2024-05-20T17:55:34.871028+0000","last_unstale":"2024-05-20T17:56:47.584782+0000","last_undegraded":"2024-05-20T17:56:47.584782+0000","last_fullsized":"2024-05-20T17:56:47.584782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_clean_scrub_stamp":"2024-05-20T17:52:08.599715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:29:40.159577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.129999+0000","last_change":"2024-05-20T17:56:45.747725+0000","last_active":"2024-05-20T17:56:48.129999+0000","last_peered":"2024-05-20T17:56:48.129999+0000","last_clean":"2024-05-20T17:56:48.129999+0000","last_became_active":"2024-05-20T17:56:45.747232+0000","last_became_peered":"2024-05-20T17:56:45.747232+0000","last_unstale":"2024-05-20T17:56:48.129999+0000","last_undegraded":"2024-05-20T17:56:48.129999+0000","last_fullsized":"2024-05-20T17:56:48.129999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:09:26.589905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585189+0000","last_change":"2024-05-20T17:56:44.721004+0000","last_active":"2024-05-20T17:56:47.585189+0000","last_peered":"2024-05-20T17:56:47.585189+0000","last_clean":"2024-05-20T17:56:47.585189+0000","last_became_active":"2024-05-20T17:56:44.720620+0000","last_became_peered":"2024-05-20T17:56:44.720620+0000","last_unstale":"2024-05-20T17:56:47.585189+0000","last_undegraded":"2024-05-20T17:56:47.585189+0000","last_fullsized":"2024-05-20T17:56:47.585189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:42:14.710663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888596+0000","last_change":"2024-05-20T17:56:44.714754+0000","last_active":"2024-05-20T17:56:46.888596+0000","last_peered":"2024-05-20T17:56:46.888596+0000","last_clean":"2024-05-20T17:56:46.888596+0000","last_became_active":"2024-05-20T17:56:44.714378+0000","last_became_peered":"2024-05-20T17:56:44.714378+0000","last_unstale":"2024-05-20T17:56:46.888596+0000","last_undegraded":"2024-05-20T17:56:46.888596+0000","last_fullsized":"2024-05-20T17:56:46.888596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:46:53.969960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130363+0000","last_change":"2024-05-20T17:56:45.742415+0000","last_active":"2024-05-20T17:56:48.130363+0000","last_peered":"2024-05-20T17:56:48.130363+0000","last_clean":"2024-05-20T17:56:48.130363+0000","last_became_active":"2024-05-20T17:56:45.741941+0000","last_became_peered":"2024-05-20T17:56:45.741941+0000","last_unstale":"2024-05-20T17:56:48.130363+0000","last_undegraded":"2024-05-20T17:56:48.130363+0000","last_fullsized":"2024-05-20T17:56:48.130363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:20:03.191021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.964038+0000","last_change":"2024-05-20T17:56:44.724361+0000","last_active":"2024-05-20T17:56:47.964038+0000","last_peered":"2024-05-20T17:56:47.964038+0000","last_clean":"2024-05-20T17:56:47.964038+0000","last_became_active":"2024-05-20T17:56:44.724033+0000","last_became_peered":"2024-05-20T17:56:44.724033+0000","last_unstale":"2024-05-20T17:56:47.964038+0000","last_undegraded":"2024-05-20T17:56:47.964038+0000","last_fullsized":"2024-05-20T17:56:47.964038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:10:13.756606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737678+0000","last_change":"2024-05-20T17:56:45.754216+0000","last_active":"2024-05-20T17:56:46.737678+0000","last_peered":"2024-05-20T17:56:46.737678+0000","last_clean":"2024-05-20T17:56:46.737678+0000","last_became_active":"2024-05-20T17:56:45.753855+0000","last_became_peered":"2024-05-20T17:56:45.753855+0000","last_unstale":"2024-05-20T17:56:46.737678+0000","last_undegraded":"2024-05-20T17:56:46.737678+0000","last_fullsized":"2024-05-20T17:56:46.737678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:14:15.664117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738850+0000","last_change":"2024-05-20T17:56:44.716296+0000","last_active":"2024-05-20T17:56:46.738850+0000","last_peered":"2024-05-20T17:56:46.738850+0000","last_clean":"2024-05-20T17:56:46.738850+0000","last_became_active":"2024-05-20T17:56:44.715952+0000","last_became_peered":"2024-05-20T17:56:44.715952+0000","last_unstale":"2024-05-20T17:56:46.738850+0000","last_undegraded":"2024-05-20T17:56:46.738850+0000","last_fullsized":"2024-05-20T17:56:46.738850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:52:37.986399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618751+0000","last_change":"2024-05-20T17:56:45.751874+0000","last_active":"2024-05-20T17:56:48.618751+0000","last_peered":"2024-05-20T17:56:48.618751+0000","last_clean":"2024-05-20T17:56:48.618751+0000","last_became_active":"2024-05-20T17:56:45.751534+0000","last_became_peered":"2024-05-20T17:56:45.751534+0000","last_unstale":"2024-05-20T17:56:48.618751+0000","last_undegraded":"2024-05-20T17:56:48.618751+0000","last_fullsized":"2024-05-20T17:56:48.618751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:05:27.248115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.917819+0000","last_change":"2024-05-20T17:56:44.724271+0000","last_active":"2024-05-20T17:56:46.917819+0000","last_peered":"2024-05-20T17:56:46.917819+0000","last_clean":"2024-05-20T17:56:46.917819+0000","last_became_active":"2024-05-20T17:56:44.723947+0000","last_became_peered":"2024-05-20T17:56:44.723947+0000","last_unstale":"2024-05-20T17:56:46.917819+0000","last_undegraded":"2024-05-20T17:56:46.917819+0000","last_fullsized":"2024-05-20T17:56:46.917819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:32:13.482059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888363+0000","last_change":"2024-05-20T17:56:45.751296+0000","last_active":"2024-05-20T17:56:46.888363+0000","last_peered":"2024-05-20T17:56:46.888363+0000","last_clean":"2024-05-20T17:56:46.888363+0000","last_became_active":"2024-05-20T17:56:45.748035+0000","last_became_peered":"2024-05-20T17:56:45.748035+0000","last_unstale":"2024-05-20T17:56:46.888363+0000","last_undegraded":"2024-05-20T17:56:46.888363+0000","last_fullsized":"2024-05-20T17:56:46.888363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:37:43.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737075+0000","last_change":"2024-05-20T17:56:44.720461+0000","last_active":"2024-05-20T17:56:46.737075+0000","last_peered":"2024-05-20T17:56:46.737075+0000","last_clean":"2024-05-20T17:56:46.737075+0000","last_became_active":"2024-05-20T17:56:44.720115+0000","last_became_peered":"2024-05-20T17:56:44.720115+0000","last_unstale":"2024-05-20T17:56:46.737075+0000","last_undegraded":"2024-05-20T17:56:46.737075+0000","last_fullsized":"2024-05-20T17:56:46.737075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:14:47.330193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.752724+0000","last_change":"2024-05-20T17:56:45.745025+0000","last_active":"2024-05-20T17:56:45.752724+0000","last_peered":"2024-05-20T17:56:45.752724+0000","last_clean":"2024-05-20T17:56:45.752724+0000","last_became_active":"2024-05-20T17:56:45.744626+0000","last_became_peered":"2024-05-20T17:56:45.744626+0000","last_unstale":"2024-05-20T17:56:45.752724+0000","last_undegraded":"2024-05-20T17:56:45.752724+0000","last_fullsized":"2024-05-20T17:56:45.752724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:31:26.096534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518264+0000","last_change":"2024-05-20T17:56:44.726099+0000","last_active":"2024-05-20T17:56:47.518264+0000","last_peered":"2024-05-20T17:56:47.518264+0000","last_clean":"2024-05-20T17:56:47.518264+0000","last_became_active":"2024-05-20T17:56:44.725714+0000","last_became_peered":"2024-05-20T17:56:44.725714+0000","last_unstale":"2024-05-20T17:56:47.518264+0000","last_undegraded":"2024-05-20T17:56:47.518264+0000","last_fullsized":"2024-05-20T17:56:47.518264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:51.326907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737317+0000","last_change":"2024-05-20T17:56:45.757376+0000","last_active":"2024-05-20T17:56:46.737317+0000","last_peered":"2024-05-20T17:56:46.737317+0000","last_clean":"2024-05-20T17:56:46.737317+0000","last_became_active":"2024-05-20T17:56:45.757053+0000","last_became_peered":"2024-05-20T17:56:45.757053+0000","last_unstale":"2024-05-20T17:56:46.737317+0000","last_undegraded":"2024-05-20T17:56:46.737317+0000","last_fullsized":"2024-05-20T17:56:46.737317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:50:27.727055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888450+0000","last_change":"2024-05-20T17:56:44.726341+0000","last_active":"2024-05-20T17:56:46.888450+0000","last_peered":"2024-05-20T17:56:46.888450+0000","last_clean":"2024-05-20T17:56:46.888450+0000","last_became_active":"2024-05-20T17:56:44.726008+0000","last_became_peered":"2024-05-20T17:56:44.726008+0000","last_unstale":"2024-05-20T17:56:46.888450+0000","last_undegraded":"2024-05-20T17:56:46.888450+0000","last_fullsized":"2024-05-20T17:56:46.888450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:25:07.520179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585496+0000","last_change":"2024-05-20T17:56:45.760469+0000","last_active":"2024-05-20T17:56:47.585496+0000","last_peered":"2024-05-20T17:56:47.585496+0000","last_clean":"2024-05-20T17:56:47.585496+0000","last_became_active":"2024-05-20T17:56:45.760100+0000","last_became_peered":"2024-05-20T17:56:45.760100+0000","last_unstale":"2024-05-20T17:56:47.585496+0000","last_undegraded":"2024-05-20T17:56:47.585496+0000","last_fullsized":"2024-05-20T17:56:47.585496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:54:06.309355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518549+0000","last_change":"2024-05-20T17:56:44.716179+0000","last_active":"2024-05-20T17:56:47.518549+0000","last_peered":"2024-05-20T17:56:47.518549+0000","last_clean":"2024-05-20T17:56:47.518549+0000","last_became_active":"2024-05-20T17:56:44.715806+0000","last_became_peered":"2024-05-20T17:56:44.715806+0000","last_unstale":"2024-05-20T17:56:47.518549+0000","last_undegraded":"2024-05-20T17:56:47.518549+0000","last_fullsized":"2024-05-20T17:56:47.518549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:07:05.319573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585593+0000","last_change":"2024-05-20T17:56:45.757302+0000","last_active":"2024-05-20T17:56:47.585593+0000","last_peered":"2024-05-20T17:56:47.585593+0000","last_clean":"2024-05-20T17:56:47.585593+0000","last_became_active":"2024-05-20T17:56:45.754169+0000","last_became_peered":"2024-05-20T17:56:45.754169+0000","last_unstale":"2024-05-20T17:56:47.585593+0000","last_undegraded":"2024-05-20T17:56:47.585593+0000","last_fullsized":"2024-05-20T17:56:47.585593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:03:10.794342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737770+0000","last_change":"2024-05-20T17:56:44.724478+0000","last_active":"2024-05-20T17:56:46.737770+0000","last_peered":"2024-05-20T17:56:46.737770+0000","last_clean":"2024-05-20T17:56:46.737770+0000","last_became_active":"2024-05-20T17:56:44.723927+0000","last_became_peered":"2024-05-20T17:56:44.723927+0000","last_unstale":"2024-05-20T17:56:46.737770+0000","last_undegraded":"2024-05-20T17:56:46.737770+0000","last_fullsized":"2024-05-20T17:56:46.737770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:04:12.003957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585348+0000","last_change":"2024-05-20T17:56:45.758135+0000","last_active":"2024-05-20T17:56:47.585348+0000","last_peered":"2024-05-20T17:56:47.585348+0000","last_clean":"2024-05-20T17:56:47.585348+0000","last_became_active":"2024-05-20T17:56:45.757358+0000","last_became_peered":"2024-05-20T17:56:45.757358+0000","last_unstale":"2024-05-20T17:56:47.585348+0000","last_undegraded":"2024-05-20T17:56:47.585348+0000","last_fullsized":"2024-05-20T17:56:47.585348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:13:06.435055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130384+0000","last_change":"2024-05-20T17:56:44.719822+0000","last_active":"2024-05-20T17:56:48.130384+0000","last_peered":"2024-05-20T17:56:48.130384+0000","last_clean":"2024-05-20T17:56:48.130384+0000","last_became_active":"2024-05-20T17:56:44.719454+0000","last_became_peered":"2024-05-20T17:56:44.719454+0000","last_unstale":"2024-05-20T17:56:48.130384+0000","last_undegraded":"2024-05-20T17:56:48.130384+0000","last_fullsized":"2024-05-20T17:56:48.130384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:03:21.048330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585584+0000","last_change":"2024-05-20T17:56:45.753510+0000","last_active":"2024-05-20T17:56:47.585584+0000","last_peered":"2024-05-20T17:56:47.585584+0000","last_clean":"2024-05-20T17:56:47.585584+0000","last_became_active":"2024-05-20T17:56:45.752718+0000","last_became_peered":"2024-05-20T17:56:45.752718+0000","last_unstale":"2024-05-20T17:56:47.585584+0000","last_undegraded":"2024-05-20T17:56:47.585584+0000","last_fullsized":"2024-05-20T17:56:47.585584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:05:44.677138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.755763+0000","last_change":"2024-05-20T17:56:45.752528+0000","last_active":"2024-05-20T17:56:45.755763+0000","last_peered":"2024-05-20T17:56:45.755763+0000","last_clean":"2024-05-20T17:56:45.755763+0000","last_became_active":"2024-05-20T17:56:45.752171+0000","last_became_peered":"2024-05-20T17:56:45.752171+0000","last_unstale":"2024-05-20T17:56:45.755763+0000","last_undegraded":"2024-05-20T17:56:45.755763+0000","last_fullsized":"2024-05-20T17:56:45.755763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:24:10.033691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518491+0000","last_change":"2024-05-20T17:56:45.749228+0000","last_active":"2024-05-20T17:56:47.518491+0000","last_peered":"2024-05-20T17:56:47.518491+0000","last_clean":"2024-05-20T17:56:47.518491+0000","last_became_active":"2024-05-20T17:56:45.748886+0000","last_became_peered":"2024-05-20T17:56:45.748886+0000","last_unstale":"2024-05-20T17:56:47.518491+0000","last_undegraded":"2024-05-20T17:56:47.518491+0000","last_fullsized":"2024-05-20T17:56:47.518491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:23:10.569720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518373+0000","last_change":"2024-05-20T17:56:45.751105+0000","last_active":"2024-05-20T17:56:47.518373+0000","last_peered":"2024-05-20T17:56:47.518373+0000","last_clean":"2024-05-20T17:56:47.518373+0000","last_became_active":"2024-05-20T17:56:45.750754+0000","last_became_peered":"2024-05-20T17:56:45.750754+0000","last_unstale":"2024-05-20T17:56:47.518373+0000","last_undegraded":"2024-05-20T17:56:47.518373+0000","last_fullsized":"2024-05-20T17:56:47.518373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:57:48.327428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618700+0000","last_change":"2024-05-20T17:56:45.752597+0000","last_active":"2024-05-20T17:56:48.618700+0000","last_peered":"2024-05-20T17:56:48.618700+0000","last_clean":"2024-05-20T17:56:48.618700+0000","last_became_active":"2024-05-20T17:56:45.752256+0000","last_became_peered":"2024-05-20T17:56:45.752256+0000","last_unstale":"2024-05-20T17:56:48.618700+0000","last_undegraded":"2024-05-20T17:56:48.618700+0000","last_fullsized":"2024-05-20T17:56:48.618700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:51:53.750906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130226+0000","last_change":"2024-05-20T17:56:45.746122+0000","last_active":"2024-05-20T17:56:48.130226+0000","last_peered":"2024-05-20T17:56:48.130226+0000","last_clean":"2024-05-20T17:56:48.130226+0000","last_became_active":"2024-05-20T17:56:45.745773+0000","last_became_peered":"2024-05-20T17:56:45.745773+0000","last_unstale":"2024-05-20T17:56:48.130226+0000","last_undegraded":"2024-05-20T17:56:48.130226+0000","last_fullsized":"2024-05-20T17:56:48.130226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:21:27.909974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.756179+0000","last_change":"2024-05-20T17:56:45.755018+0000","last_active":"2024-05-20T17:56:45.756179+0000","last_peered":"2024-05-20T17:56:45.756179+0000","last_clean":"2024-05-20T17:56:45.756179+0000","last_became_active":"2024-05-20T17:56:45.754615+0000","last_became_peered":"2024-05-20T17:56:45.754615+0000","last_unstale":"2024-05-20T17:56:45.756179+0000","last_undegraded":"2024-05-20T17:56:45.756179+0000","last_fullsized":"2024-05-20T17:56:45.756179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:33:14.425630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518777+0000","last_change":"2024-05-20T17:56:45.751091+0000","last_active":"2024-05-20T17:56:47.518777+0000","last_peered":"2024-05-20T17:56:47.518777+0000","last_clean":"2024-05-20T17:56:47.518777+0000","last_became_active":"2024-05-20T17:56:45.748573+0000","last_became_peered":"2024-05-20T17:56:45.748573+0000","last_unstale":"2024-05-20T17:56:47.518777+0000","last_undegraded":"2024-05-20T17:56:47.518777+0000","last_fullsized":"2024-05-20T17:56:47.518777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:11:16.140967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737276+0000","last_change":"2024-05-20T17:56:45.756373+0000","last_active":"2024-05-20T17:56:46.737276+0000","last_peered":"2024-05-20T17:56:46.737276+0000","last_clean":"2024-05-20T17:56:46.737276+0000","last_became_active":"2024-05-20T17:56:45.755993+0000","last_became_peered":"2024-05-20T17:56:45.755993+0000","last_unstale":"2024-05-20T17:56:46.737276+0000","last_undegraded":"2024-05-20T17:56:46.737276+0000","last_fullsized":"2024-05-20T17:56:46.737276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:34:44.491341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737916+0000","last_change":"2024-05-20T17:56:45.751866+0000","last_active":"2024-05-20T17:56:46.737916+0000","last_peered":"2024-05-20T17:56:46.737916+0000","last_clean":"2024-05-20T17:56:46.737916+0000","last_became_active":"2024-05-20T17:56:45.751518+0000","last_became_peered":"2024-05-20T17:56:45.751518+0000","last_unstale":"2024-05-20T17:56:46.737916+0000","last_undegraded":"2024-05-20T17:56:46.737916+0000","last_fullsized":"2024-05-20T17:56:46.737916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:38:18.736177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737209+0000","last_change":"2024-05-20T17:56:45.749903+0000","last_active":"2024-05-20T17:56:46.737209+0000","last_peered":"2024-05-20T17:56:46.737209+0000","last_clean":"2024-05-20T17:56:46.737209+0000","last_became_active":"2024-05-20T17:56:45.749500+0000","last_became_peered":"2024-05-20T17:56:45.749500+0000","last_unstale":"2024-05-20T17:56:46.737209+0000","last_undegraded":"2024-05-20T17:56:46.737209+0000","last_fullsized":"2024-05-20T17:56:46.737209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:39:34.875111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.918070+0000","last_change":"2024-05-20T17:56:45.747370+0000","last_active":"2024-05-20T17:56:46.918070+0000","last_peered":"2024-05-20T17:56:46.918070+0000","last_clean":"2024-05-20T17:56:46.918070+0000","last_became_active":"2024-05-20T17:56:45.747036+0000","last_became_peered":"2024-05-20T17:56:45.747036+0000","last_unstale":"2024-05-20T17:56:46.918070+0000","last_undegraded":"2024-05-20T17:56:46.918070+0000","last_fullsized":"2024-05-20T17:56:46.918070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:02:02.220522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130688+0000","last_change":"2024-05-20T17:56:45.752762+0000","last_active":"2024-05-20T17:56:48.130688+0000","last_peered":"2024-05-20T17:56:48.130688+0000","last_clean":"2024-05-20T17:56:48.130688+0000","last_became_active":"2024-05-20T17:56:45.752345+0000","last_became_peered":"2024-05-20T17:56:45.752345+0000","last_unstale":"2024-05-20T17:56:48.130688+0000","last_undegraded":"2024-05-20T17:56:48.130688+0000","last_fullsized":"2024-05-20T17:56:48.130688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:14:49.861456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.753795+0000","last_change":"2024-05-20T17:56:45.750229+0000","last_active":"2024-05-20T17:56:45.753795+0000","last_peered":"2024-05-20T17:56:45.753795+0000","last_clean":"2024-05-20T17:56:45.753795+0000","last_became_active":"2024-05-20T17:56:45.749872+0000","last_became_peered":"2024-05-20T17:56:45.749872+0000","last_unstale":"2024-05-20T17:56:45.753795+0000","last_undegraded":"2024-05-20T17:56:45.753795+0000","last_fullsized":"2024-05-20T17:56:45.753795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:17:43.464358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280272,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":913408,"data_stored":436606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1028780,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808856,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":445758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070493,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":933888,"data_stored":445780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332129,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":1036126,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":436606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":937984,"data_stored":446788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":445266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":446822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":933888,"data_stored":445280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":445198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":1027156,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-20T17:56:52.809 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-20T17:56:52.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.958+0000 7fedf8c84640 1 -- 172.21.15.129:0/462083960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf4101f80 msgr2=0x7fedf4102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.958+0000 7fedf8c84640 1 --2- 172.21.15.129:0/462083960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf4101f80 0x7fedf4102390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fede4009900 tx=0x7fede402a410 comp rx=0 tx=0).stop 2024-05-20T17:56:52.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.958+0000 7fedf8c84640 1 -- 172.21.15.129:0/462083960 shutdown_connections 2024-05-20T17:56:52.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.958+0000 7fedf8c84640 1 --2- 172.21.15.129:0/462083960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedf4102ae0 0x7fedf4102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.958+0000 7fedf8c84640 1 --2- 172.21.15.129:0/462083960 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf4103640 0x7fedf4109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.958+0000 7fedf8c84640 1 --2- 172.21.15.129:0/462083960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf4101f80 0x7fedf4102390 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.958+0000 7fedf8c84640 1 -- 172.21.15.129:0/462083960 >> 172.21.15.129:0/462083960 conn(0x7fedf40fc320 msgr2=0x7fedf40fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:52.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.959+0000 7fedf8c84640 1 -- 172.21.15.129:0/462083960 shutdown_connections 2024-05-20T17:56:52.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.959+0000 7fedf8c84640 1 -- 172.21.15.129:0/462083960 wait complete. 2024-05-20T17:56:52.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.959+0000 7fedf8c84640 1 Processor -- start 2024-05-20T17:56:52.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.959+0000 7fedf8c84640 1 -- start start 2024-05-20T17:56:52.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf8c84640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf4101f80 0x7fedf4199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf8c84640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf4102ae0 0x7fedf4199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf8c84640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedf419e500 0x7fedf41a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf8c84640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fedf410e960 con 0x7fedf4101f80 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf8c84640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fedf410e7e0 con 0x7fedf419e500 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf8c84640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fedf410eae0 con 0x7fedf4102ae0 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf4101f80 0x7fedf4199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf4101f80 0x7fedf4199040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46740/0 (socket says 172.21.15.129:46740) 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf259c640 1 -- 172.21.15.129:0/3648424912 learned_addr learned my addr 172.21.15.129:0/3648424912 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf2d9d640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedf419e500 0x7fedf41a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedebfff640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf4102ae0 0x7fedf4199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf259c640 1 -- 172.21.15.129:0/3648424912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf4102ae0 msgr2=0x7fedf4199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf259c640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf4102ae0 0x7fedf4199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf259c640 1 -- 172.21.15.129:0/3648424912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedf419e500 msgr2=0x7fedf41a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf259c640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedf419e500 0x7fedf41a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf259c640 1 -- 172.21.15.129:0/3648424912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf41a0f90 con 0x7fedf4101f80 2024-05-20T17:56:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf259c640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf4101f80 0x7fedf4199040 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fede4006010 tx=0x7fede40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedeb7fe640 1 -- 172.21.15.129:0/3648424912 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede403f030 con 0x7fedf4101f80 2024-05-20T17:56:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf8c84640 1 -- 172.21.15.129:0/3648424912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf41a1220 con 0x7fedf4101f80 2024-05-20T17:56:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedeb7fe640 1 -- 172.21.15.129:0/3648424912 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fede4003ce0 con 0x7fedf4101f80 2024-05-20T17:56:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedeb7fe640 1 -- 172.21.15.129:0/3648424912 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede4035070 con 0x7fedf4101f80 2024-05-20T17:56:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.960+0000 7fedf8c84640 1 -- 172.21.15.129:0/3648424912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf41a1710 con 0x7fedf4101f80 2024-05-20T17:56:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.961+0000 7fedf8c84640 1 -- 172.21.15.129:0/3648424912 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf406a350 con 0x7fedf4101f80 2024-05-20T17:56:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.963+0000 7fedeb7fe640 1 -- 172.21.15.129:0/3648424912 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fede4042030 con 0x7fedf4101f80 2024-05-20T17:56:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.963+0000 7fedeb7fe640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fede00a2c20 0x7fede00a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:52.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.963+0000 7fedebfff640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fede00a2c20 0x7fede00a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:52.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.963+0000 7fedeb7fe640 1 -- 172.21.15.129:0/3648424912 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fede40ef870 con 0x7fedf4101f80 2024-05-20T17:56:52.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.964+0000 7fedebfff640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fede00a2c20 0x7fede00a5130 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feddc005fd0 tx=0x7feddc005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:52.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:52.967+0000 7fedeb7fe640 1 -- 172.21.15.129:0/3648424912 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fede40bc820 con 0x7fedf4101f80 2024-05-20T17:56:53.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.141+0000 7fedf8c84640 1 -- 172.21.15.129:0/3648424912 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fedf402d590 con 0x7fede00a2c20 2024-05-20T17:56:53.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.148+0000 7fedeb7fe640 1 -- 172.21.15.129:0/3648424912 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258857 (secure 0 0 0) 0x7fedf402d590 con 0x7fede00a2c20 2024-05-20T17:56:53.149 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:53.150 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-05-20T17:56:53.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 -- 172.21.15.129:0/3648424912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fede00a2c20 msgr2=0x7fede00a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:53.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fede00a2c20 0x7fede00a5130 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feddc005fd0 tx=0x7feddc005850 comp rx=0 tx=0).stop 2024-05-20T17:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 -- 172.21.15.129:0/3648424912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf4101f80 msgr2=0x7fedf4199040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf4101f80 0x7fedf4199040 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fede4006010 tx=0x7fede40043b0 comp rx=0 tx=0).stop 2024-05-20T17:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 -- 172.21.15.129:0/3648424912 shutdown_connections 2024-05-20T17:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fede00a2c20 0x7fede00a5130 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedf419e500 0x7fedf41a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fedf4102ae0 0x7fedf4199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 --2- 172.21.15.129:0/3648424912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf4101f80 0x7fedf4199040 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 -- 172.21.15.129:0/3648424912 >> 172.21.15.129:0/3648424912 conn(0x7fedf40fc320 msgr2=0x7fedf40fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 -- 172.21.15.129:0/3648424912 shutdown_connections 2024-05-20T17:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:53.151+0000 7fedf8c84640 1 -- 172.21.15.129:0/3648424912 wait complete. 2024-05-20T17:56:53.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:53 smithi129 ceph-mon[32029]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:53.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:53 smithi129 ceph-mon[32029]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-20T17:56:53.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:53 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2949988027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:53.160 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:53 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2375148432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:53.161 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-05-20T17:56:51.706836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":137,"num_read_kb":120,"num_write":226,"num_write_kb":2952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176,"ondisk_log_size":176,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":246,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364428,"kb_used_data":11040,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124577396,"statfs":{"total":1151940427776,"available":1151567253504,"internally_reserved":0,"allocated":11304960,"data_stored":7086166,"data_compressed":106947,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.003656"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.754034+0000","last_change":"2024-05-20T17:56:45.746793+0000","last_active":"2024-05-20T17:56:45.754034+0000","last_peered":"2024-05-20T17:56:45.754034+0000","last_clean":"2024-05-20T17:56:45.754034+0000","last_became_active":"2024-05-20T17:56:45.746244+0000","last_became_peered":"2024-05-20T17:56:45.746244+0000","last_unstale":"2024-05-20T17:56:45.754034+0000","last_undegraded":"2024-05-20T17:56:45.754034+0000","last_fullsized":"2024-05-20T17:56:45.754034+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:20:36.166355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618845+0000","last_change":"2024-05-20T17:56:45.744538+0000","last_active":"2024-05-20T17:56:48.618845+0000","last_peered":"2024-05-20T17:56:48.618845+0000","last_clean":"2024-05-20T17:56:48.618845+0000","last_became_active":"2024-05-20T17:56:45.744213+0000","last_became_peered":"2024-05-20T17:56:45.744213+0000","last_unstale":"2024-05-20T17:56:48.618845+0000","last_undegraded":"2024-05-20T17:56:48.618845+0000","last_fullsized":"2024-05-20T17:56:48.618845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:19:26.163003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618644+0000","last_change":"2024-05-20T17:56:45.749725+0000","last_active":"2024-05-20T17:56:48.618644+0000","last_peered":"2024-05-20T17:56:48.618644+0000","last_clean":"2024-05-20T17:56:48.618644+0000","last_became_active":"2024-05-20T17:56:45.749392+0000","last_became_peered":"2024-05-20T17:56:45.749392+0000","last_unstale":"2024-05-20T17:56:48.618644+0000","last_undegraded":"2024-05-20T17:56:48.618644+0000","last_fullsized":"2024-05-20T17:56:48.618644+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:43:33.184192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738902+0000","last_change":"2024-05-20T17:56:45.755191+0000","last_active":"2024-05-20T17:56:46.738902+0000","last_peered":"2024-05-20T17:56:46.738902+0000","last_clean":"2024-05-20T17:56:46.738902+0000","last_became_active":"2024-05-20T17:56:45.753524+0000","last_became_peered":"2024-05-20T17:56:45.753524+0000","last_unstale":"2024-05-20T17:56:46.738902+0000","last_undegraded":"2024-05-20T17:56:46.738902+0000","last_fullsized":"2024-05-20T17:56:46.738902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:11:23.099919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888391+0000","last_change":"2024-05-20T17:56:45.756291+0000","last_active":"2024-05-20T17:56:46.888391+0000","last_peered":"2024-05-20T17:56:46.888391+0000","last_clean":"2024-05-20T17:56:46.888391+0000","last_became_active":"2024-05-20T17:56:45.755910+0000","last_became_peered":"2024-05-20T17:56:45.755910+0000","last_unstale":"2024-05-20T17:56:46.888391+0000","last_undegraded":"2024-05-20T17:56:46.888391+0000","last_fullsized":"2024-05-20T17:56:46.888391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:37:36.941012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.750755+0000","last_change":"2024-05-20T17:56:45.748756+0000","last_active":"2024-05-20T17:56:45.750755+0000","last_peered":"2024-05-20T17:56:45.750755+0000","last_clean":"2024-05-20T17:56:45.750755+0000","last_became_active":"2024-05-20T17:56:45.748406+0000","last_became_peered":"2024-05-20T17:56:45.748406+0000","last_unstale":"2024-05-20T17:56:45.750755+0000","last_undegraded":"2024-05-20T17:56:45.750755+0000","last_fullsized":"2024-05-20T17:56:45.750755+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:11:37.959445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585243+0000","last_change":"2024-05-20T17:56:45.761108+0000","last_active":"2024-05-20T17:56:47.585243+0000","last_peered":"2024-05-20T17:56:47.585243+0000","last_clean":"2024-05-20T17:56:47.585243+0000","last_became_active":"2024-05-20T17:56:45.760322+0000","last_became_peered":"2024-05-20T17:56:45.760322+0000","last_unstale":"2024-05-20T17:56:47.585243+0000","last_undegraded":"2024-05-20T17:56:47.585243+0000","last_fullsized":"2024-05-20T17:56:47.585243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:44:01.754913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737037+0000","last_change":"2024-05-20T17:56:45.757387+0000","last_active":"2024-05-20T17:56:46.737037+0000","last_peered":"2024-05-20T17:56:46.737037+0000","last_clean":"2024-05-20T17:56:46.737037+0000","last_became_active":"2024-05-20T17:56:45.756374+0000","last_became_peered":"2024-05-20T17:56:45.756374+0000","last_unstale":"2024-05-20T17:56:46.737037+0000","last_undegraded":"2024-05-20T17:56:46.737037+0000","last_fullsized":"2024-05-20T17:56:46.737037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:53:52.751019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.918434+0000","last_change":"2024-05-20T17:56:45.751933+0000","last_active":"2024-05-20T17:56:46.918434+0000","last_peered":"2024-05-20T17:56:46.918434+0000","last_clean":"2024-05-20T17:56:46.918434+0000","last_became_active":"2024-05-20T17:56:45.751414+0000","last_became_peered":"2024-05-20T17:56:45.751414+0000","last_unstale":"2024-05-20T17:56:46.918434+0000","last_undegraded":"2024-05-20T17:56:46.918434+0000","last_fullsized":"2024-05-20T17:56:46.918434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:56:52.974830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.750734+0000","last_change":"2024-05-20T17:56:45.749511+0000","last_active":"2024-05-20T17:56:45.750734+0000","last_peered":"2024-05-20T17:56:45.750734+0000","last_clean":"2024-05-20T17:56:45.750734+0000","last_became_active":"2024-05-20T17:56:45.749211+0000","last_became_peered":"2024-05-20T17:56:45.749211+0000","last_unstale":"2024-05-20T17:56:45.750734+0000","last_undegraded":"2024-05-20T17:56:45.750734+0000","last_fullsized":"2024-05-20T17:56:45.750734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:22:35.456052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585399+0000","last_change":"2024-05-20T17:56:45.753743+0000","last_active":"2024-05-20T17:56:47.585399+0000","last_peered":"2024-05-20T17:56:47.585399+0000","last_clean":"2024-05-20T17:56:47.585399+0000","last_became_active":"2024-05-20T17:56:45.753378+0000","last_became_peered":"2024-05-20T17:56:45.753378+0000","last_unstale":"2024-05-20T17:56:47.585399+0000","last_undegraded":"2024-05-20T17:56:47.585399+0000","last_fullsized":"2024-05-20T17:56:47.585399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:54.563895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738825+0000","last_change":"2024-05-20T17:56:45.754890+0000","last_active":"2024-05-20T17:56:46.738825+0000","last_peered":"2024-05-20T17:56:46.738825+0000","last_clean":"2024-05-20T17:56:46.738825+0000","last_became_active":"2024-05-20T17:56:45.754333+0000","last_became_peered":"2024-05-20T17:56:45.754333+0000","last_unstale":"2024-05-20T17:56:46.738825+0000","last_undegraded":"2024-05-20T17:56:46.738825+0000","last_fullsized":"2024-05-20T17:56:46.738825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:51.025516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130635+0000","last_change":"2024-05-20T17:56:45.745662+0000","last_active":"2024-05-20T17:56:48.130635+0000","last_peered":"2024-05-20T17:56:48.130635+0000","last_clean":"2024-05-20T17:56:48.130635+0000","last_became_active":"2024-05-20T17:56:45.745281+0000","last_became_peered":"2024-05-20T17:56:45.745281+0000","last_unstale":"2024-05-20T17:56:48.130635+0000","last_undegraded":"2024-05-20T17:56:48.130635+0000","last_fullsized":"2024-05-20T17:56:48.130635+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:38:57.057301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618507+0000","last_change":"2024-05-20T17:56:45.749164+0000","last_active":"2024-05-20T17:56:48.618507+0000","last_peered":"2024-05-20T17:56:48.618507+0000","last_clean":"2024-05-20T17:56:48.618507+0000","last_became_active":"2024-05-20T17:56:45.748822+0000","last_became_peered":"2024-05-20T17:56:45.748822+0000","last_unstale":"2024-05-20T17:56:48.618507+0000","last_undegraded":"2024-05-20T17:56:48.618507+0000","last_fullsized":"2024-05-20T17:56:48.618507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:45:47.487116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618461+0000","last_change":"2024-05-20T17:56:45.751185+0000","last_active":"2024-05-20T17:56:48.618461+0000","last_peered":"2024-05-20T17:56:48.618461+0000","last_clean":"2024-05-20T17:56:48.618461+0000","last_became_active":"2024-05-20T17:56:45.750851+0000","last_became_peered":"2024-05-20T17:56:45.750851+0000","last_unstale":"2024-05-20T17:56:48.618461+0000","last_undegraded":"2024-05-20T17:56:48.618461+0000","last_fullsized":"2024-05-20T17:56:48.618461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:31:07.792717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.917837+0000","last_change":"2024-05-20T17:56:45.748101+0000","last_active":"2024-05-20T17:56:46.917837+0000","last_peered":"2024-05-20T17:56:46.917837+0000","last_clean":"2024-05-20T17:56:46.917837+0000","last_became_active":"2024-05-20T17:56:45.747754+0000","last_became_peered":"2024-05-20T17:56:45.747754+0000","last_unstale":"2024-05-20T17:56:46.917837+0000","last_undegraded":"2024-05-20T17:56:46.917837+0000","last_fullsized":"2024-05-20T17:56:46.917837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:24:54.702601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.754059+0000","last_change":"2024-05-20T17:56:45.747497+0000","last_active":"2024-05-20T17:56:45.754059+0000","last_peered":"2024-05-20T17:56:45.754059+0000","last_clean":"2024-05-20T17:56:45.754059+0000","last_became_active":"2024-05-20T17:56:45.747139+0000","last_became_peered":"2024-05-20T17:56:45.747139+0000","last_unstale":"2024-05-20T17:56:45.754059+0000","last_undegraded":"2024-05-20T17:56:45.754059+0000","last_fullsized":"2024-05-20T17:56:45.754059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:41:17.652537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.963938+0000","last_change":"2024-05-20T17:56:45.746194+0000","last_active":"2024-05-20T17:56:47.963938+0000","last_peered":"2024-05-20T17:56:47.963938+0000","last_clean":"2024-05-20T17:56:47.963938+0000","last_became_active":"2024-05-20T17:56:45.745809+0000","last_became_peered":"2024-05-20T17:56:45.745809+0000","last_unstale":"2024-05-20T17:56:47.963938+0000","last_undegraded":"2024-05-20T17:56:47.963938+0000","last_fullsized":"2024-05-20T17:56:47.963938+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:26:29.133790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518565+0000","last_change":"2024-05-20T17:56:45.749918+0000","last_active":"2024-05-20T17:56:47.518565+0000","last_peered":"2024-05-20T17:56:47.518565+0000","last_clean":"2024-05-20T17:56:47.518565+0000","last_became_active":"2024-05-20T17:56:45.749560+0000","last_became_peered":"2024-05-20T17:56:45.749560+0000","last_unstale":"2024-05-20T17:56:47.518565+0000","last_undegraded":"2024-05-20T17:56:47.518565+0000","last_fullsized":"2024-05-20T17:56:47.518565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:33:12.737206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.964018+0000","last_change":"2024-05-20T17:56:44.718812+0000","last_active":"2024-05-20T17:56:47.964018+0000","last_peered":"2024-05-20T17:56:47.964018+0000","last_clean":"2024-05-20T17:56:47.964018+0000","last_became_active":"2024-05-20T17:56:44.718496+0000","last_became_peered":"2024-05-20T17:56:44.718496+0000","last_unstale":"2024-05-20T17:56:47.964018+0000","last_undegraded":"2024-05-20T17:56:47.964018+0000","last_fullsized":"2024-05-20T17:56:47.964018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:40:50.059357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737470+0000","last_change":"2024-05-20T17:56:45.758330+0000","last_active":"2024-05-20T17:56:46.737470+0000","last_peered":"2024-05-20T17:56:46.737470+0000","last_clean":"2024-05-20T17:56:46.737470+0000","last_became_active":"2024-05-20T17:56:45.755873+0000","last_became_peered":"2024-05-20T17:56:45.755873+0000","last_unstale":"2024-05-20T17:56:46.737470+0000","last_undegraded":"2024-05-20T17:56:46.737470+0000","last_fullsized":"2024-05-20T17:56:46.737470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:53:31.653497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737626+0000","last_change":"2024-05-20T17:56:44.728508+0000","last_active":"2024-05-20T17:56:46.737626+0000","last_peered":"2024-05-20T17:56:46.737626+0000","last_clean":"2024-05-20T17:56:46.737626+0000","last_became_active":"2024-05-20T17:56:44.727875+0000","last_became_peered":"2024-05-20T17:56:44.727875+0000","last_unstale":"2024-05-20T17:56:46.737626+0000","last_undegraded":"2024-05-20T17:56:46.737626+0000","last_fullsized":"2024-05-20T17:56:46.737626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:12:14.349039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618930+0000","last_change":"2024-05-20T17:56:45.750000+0000","last_active":"2024-05-20T17:56:48.618930+0000","last_peered":"2024-05-20T17:56:48.618930+0000","last_clean":"2024-05-20T17:56:48.618930+0000","last_became_active":"2024-05-20T17:56:45.749644+0000","last_became_peered":"2024-05-20T17:56:45.749644+0000","last_unstale":"2024-05-20T17:56:48.618930+0000","last_undegraded":"2024-05-20T17:56:48.618930+0000","last_fullsized":"2024-05-20T17:56:48.618930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:32:46.085081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888739+0000","last_change":"2024-05-20T17:56:44.722692+0000","last_active":"2024-05-20T17:56:46.888739+0000","last_peered":"2024-05-20T17:56:46.888739+0000","last_clean":"2024-05-20T17:56:46.888739+0000","last_became_active":"2024-05-20T17:56:44.722170+0000","last_became_peered":"2024-05-20T17:56:44.722170+0000","last_unstale":"2024-05-20T17:56:46.888739+0000","last_undegraded":"2024-05-20T17:56:46.888739+0000","last_fullsized":"2024-05-20T17:56:46.888739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:38:03.679562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.918091+0000","last_change":"2024-05-20T17:56:45.751666+0000","last_active":"2024-05-20T17:56:46.918091+0000","last_peered":"2024-05-20T17:56:46.918091+0000","last_clean":"2024-05-20T17:56:46.918091+0000","last_became_active":"2024-05-20T17:56:45.751291+0000","last_became_peered":"2024-05-20T17:56:45.751291+0000","last_unstale":"2024-05-20T17:56:46.918091+0000","last_undegraded":"2024-05-20T17:56:46.918091+0000","last_fullsized":"2024-05-20T17:56:46.918091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:29:51.517363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888590+0000","last_change":"2024-05-20T17:56:44.721121+0000","last_active":"2024-05-20T17:56:46.888590+0000","last_peered":"2024-05-20T17:56:46.888590+0000","last_clean":"2024-05-20T17:56:46.888590+0000","last_became_active":"2024-05-20T17:56:44.720693+0000","last_became_peered":"2024-05-20T17:56:44.720693+0000","last_unstale":"2024-05-20T17:56:46.888590+0000","last_undegraded":"2024-05-20T17:56:46.888590+0000","last_fullsized":"2024-05-20T17:56:46.888590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:44.223828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738200+0000","last_change":"2024-05-20T17:56:45.745925+0000","last_active":"2024-05-20T17:56:46.738200+0000","last_peered":"2024-05-20T17:56:46.738200+0000","last_clean":"2024-05-20T17:56:46.738200+0000","last_became_active":"2024-05-20T17:56:45.745303+0000","last_became_peered":"2024-05-20T17:56:45.745303+0000","last_unstale":"2024-05-20T17:56:46.738200+0000","last_undegraded":"2024-05-20T17:56:46.738200+0000","last_fullsized":"2024-05-20T17:56:46.738200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:41:00.230681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737929+0000","last_change":"2024-05-20T17:56:44.719048+0000","last_active":"2024-05-20T17:56:46.737929+0000","last_peered":"2024-05-20T17:56:46.737929+0000","last_clean":"2024-05-20T17:56:46.737929+0000","last_became_active":"2024-05-20T17:56:44.718397+0000","last_became_peered":"2024-05-20T17:56:44.718397+0000","last_unstale":"2024-05-20T17:56:46.737929+0000","last_undegraded":"2024-05-20T17:56:46.737929+0000","last_fullsized":"2024-05-20T17:56:46.737929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:11:53.898226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888498+0000","last_change":"2024-05-20T17:56:45.752395+0000","last_active":"2024-05-20T17:56:46.888498+0000","last_peered":"2024-05-20T17:56:46.888498+0000","last_clean":"2024-05-20T17:56:46.888498+0000","last_became_active":"2024-05-20T17:56:45.751940+0000","last_became_peered":"2024-05-20T17:56:45.751940+0000","last_unstale":"2024-05-20T17:56:46.888498+0000","last_undegraded":"2024-05-20T17:56:46.888498+0000","last_fullsized":"2024-05-20T17:56:46.888498+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:33:57.780042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.968759+0000","last_change":"2024-05-20T17:56:44.714712+0000","last_active":"2024-05-20T17:56:45.968759+0000","last_peered":"2024-05-20T17:56:45.968759+0000","last_clean":"2024-05-20T17:56:45.968759+0000","last_became_active":"2024-05-20T17:56:44.714396+0000","last_became_peered":"2024-05-20T17:56:44.714396+0000","last_unstale":"2024-05-20T17:56:45.968759+0000","last_undegraded":"2024-05-20T17:56:45.968759+0000","last_fullsized":"2024-05-20T17:56:45.968759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:21:04.831289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738281+0000","last_change":"2024-05-20T17:56:45.758160+0000","last_active":"2024-05-20T17:56:46.738281+0000","last_peered":"2024-05-20T17:56:46.738281+0000","last_clean":"2024-05-20T17:56:46.738281+0000","last_became_active":"2024-05-20T17:56:45.757405+0000","last_became_peered":"2024-05-20T17:56:45.757405+0000","last_unstale":"2024-05-20T17:56:46.738281+0000","last_undegraded":"2024-05-20T17:56:46.738281+0000","last_fullsized":"2024-05-20T17:56:46.738281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:17.625246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130033+0000","last_change":"2024-05-20T17:56:44.716342+0000","last_active":"2024-05-20T17:56:48.130033+0000","last_peered":"2024-05-20T17:56:48.130033+0000","last_clean":"2024-05-20T17:56:48.130033+0000","last_became_active":"2024-05-20T17:56:44.716010+0000","last_became_peered":"2024-05-20T17:56:44.716010+0000","last_unstale":"2024-05-20T17:56:48.130033+0000","last_undegraded":"2024-05-20T17:56:48.130033+0000","last_fullsized":"2024-05-20T17:56:48.130033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:00:25.073312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737356+0000","last_change":"2024-05-20T17:56:45.755288+0000","last_active":"2024-05-20T17:56:46.737356+0000","last_peered":"2024-05-20T17:56:46.737356+0000","last_clean":"2024-05-20T17:56:46.737356+0000","last_became_active":"2024-05-20T17:56:45.754952+0000","last_became_peered":"2024-05-20T17:56:45.754952+0000","last_unstale":"2024-05-20T17:56:46.737356+0000","last_undegraded":"2024-05-20T17:56:46.737356+0000","last_fullsized":"2024-05-20T17:56:46.737356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:44:27.631722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737462+0000","last_change":"2024-05-20T17:56:44.717887+0000","last_active":"2024-05-20T17:56:46.737462+0000","last_peered":"2024-05-20T17:56:46.737462+0000","last_clean":"2024-05-20T17:56:46.737462+0000","last_became_active":"2024-05-20T17:56:44.717581+0000","last_became_peered":"2024-05-20T17:56:44.717581+0000","last_unstale":"2024-05-20T17:56:46.737462+0000","last_undegraded":"2024-05-20T17:56:46.737462+0000","last_fullsized":"2024-05-20T17:56:46.737462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:42:54.703112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.964000+0000","last_change":"2024-05-20T17:56:45.748448+0000","last_active":"2024-05-20T17:56:47.964000+0000","last_peered":"2024-05-20T17:56:47.964000+0000","last_clean":"2024-05-20T17:56:47.964000+0000","last_became_active":"2024-05-20T17:56:45.748016+0000","last_became_peered":"2024-05-20T17:56:45.748016+0000","last_unstale":"2024-05-20T17:56:47.964000+0000","last_undegraded":"2024-05-20T17:56:47.964000+0000","last_fullsized":"2024-05-20T17:56:47.964000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:31:39.350477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.737414+0000","last_change":"2024-05-20T17:56:44.709483+0000","last_active":"2024-05-20T17:56:45.737414+0000","last_peered":"2024-05-20T17:56:45.737414+0000","last_clean":"2024-05-20T17:56:45.737414+0000","last_became_active":"2024-05-20T17:56:44.708649+0000","last_became_peered":"2024-05-20T17:56:44.708649+0000","last_unstale":"2024-05-20T17:56:45.737414+0000","last_undegraded":"2024-05-20T17:56:45.737414+0000","last_fullsized":"2024-05-20T17:56:45.737414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:31:46.580256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.918300+0000","last_change":"2024-05-20T17:56:45.752279+0000","last_active":"2024-05-20T17:56:46.918300+0000","last_peered":"2024-05-20T17:56:46.918300+0000","last_clean":"2024-05-20T17:56:46.918300+0000","last_became_active":"2024-05-20T17:56:45.751790+0000","last_became_peered":"2024-05-20T17:56:45.751790+0000","last_unstale":"2024-05-20T17:56:46.918300+0000","last_undegraded":"2024-05-20T17:56:46.918300+0000","last_fullsized":"2024-05-20T17:56:46.918300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:06:38.113559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585434+0000","last_change":"2024-05-20T17:56:44.726040+0000","last_active":"2024-05-20T17:56:47.585434+0000","last_peered":"2024-05-20T17:56:47.585434+0000","last_clean":"2024-05-20T17:56:47.585434+0000","last_became_active":"2024-05-20T17:56:44.725714+0000","last_became_peered":"2024-05-20T17:56:44.725714+0000","last_unstale":"2024-05-20T17:56:47.585434+0000","last_undegraded":"2024-05-20T17:56:47.585434+0000","last_fullsized":"2024-05-20T17:56:47.585434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:22:59.081600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618909+0000","last_change":"2024-05-20T17:56:45.756464+0000","last_active":"2024-05-20T17:56:48.618909+0000","last_peered":"2024-05-20T17:56:48.618909+0000","last_clean":"2024-05-20T17:56:48.618909+0000","last_became_active":"2024-05-20T17:56:45.755922+0000","last_became_peered":"2024-05-20T17:56:45.755922+0000","last_unstale":"2024-05-20T17:56:48.618909+0000","last_undegraded":"2024-05-20T17:56:48.618909+0000","last_fullsized":"2024-05-20T17:56:48.618909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:01:53.689939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738135+0000","last_change":"2024-05-20T17:56:44.709548+0000","last_active":"2024-05-20T17:56:46.738135+0000","last_peered":"2024-05-20T17:56:46.738135+0000","last_clean":"2024-05-20T17:56:46.738135+0000","last_became_active":"2024-05-20T17:56:44.709209+0000","last_became_peered":"2024-05-20T17:56:44.709209+0000","last_unstale":"2024-05-20T17:56:46.738135+0000","last_undegraded":"2024-05-20T17:56:46.738135+0000","last_fullsized":"2024-05-20T17:56:46.738135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:28:13.118336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.918263+0000","last_change":"2024-05-20T17:56:45.755141+0000","last_active":"2024-05-20T17:56:46.918263+0000","last_peered":"2024-05-20T17:56:46.918263+0000","last_clean":"2024-05-20T17:56:46.918263+0000","last_became_active":"2024-05-20T17:56:45.754786+0000","last_became_peered":"2024-05-20T17:56:45.754786+0000","last_unstale":"2024-05-20T17:56:46.918263+0000","last_undegraded":"2024-05-20T17:56:46.918263+0000","last_fullsized":"2024-05-20T17:56:46.918263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:45:21.044602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'6","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.972399+0000","last_change":"2024-05-20T17:56:44.713578+0000","last_active":"2024-05-20T17:56:45.972399+0000","last_peered":"2024-05-20T17:56:45.972399+0000","last_clean":"2024-05-20T17:56:45.972399+0000","last_became_active":"2024-05-20T17:56:44.713203+0000","last_became_peered":"2024-05-20T17:56:44.713203+0000","last_unstale":"2024-05-20T17:56:45.972399+0000","last_undegraded":"2024-05-20T17:56:45.972399+0000","last_fullsized":"2024-05-20T17:56:45.972399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:36:54.386036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888624+0000","last_change":"2024-05-20T17:56:45.752023+0000","last_active":"2024-05-20T17:56:46.888624+0000","last_peered":"2024-05-20T17:56:46.888624+0000","last_clean":"2024-05-20T17:56:46.888624+0000","last_became_active":"2024-05-20T17:56:45.751672+0000","last_became_peered":"2024-05-20T17:56:45.751672+0000","last_unstale":"2024-05-20T17:56:46.888624+0000","last_undegraded":"2024-05-20T17:56:46.888624+0000","last_fullsized":"2024-05-20T17:56:46.888624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:23:49.127016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518208+0000","last_change":"2024-05-20T17:56:44.717336+0000","last_active":"2024-05-20T17:56:47.518208+0000","last_peered":"2024-05-20T17:56:47.518208+0000","last_clean":"2024-05-20T17:56:47.518208+0000","last_became_active":"2024-05-20T17:56:44.717016+0000","last_became_peered":"2024-05-20T17:56:44.717016+0000","last_unstale":"2024-05-20T17:56:47.518208+0000","last_undegraded":"2024-05-20T17:56:47.518208+0000","last_fullsized":"2024-05-20T17:56:47.518208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:50:46.433461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130207+0000","last_change":"2024-05-20T17:56:45.752411+0000","last_active":"2024-05-20T17:56:48.130207+0000","last_peered":"2024-05-20T17:56:48.130207+0000","last_clean":"2024-05-20T17:56:48.130207+0000","last_became_active":"2024-05-20T17:56:45.752078+0000","last_became_peered":"2024-05-20T17:56:45.752078+0000","last_unstale":"2024-05-20T17:56:48.130207+0000","last_undegraded":"2024-05-20T17:56:48.130207+0000","last_fullsized":"2024-05-20T17:56:48.130207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:22:15.560666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737897+0000","last_change":"2024-05-20T17:56:44.723262+0000","last_active":"2024-05-20T17:56:46.737897+0000","last_peered":"2024-05-20T17:56:46.737897+0000","last_clean":"2024-05-20T17:56:46.737897+0000","last_became_active":"2024-05-20T17:56:44.722865+0000","last_became_peered":"2024-05-20T17:56:44.722865+0000","last_unstale":"2024-05-20T17:56:46.737897+0000","last_undegraded":"2024-05-20T17:56:46.737897+0000","last_fullsized":"2024-05-20T17:56:46.737897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:15:02.820718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585443+0000","last_change":"2024-05-20T17:56:45.751126+0000","last_active":"2024-05-20T17:56:47.585443+0000","last_peered":"2024-05-20T17:56:47.585443+0000","last_clean":"2024-05-20T17:56:47.585443+0000","last_became_active":"2024-05-20T17:56:45.750042+0000","last_became_peered":"2024-05-20T17:56:45.750042+0000","last_unstale":"2024-05-20T17:56:47.585443+0000","last_undegraded":"2024-05-20T17:56:47.585443+0000","last_fullsized":"2024-05-20T17:56:47.585443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:48:35.371790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738600+0000","last_change":"2024-05-20T17:56:44.718396+0000","last_active":"2024-05-20T17:56:46.738600+0000","last_peered":"2024-05-20T17:56:46.738600+0000","last_clean":"2024-05-20T17:56:46.738600+0000","last_became_active":"2024-05-20T17:56:44.718031+0000","last_became_peered":"2024-05-20T17:56:44.718031+0000","last_unstale":"2024-05-20T17:56:46.738600+0000","last_undegraded":"2024-05-20T17:56:46.738600+0000","last_fullsized":"2024-05-20T17:56:46.738600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:42:40.955520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.584993+0000","last_change":"2024-05-20T17:56:45.759302+0000","last_active":"2024-05-20T17:56:47.584993+0000","last_peered":"2024-05-20T17:56:47.584993+0000","last_clean":"2024-05-20T17:56:47.584993+0000","last_became_active":"2024-05-20T17:56:45.758889+0000","last_became_peered":"2024-05-20T17:56:45.758889+0000","last_unstale":"2024-05-20T17:56:47.584993+0000","last_undegraded":"2024-05-20T17:56:47.584993+0000","last_fullsized":"2024-05-20T17:56:47.584993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:40.685471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888157+0000","last_change":"2024-05-20T17:56:44.726502+0000","last_active":"2024-05-20T17:56:46.888157+0000","last_peered":"2024-05-20T17:56:46.888157+0000","last_clean":"2024-05-20T17:56:46.888157+0000","last_became_active":"2024-05-20T17:56:44.726168+0000","last_became_peered":"2024-05-20T17:56:44.726168+0000","last_unstale":"2024-05-20T17:56:46.888157+0000","last_undegraded":"2024-05-20T17:56:46.888157+0000","last_fullsized":"2024-05-20T17:56:46.888157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:45:14.523968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737716+0000","last_change":"2024-05-20T17:56:45.759355+0000","last_active":"2024-05-20T17:56:46.737716+0000","last_peered":"2024-05-20T17:56:46.737716+0000","last_clean":"2024-05-20T17:56:46.737716+0000","last_became_active":"2024-05-20T17:56:45.758920+0000","last_became_peered":"2024-05-20T17:56:45.758920+0000","last_unstale":"2024-05-20T17:56:46.737716+0000","last_undegraded":"2024-05-20T17:56:46.737716+0000","last_fullsized":"2024-05-20T17:56:46.737716+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:46:02.372250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888717+0000","last_change":"2024-05-20T17:56:44.718507+0000","last_active":"2024-05-20T17:56:46.888717+0000","last_peered":"2024-05-20T17:56:46.888717+0000","last_clean":"2024-05-20T17:56:46.888717+0000","last_became_active":"2024-05-20T17:56:44.718188+0000","last_became_peered":"2024-05-20T17:56:44.718188+0000","last_unstale":"2024-05-20T17:56:46.888717+0000","last_undegraded":"2024-05-20T17:56:46.888717+0000","last_fullsized":"2024-05-20T17:56:46.888717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:43:59.792942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.754182+0000","last_change":"2024-05-20T17:56:45.751361+0000","last_active":"2024-05-20T17:56:45.754182+0000","last_peered":"2024-05-20T17:56:45.754182+0000","last_clean":"2024-05-20T17:56:45.754182+0000","last_became_active":"2024-05-20T17:56:45.750032+0000","last_became_peered":"2024-05-20T17:56:45.750032+0000","last_unstale":"2024-05-20T17:56:45.754182+0000","last_undegraded":"2024-05-20T17:56:45.754182+0000","last_fullsized":"2024-05-20T17:56:45.754182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:24:43.801133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.963937+0000","last_change":"2024-05-20T17:56:44.714404+0000","last_active":"2024-05-20T17:56:47.963937+0000","last_peered":"2024-05-20T17:56:47.963937+0000","last_clean":"2024-05-20T17:56:47.963937+0000","last_became_active":"2024-05-20T17:56:44.714072+0000","last_became_peered":"2024-05-20T17:56:44.714072+0000","last_unstale":"2024-05-20T17:56:47.963937+0000","last_undegraded":"2024-05-20T17:56:47.963937+0000","last_fullsized":"2024-05-20T17:56:47.963937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:39:53.983513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.750849+0000","last_change":"2024-05-20T17:56:45.749785+0000","last_active":"2024-05-20T17:56:45.750849+0000","last_peered":"2024-05-20T17:56:45.750849+0000","last_clean":"2024-05-20T17:56:45.750849+0000","last_became_active":"2024-05-20T17:56:45.749457+0000","last_became_peered":"2024-05-20T17:56:45.749457+0000","last_unstale":"2024-05-20T17:56:45.750849+0000","last_undegraded":"2024-05-20T17:56:45.750849+0000","last_fullsized":"2024-05-20T17:56:45.750849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:00:16.303032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585282+0000","last_change":"2024-05-20T17:56:44.722787+0000","last_active":"2024-05-20T17:56:47.585282+0000","last_peered":"2024-05-20T17:56:47.585282+0000","last_clean":"2024-05-20T17:56:47.585282+0000","last_became_active":"2024-05-20T17:56:44.722214+0000","last_became_peered":"2024-05-20T17:56:44.722214+0000","last_unstale":"2024-05-20T17:56:47.585282+0000","last_undegraded":"2024-05-20T17:56:47.585282+0000","last_fullsized":"2024-05-20T17:56:47.585282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:40:16.705718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.758042+0000","last_change":"2024-05-20T17:56:45.757515+0000","last_active":"2024-05-20T17:56:45.758042+0000","last_peered":"2024-05-20T17:56:45.758042+0000","last_clean":"2024-05-20T17:56:45.758042+0000","last_became_active":"2024-05-20T17:56:45.757174+0000","last_became_peered":"2024-05-20T17:56:45.757174+0000","last_unstale":"2024-05-20T17:56:45.758042+0000","last_undegraded":"2024-05-20T17:56:45.758042+0000","last_fullsized":"2024-05-20T17:56:45.758042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:36:59.941562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737726+0000","last_change":"2024-05-20T17:56:45.759018+0000","last_active":"2024-05-20T17:56:46.737726+0000","last_peered":"2024-05-20T17:56:46.737726+0000","last_clean":"2024-05-20T17:56:46.737726+0000","last_became_active":"2024-05-20T17:56:45.757901+0000","last_became_peered":"2024-05-20T17:56:45.757901+0000","last_unstale":"2024-05-20T17:56:46.737726+0000","last_undegraded":"2024-05-20T17:56:46.737726+0000","last_fullsized":"2024-05-20T17:56:46.737726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:25:13.554683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.129927+0000","last_change":"2024-05-20T17:56:44.724642+0000","last_active":"2024-05-20T17:56:48.129927+0000","last_peered":"2024-05-20T17:56:48.129927+0000","last_clean":"2024-05-20T17:56:48.129927+0000","last_became_active":"2024-05-20T17:56:44.724311+0000","last_became_peered":"2024-05-20T17:56:44.724311+0000","last_unstale":"2024-05-20T17:56:48.129927+0000","last_undegraded":"2024-05-20T17:56:48.129927+0000","last_fullsized":"2024-05-20T17:56:48.129927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:47:35.336236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737319+0000","last_change":"2024-05-20T17:56:45.758466+0000","last_active":"2024-05-20T17:56:46.737319+0000","last_peered":"2024-05-20T17:56:46.737319+0000","last_clean":"2024-05-20T17:56:46.737319+0000","last_became_active":"2024-05-20T17:56:45.758126+0000","last_became_peered":"2024-05-20T17:56:45.758126+0000","last_unstale":"2024-05-20T17:56:46.737319+0000","last_undegraded":"2024-05-20T17:56:46.737319+0000","last_fullsized":"2024-05-20T17:56:46.737319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:36.830828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738744+0000","last_change":"2024-05-20T17:56:44.711173+0000","last_active":"2024-05-20T17:56:46.738744+0000","last_peered":"2024-05-20T17:56:46.738744+0000","last_clean":"2024-05-20T17:56:46.738744+0000","last_became_active":"2024-05-20T17:56:44.710778+0000","last_became_peered":"2024-05-20T17:56:44.710778+0000","last_unstale":"2024-05-20T17:56:46.738744+0000","last_undegraded":"2024-05-20T17:56:46.738744+0000","last_fullsized":"2024-05-20T17:56:46.738744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:20:14.150346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.584782+0000","last_change":"2024-05-20T17:55:34.871483+0000","last_active":"2024-05-20T17:56:47.584782+0000","last_peered":"2024-05-20T17:56:47.584782+0000","last_clean":"2024-05-20T17:56:47.584782+0000","last_became_active":"2024-05-20T17:55:34.871028+0000","last_became_peered":"2024-05-20T17:55:34.871028+0000","last_unstale":"2024-05-20T17:56:47.584782+0000","last_undegraded":"2024-05-20T17:56:47.584782+0000","last_fullsized":"2024-05-20T17:56:47.584782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_clean_scrub_stamp":"2024-05-20T17:52:08.599715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:29:40.159577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.129999+0000","last_change":"2024-05-20T17:56:45.747725+0000","last_active":"2024-05-20T17:56:48.129999+0000","last_peered":"2024-05-20T17:56:48.129999+0000","last_clean":"2024-05-20T17:56:48.129999+0000","last_became_active":"2024-05-20T17:56:45.747232+0000","last_became_peered":"2024-05-20T17:56:45.747232+0000","last_unstale":"2024-05-20T17:56:48.129999+0000","last_undegraded":"2024-05-20T17:56:48.129999+0000","last_fullsized":"2024-05-20T17:56:48.129999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:09:26.589905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585189+0000","last_change":"2024-05-20T17:56:44.721004+0000","last_active":"2024-05-20T17:56:47.585189+0000","last_peered":"2024-05-20T17:56:47.585189+0000","last_clean":"2024-05-20T17:56:47.585189+0000","last_became_active":"2024-05-20T17:56:44.720620+0000","last_became_peered":"2024-05-20T17:56:44.720620+0000","last_unstale":"2024-05-20T17:56:47.585189+0000","last_undegraded":"2024-05-20T17:56:47.585189+0000","last_fullsized":"2024-05-20T17:56:47.585189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:42:14.710663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888596+0000","last_change":"2024-05-20T17:56:44.714754+0000","last_active":"2024-05-20T17:56:46.888596+0000","last_peered":"2024-05-20T17:56:46.888596+0000","last_clean":"2024-05-20T17:56:46.888596+0000","last_became_active":"2024-05-20T17:56:44.714378+0000","last_became_peered":"2024-05-20T17:56:44.714378+0000","last_unstale":"2024-05-20T17:56:46.888596+0000","last_undegraded":"2024-05-20T17:56:46.888596+0000","last_fullsized":"2024-05-20T17:56:46.888596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:46:53.969960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130363+0000","last_change":"2024-05-20T17:56:45.742415+0000","last_active":"2024-05-20T17:56:48.130363+0000","last_peered":"2024-05-20T17:56:48.130363+0000","last_clean":"2024-05-20T17:56:48.130363+0000","last_became_active":"2024-05-20T17:56:45.741941+0000","last_became_peered":"2024-05-20T17:56:45.741941+0000","last_unstale":"2024-05-20T17:56:48.130363+0000","last_undegraded":"2024-05-20T17:56:48.130363+0000","last_fullsized":"2024-05-20T17:56:48.130363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:20:03.191021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.964038+0000","last_change":"2024-05-20T17:56:44.724361+0000","last_active":"2024-05-20T17:56:47.964038+0000","last_peered":"2024-05-20T17:56:47.964038+0000","last_clean":"2024-05-20T17:56:47.964038+0000","last_became_active":"2024-05-20T17:56:44.724033+0000","last_became_peered":"2024-05-20T17:56:44.724033+0000","last_unstale":"2024-05-20T17:56:47.964038+0000","last_undegraded":"2024-05-20T17:56:47.964038+0000","last_fullsized":"2024-05-20T17:56:47.964038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:10:13.756606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737678+0000","last_change":"2024-05-20T17:56:45.754216+0000","last_active":"2024-05-20T17:56:46.737678+0000","last_peered":"2024-05-20T17:56:46.737678+0000","last_clean":"2024-05-20T17:56:46.737678+0000","last_became_active":"2024-05-20T17:56:45.753855+0000","last_became_peered":"2024-05-20T17:56:45.753855+0000","last_unstale":"2024-05-20T17:56:46.737678+0000","last_undegraded":"2024-05-20T17:56:46.737678+0000","last_fullsized":"2024-05-20T17:56:46.737678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:14:15.664117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.738850+0000","last_change":"2024-05-20T17:56:44.716296+0000","last_active":"2024-05-20T17:56:46.738850+0000","last_peered":"2024-05-20T17:56:46.738850+0000","last_clean":"2024-05-20T17:56:46.738850+0000","last_became_active":"2024-05-20T17:56:44.715952+0000","last_became_peered":"2024-05-20T17:56:44.715952+0000","last_unstale":"2024-05-20T17:56:46.738850+0000","last_undegraded":"2024-05-20T17:56:46.738850+0000","last_fullsized":"2024-05-20T17:56:46.738850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:52:37.986399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618751+0000","last_change":"2024-05-20T17:56:45.751874+0000","last_active":"2024-05-20T17:56:48.618751+0000","last_peered":"2024-05-20T17:56:48.618751+0000","last_clean":"2024-05-20T17:56:48.618751+0000","last_became_active":"2024-05-20T17:56:45.751534+0000","last_became_peered":"2024-05-20T17:56:45.751534+0000","last_unstale":"2024-05-20T17:56:48.618751+0000","last_undegraded":"2024-05-20T17:56:48.618751+0000","last_fullsized":"2024-05-20T17:56:48.618751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:05:27.248115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.917819+0000","last_change":"2024-05-20T17:56:44.724271+0000","last_active":"2024-05-20T17:56:46.917819+0000","last_peered":"2024-05-20T17:56:46.917819+0000","last_clean":"2024-05-20T17:56:46.917819+0000","last_became_active":"2024-05-20T17:56:44.723947+0000","last_became_peered":"2024-05-20T17:56:44.723947+0000","last_unstale":"2024-05-20T17:56:46.917819+0000","last_undegraded":"2024-05-20T17:56:46.917819+0000","last_fullsized":"2024-05-20T17:56:46.917819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:32:13.482059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888363+0000","last_change":"2024-05-20T17:56:45.751296+0000","last_active":"2024-05-20T17:56:46.888363+0000","last_peered":"2024-05-20T17:56:46.888363+0000","last_clean":"2024-05-20T17:56:46.888363+0000","last_became_active":"2024-05-20T17:56:45.748035+0000","last_became_peered":"2024-05-20T17:56:45.748035+0000","last_unstale":"2024-05-20T17:56:46.888363+0000","last_undegraded":"2024-05-20T17:56:46.888363+0000","last_fullsized":"2024-05-20T17:56:46.888363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:37:43.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737075+0000","last_change":"2024-05-20T17:56:44.720461+0000","last_active":"2024-05-20T17:56:46.737075+0000","last_peered":"2024-05-20T17:56:46.737075+0000","last_clean":"2024-05-20T17:56:46.737075+0000","last_became_active":"2024-05-20T17:56:44.720115+0000","last_became_peered":"2024-05-20T17:56:44.720115+0000","last_unstale":"2024-05-20T17:56:46.737075+0000","last_undegraded":"2024-05-20T17:56:46.737075+0000","last_fullsized":"2024-05-20T17:56:46.737075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:14:47.330193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.752724+0000","last_change":"2024-05-20T17:56:45.745025+0000","last_active":"2024-05-20T17:56:45.752724+0000","last_peered":"2024-05-20T17:56:45.752724+0000","last_clean":"2024-05-20T17:56:45.752724+0000","last_became_active":"2024-05-20T17:56:45.744626+0000","last_became_peered":"2024-05-20T17:56:45.744626+0000","last_unstale":"2024-05-20T17:56:45.752724+0000","last_undegraded":"2024-05-20T17:56:45.752724+0000","last_fullsized":"2024-05-20T17:56:45.752724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:31:26.096534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518264+0000","last_change":"2024-05-20T17:56:44.726099+0000","last_active":"2024-05-20T17:56:47.518264+0000","last_peered":"2024-05-20T17:56:47.518264+0000","last_clean":"2024-05-20T17:56:47.518264+0000","last_became_active":"2024-05-20T17:56:44.725714+0000","last_became_peered":"2024-05-20T17:56:44.725714+0000","last_unstale":"2024-05-20T17:56:47.518264+0000","last_undegraded":"2024-05-20T17:56:47.518264+0000","last_fullsized":"2024-05-20T17:56:47.518264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:51.326907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737317+0000","last_change":"2024-05-20T17:56:45.757376+0000","last_active":"2024-05-20T17:56:46.737317+0000","last_peered":"2024-05-20T17:56:46.737317+0000","last_clean":"2024-05-20T17:56:46.737317+0000","last_became_active":"2024-05-20T17:56:45.757053+0000","last_became_peered":"2024-05-20T17:56:45.757053+0000","last_unstale":"2024-05-20T17:56:46.737317+0000","last_undegraded":"2024-05-20T17:56:46.737317+0000","last_fullsized":"2024-05-20T17:56:46.737317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:50:27.727055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.888450+0000","last_change":"2024-05-20T17:56:44.726341+0000","last_active":"2024-05-20T17:56:46.888450+0000","last_peered":"2024-05-20T17:56:46.888450+0000","last_clean":"2024-05-20T17:56:46.888450+0000","last_became_active":"2024-05-20T17:56:44.726008+0000","last_became_peered":"2024-05-20T17:56:44.726008+0000","last_unstale":"2024-05-20T17:56:46.888450+0000","last_undegraded":"2024-05-20T17:56:46.888450+0000","last_fullsized":"2024-05-20T17:56:46.888450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:25:07.520179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585496+0000","last_change":"2024-05-20T17:56:45.760469+0000","last_active":"2024-05-20T17:56:47.585496+0000","last_peered":"2024-05-20T17:56:47.585496+0000","last_clean":"2024-05-20T17:56:47.585496+0000","last_became_active":"2024-05-20T17:56:45.760100+0000","last_became_peered":"2024-05-20T17:56:45.760100+0000","last_unstale":"2024-05-20T17:56:47.585496+0000","last_undegraded":"2024-05-20T17:56:47.585496+0000","last_fullsized":"2024-05-20T17:56:47.585496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:54:06.309355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518549+0000","last_change":"2024-05-20T17:56:44.716179+0000","last_active":"2024-05-20T17:56:47.518549+0000","last_peered":"2024-05-20T17:56:47.518549+0000","last_clean":"2024-05-20T17:56:47.518549+0000","last_became_active":"2024-05-20T17:56:44.715806+0000","last_became_peered":"2024-05-20T17:56:44.715806+0000","last_unstale":"2024-05-20T17:56:47.518549+0000","last_undegraded":"2024-05-20T17:56:47.518549+0000","last_fullsized":"2024-05-20T17:56:47.518549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:07:05.319573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585593+0000","last_change":"2024-05-20T17:56:45.757302+0000","last_active":"2024-05-20T17:56:47.585593+0000","last_peered":"2024-05-20T17:56:47.585593+0000","last_clean":"2024-05-20T17:56:47.585593+0000","last_became_active":"2024-05-20T17:56:45.754169+0000","last_became_peered":"2024-05-20T17:56:45.754169+0000","last_unstale":"2024-05-20T17:56:47.585593+0000","last_undegraded":"2024-05-20T17:56:47.585593+0000","last_fullsized":"2024-05-20T17:56:47.585593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:03:10.794342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737770+0000","last_change":"2024-05-20T17:56:44.724478+0000","last_active":"2024-05-20T17:56:46.737770+0000","last_peered":"2024-05-20T17:56:46.737770+0000","last_clean":"2024-05-20T17:56:46.737770+0000","last_became_active":"2024-05-20T17:56:44.723927+0000","last_became_peered":"2024-05-20T17:56:44.723927+0000","last_unstale":"2024-05-20T17:56:46.737770+0000","last_undegraded":"2024-05-20T17:56:46.737770+0000","last_fullsized":"2024-05-20T17:56:46.737770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:04:12.003957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585348+0000","last_change":"2024-05-20T17:56:45.758135+0000","last_active":"2024-05-20T17:56:47.585348+0000","last_peered":"2024-05-20T17:56:47.585348+0000","last_clean":"2024-05-20T17:56:47.585348+0000","last_became_active":"2024-05-20T17:56:45.757358+0000","last_became_peered":"2024-05-20T17:56:45.757358+0000","last_unstale":"2024-05-20T17:56:47.585348+0000","last_undegraded":"2024-05-20T17:56:47.585348+0000","last_fullsized":"2024-05-20T17:56:47.585348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:13:06.435055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130384+0000","last_change":"2024-05-20T17:56:44.719822+0000","last_active":"2024-05-20T17:56:48.130384+0000","last_peered":"2024-05-20T17:56:48.130384+0000","last_clean":"2024-05-20T17:56:48.130384+0000","last_became_active":"2024-05-20T17:56:44.719454+0000","last_became_peered":"2024-05-20T17:56:44.719454+0000","last_unstale":"2024-05-20T17:56:48.130384+0000","last_undegraded":"2024-05-20T17:56:48.130384+0000","last_fullsized":"2024-05-20T17:56:48.130384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:03:21.048330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.585584+0000","last_change":"2024-05-20T17:56:45.753510+0000","last_active":"2024-05-20T17:56:47.585584+0000","last_peered":"2024-05-20T17:56:47.585584+0000","last_clean":"2024-05-20T17:56:47.585584+0000","last_became_active":"2024-05-20T17:56:45.752718+0000","last_became_peered":"2024-05-20T17:56:45.752718+0000","last_unstale":"2024-05-20T17:56:47.585584+0000","last_undegraded":"2024-05-20T17:56:47.585584+0000","last_fullsized":"2024-05-20T17:56:47.585584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:05:44.677138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.755763+0000","last_change":"2024-05-20T17:56:45.752528+0000","last_active":"2024-05-20T17:56:45.755763+0000","last_peered":"2024-05-20T17:56:45.755763+0000","last_clean":"2024-05-20T17:56:45.755763+0000","last_became_active":"2024-05-20T17:56:45.752171+0000","last_became_peered":"2024-05-20T17:56:45.752171+0000","last_unstale":"2024-05-20T17:56:45.755763+0000","last_undegraded":"2024-05-20T17:56:45.755763+0000","last_fullsized":"2024-05-20T17:56:45.755763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:24:10.033691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518491+0000","last_change":"2024-05-20T17:56:45.749228+0000","last_active":"2024-05-20T17:56:47.518491+0000","last_peered":"2024-05-20T17:56:47.518491+0000","last_clean":"2024-05-20T17:56:47.518491+0000","last_became_active":"2024-05-20T17:56:45.748886+0000","last_became_peered":"2024-05-20T17:56:45.748886+0000","last_unstale":"2024-05-20T17:56:47.518491+0000","last_undegraded":"2024-05-20T17:56:47.518491+0000","last_fullsized":"2024-05-20T17:56:47.518491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:23:10.569720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518373+0000","last_change":"2024-05-20T17:56:45.751105+0000","last_active":"2024-05-20T17:56:47.518373+0000","last_peered":"2024-05-20T17:56:47.518373+0000","last_clean":"2024-05-20T17:56:47.518373+0000","last_became_active":"2024-05-20T17:56:45.750754+0000","last_became_peered":"2024-05-20T17:56:45.750754+0000","last_unstale":"2024-05-20T17:56:47.518373+0000","last_undegraded":"2024-05-20T17:56:47.518373+0000","last_fullsized":"2024-05-20T17:56:47.518373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:57:48.327428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.618700+0000","last_change":"2024-05-20T17:56:45.752597+0000","last_active":"2024-05-20T17:56:48.618700+0000","last_peered":"2024-05-20T17:56:48.618700+0000","last_clean":"2024-05-20T17:56:48.618700+0000","last_became_active":"2024-05-20T17:56:45.752256+0000","last_became_peered":"2024-05-20T17:56:45.752256+0000","last_unstale":"2024-05-20T17:56:48.618700+0000","last_undegraded":"2024-05-20T17:56:48.618700+0000","last_fullsized":"2024-05-20T17:56:48.618700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:51:53.750906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130226+0000","last_change":"2024-05-20T17:56:45.746122+0000","last_active":"2024-05-20T17:56:48.130226+0000","last_peered":"2024-05-20T17:56:48.130226+0000","last_clean":"2024-05-20T17:56:48.130226+0000","last_became_active":"2024-05-20T17:56:45.745773+0000","last_became_peered":"2024-05-20T17:56:45.745773+0000","last_unstale":"2024-05-20T17:56:48.130226+0000","last_undegraded":"2024-05-20T17:56:48.130226+0000","last_fullsized":"2024-05-20T17:56:48.130226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:21:27.909974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.756179+0000","last_change":"2024-05-20T17:56:45.755018+0000","last_active":"2024-05-20T17:56:45.756179+0000","last_peered":"2024-05-20T17:56:45.756179+0000","last_clean":"2024-05-20T17:56:45.756179+0000","last_became_active":"2024-05-20T17:56:45.754615+0000","last_became_peered":"2024-05-20T17:56:45.754615+0000","last_unstale":"2024-05-20T17:56:45.756179+0000","last_undegraded":"2024-05-20T17:56:45.756179+0000","last_fullsized":"2024-05-20T17:56:45.756179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:33:14.425630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:47.518777+0000","last_change":"2024-05-20T17:56:45.751091+0000","last_active":"2024-05-20T17:56:47.518777+0000","last_peered":"2024-05-20T17:56:47.518777+0000","last_clean":"2024-05-20T17:56:47.518777+0000","last_became_active":"2024-05-20T17:56:45.748573+0000","last_became_peered":"2024-05-20T17:56:45.748573+0000","last_unstale":"2024-05-20T17:56:47.518777+0000","last_undegraded":"2024-05-20T17:56:47.518777+0000","last_fullsized":"2024-05-20T17:56:47.518777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:11:16.140967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737276+0000","last_change":"2024-05-20T17:56:45.756373+0000","last_active":"2024-05-20T17:56:46.737276+0000","last_peered":"2024-05-20T17:56:46.737276+0000","last_clean":"2024-05-20T17:56:46.737276+0000","last_became_active":"2024-05-20T17:56:45.755993+0000","last_became_peered":"2024-05-20T17:56:45.755993+0000","last_unstale":"2024-05-20T17:56:46.737276+0000","last_undegraded":"2024-05-20T17:56:46.737276+0000","last_fullsized":"2024-05-20T17:56:46.737276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:34:44.491341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737916+0000","last_change":"2024-05-20T17:56:45.751866+0000","last_active":"2024-05-20T17:56:46.737916+0000","last_peered":"2024-05-20T17:56:46.737916+0000","last_clean":"2024-05-20T17:56:46.737916+0000","last_became_active":"2024-05-20T17:56:45.751518+0000","last_became_peered":"2024-05-20T17:56:45.751518+0000","last_unstale":"2024-05-20T17:56:46.737916+0000","last_undegraded":"2024-05-20T17:56:46.737916+0000","last_fullsized":"2024-05-20T17:56:46.737916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:38:18.736177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.737209+0000","last_change":"2024-05-20T17:56:45.749903+0000","last_active":"2024-05-20T17:56:46.737209+0000","last_peered":"2024-05-20T17:56:46.737209+0000","last_clean":"2024-05-20T17:56:46.737209+0000","last_became_active":"2024-05-20T17:56:45.749500+0000","last_became_peered":"2024-05-20T17:56:45.749500+0000","last_unstale":"2024-05-20T17:56:46.737209+0000","last_undegraded":"2024-05-20T17:56:46.737209+0000","last_fullsized":"2024-05-20T17:56:46.737209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:39:34.875111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:46.918070+0000","last_change":"2024-05-20T17:56:45.747370+0000","last_active":"2024-05-20T17:56:46.918070+0000","last_peered":"2024-05-20T17:56:46.918070+0000","last_clean":"2024-05-20T17:56:46.918070+0000","last_became_active":"2024-05-20T17:56:45.747036+0000","last_became_peered":"2024-05-20T17:56:45.747036+0000","last_unstale":"2024-05-20T17:56:46.918070+0000","last_undegraded":"2024-05-20T17:56:46.918070+0000","last_fullsized":"2024-05-20T17:56:46.918070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:02:02.220522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-20T17:56:48.130688+0000","last_change":"2024-05-20T17:56:45.752762+0000","last_active":"2024-05-20T17:56:48.130688+0000","last_peered":"2024-05-20T17:56:48.130688+0000","last_clean":"2024-05-20T17:56:48.130688+0000","last_became_active":"2024-05-20T17:56:45.752345+0000","last_became_peered":"2024-05-20T17:56:45.752345+0000","last_unstale":"2024-05-20T17:56:48.130688+0000","last_undegraded":"2024-05-20T17:56:48.130688+0000","last_fullsized":"2024-05-20T17:56:48.130688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:14:49.861456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-05-20T17:56:45.753795+0000","last_change":"2024-05-20T17:56:45.750229+0000","last_active":"2024-05-20T17:56:45.753795+0000","last_peered":"2024-05-20T17:56:45.753795+0000","last_clean":"2024-05-20T17:56:45.753795+0000","last_became_active":"2024-05-20T17:56:45.749872+0000","last_became_peered":"2024-05-20T17:56:45.749872+0000","last_unstale":"2024-05-20T17:56:45.753795+0000","last_undegraded":"2024-05-20T17:56:45.753795+0000","last_fullsized":"2024-05-20T17:56:45.753795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:17:43.464358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280272,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":913408,"data_stored":436606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1028780,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808856,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":445758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070493,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":933888,"data_stored":445780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332129,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":1036126,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":436606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":937984,"data_stored":446788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":445266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":446822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":933888,"data_stored":445280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":445198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":1027156,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-20T17:56:53.165 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-20T17:56:53.166 INFO:tasks.cephadm:Setting up client nodes... 2024-05-20T17:56:53.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-20T17:56:53.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:53 smithi154 ceph-mon[36482]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:53.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:53 smithi154 ceph-mon[36482]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-20T17:56:53.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:53 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2949988027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:53.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:53 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2375148432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:53.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:53 smithi204 ceph-mon[37076]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:56:53.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:53 smithi204 ceph-mon[37076]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-20T17:56:53.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:53 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2949988027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:56:53.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:53 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2375148432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:56:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:56:54.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:54 smithi129 ceph-mon[32029]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:56:54.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:54 smithi129 ceph-mon[32029]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:56:54.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:54 smithi129 ceph-mon[32029]: pgmap v289: 97 pgs: 97 active+clean; 615 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-05-20T17:56:54.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:54 smithi154 ceph-mon[36482]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:56:54.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:54 smithi154 ceph-mon[36482]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:56:54.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:54 smithi154 ceph-mon[36482]: pgmap v289: 97 pgs: 97 active+clean; 615 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-05-20T17:56:54.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:54 smithi204 ceph-mon[37076]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:56:54.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:54 smithi204 ceph-mon[37076]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:56:54.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:54 smithi204 ceph-mon[37076]: pgmap v289: 97 pgs: 97 active+clean; 615 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6.7 KiB/s wr, 9 op/s 2024-05-20T17:56:55.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.081+0000 7f1fafa52700 1 -- 172.21.15.129:0/1895453741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fa8104780 msgr2=0x7f1fa8104b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:55.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.081+0000 7f1fafa52700 1 --2- 172.21.15.129:0/1895453741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fa8104780 0x7f1fa8104b90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009990 tx=0x7f1fa4009ca0 comp rx=0 tx=0).stop 2024-05-20T17:56:55.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.081+0000 7f1fafa52700 1 -- 172.21.15.129:0/1895453741 shutdown_connections 2024-05-20T17:56:55.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.081+0000 7f1fafa52700 1 --2- 172.21.15.129:0/1895453741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fa8101b20 0x7f1fa810b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:55.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.081+0000 7f1fafa52700 1 --2- 172.21.15.129:0/1895453741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fa8104780 0x7f1fa8104b90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:55.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.081+0000 7f1fafa52700 1 --2- 172.21.15.129:0/1895453741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fa81011d0 0x7f1fa81015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:55.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.081+0000 7f1fafa52700 1 -- 172.21.15.129:0/1895453741 >> 172.21.15.129:0/1895453741 conn(0x7f1fa80fcea0 msgr2=0x7f1fa80ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:55.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.081+0000 7f1fafa52700 1 -- 172.21.15.129:0/1895453741 shutdown_connections 2024-05-20T17:56:55.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.082+0000 7f1fafa52700 1 -- 172.21.15.129:0/1895453741 wait complete. 2024-05-20T17:56:55.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.082+0000 7f1fafa52700 1 Processor -- start 2024-05-20T17:56:55.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.082+0000 7f1fafa52700 1 -- start start 2024-05-20T17:56:55.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1fafa52700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fa81011d0 0x7f1fa8103d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:55.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1fafa52700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fa8101b20 0x7f1fa8102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:55.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1fafa52700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fa8104780 0x7f1fa8102940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:55.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1fafa52700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa810ed80 con 0x7f1fa8101b20 2024-05-20T17:56:55.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1fafa52700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa810ec00 con 0x7f1fa81011d0 2024-05-20T17:56:55.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1fafa52700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa810ef00 con 0x7f1fa8104780 2024-05-20T17:56:55.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1facfed700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fa8101b20 0x7f1fa8102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:55.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1facfed700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fa8101b20 0x7f1fa8102400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46754/0 (socket says 172.21.15.129:46754) 2024-05-20T17:56:55.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1facfed700 1 -- 172.21.15.129:0/778307169 learned_addr learned my addr 172.21.15.129:0/778307169 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:55.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1fad7ee700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fa81011d0 0x7f1fa8103d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:55.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1fadfef700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fa8104780 0x7f1fa8102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1facfed700 1 -- 172.21.15.129:0/778307169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fa8104780 msgr2=0x7f1fa8102940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1facfed700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fa8104780 0x7f1fa8102940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1facfed700 1 -- 172.21.15.129:0/778307169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fa81011d0 msgr2=0x7f1fa8103d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1facfed700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fa81011d0 0x7f1fa8103d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1facfed700 1 -- 172.21.15.129:0/778307169 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fa81030a0 con 0x7f1fa8101b20 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1fadfef700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fa8104780 0x7f1fa8102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1facfed700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fa8101b20 0x7f1fa8102400 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1f9800d950 tx=0x7f1f9800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1f9e7fc700 1 -- 172.21.15.129:0/778307169 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f98016070 con 0x7f1fa8101b20 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1fafa52700 1 -- 172.21.15.129:0/778307169 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fa81a3930 con 0x7f1fa8101b20 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1f9e7fc700 1 -- 172.21.15.129:0/778307169 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f98009e20 con 0x7f1fa8101b20 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1f9e7fc700 1 -- 172.21.15.129:0/778307169 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9800f620 con 0x7f1fa8101b20 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.083+0000 7f1fafa52700 1 -- 172.21.15.129:0/778307169 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fa81a3e60 con 0x7f1fa8101b20 2024-05-20T17:56:55.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.084+0000 7f1fafa52700 1 -- 172.21.15.129:0/778307169 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fa802dc60 con 0x7f1fa8101b20 2024-05-20T17:56:55.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.086+0000 7f1f9e7fc700 1 -- 172.21.15.129:0/778307169 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f1f98019070 con 0x7f1fa8101b20 2024-05-20T17:56:55.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.086+0000 7f1f9e7fc700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1f940a2cb0 0x7f1f940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:55.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.086+0000 7f1fad7ee700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1f940a2cb0 0x7f1f940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:55.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.087+0000 7f1f9e7fc700 1 -- 172.21.15.129:0/778307169 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f1f980cebe0 con 0x7f1fa8101b20 2024-05-20T17:56:55.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.087+0000 7f1fad7ee700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1f940a2cb0 0x7f1f940a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40098e0 tx=0x7f1fa401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:55.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.090+0000 7f1f9e7fc700 1 -- 172.21.15.129:0/778307169 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1f9809bb90 con 0x7f1fa8101b20 2024-05-20T17:56:55.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.366+0000 7f1fafa52700 1 -- 172.21.15.129:0/778307169 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f1fa802de30 con 0x7f1fa8101b20 2024-05-20T17:56:55.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.369+0000 7f1f9e7fc700 1 -- 172.21.15.129:0/778307169 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f1f9809b490 con 0x7f1fa8101b20 2024-05-20T17:56:55.368 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-05-20T17:56:55.368 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDnjktmKoLzFRAAGyw2NLx+EMG8S0GnIrW2gw== 2024-05-20T17:56:55.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.371+0000 7f1fafa52700 1 -- 172.21.15.129:0/778307169 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1f940a2cb0 msgr2=0x7f1f940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:55.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.371+0000 7f1fafa52700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1f940a2cb0 0x7f1f940a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40098e0 tx=0x7f1fa401a040 comp rx=0 tx=0).stop 2024-05-20T17:56:55.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.371+0000 7f1fafa52700 1 -- 172.21.15.129:0/778307169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fa8101b20 msgr2=0x7f1fa8102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:55.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.371+0000 7f1fafa52700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fa8101b20 0x7f1fa8102400 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1f9800d950 tx=0x7f1f9800dc60 comp rx=0 tx=0).stop 2024-05-20T17:56:55.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.371+0000 7f1fafa52700 1 -- 172.21.15.129:0/778307169 shutdown_connections 2024-05-20T17:56:55.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.371+0000 7f1fafa52700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1f940a2cb0 0x7f1f940a51c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:55.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.371+0000 7f1fafa52700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fa81011d0 0x7f1fa8103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:55.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.371+0000 7f1fafa52700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fa8104780 0x7f1fa8102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:55.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.371+0000 7f1fafa52700 1 --2- 172.21.15.129:0/778307169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fa8101b20 0x7f1fa8102400 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:55.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.371+0000 7f1fafa52700 1 -- 172.21.15.129:0/778307169 >> 172.21.15.129:0/778307169 conn(0x7f1fa80fcea0 msgr2=0x7f1fa8107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:55.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.371+0000 7f1fafa52700 1 -- 172.21.15.129:0/778307169 shutdown_connections 2024-05-20T17:56:55.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:55.372+0000 7f1fafa52700 1 -- 172.21.15.129:0/778307169 wait complete. 2024-05-20T17:56:55.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:55 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/778307169' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-20T17:56:55.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:55 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/778307169' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-20T17:56:55.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:55 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/778307169' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-20T17:56:55.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:55 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/778307169' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-20T17:56:55.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:55 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/778307169' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-20T17:56:55.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:55 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/778307169' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-20T17:56:56.008 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:56:56.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-20T17:56:56.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-20T17:56:56.045 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-20T17:56:56.045 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-20T17:56:56.046 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph mgr dump --format=json 2024-05-20T17:56:56.325 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:56:56.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:56 smithi204 ceph-mon[37076]: pgmap v290: 97 pgs: 97 active+clean; 615 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5.4 KiB/s wr, 7 op/s 2024-05-20T17:56:56.750 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:56 smithi129 ceph-mon[32029]: pgmap v290: 97 pgs: 97 active+clean; 615 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5.4 KiB/s wr, 7 op/s 2024-05-20T17:56:56.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:56 smithi154 ceph-mon[36482]: pgmap v290: 97 pgs: 97 active+clean; 615 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5.4 KiB/s wr, 7 op/s 2024-05-20T17:56:58.010 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:57 smithi129 ceph-mon[32029]: pgmap v291: 97 pgs: 97 active+clean; 620 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.4 KiB/s wr, 7 op/s 2024-05-20T17:56:58.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.017+0000 7fc428906700 1 -- 172.21.15.129:0/3807076294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc424101f60 msgr2=0x7fc424106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:58.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.017+0000 7fc428906700 1 --2- 172.21.15.129:0/3807076294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc424101f60 0x7fc424106da0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00b0a0 tx=0x7fc41c00b3b0 comp rx=0 tx=0).stop 2024-05-20T17:56:58.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.017+0000 7fc428906700 1 -- 172.21.15.129:0/3807076294 shutdown_connections 2024-05-20T17:56:58.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.017+0000 7fc428906700 1 --2- 172.21.15.129:0/3807076294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc424101f60 0x7fc424106da0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:58.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.017+0000 7fc428906700 1 --2- 172.21.15.129:0/3807076294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4241080d0 0x7fc4241084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:58.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.017+0000 7fc428906700 1 --2- 172.21.15.129:0/3807076294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc424101610 0x7fc424101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:58.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.017+0000 7fc428906700 1 -- 172.21.15.129:0/3807076294 >> 172.21.15.129:0/3807076294 conn(0x7fc4240fd3a0 msgr2=0x7fc4240ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:58.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.017+0000 7fc428906700 1 -- 172.21.15.129:0/3807076294 shutdown_connections 2024-05-20T17:56:58.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.017+0000 7fc428906700 1 -- 172.21.15.129:0/3807076294 wait complete. 2024-05-20T17:56:58.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.018+0000 7fc428906700 1 Processor -- start 2024-05-20T17:56:58.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.018+0000 7fc428906700 1 -- start start 2024-05-20T17:56:58.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.018+0000 7fc428906700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc424101610 0x7fc4241a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:58.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.018+0000 7fc428906700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc424101f60 0x7fc4241a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:58.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.018+0000 7fc428906700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4241080d0 0x7fc4241a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:58.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.018+0000 7fc428906700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc424119a50 con 0x7fc4241080d0 2024-05-20T17:56:58.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.018+0000 7fc428906700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc4241198d0 con 0x7fc424101f60 2024-05-20T17:56:58.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.018+0000 7fc428906700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc424119bd0 con 0x7fc424101610 2024-05-20T17:56:58.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.018+0000 7fc422d9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4241080d0 0x7fc4241a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:58.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.018+0000 7fc422d9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4241080d0 0x7fc4241a4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46766/0 (socket says 172.21.15.129:46766) 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.018+0000 7fc422d9d700 1 -- 172.21.15.129:0/2561711391 learned_addr learned my addr 172.21.15.129:0/2561711391 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc42259c700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc424101610 0x7fc4241a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc421d9b700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc424101f60 0x7fc4241a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc422d9d700 1 -- 172.21.15.129:0/2561711391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc424101610 msgr2=0x7fc4241a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc422d9d700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc424101610 0x7fc4241a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc422d9d700 1 -- 172.21.15.129:0/2561711391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc424101f60 msgr2=0x7fc4241a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc422d9d700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc424101f60 0x7fc4241a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc422d9d700 1 -- 172.21.15.129:0/2561711391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4241a50a0 con 0x7fc4241080d0 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc421d9b700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc424101f60 0x7fc4241a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc422d9d700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4241080d0 0x7fc4241a4ab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0062a0 tx=0x7fc41c004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc4137fe700 1 -- 172.21.15.129:0/2561711391 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c021070 con 0x7fc4241080d0 2024-05-20T17:56:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc428906700 1 -- 172.21.15.129:0/2561711391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4241a5330 con 0x7fc4241080d0 2024-05-20T17:56:58.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc4137fe700 1 -- 172.21.15.129:0/2561711391 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc41c007c00 con 0x7fc4241080d0 2024-05-20T17:56:58.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc4137fe700 1 -- 172.21.15.129:0/2561711391 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41c01d450 con 0x7fc4241080d0 2024-05-20T17:56:58.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.019+0000 7fc428906700 1 -- 172.21.15.129:0/2561711391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4241a5870 con 0x7fc4241080d0 2024-05-20T17:56:58.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.020+0000 7fc428906700 1 -- 172.21.15.129:0/2561711391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc424050170 con 0x7fc4241080d0 2024-05-20T17:56:58.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.022+0000 7fc4137fe700 1 -- 172.21.15.129:0/2561711391 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc41c004000 con 0x7fc4241080d0 2024-05-20T17:56:58.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.022+0000 7fc4137fe700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc40c0a2cb0 0x7fc40c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:56:58.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.023+0000 7fc42259c700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc40c0a2cb0 0x7fc40c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:56:58.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.023+0000 7fc4137fe700 1 -- 172.21.15.129:0/2561711391 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc41c0d00f0 con 0x7fc4241080d0 2024-05-20T17:56:58.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.023+0000 7fc42259c700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc40c0a2cb0 0x7fc40c0a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc42406ae80 tx=0x7fc41400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:56:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.026+0000 7fc4137fe700 1 -- 172.21.15.129:0/2561711391 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc41c09d130 con 0x7fc4241080d0 2024-05-20T17:56:58.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:57 smithi154 ceph-mon[36482]: pgmap v291: 97 pgs: 97 active+clean; 620 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.4 KiB/s wr, 7 op/s 2024-05-20T17:56:58.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:57 smithi204 ceph-mon[37076]: pgmap v291: 97 pgs: 97 active+clean; 620 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.4 KiB/s wr, 7 op/s 2024-05-20T17:56:58.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.283+0000 7fc428906700 1 -- 172.21.15.129:0/2561711391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fc4241ace80 con 0x7fc4241080d0 2024-05-20T17:56:58.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.291+0000 7fc4137fe700 1 -- 172.21.15.129:0/2561711391 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263209 (secure 0 0 0) 0x7fc41c09ca30 con 0x7fc4241080d0 2024-05-20T17:56:58.293 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:56:58.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 -- 172.21.15.129:0/2561711391 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc40c0a2cb0 msgr2=0x7fc40c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:58.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc40c0a2cb0 0x7fc40c0a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc42406ae80 tx=0x7fc41400b410 comp rx=0 tx=0).stop 2024-05-20T17:56:58.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 -- 172.21.15.129:0/2561711391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4241080d0 msgr2=0x7fc4241a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:56:58.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4241080d0 0x7fc4241a4ab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0062a0 tx=0x7fc41c004d70 comp rx=0 tx=0).stop 2024-05-20T17:56:58.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 -- 172.21.15.129:0/2561711391 shutdown_connections 2024-05-20T17:56:58.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc40c0a2cb0 0x7fc40c0a51c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:58.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4241080d0 0x7fc4241a4ab0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:58.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc424101610 0x7fc4241a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:58.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 --2- 172.21.15.129:0/2561711391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc424101f60 0x7fc4241a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:56:58.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 -- 172.21.15.129:0/2561711391 >> 172.21.15.129:0/2561711391 conn(0x7fc4240fd3a0 msgr2=0x7fc424104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:56:58.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 -- 172.21.15.129:0/2561711391 shutdown_connections 2024-05-20T17:56:58.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:56:58.294+0000 7fc428906700 1 -- 172.21.15.129:0/2561711391 wait complete. 2024-05-20T17:56:58.974 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:58 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2561711391' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-20T17:56:59.003 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2870089245},{"type":"v1","addr":"172.21.15.129:6801","nonce":2870089245}]},"active_addr":"172.21.15.129:6801/2870089245","active_change":"2024-05-20T17:49:47.618799+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.129:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":2249940021}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":1943655536}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":549964701}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":1369259428}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":3503592989}]}]} 2024-05-20T17:56:59.009 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-20T17:56:59.009 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-20T17:56:59.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd dump --format=json 2024-05-20T17:56:59.044 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:58 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2561711391' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-20T17:56:59.081 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:58 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2561711391' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-20T17:56:59.249 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:00.057 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:56:59 smithi154 ceph-mon[36482]: pgmap v292: 97 pgs: 97 active+clean; 620 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-05-20T17:57:00.057 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:56:59 smithi129 ceph-mon[32029]: pgmap v292: 97 pgs: 97 active+clean; 620 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-05-20T17:57:00.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:56:59 smithi204 ceph-mon[37076]: pgmap v292: 97 pgs: 97 active+clean; 620 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-05-20T17:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.911+0000 7fdce171b700 1 -- 172.21.15.129:0/1433116270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcdc103130 msgr2=0x7fdcdc10eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.911+0000 7fdce171b700 1 --2- 172.21.15.129:0/1433116270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcdc103130 0x7fdcdc10eef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdcd400b0a0 tx=0x7fdcd400b3b0 comp rx=0 tx=0).stop 2024-05-20T17:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.911+0000 7fdce171b700 1 -- 172.21.15.129:0/1433116270 shutdown_connections 2024-05-20T17:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.911+0000 7fdce171b700 1 --2- 172.21.15.129:0/1433116270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcdc103130 0x7fdcdc10eef0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.911+0000 7fdce171b700 1 --2- 172.21.15.129:0/1433116270 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcdc107080 0x7fdcdc107490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.911+0000 7fdce171b700 1 --2- 172.21.15.129:0/1433116270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcdc1027e0 0x7fdcdc102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.911+0000 7fdce171b700 1 -- 172.21.15.129:0/1433116270 >> 172.21.15.129:0/1433116270 conn(0x7fdcdc0fd360 msgr2=0x7fdcdc0ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.911+0000 7fdce171b700 1 -- 172.21.15.129:0/1433116270 shutdown_connections 2024-05-20T17:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.911+0000 7fdce171b700 1 -- 172.21.15.129:0/1433116270 wait complete. 2024-05-20T17:57:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.912+0000 7fdce171b700 1 Processor -- start 2024-05-20T17:57:00.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.912+0000 7fdce171b700 1 -- start start 2024-05-20T17:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.912+0000 7fdce171b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcdc1027e0 0x7fdcdc19e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.912+0000 7fdce171b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcdc103130 0x7fdcdc19ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdce171b700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcdc107080 0x7fdcdc1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdce171b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc112380 con 0x7fdcdc1027e0 2024-05-20T17:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdce171b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc112200 con 0x7fdcdc103130 2024-05-20T17:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdce171b700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc112500 con 0x7fdcdc107080 2024-05-20T17:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcdaffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcdc1027e0 0x7fdcdc19e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcda7fc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcdc103130 0x7fdcdc19ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcdaffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcdc1027e0 0x7fdcdc19e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43322/0 (socket says 172.21.15.129:43322) 2024-05-20T17:57:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcda7fc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcdc103130 0x7fdcdc19ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:47764/0 (socket says 172.21.15.129:47764) 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcdaffd700 1 -- 172.21.15.129:0/912860057 learned_addr learned my addr 172.21.15.129:0/912860057 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcdb7fe700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcdc107080 0x7fdcdc1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcdaffd700 1 -- 172.21.15.129:0/912860057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcdc107080 msgr2=0x7fdcdc1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcdaffd700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcdc107080 0x7fdcdc1989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcdaffd700 1 -- 172.21.15.129:0/912860057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcdc103130 msgr2=0x7fdcdc19ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcdaffd700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcdc103130 0x7fdcdc19ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcdaffd700 1 -- 172.21.15.129:0/912860057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcdc199070 con 0x7fdcdc1027e0 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcda7fc700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcdc103130 0x7fdcdc19ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcdaffd700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcdc1027e0 0x7fdcdc19e7a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00e9f0 tx=0x7fdccc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcc3fff700 1 -- 172.21.15.129:0/912860057 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc004230 con 0x7fdcdc1027e0 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdce171b700 1 -- 172.21.15.129:0/912860057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcdc199360 con 0x7fdcdc1027e0 2024-05-20T17:57:00.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcc3fff700 1 -- 172.21.15.129:0/912860057 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdccc0043d0 con 0x7fdcdc1027e0 2024-05-20T17:57:00.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdcc3fff700 1 -- 172.21.15.129:0/912860057 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc003d50 con 0x7fdcdc1027e0 2024-05-20T17:57:00.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.913+0000 7fdce171b700 1 -- 172.21.15.129:0/912860057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcdc1a5630 con 0x7fdcdc1027e0 2024-05-20T17:57:00.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.914+0000 7fdce171b700 1 -- 172.21.15.129:0/912860057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcdc050170 con 0x7fdcdc1027e0 2024-05-20T17:57:00.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.917+0000 7fdcc3fff700 1 -- 172.21.15.129:0/912860057 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fdccc00c7c0 con 0x7fdcdc1027e0 2024-05-20T17:57:00.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.917+0000 7fdcc3fff700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdcc40a2cb0 0x7fdcc40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:00.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.917+0000 7fdcda7fc700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdcc40a2cb0 0x7fdcc40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:00.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.917+0000 7fdcc3fff700 1 -- 172.21.15.129:0/912860057 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fdccc0cf2b0 con 0x7fdcdc1027e0 2024-05-20T17:57:00.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.918+0000 7fdcda7fc700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdcc40a2cb0 0x7fdcc40a51c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc100d20 tx=0x7fdcd0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:00.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:00.921+0000 7fdcc3fff700 1 -- 172.21.15.129:0/912860057 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdccc09c260 con 0x7fdcdc1027e0 2024-05-20T17:57:01.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.119+0000 7fdce171b700 1 -- 172.21.15.129:0/912860057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdcdc199ef0 con 0x7fdcdc1027e0 2024-05-20T17:57:01.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.120+0000 7fdcc3fff700 1 -- 172.21.15.129:0/912860057 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22240 (secure 0 0 0) 0x7fdccc09bb60 con 0x7fdcdc1027e0 2024-05-20T17:57:01.119 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:57:01.120 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":86,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:56:46.728647+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:57:01.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 -- 172.21.15.129:0/912860057 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdcc40a2cb0 msgr2=0x7fdcc40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdcc40a2cb0 0x7fdcc40a51c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc100d20 tx=0x7fdcd0009260 comp rx=0 tx=0).stop 2024-05-20T17:57:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 -- 172.21.15.129:0/912860057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcdc1027e0 msgr2=0x7fdcdc19e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcdc1027e0 0x7fdcdc19e7a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00e9f0 tx=0x7fdccc00ed00 comp rx=0 tx=0).stop 2024-05-20T17:57:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 -- 172.21.15.129:0/912860057 shutdown_connections 2024-05-20T17:57:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdcc40a2cb0 0x7fdcc40a51c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdcdc107080 0x7fdcdc1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcdc1027e0 0x7fdcdc19e7a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 --2- 172.21.15.129:0/912860057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcdc103130 0x7fdcdc19ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 -- 172.21.15.129:0/912860057 >> 172.21.15.129:0/912860057 conn(0x7fdcdc0fd360 msgr2=0x7fdcdc106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 -- 172.21.15.129:0/912860057 shutdown_connections 2024-05-20T17:57:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:01.123+0000 7fdce171b700 1 -- 172.21.15.129:0/912860057 wait complete. 2024-05-20T17:57:01.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:01 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/912860057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:57:01.484 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:01 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/912860057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:57:01.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:01 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/912860057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:57:01.812 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-20T17:57:01.813 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd dump --format=json 2024-05-20T17:57:02.050 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:02.423 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:02 smithi129 ceph-mon[32029]: pgmap v293: 97 pgs: 97 active+clean; 620 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-05-20T17:57:02.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:02 smithi204 ceph-mon[37076]: pgmap v293: 97 pgs: 97 active+clean; 620 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-05-20T17:57:02.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:02 smithi154 ceph-mon[36482]: pgmap v293: 97 pgs: 97 active+clean; 620 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-05-20T17:57:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.746+0000 7fd1f2e53700 1 -- 172.21.15.129:0/3692689692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1ec102340 msgr2=0x7fd1ec102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.746+0000 7fd1f2e53700 1 --2- 172.21.15.129:0/3692689692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1ec102340 0x7fd1ec102750 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00099c0 tx=0x7fd1e0009cd0 comp rx=0 tx=0).stop 2024-05-20T17:57:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.746+0000 7fd1f2e53700 1 -- 172.21.15.129:0/3692689692 shutdown_connections 2024-05-20T17:57:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.746+0000 7fd1f2e53700 1 --2- 172.21.15.129:0/3692689692 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1ec1075c0 0x7fd1ec10ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.746+0000 7fd1f2e53700 1 --2- 172.21.15.129:0/3692689692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1ec102340 0x7fd1ec102750 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.746+0000 7fd1f2e53700 1 --2- 172.21.15.129:0/3692689692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1ec106c70 0x7fd1ec107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.746+0000 7fd1f2e53700 1 -- 172.21.15.129:0/3692689692 >> 172.21.15.129:0/3692689692 conn(0x7fd1ec0fdc40 msgr2=0x7fd1ec100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.747+0000 7fd1f2e53700 1 -- 172.21.15.129:0/3692689692 shutdown_connections 2024-05-20T17:57:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.747+0000 7fd1f2e53700 1 -- 172.21.15.129:0/3692689692 wait complete. 2024-05-20T17:57:03.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.747+0000 7fd1f2e53700 1 Processor -- start 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.747+0000 7fd1f2e53700 1 -- start start 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.747+0000 7fd1f2e53700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1ec102340 0x7fd1ec19bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.747+0000 7fd1f2e53700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1ec106c70 0x7fd1ec19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.747+0000 7fd1f2e53700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1ec1075c0 0x7fd1ec1a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.747+0000 7fd1f2e53700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec112390 con 0x7fd1ec106c70 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.747+0000 7fd1f2e53700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec112210 con 0x7fd1ec1075c0 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.747+0000 7fd1f2e53700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec112510 con 0x7fd1ec102340 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1ebfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1ec106c70 0x7fd1ec19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1ebfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1ec106c70 0x7fd1ec19c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43340/0 (socket says 172.21.15.129:43340) 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1ebfff700 1 -- 172.21.15.129:0/4104836617 learned_addr learned my addr 172.21.15.129:0/4104836617 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1f13f0700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1ec1075c0 0x7fd1ec1a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1f0bef700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1ec102340 0x7fd1ec19bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1ebfff700 1 -- 172.21.15.129:0/4104836617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1ec102340 msgr2=0x7fd1ec19bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:03.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1ebfff700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1ec102340 0x7fd1ec19bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:03.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1ebfff700 1 -- 172.21.15.129:0/4104836617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1ec1075c0 msgr2=0x7fd1ec1a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:03.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1ebfff700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1ec1075c0 0x7fd1ec1a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:03.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1ebfff700 1 -- 172.21.15.129:0/4104836617 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ec1a0e70 con 0x7fd1ec106c70 2024-05-20T17:57:03.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1ebfff700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1ec106c70 0x7fd1ec19c540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800e970 tx=0x7fd1d800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:03.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1e9ffb700 1 -- 172.21.15.129:0/4104836617 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d800cbd0 con 0x7fd1ec106c70 2024-05-20T17:57:03.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1f2e53700 1 -- 172.21.15.129:0/4104836617 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1ec1a1160 con 0x7fd1ec106c70 2024-05-20T17:57:03.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1e9ffb700 1 -- 172.21.15.129:0/4104836617 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1d800cd70 con 0x7fd1ec106c70 2024-05-20T17:57:03.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1e9ffb700 1 -- 172.21.15.129:0/4104836617 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8010430 con 0x7fd1ec106c70 2024-05-20T17:57:03.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.748+0000 7fd1f2e53700 1 -- 172.21.15.129:0/4104836617 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1ec1a8a80 con 0x7fd1ec106c70 2024-05-20T17:57:03.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.750+0000 7fd1f2e53700 1 -- 172.21.15.129:0/4104836617 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1ec050170 con 0x7fd1ec106c70 2024-05-20T17:57:03.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.751+0000 7fd1e9ffb700 1 -- 172.21.15.129:0/4104836617 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fd1d8004750 con 0x7fd1ec106c70 2024-05-20T17:57:03.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.751+0000 7fd1e9ffb700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd1dc0a2cb0 0x7fd1dc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:03.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.751+0000 7fd1e9ffb700 1 -- 172.21.15.129:0/4104836617 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fd1d80cf3d0 con 0x7fd1ec106c70 2024-05-20T17:57:03.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.751+0000 7fd1f0bef700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd1dc0a2cb0 0x7fd1dc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:03.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.752+0000 7fd1f0bef700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd1dc0a2cb0 0x7fd1dc0a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0003800 tx=0x7fd1e001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:03.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.756+0000 7fd1e9ffb700 1 -- 172.21.15.129:0/4104836617 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd1d809c380 con 0x7fd1ec106c70 2024-05-20T17:57:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.957+0000 7fd1f2e53700 1 -- 172.21.15.129:0/4104836617 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd1ec19d300 con 0x7fd1ec106c70 2024-05-20T17:57:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.959+0000 7fd1e9ffb700 1 -- 172.21.15.129:0/4104836617 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22240 (secure 0 0 0) 0x7fd1d809bc80 con 0x7fd1ec106c70 2024-05-20T17:57:03.958 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:57:03.958 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":86,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:56:46.728647+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:57:03.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 -- 172.21.15.129:0/4104836617 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd1dc0a2cb0 msgr2=0x7fd1dc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:03.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd1dc0a2cb0 0x7fd1dc0a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0003800 tx=0x7fd1e001a040 comp rx=0 tx=0).stop 2024-05-20T17:57:03.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 -- 172.21.15.129:0/4104836617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1ec106c70 msgr2=0x7fd1ec19c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:03.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1ec106c70 0x7fd1ec19c540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd1d800e970 tx=0x7fd1d800ed30 comp rx=0 tx=0).stop 2024-05-20T17:57:03.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 -- 172.21.15.129:0/4104836617 shutdown_connections 2024-05-20T17:57:03.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd1dc0a2cb0 0x7fd1dc0a51c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:03.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1ec1075c0 0x7fd1ec1a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:03.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1ec102340 0x7fd1ec19bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:03.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 --2- 172.21.15.129:0/4104836617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1ec106c70 0x7fd1ec19c540 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:03.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 -- 172.21.15.129:0/4104836617 >> 172.21.15.129:0/4104836617 conn(0x7fd1ec0fdc40 msgr2=0x7fd1ec1052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:03.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 -- 172.21.15.129:0/4104836617 shutdown_connections 2024-05-20T17:57:03.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:03.961+0000 7fd1f2e53700 1 -- 172.21.15.129:0/4104836617 wait complete. 2024-05-20T17:57:04.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:03 smithi154 ceph-mon[36482]: pgmap v294: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-05-20T17:57:04.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:03 smithi129 ceph-mon[32029]: pgmap v294: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-05-20T17:57:04.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:03 smithi204 ceph-mon[37076]: pgmap v294: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-05-20T17:57:04.634 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-20T17:57:04.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-20T17:57:04.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-20T17:57:04.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-20T17:57:04.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-20T17:57:04.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-20T17:57:04.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-20T17:57:04.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-20T17:57:04.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-20T17:57:04.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-20T17:57:04.639 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-20T17:57:04.639 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-20T17:57:04.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:04 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4104836617' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:57:05.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:04 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/4104836617' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:57:05.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:04 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/4104836617' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:57:05.215 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:05.521 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:05.539 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:05.551 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:05.551 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:06.094 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:05 smithi129 ceph-mon[32029]: pgmap v295: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-05-20T17:57:06.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:05 smithi154 ceph-mon[36482]: pgmap v295: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-05-20T17:57:06.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:05 smithi204 ceph-mon[37076]: pgmap v295: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-05-20T17:57:06.472 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:07.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:07 smithi129 ceph-mon[32029]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2024-05-20T17:57:08.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:07 smithi154 ceph-mon[36482]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2024-05-20T17:57:08.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:07 smithi204 ceph-mon[37076]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2024-05-20T17:57:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.928+0000 7f5177cf6700 1 -- 172.21.15.129:0/2757730259 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5170101840 msgr2=0x7f5170101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.928+0000 7f5177cf6700 1 --2- 172.21.15.129:0/2757730259 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5170101840 0x7f5170101c50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5160009990 tx=0x7f5160009ca0 comp rx=0 tx=0).stop 2024-05-20T17:57:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.928+0000 7f5177cf6700 1 -- 172.21.15.129:0/2757730259 shutdown_connections 2024-05-20T17:57:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.928+0000 7f5177cf6700 1 --2- 172.21.15.129:0/2757730259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5170107870 0x7f5170107c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.928+0000 7f5177cf6700 1 --2- 172.21.15.129:0/2757730259 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5170102190 0x7f517010e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.928+0000 7f5177cf6700 1 --2- 172.21.15.129:0/2757730259 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5170101840 0x7f5170101c50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.928+0000 7f5177cf6700 1 -- 172.21.15.129:0/2757730259 >> 172.21.15.129:0/2757730259 conn(0x7f51700fd3a0 msgr2=0x7f51700ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.928+0000 7f5177cf6700 1 -- 172.21.15.129:0/2757730259 shutdown_connections 2024-05-20T17:57:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.928+0000 7f5177cf6700 1 -- 172.21.15.129:0/2757730259 wait complete. 2024-05-20T17:57:09.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.929+0000 7f5177cf6700 1 Processor -- start 2024-05-20T17:57:09.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.929+0000 7f5177cf6700 1 -- start start 2024-05-20T17:57:09.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.929+0000 7f5177cf6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5170101840 0x7f5170199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:09.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.929+0000 7f5177cf6700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5170102190 0x7f517019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.929+0000 7f5177cf6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5170107870 0x7f517019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.929+0000 7f5177cf6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5170111b30 con 0x7f5170107870 2024-05-20T17:57:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.929+0000 7f5177cf6700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f51701119b0 con 0x7f5170101840 2024-05-20T17:57:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.929+0000 7f5177cf6700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5170111cb0 con 0x7f5170102190 2024-05-20T17:57:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5176293700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5170107870 0x7f517019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5175291700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5170102190 0x7f517019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5176293700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5170107870 0x7f517019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54296/0 (socket says 172.21.15.129:54296) 2024-05-20T17:57:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5176293700 1 -- 172.21.15.129:0/3890002492 learned_addr learned my addr 172.21.15.129:0/3890002492 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5175a92700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5170101840 0x7f5170199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5176293700 1 -- 172.21.15.129:0/3890002492 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5170102190 msgr2=0x7f517019a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5176293700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5170102190 0x7f517019a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5176293700 1 -- 172.21.15.129:0/3890002492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5170101840 msgr2=0x7f5170199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5176293700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5170101840 0x7f5170199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5176293700 1 -- 172.21.15.129:0/3890002492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517019ed10 con 0x7f5170107870 2024-05-20T17:57:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5175291700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5170102190 0x7f517019a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:57:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5176293700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5170107870 0x7f517019e670 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f516800ada0 tx=0x7f516800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5166ffd700 1 -- 172.21.15.129:0/3890002492 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168017020 con 0x7f5170107870 2024-05-20T17:57:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5177cf6700 1 -- 172.21.15.129:0/3890002492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f517019f000 con 0x7f5170107870 2024-05-20T17:57:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5166ffd700 1 -- 172.21.15.129:0/3890002492 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f516800cdb0 con 0x7f5170107870 2024-05-20T17:57:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5166ffd700 1 -- 172.21.15.129:0/3890002492 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51680053e0 con 0x7f5170107870 2024-05-20T17:57:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.930+0000 7f5177cf6700 1 -- 172.21.15.129:0/3890002492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51701a6920 con 0x7f5170107870 2024-05-20T17:57:09.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.931+0000 7f5177cf6700 1 -- 172.21.15.129:0/3890002492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5170062d50 con 0x7f5170107870 2024-05-20T17:57:09.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.933+0000 7f5166ffd700 1 -- 172.21.15.129:0/3890002492 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f516800c8d0 con 0x7f5170107870 2024-05-20T17:57:09.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.933+0000 7f5166ffd700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f515c0a2cb0 0x7f515c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:09.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.933+0000 7f5175a92700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f515c0a2cb0 0x7f515c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:09.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.934+0000 7f5166ffd700 1 -- 172.21.15.129:0/3890002492 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f51680cf490 con 0x7f5170107870 2024-05-20T17:57:09.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.934+0000 7f5166ffd700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593] conn(0x7f515c0ac4e0 0x7f515c0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:09.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.934+0000 7f5166ffd700 1 -- 172.21.15.129:0/3890002492 --> [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f516800fe50 con 0x7f515c0ac4e0 2024-05-20T17:57:09.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.934+0000 7f5166ffd700 1 -- 172.21.15.129:0/3890002492 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f51680cf970 con 0x7f5170107870 2024-05-20T17:57:09.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.934+0000 7f5175a92700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f515c0a2cb0 0x7f515c0a51c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f516c005950 tx=0x7f516c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:09.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.934+0000 7f5175291700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593] conn(0x7f515c0ac4e0 0x7f515c0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:09.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.934+0000 7f5175291700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593] conn(0x7f515c0ac4e0 0x7f515c0ae930 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:09.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.935+0000 7f5166ffd700 1 -- 172.21.15.129:0/3890002492 <== osd.4 v2:172.21.15.154:6808/3850885593 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f516800fe50 con 0x7f515c0ac4e0 2024-05-20T17:57:09.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.951+0000 7f5177cf6700 1 -- 172.21.15.129:0/3890002492 --> [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5170063150 con 0x7f515c0ac4e0 2024-05-20T17:57:09.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5166ffd700 1 -- 172.21.15.129:0/3890002492 <== osd.4 v2:172.21.15.154:6808/3850885593 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5170063150 con 0x7f515c0ac4e0 2024-05-20T17:57:09.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5177cf6700 1 -- 172.21.15.129:0/3890002492 >> [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593] conn(0x7f515c0ac4e0 msgr2=0x7f515c0ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:09.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5177cf6700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593] conn(0x7f515c0ac4e0 0x7f515c0ae930 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:09.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5177cf6700 1 -- 172.21.15.129:0/3890002492 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f515c0a2cb0 msgr2=0x7f515c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:09.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5177cf6700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f515c0a2cb0 0x7f515c0a51c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f516c005950 tx=0x7f516c00b410 comp rx=0 tx=0).stop 2024-05-20T17:57:09.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5177cf6700 1 -- 172.21.15.129:0/3890002492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5170107870 msgr2=0x7f517019e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:09.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5177cf6700 1 --2- 172.21.15.129:0/3890002492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5170107870 0x7f517019e670 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f516800ada0 tx=0x7f516800c330 comp rx=0 tx=0).stop 2024-05-20T17:57:09.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5176293700 1 -- 172.21.15.129:0/3890002492 reap_dead start 2024-05-20T17:57:09.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5177cf6700 1 -- 172.21.15.129:0/3890002492 shutdown_connections 2024-05-20T17:57:09.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5177cf6700 1 -- 172.21.15.129:0/3890002492 >> 172.21.15.129:0/3890002492 conn(0x7f51700fd3a0 msgr2=0x7f5170108650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:09.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5177cf6700 1 -- 172.21.15.129:0/3890002492 shutdown_connections 2024-05-20T17:57:09.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:09.952+0000 7f5177cf6700 1 -- 172.21.15.129:0/3890002492 wait complete. 2024-05-20T17:57:10.032 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:09 smithi129 ceph-mon[32029]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-05-20T17:57:10.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:09 smithi154 ceph-mon[36482]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-05-20T17:57:10.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:09 smithi204 ceph-mon[37076]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-05-20T17:57:11.502 INFO:teuthology.orchestra.run.smithi129.stdout:141733920822 2024-05-20T17:57:11.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-20T17:57:12.015 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:11 smithi129 ceph-mon[32029]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-05-20T17:57:12.035 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:12.037 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:12.038 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:12.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.057+0000 7f274bfff700 1 -- 172.21.15.129:0/1107384180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2744095cf0 msgr2=0x7f27440a7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:12.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.057+0000 7f274bfff700 1 --2- 172.21.15.129:0/1107384180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2744095cf0 0x7f27440a7200 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f27400098f0 tx=0x7f2740009c00 comp rx=0 tx=0).stop 2024-05-20T17:57:12.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.057+0000 7f274bfff700 1 -- 172.21.15.129:0/1107384180 shutdown_connections 2024-05-20T17:57:12.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.057+0000 7f274bfff700 1 --2- 172.21.15.129:0/1107384180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f274409b570 0x7f27440a9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:12.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.057+0000 7f274bfff700 1 --2- 172.21.15.129:0/1107384180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f27440953a0 0x7f27440957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:12.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.057+0000 7f274bfff700 1 --2- 172.21.15.129:0/1107384180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2744095cf0 0x7f27440a7200 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:12.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.057+0000 7f274bfff700 1 -- 172.21.15.129:0/1107384180 >> 172.21.15.129:0/1107384180 conn(0x7f274400b050 msgr2=0x7f274400b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:12.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.057+0000 7f274bfff700 1 -- 172.21.15.129:0/1107384180 shutdown_connections 2024-05-20T17:57:12.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.057+0000 7f274bfff700 1 -- 172.21.15.129:0/1107384180 wait complete. 2024-05-20T17:57:12.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.058+0000 7f274bfff700 1 Processor -- start 2024-05-20T17:57:12.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.058+0000 7f274bfff700 1 -- start start 2024-05-20T17:57:12.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274bfff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27440953a0 0x7f27441394c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:12.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2744095cf0 0x7f2744139a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:12.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274bfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f274409b570 0x7f2744133650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:12.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274bfff700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f27440ace50 con 0x7f2744095cf0 2024-05-20T17:57:12.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274bfff700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f27440accd0 con 0x7f274409b570 2024-05-20T17:57:12.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274bfff700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f27440acfd0 con 0x7f27440953a0 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274a7fc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2744095cf0 0x7f2744139a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274a7fc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2744095cf0 0x7f2744139a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54314/0 (socket says 172.21.15.129:54314) 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274a7fc700 1 -- 172.21.15.129:0/1264762091 learned_addr learned my addr 172.21.15.129:0/1264762091 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274affd700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27440953a0 0x7f27441394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274b7fe700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f274409b570 0x7f2744133650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274a7fc700 1 -- 172.21.15.129:0/1264762091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27440953a0 msgr2=0x7f27441394c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274a7fc700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27440953a0 0x7f27441394c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274a7fc700 1 -- 172.21.15.129:0/1264762091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f274409b570 msgr2=0x7f2744133650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274a7fc700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f274409b570 0x7f2744133650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274a7fc700 1 -- 172.21.15.129:0/1264762091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2744133d80 con 0x7f2744095cf0 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274b7fe700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f274409b570 0x7f2744133650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:57:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.059+0000 7f274a7fc700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2744095cf0 0x7f2744139a00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2740009fd0 tx=0x7f274000fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.060+0000 7f2733fff700 1 -- 172.21.15.129:0/1264762091 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27400052b0 con 0x7f2744095cf0 2024-05-20T17:57:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.060+0000 7f2733fff700 1 -- 172.21.15.129:0/1264762091 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f274000ba40 con 0x7f2744095cf0 2024-05-20T17:57:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.060+0000 7f2733fff700 1 -- 172.21.15.129:0/1264762091 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2740003ba0 con 0x7f2744095cf0 2024-05-20T17:57:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.060+0000 7f274bfff700 1 -- 172.21.15.129:0/1264762091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2744134010 con 0x7f2744095cf0 2024-05-20T17:57:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.060+0000 7f274bfff700 1 -- 172.21.15.129:0/1264762091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27441343e0 con 0x7f2744095cf0 2024-05-20T17:57:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.061+0000 7f274bfff700 1 -- 172.21.15.129:0/1264762091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f27440a8900 con 0x7f2744095cf0 2024-05-20T17:57:12.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.063+0000 7f2733fff700 1 -- 172.21.15.129:0/1264762091 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2740005450 con 0x7f2744095cf0 2024-05-20T17:57:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.064+0000 7f2733fff700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f27340a2d30 0x7f27340a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.064+0000 7f274affd700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f27340a2d30 0x7f27340a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.064+0000 7f2733fff700 1 -- 172.21.15.129:0/1264762091 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f27400d0780 con 0x7f2744095cf0 2024-05-20T17:57:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.064+0000 7f2733fff700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578] conn(0x7f27340ac560 0x7f27340ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.064+0000 7f2733fff700 1 -- 172.21.15.129:0/1264762091 --> [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27340af080 con 0x7f27340ac560 2024-05-20T17:57:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.064+0000 7f2733fff700 1 -- 172.21.15.129:0/1264762091 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f27400d0c60 con 0x7f2744095cf0 2024-05-20T17:57:12.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.065+0000 7f274b7fe700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578] conn(0x7f27340ac560 0x7f27340ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:12.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.065+0000 7f274affd700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f27340a2d30 0x7f27340a5240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f273c005fd0 tx=0x7f273c009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:12.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.065+0000 7f274b7fe700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578] conn(0x7f27340ac560 0x7f27340ae9b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:12.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.066+0000 7f2733fff700 1 -- 172.21.15.129:0/1264762091 <== osd.5 v2:172.21.15.204:6808/12527578 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f27340af080 con 0x7f27340ac560 2024-05-20T17:57:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.082+0000 7f274bfff700 1 -- 172.21.15.129:0/1264762091 --> [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f27440a8b10 con 0x7f27340ac560 2024-05-20T17:57:12.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f2733fff700 1 -- 172.21.15.129:0/1264762091 <== osd.5 v2:172.21.15.204:6808/12527578 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f27440a8b10 con 0x7f27340ac560 2024-05-20T17:57:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f274bfff700 1 -- 172.21.15.129:0/1264762091 >> [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578] conn(0x7f27340ac560 msgr2=0x7f27340ae9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f274bfff700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578] conn(0x7f27340ac560 0x7f27340ae9b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f274bfff700 1 -- 172.21.15.129:0/1264762091 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f27340a2d30 msgr2=0x7f27340a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f274bfff700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f27340a2d30 0x7f27340a5240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f273c005fd0 tx=0x7f273c009310 comp rx=0 tx=0).stop 2024-05-20T17:57:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f274bfff700 1 -- 172.21.15.129:0/1264762091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2744095cf0 msgr2=0x7f2744139a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f274bfff700 1 --2- 172.21.15.129:0/1264762091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2744095cf0 0x7f2744139a00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2740009fd0 tx=0x7f274000fb20 comp rx=0 tx=0).stop 2024-05-20T17:57:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f274b7fe700 1 -- 172.21.15.129:0/1264762091 reap_dead start 2024-05-20T17:57:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f274bfff700 1 -- 172.21.15.129:0/1264762091 shutdown_connections 2024-05-20T17:57:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f274bfff700 1 -- 172.21.15.129:0/1264762091 >> 172.21.15.129:0/1264762091 conn(0x7f274400b050 msgr2=0x7f27440a7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f274bfff700 1 -- 172.21.15.129:0/1264762091 shutdown_connections 2024-05-20T17:57:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:12.083+0000 7f274bfff700 1 -- 172.21.15.129:0/1264762091 wait complete. 2024-05-20T17:57:12.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:11 smithi154 ceph-mon[36482]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-05-20T17:57:12.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:11 smithi204 ceph-mon[37076]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-05-20T17:57:13.422 INFO:teuthology.orchestra.run.smithi129.stdout:163208757296 2024-05-20T17:57:13.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-20T17:57:14.034 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:13 smithi129 ceph-mon[32029]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-20T17:57:14.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:13 smithi154 ceph-mon[36482]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-20T17:57:14.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:13 smithi204 ceph-mon[37076]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-20T17:57:15.909 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:16.010 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:15 smithi129 ceph-mon[32029]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-05-20T17:57:16.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:15 smithi154 ceph-mon[36482]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-05-20T17:57:16.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:15 smithi204 ceph-mon[37076]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-05-20T17:57:17.841 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:17 smithi129 ceph-mon[32029]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:57:18.072 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:18.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:17 smithi154 ceph-mon[36482]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:57:18.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:17 smithi204 ceph-mon[37076]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:57:19.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.799+0000 7f66802cd700 1 -- 172.21.15.129:0/3104083973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6678102340 msgr2=0x7f6678102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:19.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.799+0000 7f66802cd700 1 --2- 172.21.15.129:0/3104083973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6678102340 0x7f6678102750 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f66740099c0 tx=0x7f6674009cd0 comp rx=0 tx=0).stop 2024-05-20T17:57:19.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.799+0000 7f66802cd700 1 -- 172.21.15.129:0/3104083973 shutdown_connections 2024-05-20T17:57:19.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.799+0000 7f66802cd700 1 --2- 172.21.15.129:0/3104083973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66781075c0 0x7f667810ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:19.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.799+0000 7f66802cd700 1 --2- 172.21.15.129:0/3104083973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6678102340 0x7f6678102750 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:19.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.799+0000 7f66802cd700 1 --2- 172.21.15.129:0/3104083973 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6678106c70 0x7f6678107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:19.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.799+0000 7f66802cd700 1 -- 172.21.15.129:0/3104083973 >> 172.21.15.129:0/3104083973 conn(0x7f66780fdc70 msgr2=0x7f6678100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:19.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.799+0000 7f66802cd700 1 -- 172.21.15.129:0/3104083973 shutdown_connections 2024-05-20T17:57:19.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.799+0000 7f66802cd700 1 -- 172.21.15.129:0/3104083973 wait complete. 2024-05-20T17:57:19.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.800+0000 7f66802cd700 1 Processor -- start 2024-05-20T17:57:19.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.800+0000 7f66802cd700 1 -- start start 2024-05-20T17:57:19.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.800+0000 7f66802cd700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6678102340 0x7f66780fe150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:19.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.800+0000 7f66802cd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6678106c70 0x7f66780fe690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:19.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.800+0000 7f66802cd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66781075c0 0x7f66780febd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:19.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.800+0000 7f66802cd700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6678112340 con 0x7f66781075c0 2024-05-20T17:57:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.800+0000 7f66802cd700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f66781121c0 con 0x7f6678106c70 2024-05-20T17:57:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.800+0000 7f66802cd700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f66781124c0 con 0x7f6678102340 2024-05-20T17:57:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f667e86a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66781075c0 0x7f66780febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f667e86a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66781075c0 0x7f66780febd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39294/0 (socket says 172.21.15.129:39294) 2024-05-20T17:57:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f667e86a700 1 -- 172.21.15.129:0/440285670 learned_addr learned my addr 172.21.15.129:0/440285670 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f667e069700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6678102340 0x7f66780fe150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:19.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f667d868700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6678106c70 0x7f66780fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:19.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f667e86a700 1 -- 172.21.15.129:0/440285670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6678102340 msgr2=0x7f66780fe150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:19.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f667e86a700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6678102340 0x7f66780fe150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:19.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f667e86a700 1 -- 172.21.15.129:0/440285670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6678106c70 msgr2=0x7f66780fe690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:19.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f667e86a700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6678106c70 0x7f66780fe690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:19.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f667e86a700 1 -- 172.21.15.129:0/440285670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66781a8120 con 0x7f66781075c0 2024-05-20T17:57:19.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f667e86a700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66781075c0 0x7f66780febd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f667000ada0 tx=0x7f667000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:19.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f666f7fe700 1 -- 172.21.15.129:0/440285670 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670017020 con 0x7f66781075c0 2024-05-20T17:57:19.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f666f7fe700 1 -- 172.21.15.129:0/440285670 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f667000cdb0 con 0x7f66781075c0 2024-05-20T17:57:19.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f66802cd700 1 -- 172.21.15.129:0/440285670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66781a8410 con 0x7f66781075c0 2024-05-20T17:57:19.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f666f7fe700 1 -- 172.21.15.129:0/440285670 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66700053e0 con 0x7f66781075c0 2024-05-20T17:57:19.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.801+0000 7f66802cd700 1 -- 172.21.15.129:0/440285670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66781a8950 con 0x7f66781075c0 2024-05-20T17:57:19.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.802+0000 7f66802cd700 1 -- 172.21.15.129:0/440285670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6678062d50 con 0x7f66781075c0 2024-05-20T17:57:19.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.804+0000 7f666f7fe700 1 -- 172.21.15.129:0/440285670 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f667000c8d0 con 0x7f66781075c0 2024-05-20T17:57:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.804+0000 7f666f7fe700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f66640a2d30 0x7f66640a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.804+0000 7f667e069700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f66640a2d30 0x7f66640a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.805+0000 7f666f7fe700 1 -- 172.21.15.129:0/440285670 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f66700cf060 con 0x7f66781075c0 2024-05-20T17:57:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.805+0000 7f666f7fe700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067] conn(0x7f66640ac560 0x7f66640ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.805+0000 7f666f7fe700 1 -- 172.21.15.129:0/440285670 --> [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f667001ee50 con 0x7f66640ac560 2024-05-20T17:57:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.805+0000 7f666f7fe700 1 -- 172.21.15.129:0/440285670 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f66700cf540 con 0x7f66781075c0 2024-05-20T17:57:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.805+0000 7f667d868700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067] conn(0x7f66640ac560 0x7f66640ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:19.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.805+0000 7f667e069700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f66640a2d30 0x7f66640a5240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6674000c00 tx=0x7f667401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:19.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.805+0000 7f667d868700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067] conn(0x7f66640ac560 0x7f66640ae9b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:19.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.806+0000 7f666f7fe700 1 -- 172.21.15.129:0/440285670 <== osd.6 v2:172.21.15.129:6818/2994413067 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f667001ee50 con 0x7f66640ac560 2024-05-20T17:57:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.822+0000 7f66802cd700 1 -- 172.21.15.129:0/440285670 --> [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f66780631d0 con 0x7f66640ac560 2024-05-20T17:57:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.822+0000 7f666f7fe700 1 -- 172.21.15.129:0/440285670 <== osd.6 v2:172.21.15.129:6818/2994413067 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f66780631d0 con 0x7f66640ac560 2024-05-20T17:57:19.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.823+0000 7f66802cd700 1 -- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067] conn(0x7f66640ac560 msgr2=0x7f66640ae9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:19.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.823+0000 7f66802cd700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067] conn(0x7f66640ac560 0x7f66640ae9b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:19.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.823+0000 7f66802cd700 1 -- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f66640a2d30 msgr2=0x7f66640a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:19.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.823+0000 7f66802cd700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f66640a2d30 0x7f66640a5240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6674000c00 tx=0x7f667401a040 comp rx=0 tx=0).stop 2024-05-20T17:57:19.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.823+0000 7f66802cd700 1 -- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66781075c0 msgr2=0x7f66780febd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:19.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.823+0000 7f66802cd700 1 --2- 172.21.15.129:0/440285670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66781075c0 0x7f66780febd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f667000ada0 tx=0x7f667000c330 comp rx=0 tx=0).stop 2024-05-20T17:57:19.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.823+0000 7f667e86a700 1 -- 172.21.15.129:0/440285670 reap_dead start 2024-05-20T17:57:19.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.823+0000 7f66802cd700 1 -- 172.21.15.129:0/440285670 shutdown_connections 2024-05-20T17:57:19.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.823+0000 7f66802cd700 1 -- 172.21.15.129:0/440285670 >> 172.21.15.129:0/440285670 conn(0x7f66780fdc70 msgr2=0x7f6678106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:19.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.823+0000 7f66802cd700 1 -- 172.21.15.129:0/440285670 shutdown_connections 2024-05-20T17:57:19.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:19.823+0000 7f66802cd700 1 -- 172.21.15.129:0/440285670 wait complete. 2024-05-20T17:57:19.832 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:19 smithi129 ceph-mon[32029]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-05-20T17:57:20.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.007+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/1139173194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1cc1075c0 msgr2=0x7ff1cc10ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.007+0000 7ff1d3bd4700 1 --2- 172.21.15.129:0/1139173194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1cc1075c0 0x7ff1cc10ef20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400b0d0 tx=0x7ff1c400b3e0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.007+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/1139173194 shutdown_connections 2024-05-20T17:57:20.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.007+0000 7ff1d3bd4700 1 --2- 172.21.15.129:0/1139173194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1cc1075c0 0x7ff1cc10ef20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.007+0000 7ff1d3bd4700 1 --2- 172.21.15.129:0/1139173194 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1cc102340 0x7ff1cc102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.007+0000 7ff1d3bd4700 1 --2- 172.21.15.129:0/1139173194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc106c70 0x7ff1cc107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.007+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/1139173194 >> 172.21.15.129:0/1139173194 conn(0x7ff1cc0fdc40 msgr2=0x7ff1cc100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:20.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.007+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/1139173194 shutdown_connections 2024-05-20T17:57:20.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.007+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/1139173194 wait complete. 2024-05-20T17:57:20.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.008+0000 7ff1d3bd4700 1 Processor -- start 2024-05-20T17:57:20.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.008+0000 7ff1d3bd4700 1 -- start start 2024-05-20T17:57:20.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.008+0000 7ff1d3bd4700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1cc102340 0x7ff1cc19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.008+0000 7ff1d3bd4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc106c70 0x7ff1cc19b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.008+0000 7ff1d3bd4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1cc1075c0 0x7ff1cc19b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.008+0000 7ff1d3bd4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc112360 con 0x7ff1cc1075c0 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.008+0000 7ff1d3bd4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc1121e0 con 0x7ff1cc106c70 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.008+0000 7ff1d3bd4700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc1124e0 con 0x7ff1cc102340 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.008+0000 7ff1d2171700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1cc1075c0 0x7ff1cc19b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.008+0000 7ff1d2171700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1cc1075c0 0x7ff1cc19b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39314/0 (socket says 172.21.15.129:39314) 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.008+0000 7ff1d2171700 1 -- 172.21.15.129:0/3720004042 learned_addr learned my addr 172.21.15.129:0/3720004042 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1d1970700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1cc102340 0x7ff1cc19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1d116f700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc106c70 0x7ff1cc19b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1d2171700 1 -- 172.21.15.129:0/3720004042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1cc102340 msgr2=0x7ff1cc19aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1d2171700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1cc102340 0x7ff1cc19aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1d2171700 1 -- 172.21.15.129:0/3720004042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc106c70 msgr2=0x7ff1cc19b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1d2171700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc106c70 0x7ff1cc19b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1d2171700 1 -- 172.21.15.129:0/3720004042 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1cc1a5ef0 con 0x7ff1cc1075c0 2024-05-20T17:57:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1d1970700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1cc102340 0x7ff1cc19aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:57:20.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1d116f700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc106c70 0x7ff1cc19b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:57:20.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1d2171700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1cc1075c0 0x7ff1cc19b640 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4006070 tx=0x7ff1c4004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:20.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1c2ffd700 1 -- 172.21.15.129:0/3720004042 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4021070 con 0x7ff1cc1075c0 2024-05-20T17:57:20.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1c2ffd700 1 -- 172.21.15.129:0/3720004042 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1c4007c80 con 0x7ff1cc1075c0 2024-05-20T17:57:20.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/3720004042 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1cc1a6180 con 0x7ff1cc1075c0 2024-05-20T17:57:20.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.009+0000 7ff1c2ffd700 1 -- 172.21.15.129:0/3720004042 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40053c0 con 0x7ff1cc1075c0 2024-05-20T17:57:20.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.010+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/3720004042 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1cc1a6630 con 0x7ff1cc1075c0 2024-05-20T17:57:20.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.010+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/3720004042 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff1cc062d50 con 0x7ff1cc1075c0 2024-05-20T17:57:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.012+0000 7ff1c2ffd700 1 -- 172.21.15.129:0/3720004042 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff1c4024070 con 0x7ff1cc1075c0 2024-05-20T17:57:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.012+0000 7ff1c2ffd700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff1b80a2cb0 0x7ff1b80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.012+0000 7ff1d1970700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff1b80a2cb0 0x7ff1b80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.013+0000 7ff1c2ffd700 1 -- 172.21.15.129:0/3720004042 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff1c400f020 con 0x7ff1cc1075c0 2024-05-20T17:57:20.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.013+0000 7ff1c2ffd700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271] conn(0x7ff1b80ac4e0 0x7ff1b80ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.013+0000 7ff1c2ffd700 1 -- 172.21.15.129:0/3720004042 --> [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1c4026e50 con 0x7ff1b80ac4e0 2024-05-20T17:57:20.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.013+0000 7ff1c2ffd700 1 -- 172.21.15.129:0/3720004042 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7ff1c40d0e90 con 0x7ff1cc1075c0 2024-05-20T17:57:20.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.013+0000 7ff1d1970700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff1b80a2cb0 0x7ff1b80a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8005fd0 tx=0x7ff1c8005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:20.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.013+0000 7ff1d116f700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271] conn(0x7ff1b80ac4e0 0x7ff1b80ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.014+0000 7ff1d116f700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271] conn(0x7ff1b80ac4e0 0x7ff1b80ae930 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:20.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.014+0000 7ff1c2ffd700 1 -- 172.21.15.129:0/3720004042 <== osd.7 v2:172.21.15.154:6816/2792653271 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1c4026e50 con 0x7ff1b80ac4e0 2024-05-20T17:57:20.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.031+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/3720004042 --> [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff1cc0631d0 con 0x7ff1b80ac4e0 2024-05-20T17:57:20.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.031+0000 7ff1c2ffd700 1 -- 172.21.15.129:0/3720004042 <== osd.7 v2:172.21.15.154:6816/2792653271 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff1cc0631d0 con 0x7ff1b80ac4e0 2024-05-20T17:57:20.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.032+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/3720004042 >> [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271] conn(0x7ff1b80ac4e0 msgr2=0x7ff1b80ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.032+0000 7ff1d3bd4700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271] conn(0x7ff1b80ac4e0 0x7ff1b80ae930 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.032+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/3720004042 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff1b80a2cb0 msgr2=0x7ff1b80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.032+0000 7ff1d3bd4700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff1b80a2cb0 0x7ff1b80a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8005fd0 tx=0x7ff1c8005ee0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.032+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/3720004042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1cc1075c0 msgr2=0x7ff1cc19b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.032+0000 7ff1d3bd4700 1 --2- 172.21.15.129:0/3720004042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1cc1075c0 0x7ff1cc19b640 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4006070 tx=0x7ff1c4004db0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.032+0000 7ff1d2171700 1 -- 172.21.15.129:0/3720004042 reap_dead start 2024-05-20T17:57:20.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.032+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/3720004042 shutdown_connections 2024-05-20T17:57:20.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.032+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/3720004042 >> 172.21.15.129:0/3720004042 conn(0x7ff1cc0fdc40 msgr2=0x7ff1cc1063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:20.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.032+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/3720004042 shutdown_connections 2024-05-20T17:57:20.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.032+0000 7ff1d3bd4700 1 -- 172.21.15.129:0/3720004042 wait complete. 2024-05-20T17:57:20.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:19 smithi154 ceph-mon[36482]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-05-20T17:57:20.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:19 smithi204 ceph-mon[37076]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-05-20T17:57:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.761+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/3141131580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf38103460 msgr2=0x7fcf38103870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.761+0000 7fcf3c8dc700 1 --2- 172.21.15.129:0/3141131580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf38103460 0x7fcf38103870 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcf280099c0 tx=0x7fcf28009cd0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.761+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/3141131580 shutdown_connections 2024-05-20T17:57:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.761+0000 7fcf3c8dc700 1 --2- 172.21.15.129:0/3141131580 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf381086e0 0x7fcf38110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.761+0000 7fcf3c8dc700 1 --2- 172.21.15.129:0/3141131580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf38103460 0x7fcf38103870 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.761+0000 7fcf3c8dc700 1 --2- 172.21.15.129:0/3141131580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf38107d90 0x7fcf381081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.761+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/3141131580 >> 172.21.15.129:0/3141131580 conn(0x7fcf380fecf0 msgr2=0x7fcf38101140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.761+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/3141131580 shutdown_connections 2024-05-20T17:57:20.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.761+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/3141131580 wait complete. 2024-05-20T17:57:20.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.762+0000 7fcf3c8dc700 1 Processor -- start 2024-05-20T17:57:20.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.762+0000 7fcf3c8dc700 1 -- start start 2024-05-20T17:57:20.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.762+0000 7fcf3c8dc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf38103460 0x7fcf38199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.762+0000 7fcf3c8dc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf38107d90 0x7fcf3819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.762+0000 7fcf3c8dc700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf381086e0 0x7fcf3819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.762+0000 7fcf3c8dc700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcf38113480 con 0x7fcf38103460 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.762+0000 7fcf3c8dc700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcf38113300 con 0x7fcf38107d90 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.762+0000 7fcf3c8dc700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcf38113600 con 0x7fcf381086e0 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.762+0000 7fcf3659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf38103460 0x7fcf38199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.762+0000 7fcf3659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf38103460 0x7fcf38199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39356/0 (socket says 172.21.15.129:39356) 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf3659c700 1 -- 172.21.15.129:0/2114024316 learned_addr learned my addr 172.21.15.129:0/2114024316 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf36d9d700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf381086e0 0x7fcf3819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf35d9b700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf38107d90 0x7fcf3819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf3659c700 1 -- 172.21.15.129:0/2114024316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf381086e0 msgr2=0x7fcf3819e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf3659c700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf381086e0 0x7fcf3819e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf3659c700 1 -- 172.21.15.129:0/2114024316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf38107d90 msgr2=0x7fcf3819a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf3659c700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf38107d90 0x7fcf3819a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf3659c700 1 -- 172.21.15.129:0/2114024316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3819eca0 con 0x7fcf38103460 2024-05-20T17:57:20.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf35d9b700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf38107d90 0x7fcf3819a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:57:20.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf36d9d700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf381086e0 0x7fcf3819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:57:20.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf3659c700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf38103460 0x7fcf38199dd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcf280059d0 tx=0x7fcf2800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:20.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf277fe700 1 -- 172.21.15.129:0/2114024316 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf28027070 con 0x7fcf38103460 2024-05-20T17:57:20.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/2114024316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf3819ef30 con 0x7fcf38103460 2024-05-20T17:57:20.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf277fe700 1 -- 172.21.15.129:0/2114024316 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf28003680 con 0x7fcf38103460 2024-05-20T17:57:20.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf277fe700 1 -- 172.21.15.129:0/2114024316 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf28017650 con 0x7fcf38103460 2024-05-20T17:57:20.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.763+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/2114024316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf38074870 con 0x7fcf38103460 2024-05-20T17:57:20.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.764+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/2114024316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcf3819f460 con 0x7fcf38103460 2024-05-20T17:57:20.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.766+0000 7fcf277fe700 1 -- 172.21.15.129:0/2114024316 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fcf2802a070 con 0x7fcf38103460 2024-05-20T17:57:20.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.767+0000 7fcf277fe700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fcf200a70c0 0x7fcf200a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.767+0000 7fcf277fe700 1 -- 172.21.15.129:0/2114024316 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fcf280d0b10 con 0x7fcf38103460 2024-05-20T17:57:20.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.767+0000 7fcf35d9b700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fcf200a70c0 0x7fcf200a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.767+0000 7fcf277fe700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746] conn(0x7fcf200b08f0 0x7fcf200b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.767+0000 7fcf277fe700 1 -- 172.21.15.129:0/2114024316 --> [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf280056d0 con 0x7fcf200b08f0 2024-05-20T17:57:20.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.767+0000 7fcf277fe700 1 -- 172.21.15.129:0/2114024316 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fcf280d0ff0 con 0x7fcf38103460 2024-05-20T17:57:20.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.767+0000 7fcf36d9d700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746] conn(0x7fcf200b08f0 0x7fcf200b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.768+0000 7fcf35d9b700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fcf200a70c0 0x7fcf200a95d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcf381025d0 tx=0x7fcf2c00b5f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:20.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.768+0000 7fcf36d9d700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746] conn(0x7fcf200b08f0 0x7fcf200b2d40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:20.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.769+0000 7fcf277fe700 1 -- 172.21.15.129:0/2114024316 <== osd.0 v2:172.21.15.129:6802/4065863746 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf280056d0 con 0x7fcf200b08f0 2024-05-20T17:57:20.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.785+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/2114024316 --> [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcf38062fa0 con 0x7fcf200b08f0 2024-05-20T17:57:20.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.785+0000 7fcf277fe700 1 -- 172.21.15.129:0/2114024316 <== osd.0 v2:172.21.15.129:6802/4065863746 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcf38062fa0 con 0x7fcf200b08f0 2024-05-20T17:57:20.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.785+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746] conn(0x7fcf200b08f0 msgr2=0x7fcf200b2d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.785+0000 7fcf3c8dc700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746] conn(0x7fcf200b08f0 0x7fcf200b2d40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.785+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fcf200a70c0 msgr2=0x7fcf200a95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.785+0000 7fcf3c8dc700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fcf200a70c0 0x7fcf200a95d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcf381025d0 tx=0x7fcf2c00b5f0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.785+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf38103460 msgr2=0x7fcf38199dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.785+0000 7fcf3c8dc700 1 --2- 172.21.15.129:0/2114024316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf38103460 0x7fcf38199dd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcf280059d0 tx=0x7fcf2800fa00 comp rx=0 tx=0).stop 2024-05-20T17:57:20.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.785+0000 7fcf36d9d700 1 -- 172.21.15.129:0/2114024316 reap_dead start 2024-05-20T17:57:20.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.785+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/2114024316 shutdown_connections 2024-05-20T17:57:20.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.785+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/2114024316 >> 172.21.15.129:0/2114024316 conn(0x7fcf380fecf0 msgr2=0x7fcf38107500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:20.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.786+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/2114024316 shutdown_connections 2024-05-20T17:57:20.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.786+0000 7fcf3c8dc700 1 -- 172.21.15.129:0/2114024316 wait complete. 2024-05-20T17:57:20.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.972+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/1406585913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e64104ae0 msgr2=0x7f5e6410d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.972+0000 7f5e6a7a2700 1 --2- 172.21.15.129:0/1406585913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e64104ae0 0x7f5e6410d030 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009990 tx=0x7f5e58009ca0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.972+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/1406585913 shutdown_connections 2024-05-20T17:57:20.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.972+0000 7f5e6a7a2700 1 --2- 172.21.15.129:0/1406585913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e6410d5c0 0x7f5e6410fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.972+0000 7f5e6a7a2700 1 --2- 172.21.15.129:0/1406585913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e64104190 0x7f5e641045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.972+0000 7f5e6a7a2700 1 --2- 172.21.15.129:0/1406585913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e64104ae0 0x7f5e6410d030 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.972+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/1406585913 >> 172.21.15.129:0/1406585913 conn(0x7f5e64077940 msgr2=0x7f5e64077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:20.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.972+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/1406585913 shutdown_connections 2024-05-20T17:57:20.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.973+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/1406585913 wait complete. 2024-05-20T17:57:20.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.973+0000 7f5e6a7a2700 1 Processor -- start 2024-05-20T17:57:20.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.973+0000 7f5e6a7a2700 1 -- start start 2024-05-20T17:57:20.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.973+0000 7f5e6a7a2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e64104190 0x7f5e6419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e6a7a2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e64104ae0 0x7f5e6419fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e6a7a2700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6410d5c0 0x7f5e64199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e6a7a2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5e64113230 con 0x7f5e64104190 2024-05-20T17:57:20.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e6a7a2700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e641130b0 con 0x7f5e64104ae0 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e6a7a2700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5e641133b0 con 0x7f5e6410d5c0 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e697a0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e64104190 0x7f5e6419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e697a0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e64104190 0x7f5e6419f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39386/0 (socket says 172.21.15.129:39386) 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e697a0700 1 -- 172.21.15.129:0/878074851 learned_addr learned my addr 172.21.15.129:0/878074851 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e68f9f700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e64104ae0 0x7f5e6419fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e69fa1700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6410d5c0 0x7f5e64199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e697a0700 1 -- 172.21.15.129:0/878074851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6410d5c0 msgr2=0x7f5e64199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e697a0700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5e6410d5c0 0x7f5e64199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e697a0700 1 -- 172.21.15.129:0/878074851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e64104ae0 msgr2=0x7f5e6419fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e697a0700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e64104ae0 0x7f5e6419fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e697a0700 1 -- 172.21.15.129:0/878074851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6419a0c0 con 0x7f5e64104190 2024-05-20T17:57:20.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e697a0700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e64104190 0x7f5e6419f7f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5e5400ef10 tx=0x7f5e5400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:20.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e627fc700 1 -- 172.21.15.129:0/878074851 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5400ca70 con 0x7f5e64104190 2024-05-20T17:57:20.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e627fc700 1 -- 172.21.15.129:0/878074851 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e54019070 con 0x7f5e64104190 2024-05-20T17:57:20.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/878074851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e6419a350 con 0x7f5e64104190 2024-05-20T17:57:20.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e627fc700 1 -- 172.21.15.129:0/878074851 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e540146e0 con 0x7f5e64104190 2024-05-20T17:57:20.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.974+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/878074851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e641a6680 con 0x7f5e64104190 2024-05-20T17:57:20.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.975+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/878074851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5e64062d50 con 0x7f5e64104190 2024-05-20T17:57:20.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.977+0000 7f5e627fc700 1 -- 172.21.15.129:0/878074851 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f5e54012050 con 0x7f5e64104190 2024-05-20T17:57:20.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.977+0000 7f5e627fc700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5e4c0a2cb0 0x7f5e4c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.977+0000 7f5e68f9f700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5e4c0a2cb0 0x7f5e4c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.977+0000 7f5e627fc700 1 -- 172.21.15.129:0/878074851 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f5e540cfb40 con 0x7f5e64104190 2024-05-20T17:57:20.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.978+0000 7f5e627fc700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720] conn(0x7f5e4c0ac4e0 0x7f5e4c0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:20.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.978+0000 7f5e627fc700 1 -- 172.21.15.129:0/878074851 --> [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e54026e50 con 0x7f5e4c0ac4e0 2024-05-20T17:57:20.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.978+0000 7f5e627fc700 1 -- 172.21.15.129:0/878074851 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f5e540d0020 con 0x7f5e64104190 2024-05-20T17:57:20.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.978+0000 7f5e68f9f700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5e4c0a2cb0 0x7f5e4c0a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800b2e0 tx=0x7f5e58005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:20.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.978+0000 7f5e69fa1700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720] conn(0x7f5e4c0ac4e0 0x7f5e4c0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:20.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.978+0000 7f5e69fa1700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720] conn(0x7f5e4c0ac4e0 0x7f5e4c0ae930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:20.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.979+0000 7f5e627fc700 1 -- 172.21.15.129:0/878074851 <== osd.2 v2:172.21.15.204:6800/278897720 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5e54026e50 con 0x7f5e4c0ac4e0 2024-05-20T17:57:20.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.995+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/878074851 --> [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5e640631d0 con 0x7f5e4c0ac4e0 2024-05-20T17:57:20.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e627fc700 1 -- 172.21.15.129:0/878074851 <== osd.2 v2:172.21.15.204:6800/278897720 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5e640631d0 con 0x7f5e4c0ac4e0 2024-05-20T17:57:20.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/878074851 >> [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720] conn(0x7f5e4c0ac4e0 msgr2=0x7f5e4c0ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e6a7a2700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720] conn(0x7f5e4c0ac4e0 0x7f5e4c0ae930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/878074851 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5e4c0a2cb0 msgr2=0x7f5e4c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e6a7a2700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5e4c0a2cb0 0x7f5e4c0a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800b2e0 tx=0x7f5e58005960 comp rx=0 tx=0).stop 2024-05-20T17:57:20.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/878074851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e64104190 msgr2=0x7f5e6419f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:20.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e6a7a2700 1 --2- 172.21.15.129:0/878074851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e64104190 0x7f5e6419f7f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5e5400ef10 tx=0x7f5e5400c3a0 comp rx=0 tx=0).stop 2024-05-20T17:57:20.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e69fa1700 1 -- 172.21.15.129:0/878074851 reap_dead start 2024-05-20T17:57:20.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/878074851 shutdown_connections 2024-05-20T17:57:20.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/878074851 >> 172.21.15.129:0/878074851 conn(0x7f5e64077940 msgr2=0x7f5e640fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:20.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/878074851 shutdown_connections 2024-05-20T17:57:20.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.996+0000 7f5e6a7a2700 1 -- 172.21.15.129:0/878074851 wait complete. 2024-05-20T17:57:21.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.570+0000 7fdb48200700 1 -- 172.21.15.129:0/3877308013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb40101840 msgr2=0x7fdb40101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.570+0000 7fdb48200700 1 --2- 172.21.15.129:0/3877308013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb40101840 0x7fdb40101c50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdb30009990 tx=0x7fdb30009ca0 comp rx=0 tx=0).stop 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.570+0000 7fdb48200700 1 -- 172.21.15.129:0/3877308013 shutdown_connections 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.570+0000 7fdb48200700 1 --2- 172.21.15.129:0/3877308013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb40102190 0x7fdb4010e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.570+0000 7fdb48200700 1 --2- 172.21.15.129:0/3877308013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb40107870 0x7fdb40107c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.570+0000 7fdb48200700 1 --2- 172.21.15.129:0/3877308013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb40101840 0x7fdb40101c50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.570+0000 7fdb48200700 1 -- 172.21.15.129:0/3877308013 >> 172.21.15.129:0/3877308013 conn(0x7fdb400fd570 msgr2=0x7fdb400ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.570+0000 7fdb48200700 1 -- 172.21.15.129:0/3877308013 shutdown_connections 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.570+0000 7fdb48200700 1 -- 172.21.15.129:0/3877308013 wait complete. 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.571+0000 7fdb48200700 1 Processor -- start 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.571+0000 7fdb48200700 1 -- start start 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.571+0000 7fdb48200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb40101840 0x7fdb40199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:21.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.571+0000 7fdb48200700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb40102190 0x7fdb4019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.571+0000 7fdb48200700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb40107870 0x7fdb4019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.571+0000 7fdb48200700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdb40111310 con 0x7fdb40107870 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.571+0000 7fdb48200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb40111190 con 0x7fdb40101840 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.571+0000 7fdb48200700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdb40111490 con 0x7fdb40102190 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.571+0000 7fdb4679d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb40107870 0x7fdb4019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb4679d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb40107870 0x7fdb4019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39334/0 (socket says 172.21.15.129:39334) 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb4679d700 1 -- 172.21.15.129:0/2363500190 learned_addr learned my addr 172.21.15.129:0/2363500190 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb45f9c700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb40101840 0x7fdb40199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb4579b700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb40102190 0x7fdb4019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb4679d700 1 -- 172.21.15.129:0/2363500190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb40102190 msgr2=0x7fdb4019a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb4679d700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb40102190 0x7fdb4019a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb4679d700 1 -- 172.21.15.129:0/2363500190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb40101840 msgr2=0x7fdb40199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:21.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb4679d700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb40101840 0x7fdb40199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb4679d700 1 -- 172.21.15.129:0/2363500190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4019ed20 con 0x7fdb40107870 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb4579b700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb40102190 0x7fdb4019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb4679d700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb40107870 0x7fdb4019e680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009690 tx=0x7fdb38007f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb36ffd700 1 -- 172.21.15.129:0/2363500190 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb38018030 con 0x7fdb40107870 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb36ffd700 1 -- 172.21.15.129:0/2363500190 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb3800ea30 con 0x7fdb40107870 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb36ffd700 1 -- 172.21.15.129:0/2363500190 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb380047a0 con 0x7fdb40107870 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb48200700 1 -- 172.21.15.129:0/2363500190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb4019f010 con 0x7fdb40107870 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.572+0000 7fdb48200700 1 -- 172.21.15.129:0/2363500190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb401a6930 con 0x7fdb40107870 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.573+0000 7fdb48200700 1 -- 172.21.15.129:0/2363500190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb40062dd0 con 0x7fdb40107870 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.575+0000 7fdb36ffd700 1 -- 172.21.15.129:0/2363500190 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fdb3801b070 con 0x7fdb40107870 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.576+0000 7fdb36ffd700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdb2c0a2cb0 0x7fdb2c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:21.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.576+0000 7fdb36ffd700 1 -- 172.21.15.129:0/2363500190 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fdb380cf4b0 con 0x7fdb40107870 2024-05-20T17:57:21.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.576+0000 7fdb45f9c700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdb2c0a2cb0 0x7fdb2c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:21.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.576+0000 7fdb36ffd700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241] conn(0x7fdb2c0ac4e0 0x7fdb2c0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:21.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.576+0000 7fdb36ffd700 1 -- 172.21.15.129:0/2363500190 --> [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb38018ec0 con 0x7fdb2c0ac4e0 2024-05-20T17:57:21.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.576+0000 7fdb36ffd700 1 -- 172.21.15.129:0/2363500190 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fdb380cf990 con 0x7fdb40107870 2024-05-20T17:57:21.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.576+0000 7fdb45f9c700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdb2c0a2cb0 0x7fdb2c0a51c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdb40102980 tx=0x7fdb3c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:21.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.577+0000 7fdb4579b700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241] conn(0x7fdb2c0ac4e0 0x7fdb2c0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:21.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.577+0000 7fdb4579b700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241] conn(0x7fdb2c0ac4e0 0x7fdb2c0ae930 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.578+0000 7fdb36ffd700 1 -- 172.21.15.129:0/2363500190 <== osd.11 v2:172.21.15.204:6824/329726241 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb38018ec0 con 0x7fdb2c0ac4e0 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.594+0000 7fdb48200700 1 -- 172.21.15.129:0/2363500190 --> [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb40063250 con 0x7fdb2c0ac4e0 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb36ffd700 1 -- 172.21.15.129:0/2363500190 <== osd.11 v2:172.21.15.204:6824/329726241 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdb40063250 con 0x7fdb2c0ac4e0 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb48200700 1 -- 172.21.15.129:0/2363500190 >> [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241] conn(0x7fdb2c0ac4e0 msgr2=0x7fdb2c0ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb48200700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241] conn(0x7fdb2c0ac4e0 0x7fdb2c0ae930 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb48200700 1 -- 172.21.15.129:0/2363500190 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdb2c0a2cb0 msgr2=0x7fdb2c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb48200700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdb2c0a2cb0 0x7fdb2c0a51c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdb40102980 tx=0x7fdb3c009260 comp rx=0 tx=0).stop 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb48200700 1 -- 172.21.15.129:0/2363500190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb40107870 msgr2=0x7fdb4019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb48200700 1 --2- 172.21.15.129:0/2363500190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdb40107870 0x7fdb4019e680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009690 tx=0x7fdb38007f80 comp rx=0 tx=0).stop 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb4679d700 1 -- 172.21.15.129:0/2363500190 reap_dead start 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb48200700 1 -- 172.21.15.129:0/2363500190 shutdown_connections 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb48200700 1 -- 172.21.15.129:0/2363500190 >> 172.21.15.129:0/2363500190 conn(0x7fdb400fd570 msgr2=0x7fdb40106b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb48200700 1 -- 172.21.15.129:0/2363500190 shutdown_connections 2024-05-20T17:57:21.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:20.595+0000 7fdb48200700 1 -- 172.21.15.129:0/2363500190 wait complete. 2024-05-20T17:57:22.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:21 smithi154 ceph-mon[36482]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-05-20T17:57:22.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:21 smithi129 ceph-mon[32029]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-05-20T17:57:22.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:21 smithi204 ceph-mon[37076]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-05-20T17:57:22.588 INFO:teuthology.orchestra.run.smithi129.stdout:219043332136 2024-05-20T17:57:22.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-20T17:57:23.195 INFO:teuthology.orchestra.run.smithi129.stdout:34359738443 2024-05-20T17:57:23.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-20T17:57:23.486 INFO:teuthology.orchestra.run.smithi129.stdout:347892350996 2024-05-20T17:57:23.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-20T17:57:24.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:23 smithi154 ceph-mon[36482]: pgmap v304: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:57:24.148 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:23 smithi129 ceph-mon[32029]: pgmap v304: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:57:24.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:23 smithi204 ceph-mon[37076]: pgmap v304: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:57:24.698 INFO:teuthology.orchestra.run.smithi129.stdout:77309411394 2024-05-20T17:57:24.698 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-20T17:57:24.716 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:25.161 INFO:teuthology.orchestra.run.smithi129.stdout:197568495662 2024-05-20T17:57:25.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-20T17:57:25.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:25 smithi129 ceph-mon[32029]: pgmap v305: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:26.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.024+0000 7f8e7d44d700 1 -- 172.21.15.129:0/3969803581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e78107d80 msgr2=0x7f8e78108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:26.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.024+0000 7f8e7d44d700 1 --2- 172.21.15.129:0/3969803581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e78107d80 0x7f8e78108190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0099c0 tx=0x7f8e6c009cd0 comp rx=0 tx=0).stop 2024-05-20T17:57:26.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.024+0000 7f8e7d44d700 1 -- 172.21.15.129:0/3969803581 shutdown_connections 2024-05-20T17:57:26.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.024+0000 7f8e7d44d700 1 --2- 172.21.15.129:0/3969803581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e78103450 0x7f8e78103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:26.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.024+0000 7f8e7d44d700 1 --2- 172.21.15.129:0/3969803581 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e781086d0 0x7f8e78110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:26.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.024+0000 7f8e7d44d700 1 --2- 172.21.15.129:0/3969803581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e78107d80 0x7f8e78108190 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:26.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.024+0000 7f8e7d44d700 1 -- 172.21.15.129:0/3969803581 >> 172.21.15.129:0/3969803581 conn(0x7f8e780fed20 msgr2=0x7f8e78101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:26.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.024+0000 7f8e7d44d700 1 -- 172.21.15.129:0/3969803581 shutdown_connections 2024-05-20T17:57:26.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.024+0000 7f8e7d44d700 1 -- 172.21.15.129:0/3969803581 wait complete. 2024-05-20T17:57:26.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e7d44d700 1 Processor -- start 2024-05-20T17:57:26.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e7d44d700 1 -- start start 2024-05-20T17:57:26.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e7d44d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e78103450 0x7f8e78199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:26.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e7d44d700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e78107d80 0x7f8e7819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:26.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e7d44d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e781086d0 0x7f8e7819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:26.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e7d44d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8e78113480 con 0x7f8e78103450 2024-05-20T17:57:26.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e7d44d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8e78113300 con 0x7f8e781086d0 2024-05-20T17:57:26.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e7d44d700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8e78113600 con 0x7f8e78107d80 2024-05-20T17:57:26.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e76ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e78103450 0x7f8e78199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:26.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e76ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e78103450 0x7f8e78199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39404/0 (socket says 172.21.15.129:39404) 2024-05-20T17:57:26.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e76ffd700 1 -- 172.21.15.129:0/1777384402 learned_addr learned my addr 172.21.15.129:0/1777384402 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:26.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.025+0000 7f8e777fe700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e781086d0 0x7f8e7819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:26.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e767fc700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e78107d80 0x7f8e7819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:26.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e76ffd700 1 -- 172.21.15.129:0/1777384402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e78107d80 msgr2=0x7f8e7819a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:26.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e76ffd700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8e78107d80 0x7f8e7819a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:26.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e76ffd700 1 -- 172.21.15.129:0/1777384402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e781086d0 msgr2=0x7f8e7819e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:26.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e76ffd700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e781086d0 0x7f8e7819e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:26.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e76ffd700 1 -- 172.21.15.129:0/1777384402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7819ec80 con 0x7f8e78103450 2024-05-20T17:57:26.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e76ffd700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e78103450 0x7f8e78199db0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800e9a0 tx=0x7f8e6800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:26.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e5ffff700 1 -- 172.21.15.129:0/1777384402 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6800cbd0 con 0x7f8e78103450 2024-05-20T17:57:26.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e7d44d700 1 -- 172.21.15.129:0/1777384402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7819ef70 con 0x7f8e78103450 2024-05-20T17:57:26.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e5ffff700 1 -- 172.21.15.129:0/1777384402 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e6800cd70 con 0x7f8e78103450 2024-05-20T17:57:26.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e5ffff700 1 -- 172.21.15.129:0/1777384402 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68018750 con 0x7f8e78103450 2024-05-20T17:57:26.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.026+0000 7f8e7d44d700 1 -- 172.21.15.129:0/1777384402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e781a6890 con 0x7f8e78103450 2024-05-20T17:57:26.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.027+0000 7f8e7d44d700 1 -- 172.21.15.129:0/1777384402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8e78062d90 con 0x7f8e78103450 2024-05-20T17:57:26.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.028+0000 7f8e5ffff700 1 -- 172.21.15.129:0/1777384402 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f8e68010430 con 0x7f8e78103450 2024-05-20T17:57:26.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.029+0000 7f8e5ffff700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8e600a2cb0 0x7f8e600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:26.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.029+0000 7f8e767fc700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8e600a2cb0 0x7f8e600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:26.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.029+0000 7f8e5ffff700 1 -- 172.21.15.129:0/1777384402 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f8e680cfe70 con 0x7f8e78103450 2024-05-20T17:57:26.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.029+0000 7f8e5ffff700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880] conn(0x7f8e600ac4e0 0x7f8e600ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:26.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.029+0000 7f8e5ffff700 1 -- 172.21.15.129:0/1777384402 --> [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e6801bec0 con 0x7f8e600ac4e0 2024-05-20T17:57:26.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.029+0000 7f8e5ffff700 1 -- 172.21.15.129:0/1777384402 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f8e680d0350 con 0x7f8e78103450 2024-05-20T17:57:26.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.029+0000 7f8e767fc700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8e600a2cb0 0x7f8e600a51c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0098e0 tx=0x7f8e6c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:26.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.030+0000 7f8e777fe700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880] conn(0x7f8e600ac4e0 0x7f8e600ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:26.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.030+0000 7f8e777fe700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880] conn(0x7f8e600ac4e0 0x7f8e600ae930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:26.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.031+0000 7f8e5ffff700 1 -- 172.21.15.129:0/1777384402 <== osd.10 v2:172.21.15.154:6824/1707430880 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8e6801bec0 con 0x7f8e600ac4e0 2024-05-20T17:57:26.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.047+0000 7f8e7d44d700 1 -- 172.21.15.129:0/1777384402 --> [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8e78063190 con 0x7f8e600ac4e0 2024-05-20T17:57:26.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e5ffff700 1 -- 172.21.15.129:0/1777384402 <== osd.10 v2:172.21.15.154:6824/1707430880 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8e78063190 con 0x7f8e600ac4e0 2024-05-20T17:57:26.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e7d44d700 1 -- 172.21.15.129:0/1777384402 >> [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880] conn(0x7f8e600ac4e0 msgr2=0x7f8e600ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:26.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e7d44d700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880] conn(0x7f8e600ac4e0 0x7f8e600ae930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:26.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e7d44d700 1 -- 172.21.15.129:0/1777384402 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8e600a2cb0 msgr2=0x7f8e600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:26.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e7d44d700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8e600a2cb0 0x7f8e600a51c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0098e0 tx=0x7f8e6c000bc0 comp rx=0 tx=0).stop 2024-05-20T17:57:26.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e7d44d700 1 -- 172.21.15.129:0/1777384402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e78103450 msgr2=0x7f8e78199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:26.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e7d44d700 1 --2- 172.21.15.129:0/1777384402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e78103450 0x7f8e78199db0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800e9a0 tx=0x7f8e6800ecb0 comp rx=0 tx=0).stop 2024-05-20T17:57:26.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e777fe700 1 -- 172.21.15.129:0/1777384402 reap_dead start 2024-05-20T17:57:26.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e7d44d700 1 -- 172.21.15.129:0/1777384402 shutdown_connections 2024-05-20T17:57:26.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e7d44d700 1 -- 172.21.15.129:0/1777384402 >> 172.21.15.129:0/1777384402 conn(0x7f8e780fed20 msgr2=0x7f8e781074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:26.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e7d44d700 1 -- 172.21.15.129:0/1777384402 shutdown_connections 2024-05-20T17:57:26.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:26.048+0000 7f8e7d44d700 1 -- 172.21.15.129:0/1777384402 wait complete. 2024-05-20T17:57:26.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:25 smithi154 ceph-mon[36482]: pgmap v305: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:26.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:25 smithi204 ceph-mon[37076]: pgmap v305: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:27.100 INFO:teuthology.orchestra.run.smithi129.stdout:313532612634 2024-05-20T17:57:27.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-20T17:57:27.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.772+0000 7fce80b1c700 1 -- 172.21.15.129:0/2068742215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7c10a8e0 msgr2=0x7fce7c112240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:27.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.772+0000 7fce80b1c700 1 --2- 172.21.15.129:0/2068742215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7c10a8e0 0x7fce7c112240 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fce7400b0d0 tx=0x7fce7400b3e0 comp rx=0 tx=0).stop 2024-05-20T17:57:27.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.772+0000 7fce80b1c700 1 -- 172.21.15.129:0/2068742215 shutdown_connections 2024-05-20T17:57:27.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.772+0000 7fce80b1c700 1 --2- 172.21.15.129:0/2068742215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce7c105660 0x7fce7c105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:27.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.772+0000 7fce80b1c700 1 --2- 172.21.15.129:0/2068742215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7c10a8e0 0x7fce7c112240 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:27.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.772+0000 7fce80b1c700 1 --2- 172.21.15.129:0/2068742215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce7c109f90 0x7fce7c10a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:27.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.772+0000 7fce80b1c700 1 -- 172.21.15.129:0/2068742215 >> 172.21.15.129:0/2068742215 conn(0x7fce7c100ed0 msgr2=0x7fce7c103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:27.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.772+0000 7fce80b1c700 1 -- 172.21.15.129:0/2068742215 shutdown_connections 2024-05-20T17:57:27.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.772+0000 7fce80b1c700 1 -- 172.21.15.129:0/2068742215 wait complete. 2024-05-20T17:57:27.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.773+0000 7fce80b1c700 1 Processor -- start 2024-05-20T17:57:27.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.773+0000 7fce80b1c700 1 -- start start 2024-05-20T17:57:27.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.773+0000 7fce80b1c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7c105660 0x7fce7c19bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:27.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.773+0000 7fce80b1c700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce7c109f90 0x7fce7c19c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.773+0000 7fce80b1c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce7c10a8e0 0x7fce7c1a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.773+0000 7fce80b1c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c115680 con 0x7fce7c105660 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.773+0000 7fce80b1c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c115500 con 0x7fce7c10a8e0 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.773+0000 7fce80b1c700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c115800 con 0x7fce7c109f90 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce7a59c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7c105660 0x7fce7c19bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce7a59c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7c105660 0x7fce7c19bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39428/0 (socket says 172.21.15.129:39428) 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce7a59c700 1 -- 172.21.15.129:0/2725399953 learned_addr learned my addr 172.21.15.129:0/2725399953 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce7ad9d700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce7c10a8e0 0x7fce7c1a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce79d9b700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce7c109f90 0x7fce7c19c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce7a59c700 1 -- 172.21.15.129:0/2725399953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce7c109f90 msgr2=0x7fce7c19c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce7a59c700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce7c109f90 0x7fce7c19c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:27.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce7a59c700 1 -- 172.21.15.129:0/2725399953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce7c10a8e0 msgr2=0x7fce7c1a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:27.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce7a59c700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce7c10a8e0 0x7fce7c1a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:27.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce7a59c700 1 -- 172.21.15.129:0/2725399953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7c1a0e00 con 0x7fce7c105660 2024-05-20T17:57:27.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce7a59c700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7c105660 0x7fce7c19bf30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fce6c00d980 tx=0x7fce6c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:27.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce6b7fe700 1 -- 172.21.15.129:0/2725399953 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c016070 con 0x7fce7c105660 2024-05-20T17:57:27.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce6b7fe700 1 -- 172.21.15.129:0/2725399953 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce6c004d10 con 0x7fce7c105660 2024-05-20T17:57:27.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce80b1c700 1 -- 172.21.15.129:0/2725399953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce7c1a10f0 con 0x7fce7c105660 2024-05-20T17:57:27.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce6b7fe700 1 -- 172.21.15.129:0/2725399953 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c005670 con 0x7fce7c105660 2024-05-20T17:57:27.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.774+0000 7fce80b1c700 1 -- 172.21.15.129:0/2725399953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce7c1a8a10 con 0x7fce7c105660 2024-05-20T17:57:27.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.775+0000 7fce80b1c700 1 -- 172.21.15.129:0/2725399953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fce7c103420 con 0x7fce7c105660 2024-05-20T17:57:27.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.777+0000 7fce6b7fe700 1 -- 172.21.15.129:0/2725399953 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fce6c004750 con 0x7fce7c105660 2024-05-20T17:57:27.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.777+0000 7fce6b7fe700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fce640a2d30 0x7fce640a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:27.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.777+0000 7fce6b7fe700 1 -- 172.21.15.129:0/2725399953 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fce6c0cf390 con 0x7fce7c105660 2024-05-20T17:57:27.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.777+0000 7fce79d9b700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fce640a2d30 0x7fce640a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:27.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.778+0000 7fce6b7fe700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596] conn(0x7fce640ac560 0x7fce640ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:27.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.778+0000 7fce6b7fe700 1 -- 172.21.15.129:0/2725399953 --> [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce640af080 con 0x7fce640ac560 2024-05-20T17:57:27.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.778+0000 7fce6b7fe700 1 -- 172.21.15.129:0/2725399953 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fce6c0cf870 con 0x7fce7c105660 2024-05-20T17:57:27.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.778+0000 7fce7ad9d700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596] conn(0x7fce640ac560 0x7fce640ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:27.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.778+0000 7fce79d9b700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fce640a2d30 0x7fce640a5240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fce7c102100 tx=0x7fce70009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:27.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.778+0000 7fce7ad9d700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596] conn(0x7fce640ac560 0x7fce640ae9b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:27.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.779+0000 7fce6b7fe700 1 -- 172.21.15.129:0/2725399953 <== osd.1 v2:172.21.15.154:6800/792053596 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce640af080 con 0x7fce640ac560 2024-05-20T17:57:27.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.796+0000 7fce80b1c700 1 -- 172.21.15.129:0/2725399953 --> [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fce7c106b10 con 0x7fce640ac560 2024-05-20T17:57:27.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.797+0000 7fce6b7fe700 1 -- 172.21.15.129:0/2725399953 <== osd.1 v2:172.21.15.154:6800/792053596 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fce7c106b10 con 0x7fce640ac560 2024-05-20T17:57:27.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.797+0000 7fce80b1c700 1 -- 172.21.15.129:0/2725399953 >> [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596] conn(0x7fce640ac560 msgr2=0x7fce640ae9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:27.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.797+0000 7fce80b1c700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596] conn(0x7fce640ac560 0x7fce640ae9b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:27.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.797+0000 7fce80b1c700 1 -- 172.21.15.129:0/2725399953 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fce640a2d30 msgr2=0x7fce640a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:27.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.797+0000 7fce80b1c700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fce640a2d30 0x7fce640a5240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fce7c102100 tx=0x7fce70009260 comp rx=0 tx=0).stop 2024-05-20T17:57:27.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.797+0000 7fce80b1c700 1 -- 172.21.15.129:0/2725399953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7c105660 msgr2=0x7fce7c19bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:27.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.797+0000 7fce80b1c700 1 --2- 172.21.15.129:0/2725399953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7c105660 0x7fce7c19bf30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fce6c00d980 tx=0x7fce6c00dc90 comp rx=0 tx=0).stop 2024-05-20T17:57:27.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.797+0000 7fce7ad9d700 1 -- 172.21.15.129:0/2725399953 reap_dead start 2024-05-20T17:57:27.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.797+0000 7fce80b1c700 1 -- 172.21.15.129:0/2725399953 shutdown_connections 2024-05-20T17:57:27.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.797+0000 7fce80b1c700 1 -- 172.21.15.129:0/2725399953 >> 172.21.15.129:0/2725399953 conn(0x7fce7c100ed0 msgr2=0x7fce7c1029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:27.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.797+0000 7fce80b1c700 1 -- 172.21.15.129:0/2725399953 shutdown_connections 2024-05-20T17:57:27.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:27.798+0000 7fce80b1c700 1 -- 172.21.15.129:0/2725399953 wait complete. 2024-05-20T17:57:28.039 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:27 smithi129 ceph-mon[32029]: pgmap v306: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:28.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:27 smithi154 ceph-mon[36482]: pgmap v306: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:28.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:27 smithi204 ceph-mon[37076]: pgmap v306: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.725+0000 7f2d5c904700 1 -- 172.21.15.129:0/2274584063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d58103460 msgr2=0x7f2d58103870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.725+0000 7f2d5c904700 1 --2- 172.21.15.129:0/2274584063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d58103460 0x7f2d58103870 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2d480099c0 tx=0x7f2d48009cd0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.725+0000 7f2d5c904700 1 -- 172.21.15.129:0/2274584063 shutdown_connections 2024-05-20T17:57:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.725+0000 7f2d5c904700 1 --2- 172.21.15.129:0/2274584063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d581086e0 0x7f2d58110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.725+0000 7f2d5c904700 1 --2- 172.21.15.129:0/2274584063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d58103460 0x7f2d58103870 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.725+0000 7f2d5c904700 1 --2- 172.21.15.129:0/2274584063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d58107d90 0x7f2d581081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.725+0000 7f2d5c904700 1 -- 172.21.15.129:0/2274584063 >> 172.21.15.129:0/2274584063 conn(0x7f2d580fed50 msgr2=0x7f2d581011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.725+0000 7f2d5c904700 1 -- 172.21.15.129:0/2274584063 shutdown_connections 2024-05-20T17:57:28.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.725+0000 7f2d5c904700 1 -- 172.21.15.129:0/2274584063 wait complete. 2024-05-20T17:57:28.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d5c904700 1 Processor -- start 2024-05-20T17:57:28.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d5c904700 1 -- start start 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d5c904700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d58103460 0x7f2d58199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d5c904700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d58107d90 0x7f2d5819a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d5c904700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d581086e0 0x7f2d5819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d5c904700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2d58113460 con 0x7f2d58107d90 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d5c904700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d581132e0 con 0x7f2d581086e0 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d5c904700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2d581135e0 con 0x7f2d58103460 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d55d9b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d58107d90 0x7f2d5819a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d55d9b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d58107d90 0x7f2d5819a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39432/0 (socket says 172.21.15.129:39432) 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d55d9b700 1 -- 172.21.15.129:0/2530992225 learned_addr learned my addr 172.21.15.129:0/2530992225 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d5659c700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d58103460 0x7f2d58199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d56d9d700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d581086e0 0x7f2d5819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d55d9b700 1 -- 172.21.15.129:0/2530992225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d58103460 msgr2=0x7f2d58199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d55d9b700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d58103460 0x7f2d58199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d55d9b700 1 -- 172.21.15.129:0/2530992225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d581086e0 msgr2=0x7f2d5819e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d55d9b700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d581086e0 0x7f2d5819e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.726+0000 7f2d55d9b700 1 -- 172.21.15.129:0/2530992225 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5819eb90 con 0x7f2d58107d90 2024-05-20T17:57:28.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.727+0000 7f2d55d9b700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d58107d90 0x7f2d5819a310 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00d950 tx=0x7f2d4c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:28.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.727+0000 7f2d477fe700 1 -- 172.21.15.129:0/2530992225 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c016070 con 0x7f2d58107d90 2024-05-20T17:57:28.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.727+0000 7f2d477fe700 1 -- 172.21.15.129:0/2530992225 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d4c004d10 con 0x7f2d58107d90 2024-05-20T17:57:28.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.727+0000 7f2d5c904700 1 -- 172.21.15.129:0/2530992225 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d5819ee80 con 0x7f2d58107d90 2024-05-20T17:57:28.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.727+0000 7f2d477fe700 1 -- 172.21.15.129:0/2530992225 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c005600 con 0x7f2d58107d90 2024-05-20T17:57:28.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.727+0000 7f2d5c904700 1 -- 172.21.15.129:0/2530992225 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d581a6850 con 0x7f2d58107d90 2024-05-20T17:57:28.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.728+0000 7f2d5c904700 1 -- 172.21.15.129:0/2530992225 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d58062dd0 con 0x7f2d58107d90 2024-05-20T17:57:28.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.729+0000 7f2d477fe700 1 -- 172.21.15.129:0/2530992225 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2d4c004220 con 0x7f2d58107d90 2024-05-20T17:57:28.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.730+0000 7f2d477fe700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2d400a2d30 0x7f2d400a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:28.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.730+0000 7f2d477fe700 1 -- 172.21.15.129:0/2530992225 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f2d4c0cf2d0 con 0x7f2d58107d90 2024-05-20T17:57:28.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.730+0000 7f2d5659c700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2d400a2d30 0x7f2d400a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:28.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.730+0000 7f2d477fe700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284] conn(0x7f2d400ac560 0x7f2d400ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:28.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.730+0000 7f2d477fe700 1 -- 172.21.15.129:0/2530992225 --> [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d400af080 con 0x7f2d400ac560 2024-05-20T17:57:28.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.730+0000 7f2d477fe700 1 -- 172.21.15.129:0/2530992225 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f2d4c0cf7b0 con 0x7f2d58107d90 2024-05-20T17:57:28.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.730+0000 7f2d5659c700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2d400a2d30 0x7f2d400a5240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2d48000c00 tx=0x7f2d4801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:28.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.731+0000 7f2d56d9d700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284] conn(0x7f2d400ac560 0x7f2d400ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:28.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.731+0000 7f2d56d9d700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284] conn(0x7f2d400ac560 0x7f2d400ae9b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:28.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.732+0000 7f2d477fe700 1 -- 172.21.15.129:0/2530992225 <== osd.8 v2:172.21.15.204:6816/4118068284 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2d400af080 con 0x7f2d400ac560 2024-05-20T17:57:28.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.748+0000 7f2d5c904700 1 -- 172.21.15.129:0/2530992225 --> [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2d580631d0 con 0x7f2d400ac560 2024-05-20T17:57:28.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d477fe700 1 -- 172.21.15.129:0/2530992225 <== osd.8 v2:172.21.15.204:6816/4118068284 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2d580631d0 con 0x7f2d400ac560 2024-05-20T17:57:28.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d5c904700 1 -- 172.21.15.129:0/2530992225 >> [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284] conn(0x7f2d400ac560 msgr2=0x7f2d400ae9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d5c904700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284] conn(0x7f2d400ac560 0x7f2d400ae9b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d5c904700 1 -- 172.21.15.129:0/2530992225 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2d400a2d30 msgr2=0x7f2d400a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d5c904700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2d400a2d30 0x7f2d400a5240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2d48000c00 tx=0x7f2d4801a040 comp rx=0 tx=0).stop 2024-05-20T17:57:28.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d5c904700 1 -- 172.21.15.129:0/2530992225 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d58107d90 msgr2=0x7f2d5819a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d5c904700 1 --2- 172.21.15.129:0/2530992225 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d58107d90 0x7f2d5819a310 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00d950 tx=0x7f2d4c00dc60 comp rx=0 tx=0).stop 2024-05-20T17:57:28.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d56d9d700 1 -- 172.21.15.129:0/2530992225 reap_dead start 2024-05-20T17:57:28.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d5c904700 1 -- 172.21.15.129:0/2530992225 shutdown_connections 2024-05-20T17:57:28.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d5c904700 1 -- 172.21.15.129:0/2530992225 >> 172.21.15.129:0/2530992225 conn(0x7f2d580fed50 msgr2=0x7f2d58109640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:28.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d5c904700 1 -- 172.21.15.129:0/2530992225 shutdown_connections 2024-05-20T17:57:28.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.749+0000 7f2d5c904700 1 -- 172.21.15.129:0/2530992225 wait complete. 2024-05-20T17:57:28.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.911+0000 7f50193d4700 1 -- 172.21.15.129:0/489682958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5014107d80 msgr2=0x7f5014108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.911+0000 7f50193d4700 1 --2- 172.21.15.129:0/489682958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5014107d80 0x7f5014108190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f50080099c0 tx=0x7f5008009cd0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.912+0000 7f50193d4700 1 -- 172.21.15.129:0/489682958 shutdown_connections 2024-05-20T17:57:28.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.912+0000 7f50193d4700 1 --2- 172.21.15.129:0/489682958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50141086d0 0x7f5014110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.912+0000 7f50193d4700 1 --2- 172.21.15.129:0/489682958 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5014103450 0x7f5014103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.912+0000 7f50193d4700 1 --2- 172.21.15.129:0/489682958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5014107d80 0x7f5014108190 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.912+0000 7f50193d4700 1 -- 172.21.15.129:0/489682958 >> 172.21.15.129:0/489682958 conn(0x7f50140fed20 msgr2=0x7f5014101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:28.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.912+0000 7f50193d4700 1 -- 172.21.15.129:0/489682958 shutdown_connections 2024-05-20T17:57:28.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.912+0000 7f50193d4700 1 -- 172.21.15.129:0/489682958 wait complete. 2024-05-20T17:57:28.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.912+0000 7f50193d4700 1 Processor -- start 2024-05-20T17:57:28.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50193d4700 1 -- start start 2024-05-20T17:57:28.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50193d4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5014103450 0x7f5014075a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:28.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50193d4700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5014107d80 0x7f5014075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:28.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50193d4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50141086d0 0x7f50140764d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50193d4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5014113430 con 0x7f50141086d0 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50193d4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f50141132b0 con 0x7f5014103450 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50193d4700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f50141135b0 con 0x7f5014107d80 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50137fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50141086d0 0x7f50140764d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50137fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50141086d0 0x7f50140764d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39454/0 (socket says 172.21.15.129:39454) 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50137fe700 1 -- 172.21.15.129:0/1448547336 learned_addr learned my addr 172.21.15.129:0/1448547336 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f5012ffd700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5014103450 0x7f5014075a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50127fc700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5014107d80 0x7f5014075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50137fe700 1 -- 172.21.15.129:0/1448547336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5014107d80 msgr2=0x7f5014075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50137fe700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5014107d80 0x7f5014075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50137fe700 1 -- 172.21.15.129:0/1448547336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5014103450 msgr2=0x7f5014075a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50137fe700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5014103450 0x7f5014075a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50137fe700 1 -- 172.21.15.129:0/1448547336 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50141a8120 con 0x7f50141086d0 2024-05-20T17:57:28.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.913+0000 7f50137fe700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50141086d0 0x7f50140764d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f500c007ee0 tx=0x7f500c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:28.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.914+0000 7f4ffbfff700 1 -- 172.21.15.129:0/1448547336 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c00dbd0 con 0x7f50141086d0 2024-05-20T17:57:28.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.914+0000 7f50193d4700 1 -- 172.21.15.129:0/1448547336 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50141a8410 con 0x7f50141086d0 2024-05-20T17:57:28.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.914+0000 7f4ffbfff700 1 -- 172.21.15.129:0/1448547336 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f500c00dd70 con 0x7f50141086d0 2024-05-20T17:57:28.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.914+0000 7f4ffbfff700 1 -- 172.21.15.129:0/1448547336 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c005a60 con 0x7f50141086d0 2024-05-20T17:57:28.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.914+0000 7f50193d4700 1 -- 172.21.15.129:0/1448547336 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50141a8920 con 0x7f50141086d0 2024-05-20T17:57:28.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.914+0000 7f50193d4700 1 -- 172.21.15.129:0/1448547336 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5014062d10 con 0x7f50141086d0 2024-05-20T17:57:28.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.916+0000 7f4ffbfff700 1 -- 172.21.15.129:0/1448547336 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f500c01a020 con 0x7f50141086d0 2024-05-20T17:57:28.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.917+0000 7f4ffbfff700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f4ffc0a2cb0 0x7f4ffc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:28.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.917+0000 7f5012ffd700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f4ffc0a2cb0 0x7f4ffc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:28.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.917+0000 7f4ffbfff700 1 -- 172.21.15.129:0/1448547336 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f500c0cf110 con 0x7f50141086d0 2024-05-20T17:57:28.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.917+0000 7f4ffbfff700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755] conn(0x7f4ffc0ac4e0 0x7f4ffc0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:28.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.917+0000 7f4ffbfff700 1 -- 172.21.15.129:0/1448547336 --> [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ffc0af000 con 0x7f4ffc0ac4e0 2024-05-20T17:57:28.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.917+0000 7f4ffbfff700 1 -- 172.21.15.129:0/1448547336 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f500c0cf5f0 con 0x7f50141086d0 2024-05-20T17:57:28.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.917+0000 7f50127fc700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755] conn(0x7f4ffc0ac4e0 0x7f4ffc0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:28.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.917+0000 7f5012ffd700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f4ffc0a2cb0 0x7f4ffc0a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5004005950 tx=0x7f500400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:28.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.918+0000 7f50127fc700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755] conn(0x7f4ffc0ac4e0 0x7f4ffc0ae930 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:28.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.918+0000 7f4ffbfff700 1 -- 172.21.15.129:0/1448547336 <== osd.9 v2:172.21.15.129:6826/2231439755 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4ffc0af000 con 0x7f4ffc0ac4e0 2024-05-20T17:57:28.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.934+0000 7f50193d4700 1 -- 172.21.15.129:0/1448547336 --> [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5014063110 con 0x7f4ffc0ac4e0 2024-05-20T17:57:28.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f4ffbfff700 1 -- 172.21.15.129:0/1448547336 <== osd.9 v2:172.21.15.129:6826/2231439755 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5014063110 con 0x7f4ffc0ac4e0 2024-05-20T17:57:28.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f50193d4700 1 -- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755] conn(0x7f4ffc0ac4e0 msgr2=0x7f4ffc0ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f50193d4700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755] conn(0x7f4ffc0ac4e0 0x7f4ffc0ae930 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f50193d4700 1 -- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f4ffc0a2cb0 msgr2=0x7f4ffc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f50193d4700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f4ffc0a2cb0 0x7f4ffc0a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5004005950 tx=0x7f500400b410 comp rx=0 tx=0).stop 2024-05-20T17:57:28.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f50193d4700 1 -- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50141086d0 msgr2=0x7f50140764d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:28.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f50193d4700 1 --2- 172.21.15.129:0/1448547336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50141086d0 0x7f50140764d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f500c007ee0 tx=0x7f500c00d3b0 comp rx=0 tx=0).stop 2024-05-20T17:57:28.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f50137fe700 1 -- 172.21.15.129:0/1448547336 reap_dead start 2024-05-20T17:57:28.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f50193d4700 1 -- 172.21.15.129:0/1448547336 shutdown_connections 2024-05-20T17:57:28.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f50193d4700 1 -- 172.21.15.129:0/1448547336 >> 172.21.15.129:0/1448547336 conn(0x7f50140fed20 msgr2=0x7f5014109630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:28.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f50193d4700 1 -- 172.21.15.129:0/1448547336 shutdown_connections 2024-05-20T17:57:28.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:28.935+0000 7f50193d4700 1 -- 172.21.15.129:0/1448547336 wait complete. 2024-05-20T17:57:29.658 INFO:teuthology.orchestra.run.smithi129.stdout:253403070502 2024-05-20T17:57:29.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-20T17:57:29.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:29 smithi129 ceph-mon[32029]: pgmap v307: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:30.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:29 smithi154 ceph-mon[36482]: pgmap v307: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:30.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:29 smithi204 ceph-mon[37076]: pgmap v307: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:30.346 INFO:teuthology.orchestra.run.smithi129.stdout:55834574920 2024-05-20T17:57:30.347 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-20T17:57:30.779 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:32.032 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:31 smithi129 ceph-mon[32029]: pgmap v308: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:32.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:31 smithi154 ceph-mon[36482]: pgmap v308: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:32.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:31 smithi204 ceph-mon[37076]: pgmap v308: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:32.435 INFO:teuthology.orchestra.run.smithi129.stdout:287762808864 2024-05-20T17:57:32.435 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-20T17:57:32.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.638+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1433889133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4aa4102190 msgr2=0x7f4aa410e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:32.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.638+0000 7f4aaa7e1700 1 --2- 172.21.15.129:0/1433889133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4aa4102190 0x7f4aa410e710 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00b0a0 tx=0x7f4a9c00b3b0 comp rx=0 tx=0).stop 2024-05-20T17:57:32.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.638+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1433889133 shutdown_connections 2024-05-20T17:57:32.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.638+0000 7f4aaa7e1700 1 --2- 172.21.15.129:0/1433889133 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4aa4102190 0x7f4aa410e710 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:32.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.638+0000 7f4aaa7e1700 1 --2- 172.21.15.129:0/1433889133 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4aa4107870 0x7f4aa4107c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:32.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.638+0000 7f4aaa7e1700 1 --2- 172.21.15.129:0/1433889133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4aa4101840 0x7f4aa4101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:32.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.638+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1433889133 >> 172.21.15.129:0/1433889133 conn(0x7f4aa40fd380 msgr2=0x7f4aa40ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:32.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.638+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1433889133 shutdown_connections 2024-05-20T17:57:32.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.638+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1433889133 wait complete. 2024-05-20T17:57:32.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.639+0000 7f4aaa7e1700 1 Processor -- start 2024-05-20T17:57:32.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.639+0000 7f4aaa7e1700 1 -- start start 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.639+0000 7f4aaa7e1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4aa4101840 0x7f4aa419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.639+0000 7f4aaa7e1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4aa4102190 0x7f4aa419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.639+0000 7f4aaa7e1700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4aa4107870 0x7f4aa41a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.639+0000 7f4aaa7e1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4111af0 con 0x7f4aa4101840 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.639+0000 7f4aaa7e1700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4111970 con 0x7f4aa4102190 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.639+0000 7f4aaa7e1700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4111c70 con 0x7f4aa4107870 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa3fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4aa4101840 0x7f4aa419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa3fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4aa4101840 0x7f4aa419bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60762/0 (socket says 172.21.15.129:60762) 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa3fff700 1 -- 172.21.15.129:0/1030586650 learned_addr learned my addr 172.21.15.129:0/1030586650 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa8d7e700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4aa4107870 0x7f4aa41a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa37fe700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4aa4102190 0x7f4aa419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa3fff700 1 -- 172.21.15.129:0/1030586650 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4aa4107870 msgr2=0x7f4aa41a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:32.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa3fff700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4aa4107870 0x7f4aa41a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:32.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa3fff700 1 -- 172.21.15.129:0/1030586650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4aa4102190 msgr2=0x7f4aa419c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:32.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa3fff700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4aa4102190 0x7f4aa419c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:32.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa3fff700 1 -- 172.21.15.129:0/1030586650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa41a0dd0 con 0x7f4aa4101840 2024-05-20T17:57:32.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa3fff700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4aa4101840 0x7f4aa419bf00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400d950 tx=0x7f4a9400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:32.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa17fa700 1 -- 172.21.15.129:0/1030586650 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94016070 con 0x7f4aa4101840 2024-05-20T17:57:32.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa17fa700 1 -- 172.21.15.129:0/1030586650 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a94004d10 con 0x7f4aa4101840 2024-05-20T17:57:32.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aa17fa700 1 -- 172.21.15.129:0/1030586650 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94005600 con 0x7f4aa4101840 2024-05-20T17:57:32.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1030586650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa41a10c0 con 0x7f4aa4101840 2024-05-20T17:57:32.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.640+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1030586650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa41a89e0 con 0x7f4aa4101840 2024-05-20T17:57:32.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.641+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1030586650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa4050170 con 0x7f4aa4101840 2024-05-20T17:57:32.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.643+0000 7f4aa17fa700 1 -- 172.21.15.129:0/1030586650 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f4a94004220 con 0x7f4aa4101840 2024-05-20T17:57:32.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.644+0000 7f4aa17fa700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f4a8c0a2cb0 0x7f4a8c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:32.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.644+0000 7f4aa17fa700 1 -- 172.21.15.129:0/1030586650 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f4a940cf1e0 con 0x7f4aa4101840 2024-05-20T17:57:32.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.644+0000 7f4aa37fe700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f4a8c0a2cb0 0x7f4a8c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:32.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.644+0000 7f4aa37fe700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f4a8c0a2cb0 0x7f4a8c0a51c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4105120 tx=0x7f4a98009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:32.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.648+0000 7f4aa17fa700 1 -- 172.21.15.129:0/1030586650 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4a9401f020 con 0x7f4aa4101840 2024-05-20T17:57:32.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.853+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1030586650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f4aa41a8bb0 con 0x7f4aa4101840 2024-05-20T17:57:32.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.853+0000 7f4aa17fa700 1 -- 172.21.15.129:0/1030586650 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4a9409c190 con 0x7f4aa4101840 2024-05-20T17:57:32.852 INFO:teuthology.orchestra.run.smithi129.stdout:141733920826 2024-05-20T17:57:32.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.855+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1030586650 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f4a8c0a2cb0 msgr2=0x7f4a8c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:32.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.855+0000 7f4aaa7e1700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f4a8c0a2cb0 0x7f4a8c0a51c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4105120 tx=0x7f4a98009260 comp rx=0 tx=0).stop 2024-05-20T17:57:32.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.855+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1030586650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4aa4101840 msgr2=0x7f4aa419bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:32.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.855+0000 7f4aaa7e1700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4aa4101840 0x7f4aa419bf00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4a9400d950 tx=0x7f4a9400dc60 comp rx=0 tx=0).stop 2024-05-20T17:57:32.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.855+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1030586650 shutdown_connections 2024-05-20T17:57:32.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.855+0000 7f4aaa7e1700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f4a8c0a2cb0 0x7f4a8c0a51c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:32.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.855+0000 7f4aaa7e1700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4aa4107870 0x7f4aa41a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:32.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.855+0000 7f4aaa7e1700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4aa4101840 0x7f4aa419bf00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:32.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.855+0000 7f4aaa7e1700 1 --2- 172.21.15.129:0/1030586650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4aa4102190 0x7f4aa419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:32.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.855+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1030586650 >> 172.21.15.129:0/1030586650 conn(0x7f4aa40fd380 msgr2=0x7f4aa4105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:32.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.856+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1030586650 shutdown_connections 2024-05-20T17:57:32.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:32.856+0000 7f4aaa7e1700 1 -- 172.21.15.129:0/1030586650 wait complete. 2024-05-20T17:57:33.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:32 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1030586650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-20T17:57:33.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:32 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1030586650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-20T17:57:33.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:32 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1030586650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-20T17:57:33.453 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920822 got 141733920826 for osd.4 2024-05-20T17:57:33.453 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:33.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.742+0000 7f2452bd5700 1 -- 172.21.15.129:0/1797355609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f244c103450 msgr2=0x7f244c103860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:33.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.742+0000 7f2452bd5700 1 --2- 172.21.15.129:0/1797355609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f244c103450 0x7f244c103860 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f24400099c0 tx=0x7f2440009cd0 comp rx=0 tx=0).stop 2024-05-20T17:57:33.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.742+0000 7f2452bd5700 1 -- 172.21.15.129:0/1797355609 shutdown_connections 2024-05-20T17:57:33.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.742+0000 7f2452bd5700 1 --2- 172.21.15.129:0/1797355609 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f244c1086d0 0x7f244c110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:33.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.742+0000 7f2452bd5700 1 --2- 172.21.15.129:0/1797355609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f244c103450 0x7f244c103860 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:33.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.742+0000 7f2452bd5700 1 --2- 172.21.15.129:0/1797355609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f244c107d80 0x7f244c108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:33.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.742+0000 7f2452bd5700 1 -- 172.21.15.129:0/1797355609 >> 172.21.15.129:0/1797355609 conn(0x7f244c0fed20 msgr2=0x7f244c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:33.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.742+0000 7f2452bd5700 1 -- 172.21.15.129:0/1797355609 shutdown_connections 2024-05-20T17:57:33.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.742+0000 7f2452bd5700 1 -- 172.21.15.129:0/1797355609 wait complete. 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.743+0000 7f2452bd5700 1 Processor -- start 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.743+0000 7f2452bd5700 1 -- start start 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.743+0000 7f2452bd5700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f244c103450 0x7f244c199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.743+0000 7f2452bd5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f244c107d80 0x7f244c19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.743+0000 7f2452bd5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f244c1086d0 0x7f244c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.743+0000 7f2452bd5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f244c113450 con 0x7f244c107d80 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.743+0000 7f2452bd5700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1132d0 con 0x7f244c1086d0 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.743+0000 7f2452bd5700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1135d0 con 0x7f244c103450 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f244bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f244c107d80 0x7f244c19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f244bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f244c107d80 0x7f244c19a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60784/0 (socket says 172.21.15.129:60784) 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f244bfff700 1 -- 172.21.15.129:0/764316982 learned_addr learned my addr 172.21.15.129:0/764316982 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f2450971700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f244c103450 0x7f244c199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f2451172700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f244c1086d0 0x7f244c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:33.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f244bfff700 1 -- 172.21.15.129:0/764316982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f244c103450 msgr2=0x7f244c199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:33.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f244bfff700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f244c103450 0x7f244c199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:33.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f244bfff700 1 -- 172.21.15.129:0/764316982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f244c1086d0 msgr2=0x7f244c19e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:33.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f244bfff700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f244c1086d0 0x7f244c19e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:33.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f244bfff700 1 -- 172.21.15.129:0/764316982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244c19ecb0 con 0x7f244c107d80 2024-05-20T17:57:33.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f244bfff700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f244c107d80 0x7f244c19a380 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f243800e970 tx=0x7f243800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:33.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f2449ffb700 1 -- 172.21.15.129:0/764316982 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438004a30 con 0x7f244c107d80 2024-05-20T17:57:33.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f2449ffb700 1 -- 172.21.15.129:0/764316982 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f243800ccc0 con 0x7f244c107d80 2024-05-20T17:57:33.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f2452bd5700 1 -- 172.21.15.129:0/764316982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244c19efa0 con 0x7f244c107d80 2024-05-20T17:57:33.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.744+0000 7f2449ffb700 1 -- 172.21.15.129:0/764316982 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438005650 con 0x7f244c107d80 2024-05-20T17:57:33.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.745+0000 7f2452bd5700 1 -- 172.21.15.129:0/764316982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244c1a68c0 con 0x7f244c107d80 2024-05-20T17:57:33.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.746+0000 7f2452bd5700 1 -- 172.21.15.129:0/764316982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f244c062d50 con 0x7f244c107d80 2024-05-20T17:57:33.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.747+0000 7f2449ffb700 1 -- 172.21.15.129:0/764316982 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2438004bd0 con 0x7f244c107d80 2024-05-20T17:57:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.748+0000 7f2449ffb700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f243c0a2d30 0x7f243c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.748+0000 7f2450971700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f243c0a2d30 0x7f243c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.748+0000 7f2449ffb700 1 -- 172.21.15.129:0/764316982 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f24380cfc00 con 0x7f244c107d80 2024-05-20T17:57:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.749+0000 7f2449ffb700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415] conn(0x7f243c0ac560 0x7f243c0ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.749+0000 7f2449ffb700 1 -- 172.21.15.129:0/764316982 --> [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f243c0af080 con 0x7f243c0ac560 2024-05-20T17:57:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.749+0000 7f2449ffb700 1 -- 172.21.15.129:0/764316982 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f24380d00e0 con 0x7f244c107d80 2024-05-20T17:57:33.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.749+0000 7f2451172700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415] conn(0x7f243c0ac560 0x7f243c0ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:33.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.749+0000 7f2450971700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f243c0a2d30 0x7f243c0a5240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2440000c00 tx=0x7f244001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:33.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.749+0000 7f2451172700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415] conn(0x7f243c0ac560 0x7f243c0ae9b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:33.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.750+0000 7f2449ffb700 1 -- 172.21.15.129:0/764316982 <== osd.3 v2:172.21.15.129:6810/310906415 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f243c0af080 con 0x7f243c0ac560 2024-05-20T17:57:33.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.768+0000 7f2452bd5700 1 -- 172.21.15.129:0/764316982 --> [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f244c063150 con 0x7f243c0ac560 2024-05-20T17:57:33.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.768+0000 7f2449ffb700 1 -- 172.21.15.129:0/764316982 <== osd.3 v2:172.21.15.129:6810/310906415 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f244c063150 con 0x7f243c0ac560 2024-05-20T17:57:33.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.769+0000 7f2452bd5700 1 -- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415] conn(0x7f243c0ac560 msgr2=0x7f243c0ae9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:33.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.769+0000 7f2452bd5700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415] conn(0x7f243c0ac560 0x7f243c0ae9b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:33.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.769+0000 7f2452bd5700 1 -- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f243c0a2d30 msgr2=0x7f243c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:33.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.769+0000 7f2452bd5700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f243c0a2d30 0x7f243c0a5240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2440000c00 tx=0x7f244001a040 comp rx=0 tx=0).stop 2024-05-20T17:57:33.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.769+0000 7f2452bd5700 1 -- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f244c107d80 msgr2=0x7f244c19a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:33.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.769+0000 7f2452bd5700 1 --2- 172.21.15.129:0/764316982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f244c107d80 0x7f244c19a380 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f243800e970 tx=0x7f243800ec80 comp rx=0 tx=0).stop 2024-05-20T17:57:33.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.769+0000 7f2451172700 1 -- 172.21.15.129:0/764316982 reap_dead start 2024-05-20T17:57:33.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.769+0000 7f2452bd5700 1 -- 172.21.15.129:0/764316982 shutdown_connections 2024-05-20T17:57:33.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.769+0000 7f2452bd5700 1 -- 172.21.15.129:0/764316982 >> 172.21.15.129:0/764316982 conn(0x7f244c0fed20 msgr2=0x7f244c107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:33.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.769+0000 7f2452bd5700 1 -- 172.21.15.129:0/764316982 shutdown_connections 2024-05-20T17:57:33.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:33.769+0000 7f2452bd5700 1 -- 172.21.15.129:0/764316982 wait complete. 2024-05-20T17:57:34.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:33 smithi129 ceph-mon[32029]: pgmap v309: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:34.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:33 smithi204 ceph-mon[37076]: pgmap v309: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:34.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:33 smithi154 ceph-mon[36482]: pgmap v309: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:34.756 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:34.756 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:34.757 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:34.772 INFO:teuthology.orchestra.run.smithi129.stdout:107374182464 2024-05-20T17:57:34.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-20T17:57:34.774 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:36.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:35 smithi154 ceph-mon[36482]: pgmap v310: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:36.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:35 smithi129 ceph-mon[32029]: pgmap v310: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:36.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:35 smithi204 ceph-mon[37076]: pgmap v310: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:38.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:37 smithi154 ceph-mon[36482]: pgmap v311: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:38.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:37 smithi129 ceph-mon[32029]: pgmap v311: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:38.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:37 smithi204 ceph-mon[37076]: pgmap v311: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:38.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.797+0000 7faedf774700 1 -- 172.21.15.129:0/3484303266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faed81075c0 msgr2=0x7faed810ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:38.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.797+0000 7faedf774700 1 --2- 172.21.15.129:0/3484303266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faed81075c0 0x7faed810ef20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faed000b0d0 tx=0x7faed000b3e0 comp rx=0 tx=0).stop 2024-05-20T17:57:38.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.797+0000 7faedf774700 1 -- 172.21.15.129:0/3484303266 shutdown_connections 2024-05-20T17:57:38.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.797+0000 7faedf774700 1 --2- 172.21.15.129:0/3484303266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faed81075c0 0x7faed810ef20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:38.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.797+0000 7faedf774700 1 --2- 172.21.15.129:0/3484303266 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faed8102340 0x7faed8102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:38.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.797+0000 7faedf774700 1 --2- 172.21.15.129:0/3484303266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faed8106c70 0x7faed8107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:38.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.797+0000 7faedf774700 1 -- 172.21.15.129:0/3484303266 >> 172.21.15.129:0/3484303266 conn(0x7faed80fdc40 msgr2=0x7faed8100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:38.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.798+0000 7faedf774700 1 -- 172.21.15.129:0/3484303266 shutdown_connections 2024-05-20T17:57:38.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.798+0000 7faedf774700 1 -- 172.21.15.129:0/3484303266 wait complete. 2024-05-20T17:57:38.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.798+0000 7faedf774700 1 Processor -- start 2024-05-20T17:57:38.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.798+0000 7faedf774700 1 -- start start 2024-05-20T17:57:38.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.798+0000 7faedf774700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faed8102340 0x7faed8199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:38.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedf774700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faed8106c70 0x7faed819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:38.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedf774700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faed81075c0 0x7faed819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:38.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedf774700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faed8113490 con 0x7faed8106c70 2024-05-20T17:57:38.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedf774700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faed8113310 con 0x7faed81075c0 2024-05-20T17:57:38.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedf774700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faed8113610 con 0x7faed8102340 2024-05-20T17:57:38.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedcd0f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faed8106c70 0x7faed819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedd510700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faed8102340 0x7faed8199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedcd0f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faed8106c70 0x7faed819a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60796/0 (socket says 172.21.15.129:60796) 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedcd0f700 1 -- 172.21.15.129:0/1781898903 learned_addr learned my addr 172.21.15.129:0/1781898903 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faeddd11700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faed81075c0 0x7faed819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedcd0f700 1 -- 172.21.15.129:0/1781898903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faed8102340 msgr2=0x7faed8199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedcd0f700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faed8102340 0x7faed8199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedcd0f700 1 -- 172.21.15.129:0/1781898903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faed81075c0 msgr2=0x7faed819e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedcd0f700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faed81075c0 0x7faed819e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedcd0f700 1 -- 172.21.15.129:0/1781898903 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed819eb60 con 0x7faed8106c70 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedcd0f700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faed8106c70 0x7faed819a2e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faed400c8f0 tx=0x7faed400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faece7fc700 1 -- 172.21.15.129:0/1781898903 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed40078d0 con 0x7faed8106c70 2024-05-20T17:57:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faece7fc700 1 -- 172.21.15.129:0/1781898903 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faed400f450 con 0x7faed8106c70 2024-05-20T17:57:38.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faedf774700 1 -- 172.21.15.129:0/1781898903 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed819ee50 con 0x7faed8106c70 2024-05-20T17:57:38.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.799+0000 7faece7fc700 1 -- 172.21.15.129:0/1781898903 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed4018640 con 0x7faed8106c70 2024-05-20T17:57:38.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.800+0000 7faedf774700 1 -- 172.21.15.129:0/1781898903 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed81a6820 con 0x7faed8106c70 2024-05-20T17:57:38.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.801+0000 7faedf774700 1 -- 172.21.15.129:0/1781898903 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed8050170 con 0x7faed8106c70 2024-05-20T17:57:38.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.802+0000 7faece7fc700 1 -- 172.21.15.129:0/1781898903 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7faed400f600 con 0x7faed8106c70 2024-05-20T17:57:38.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.802+0000 7faece7fc700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faec40a2d30 0x7faec40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:38.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.802+0000 7faedd510700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faec40a2d30 0x7faec40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:38.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.802+0000 7faece7fc700 1 -- 172.21.15.129:0/1781898903 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7faed40ced60 con 0x7faed8106c70 2024-05-20T17:57:38.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.803+0000 7faedd510700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faec40a2d30 0x7faec40a5240 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faec800ac50 tx=0x7faec800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:38.808 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:38.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:38.812+0000 7faece7fc700 1 -- 172.21.15.129:0/1781898903 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7faed4014030 con 0x7faed8106c70 2024-05-20T17:57:38.812 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:38.814 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:38.822 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:38.827 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:39.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.041+0000 7faedf774700 1 -- 172.21.15.129:0/1781898903 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7faed802dc60 con 0x7faed8106c70 2024-05-20T17:57:39.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.041+0000 7faece7fc700 1 -- 172.21.15.129:0/1781898903 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7faed409bd10 con 0x7faed8106c70 2024-05-20T17:57:39.040 INFO:teuthology.orchestra.run.smithi129.stdout:163208757302 2024-05-20T17:57:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 -- 172.21.15.129:0/1781898903 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faec40a2d30 msgr2=0x7faec40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faec40a2d30 0x7faec40a5240 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faec800ac50 tx=0x7faec800a380 comp rx=0 tx=0).stop 2024-05-20T17:57:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 -- 172.21.15.129:0/1781898903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faed8106c70 msgr2=0x7faed819a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faed8106c70 0x7faed819a2e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faed400c8f0 tx=0x7faed400cc00 comp rx=0 tx=0).stop 2024-05-20T17:57:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 -- 172.21.15.129:0/1781898903 shutdown_connections 2024-05-20T17:57:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7faec40a2d30 0x7faec40a5240 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faed81075c0 0x7faed819e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faed8102340 0x7faed8199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:39.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 --2- 172.21.15.129:0/1781898903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faed8106c70 0x7faed819a2e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:39.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 -- 172.21.15.129:0/1781898903 >> 172.21.15.129:0/1781898903 conn(0x7faed80fdc40 msgr2=0x7faed81063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:39.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 -- 172.21.15.129:0/1781898903 shutdown_connections 2024-05-20T17:57:39.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:39.043+0000 7faedf774700 1 -- 172.21.15.129:0/1781898903 wait complete. 2024-05-20T17:57:39.282 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:39 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1781898903' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-20T17:57:39.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:39 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1781898903' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-20T17:57:39.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:39 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1781898903' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-20T17:57:40.312 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757296 got 163208757302 for osd.5 2024-05-20T17:57:40.313 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:40.385 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:40 smithi129 ceph-mon[32029]: pgmap v312: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:40.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:40 smithi154 ceph-mon[36482]: pgmap v312: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:40.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:40 smithi204 ceph-mon[37076]: pgmap v312: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:41.159 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:57:41.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:41 smithi129 ceph-mon[32029]: pgmap v313: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:42.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:41 smithi154 ceph-mon[36482]: pgmap v313: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:42.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:41 smithi204 ceph-mon[37076]: pgmap v313: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:44.044 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:43 smithi129 ceph-mon[32029]: pgmap v314: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:44.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:43 smithi154 ceph-mon[36482]: pgmap v314: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:44.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:43 smithi204 ceph-mon[37076]: pgmap v314: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.746+0000 7f9b29df3700 1 -- 172.21.15.129:0/2438769954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b24102340 msgr2=0x7f9b24102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.746+0000 7f9b29df3700 1 --2- 172.21.15.129:0/2438769954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b24102340 0x7f9b24102750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9b140099c0 tx=0x7f9b14009cd0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.746+0000 7f9b29df3700 1 -- 172.21.15.129:0/2438769954 shutdown_connections 2024-05-20T17:57:44.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.746+0000 7f9b29df3700 1 --2- 172.21.15.129:0/2438769954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b241075c0 0x7f9b2410ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.746+0000 7f9b29df3700 1 --2- 172.21.15.129:0/2438769954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b24102340 0x7f9b24102750 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.746+0000 7f9b29df3700 1 --2- 172.21.15.129:0/2438769954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b24106c70 0x7f9b24107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.746+0000 7f9b29df3700 1 -- 172.21.15.129:0/2438769954 >> 172.21.15.129:0/2438769954 conn(0x7f9b240fdc40 msgr2=0x7f9b24100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:44.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.746+0000 7f9b29df3700 1 -- 172.21.15.129:0/2438769954 shutdown_connections 2024-05-20T17:57:44.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.746+0000 7f9b29df3700 1 -- 172.21.15.129:0/2438769954 wait complete. 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.747+0000 7f9b29df3700 1 Processor -- start 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.747+0000 7f9b29df3700 1 -- start start 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b29df3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b24102340 0x7f9b2419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b29df3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b24106c70 0x7f9b2419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b29df3700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b241075c0 0x7f9b24198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b29df3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9b24112340 con 0x7f9b24102340 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b29df3700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9b241121c0 con 0x7f9b24106c70 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b29df3700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9b241124c0 con 0x7f9b241075c0 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b237fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b24102340 0x7f9b2419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b237fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b24102340 0x7f9b2419e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48942/0 (socket says 172.21.15.129:48942) 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b237fe700 1 -- 172.21.15.129:0/3403590643 learned_addr learned my addr 172.21.15.129:0/3403590643 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b23fff700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b241075c0 0x7f9b24198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b22ffd700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b24106c70 0x7f9b2419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b237fe700 1 -- 172.21.15.129:0/3403590643 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b241075c0 msgr2=0x7f9b24198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b237fe700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b241075c0 0x7f9b24198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b237fe700 1 -- 172.21.15.129:0/3403590643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b24106c70 msgr2=0x7f9b2419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b237fe700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b24106c70 0x7f9b2419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b237fe700 1 -- 172.21.15.129:0/3403590643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b241990a0 con 0x7f9b24102340 2024-05-20T17:57:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.748+0000 7f9b237fe700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b24102340 0x7f9b2419e840 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9b140059d0 tx=0x7f9b1400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.749+0000 7f9b20ff9700 1 -- 172.21.15.129:0/3403590643 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14027070 con 0x7f9b24102340 2024-05-20T17:57:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.749+0000 7f9b20ff9700 1 -- 172.21.15.129:0/3403590643 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b14003680 con 0x7f9b24102340 2024-05-20T17:57:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.749+0000 7f9b20ff9700 1 -- 172.21.15.129:0/3403590643 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b14017650 con 0x7f9b24102340 2024-05-20T17:57:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.749+0000 7f9b29df3700 1 -- 172.21.15.129:0/3403590643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b24199330 con 0x7f9b24102340 2024-05-20T17:57:44.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.749+0000 7f9b29df3700 1 -- 172.21.15.129:0/3403590643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b241997c0 con 0x7f9b24102340 2024-05-20T17:57:44.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.750+0000 7f9b29df3700 1 -- 172.21.15.129:0/3403590643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b24050170 con 0x7f9b24102340 2024-05-20T17:57:44.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.752+0000 7f9b20ff9700 1 -- 172.21.15.129:0/3403590643 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f9b1402a070 con 0x7f9b24102340 2024-05-20T17:57:44.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.753+0000 7f9b20ff9700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9b0c0a2d30 0x7f9b0c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:44.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.753+0000 7f9b22ffd700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9b0c0a2d30 0x7f9b0c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:44.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.753+0000 7f9b20ff9700 1 -- 172.21.15.129:0/3403590643 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f9b140cff70 con 0x7f9b24102340 2024-05-20T17:57:44.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.753+0000 7f9b22ffd700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9b0c0a2d30 0x7f9b0c0a5240 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9b24105260 tx=0x7f9b18009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:44.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.757+0000 7f9b20ff9700 1 -- 172.21.15.129:0/3403590643 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9b1405e850 con 0x7f9b24102340 2024-05-20T17:57:44.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.938+0000 7fde71aa7700 1 -- 172.21.15.129:0/1424944562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde6c10d790 msgr2=0x7fde6c10fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:44.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.938+0000 7fde71aa7700 1 --2- 172.21.15.129:0/1424944562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde6c10d790 0x7fde6c10fc70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fde6400b0a0 tx=0x7fde6400b3b0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.938+0000 7fde71aa7700 1 -- 172.21.15.129:0/1424944562 shutdown_connections 2024-05-20T17:57:44.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.938+0000 7fde71aa7700 1 --2- 172.21.15.129:0/1424944562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde6c104360 0x7fde6c104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.938+0000 7fde71aa7700 1 --2- 172.21.15.129:0/1424944562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde6c10d790 0x7fde6c10fc70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.938+0000 7fde71aa7700 1 --2- 172.21.15.129:0/1424944562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde6c104cb0 0x7fde6c10d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.938+0000 7fde71aa7700 1 -- 172.21.15.129:0/1424944562 >> 172.21.15.129:0/1424944562 conn(0x7fde6c0fddd0 msgr2=0x7fde6c1001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:44.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.938+0000 7fde71aa7700 1 -- 172.21.15.129:0/1424944562 shutdown_connections 2024-05-20T17:57:44.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.938+0000 7fde71aa7700 1 -- 172.21.15.129:0/1424944562 wait complete. 2024-05-20T17:57:44.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.939+0000 7fde71aa7700 1 Processor -- start 2024-05-20T17:57:44.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.939+0000 7fde71aa7700 1 -- start start 2024-05-20T17:57:44.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.939+0000 7fde71aa7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde6c104360 0x7fde6c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:44.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.939+0000 7fde71aa7700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde6c104cb0 0x7fde6c19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:44.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.939+0000 7fde71aa7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde6c10d790 0x7fde6c19e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:44.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.939+0000 7fde71aa7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c113460 con 0x7fde6c104360 2024-05-20T17:57:44.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.939+0000 7fde71aa7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c1132e0 con 0x7fde6c10d790 2024-05-20T17:57:44.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.939+0000 7fde71aa7700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c1135e0 con 0x7fde6c104cb0 2024-05-20T17:57:44.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.939+0000 7fde6b7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde6c104360 0x7fde6c199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:44.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde6b7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde6c104360 0x7fde6c199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48962/0 (socket says 172.21.15.129:48962) 2024-05-20T17:57:44.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde6b7fe700 1 -- 172.21.15.129:0/2891454812 learned_addr learned my addr 172.21.15.129:0/2891454812 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde6bfff700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde6c10d790 0x7fde6c19e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde6affd700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde6c104cb0 0x7fde6c19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde6b7fe700 1 -- 172.21.15.129:0/2891454812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde6c104cb0 msgr2=0x7fde6c19a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde6b7fe700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde6c104cb0 0x7fde6c19a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde6b7fe700 1 -- 172.21.15.129:0/2891454812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde6c10d790 msgr2=0x7fde6c19e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde6b7fe700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde6c10d790 0x7fde6c19e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde6b7fe700 1 -- 172.21.15.129:0/2891454812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6c19ebe0 con 0x7fde6c104360 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde6b7fe700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde6c104360 0x7fde6c199d10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fde5c00eb30 tx=0x7fde5c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde68ff9700 1 -- 172.21.15.129:0/2891454812 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c00cbd0 con 0x7fde6c104360 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.940+0000 7fde71aa7700 1 -- 172.21.15.129:0/2891454812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde6c19eed0 con 0x7fde6c104360 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.941+0000 7fde68ff9700 1 -- 172.21.15.129:0/2891454812 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde5c004500 con 0x7fde6c104360 2024-05-20T17:57:44.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.941+0000 7fde71aa7700 1 -- 172.21.15.129:0/2891454812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde6c101f80 con 0x7fde6c104360 2024-05-20T17:57:44.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.941+0000 7fde68ff9700 1 -- 172.21.15.129:0/2891454812 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c010430 con 0x7fde6c104360 2024-05-20T17:57:44.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.942+0000 7fde71aa7700 1 -- 172.21.15.129:0/2891454812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde58005360 con 0x7fde6c104360 2024-05-20T17:57:44.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.944+0000 7fde68ff9700 1 -- 172.21.15.129:0/2891454812 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fde5c003680 con 0x7fde6c104360 2024-05-20T17:57:44.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.945+0000 7fde68ff9700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fde540a2cb0 0x7fde540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:44.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.945+0000 7fde68ff9700 1 -- 172.21.15.129:0/2891454812 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fde5c0cf380 con 0x7fde6c104360 2024-05-20T17:57:44.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.945+0000 7fde6affd700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fde540a2cb0 0x7fde540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:44.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.946+0000 7fde6affd700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fde540a2cb0 0x7fde540a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fde6c101180 tx=0x7fde60012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:44.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.950+0000 7fde68ff9700 1 -- 172.21.15.129:0/2891454812 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fde5c09c230 con 0x7fde6c104360 2024-05-20T17:57:44.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.967+0000 7f9b29df3700 1 -- 172.21.15.129:0/3403590643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f9b24109f60 con 0x7f9b24102340 2024-05-20T17:57:44.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.967+0000 7f9b20ff9700 1 -- 172.21.15.129:0/3403590643 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9b14013070 con 0x7f9b24102340 2024-05-20T17:57:44.966 INFO:teuthology.orchestra.run.smithi129.stdout:34359738448 2024-05-20T17:57:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.969+0000 7f9b29df3700 1 -- 172.21.15.129:0/3403590643 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9b0c0a2d30 msgr2=0x7f9b0c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.969+0000 7f9b29df3700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9b0c0a2d30 0x7f9b0c0a5240 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9b24105260 tx=0x7f9b18009260 comp rx=0 tx=0).stop 2024-05-20T17:57:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.969+0000 7f9b29df3700 1 -- 172.21.15.129:0/3403590643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b24102340 msgr2=0x7f9b2419e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.969+0000 7f9b29df3700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b24102340 0x7f9b2419e840 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9b140059d0 tx=0x7f9b1400fa00 comp rx=0 tx=0).stop 2024-05-20T17:57:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.970+0000 7f9b29df3700 1 -- 172.21.15.129:0/3403590643 shutdown_connections 2024-05-20T17:57:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.970+0000 7f9b29df3700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9b0c0a2d30 0x7f9b0c0a5240 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.970+0000 7f9b29df3700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b241075c0 0x7f9b24198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.970+0000 7f9b29df3700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b24102340 0x7f9b2419e840 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.970+0000 7f9b29df3700 1 --2- 172.21.15.129:0/3403590643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b24106c70 0x7f9b2419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.970+0000 7f9b29df3700 1 -- 172.21.15.129:0/3403590643 >> 172.21.15.129:0/3403590643 conn(0x7f9b240fdc40 msgr2=0x7f9b24100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:44.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.970+0000 7f9b29df3700 1 -- 172.21.15.129:0/3403590643 shutdown_connections 2024-05-20T17:57:44.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:44.970+0000 7f9b29df3700 1 -- 172.21.15.129:0/3403590643 wait complete. 2024-05-20T17:57:45.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:45 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3403590643' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-20T17:57:45.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.167+0000 7fde71aa7700 1 -- 172.21.15.129:0/2891454812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fde58005920 con 0x7fde6c104360 2024-05-20T17:57:45.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.167+0000 7fde68ff9700 1 -- 172.21.15.129:0/2891454812 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fde5c09bb30 con 0x7fde6c104360 2024-05-20T17:57:45.166 INFO:teuthology.orchestra.run.smithi129.stdout:219043332141 2024-05-20T17:57:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.169+0000 7fde71aa7700 1 -- 172.21.15.129:0/2891454812 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fde540a2cb0 msgr2=0x7fde540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.169+0000 7fde71aa7700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fde540a2cb0 0x7fde540a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fde6c101180 tx=0x7fde60012040 comp rx=0 tx=0).stop 2024-05-20T17:57:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.169+0000 7fde71aa7700 1 -- 172.21.15.129:0/2891454812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde6c104360 msgr2=0x7fde6c199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.169+0000 7fde71aa7700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde6c104360 0x7fde6c199d10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fde5c00eb30 tx=0x7fde5c00ee40 comp rx=0 tx=0).stop 2024-05-20T17:57:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.169+0000 7fde71aa7700 1 -- 172.21.15.129:0/2891454812 shutdown_connections 2024-05-20T17:57:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.169+0000 7fde71aa7700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fde540a2cb0 0x7fde540a51c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:45.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.169+0000 7fde71aa7700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde6c10d790 0x7fde6c19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:45.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.169+0000 7fde71aa7700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde6c104360 0x7fde6c199d10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:45.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.169+0000 7fde71aa7700 1 --2- 172.21.15.129:0/2891454812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fde6c104cb0 0x7fde6c19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:45.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.169+0000 7fde71aa7700 1 -- 172.21.15.129:0/2891454812 >> 172.21.15.129:0/2891454812 conn(0x7fde6c0fddd0 msgr2=0x7fde6c0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:45.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.170+0000 7fde71aa7700 1 -- 172.21.15.129:0/2891454812 shutdown_connections 2024-05-20T17:57:45.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.170+0000 7fde71aa7700 1 -- 172.21.15.129:0/2891454812 wait complete. 2024-05-20T17:57:45.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:45 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3403590643' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-20T17:57:45.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:45 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3403590643' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-20T17:57:45.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.984+0000 7f39cfa5b700 1 -- 172.21.15.129:0/737708051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39c81086d0 msgr2=0x7f39c8110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:45.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.984+0000 7f39cfa5b700 1 --2- 172.21.15.129:0/737708051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39c81086d0 0x7f39c8110030 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f39c000b0d0 tx=0x7f39c000b3e0 comp rx=0 tx=0).stop 2024-05-20T17:57:45.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.984+0000 7f39cfa5b700 1 -- 172.21.15.129:0/737708051 shutdown_connections 2024-05-20T17:57:45.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.984+0000 7f39cfa5b700 1 --2- 172.21.15.129:0/737708051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39c81086d0 0x7f39c8110030 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:45.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.984+0000 7f39cfa5b700 1 --2- 172.21.15.129:0/737708051 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39c8103450 0x7f39c8103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:45.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.984+0000 7f39cfa5b700 1 --2- 172.21.15.129:0/737708051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c8107d80 0x7f39c8108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:45.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.984+0000 7f39cfa5b700 1 -- 172.21.15.129:0/737708051 >> 172.21.15.129:0/737708051 conn(0x7f39c80fed20 msgr2=0x7f39c8101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:45.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.984+0000 7f39cfa5b700 1 -- 172.21.15.129:0/737708051 shutdown_connections 2024-05-20T17:57:45.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.984+0000 7f39cfa5b700 1 -- 172.21.15.129:0/737708051 wait complete. 2024-05-20T17:57:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.985+0000 7f39cfa5b700 1 Processor -- start 2024-05-20T17:57:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.985+0000 7f39cfa5b700 1 -- start start 2024-05-20T17:57:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.985+0000 7f39cfa5b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39c8103450 0x7f39c819f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.985+0000 7f39cfa5b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c8107d80 0x7f39c819fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.985+0000 7f39cfa5b700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39c81086d0 0x7f39c8199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.985+0000 7f39cfa5b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f39c8113470 con 0x7f39c8103450 2024-05-20T17:57:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.985+0000 7f39cfa5b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81132f0 con 0x7f39c8107d80 2024-05-20T17:57:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.985+0000 7f39cfa5b700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f39c81135f0 con 0x7f39c81086d0 2024-05-20T17:57:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.985+0000 7f39cd7f7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39c8103450 0x7f39c819f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.985+0000 7f39cd7f7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39c8103450 0x7f39c819f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48982/0 (socket says 172.21.15.129:48982) 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.985+0000 7f39cd7f7700 1 -- 172.21.15.129:0/488864292 learned_addr learned my addr 172.21.15.129:0/488864292 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39ccff6700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c8107d80 0x7f39c819fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39cdff8700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39c81086d0 0x7f39c8199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39cd7f7700 1 -- 172.21.15.129:0/488864292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39c81086d0 msgr2=0x7f39c8199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39cd7f7700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39c81086d0 0x7f39c8199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39cd7f7700 1 -- 172.21.15.129:0/488864292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c8107d80 msgr2=0x7f39c819fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39cd7f7700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c8107d80 0x7f39c819fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39cd7f7700 1 -- 172.21.15.129:0/488864292 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c819a260 con 0x7f39c8103450 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39cd7f7700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39c8103450 0x7f39c819f9a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f39c400d980 tx=0x7f39c400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39be7fc700 1 -- 172.21.15.129:0/488864292 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c4017070 con 0x7f39c8103450 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39be7fc700 1 -- 172.21.15.129:0/488864292 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39c4005020 con 0x7f39c8103450 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39cfa5b700 1 -- 172.21.15.129:0/488864292 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c819a550 con 0x7f39c8103450 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39be7fc700 1 -- 172.21.15.129:0/488864292 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c400fbf0 con 0x7f39c8103450 2024-05-20T17:57:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.986+0000 7f39cfa5b700 1 -- 172.21.15.129:0/488864292 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c81a6750 con 0x7f39c8103450 2024-05-20T17:57:45.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.987+0000 7f39cfa5b700 1 -- 172.21.15.129:0/488864292 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c8104900 con 0x7f39c8103450 2024-05-20T17:57:45.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.989+0000 7f39be7fc700 1 -- 172.21.15.129:0/488864292 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f39c40051d0 con 0x7f39c8103450 2024-05-20T17:57:45.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.990+0000 7f39be7fc700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39b40a2d30 0x7f39b40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:45.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.990+0000 7f39ccff6700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39b40a2d30 0x7f39b40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:45.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.990+0000 7f39be7fc700 1 -- 172.21.15.129:0/488864292 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f39c40cf1a0 con 0x7f39c8103450 2024-05-20T17:57:45.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.990+0000 7f39ccff6700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39b40a2d30 0x7f39b40a5240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f39c819b260 tx=0x7f39b800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:45.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:45.994+0000 7f39be7fc700 1 -- 172.21.15.129:0/488864292 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f39c4020020 con 0x7f39c8103450 2024-05-20T17:57:46.132 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:46 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:57:46.132 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:46 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2891454812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-20T17:57:46.132 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:46 smithi154 ceph-mon[36482]: pgmap v315: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:46.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:46 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:57:46.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:46 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2891454812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-20T17:57:46.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:46 smithi129 ceph-mon[32029]: pgmap v315: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:46.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:46 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:57:46.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:46 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2891454812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-20T17:57:46.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:46 smithi204 ceph-mon[37076]: pgmap v315: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:57:46.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.207+0000 7f39cfa5b700 1 -- 172.21.15.129:0/488864292 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f39c8050170 con 0x7f39c8103450 2024-05-20T17:57:46.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.208+0000 7f39be7fc700 1 -- 172.21.15.129:0/488864292 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f39c409c1d0 con 0x7f39c8103450 2024-05-20T17:57:46.207 INFO:teuthology.orchestra.run.smithi129.stdout:313532612638 2024-05-20T17:57:46.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.210+0000 7f39cfa5b700 1 -- 172.21.15.129:0/488864292 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39b40a2d30 msgr2=0x7f39b40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:46.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.210+0000 7f39cfa5b700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39b40a2d30 0x7f39b40a5240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f39c819b260 tx=0x7f39b800b410 comp rx=0 tx=0).stop 2024-05-20T17:57:46.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.210+0000 7f39cfa5b700 1 -- 172.21.15.129:0/488864292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39c8103450 msgr2=0x7f39c819f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:46.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.210+0000 7f39cfa5b700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39c8103450 0x7f39c819f9a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f39c400d980 tx=0x7f39c400dc90 comp rx=0 tx=0).stop 2024-05-20T17:57:46.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.210+0000 7f39cfa5b700 1 -- 172.21.15.129:0/488864292 shutdown_connections 2024-05-20T17:57:46.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.210+0000 7f39cfa5b700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f39b40a2d30 0x7f39b40a5240 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.210+0000 7f39cfa5b700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f39c81086d0 0x7f39c8199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.210+0000 7f39cfa5b700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39c8103450 0x7f39c819f9a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.210+0000 7f39cfa5b700 1 --2- 172.21.15.129:0/488864292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c8107d80 0x7f39c819fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.210+0000 7f39cfa5b700 1 -- 172.21.15.129:0/488864292 >> 172.21.15.129:0/488864292 conn(0x7f39c80fed20 msgr2=0x7f39c8107490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:46.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.211+0000 7f39cfa5b700 1 -- 172.21.15.129:0/488864292 shutdown_connections 2024-05-20T17:57:46.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.211+0000 7f39cfa5b700 1 -- 172.21.15.129:0/488864292 wait complete. 2024-05-20T17:57:46.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.658+0000 7f09d2ded700 1 -- 172.21.15.129:0/1684750137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09cc102870 msgr2=0x7f09cc10f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:46.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.658+0000 7f09d2ded700 1 --2- 172.21.15.129:0/1684750137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09cc102870 0x7f09cc10f590 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f09b8009990 tx=0x7f09b8009ca0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.658+0000 7f09d2ded700 1 -- 172.21.15.129:0/1684750137 shutdown_connections 2024-05-20T17:57:46.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.658+0000 7f09d2ded700 1 --2- 172.21.15.129:0/1684750137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09cc10fad0 0x7f09cc111ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.658+0000 7f09d2ded700 1 --2- 172.21.15.129:0/1684750137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09cc101f20 0x7f09cc102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.658+0000 7f09d2ded700 1 --2- 172.21.15.129:0/1684750137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09cc102870 0x7f09cc10f590 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.658+0000 7f09d2ded700 1 -- 172.21.15.129:0/1684750137 >> 172.21.15.129:0/1684750137 conn(0x7f09cc0fddd0 msgr2=0x7f09cc1001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:46.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.658+0000 7f09d2ded700 1 -- 172.21.15.129:0/1684750137 shutdown_connections 2024-05-20T17:57:46.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.658+0000 7f09d2ded700 1 -- 172.21.15.129:0/1684750137 wait complete. 2024-05-20T17:57:46.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d2ded700 1 Processor -- start 2024-05-20T17:57:46.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d2ded700 1 -- start start 2024-05-20T17:57:46.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d2ded700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09cc101f20 0x7f09cc19c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:46.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d2ded700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09cc102870 0x7f09cc19c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:46.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d2ded700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09cc10fad0 0x7f09cc1a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:46.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d2ded700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc115690 con 0x7f09cc101f20 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d2ded700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc115510 con 0x7f09cc10fad0 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d2ded700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc115810 con 0x7f09cc102870 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d0b89700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09cc101f20 0x7f09cc19c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d0b89700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09cc101f20 0x7f09cc19c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49010/0 (socket says 172.21.15.129:49010) 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d0b89700 1 -- 172.21.15.129:0/661392240 learned_addr learned my addr 172.21.15.129:0/661392240 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09cbfff700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09cc102870 0x7f09cc19c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.659+0000 7f09d138a700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09cc10fad0 0x7f09cc1a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.660+0000 7f09d0b89700 1 -- 172.21.15.129:0/661392240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09cc102870 msgr2=0x7f09cc19c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.660+0000 7f09d0b89700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09cc102870 0x7f09cc19c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.660+0000 7f09d0b89700 1 -- 172.21.15.129:0/661392240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09cc10fad0 msgr2=0x7f09cc1a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.660+0000 7f09d0b89700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09cc10fad0 0x7f09cc1a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.660+0000 7f09d0b89700 1 -- 172.21.15.129:0/661392240 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc1a0f70 con 0x7f09cc101f20 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.660+0000 7f09d0b89700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09cc101f20 0x7f09cc19c0a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f09c000e9a0 tx=0x7f09c000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.660+0000 7f09c9ffb700 1 -- 172.21.15.129:0/661392240 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c000ca10 con 0x7f09cc101f20 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.660+0000 7f09c9ffb700 1 -- 172.21.15.129:0/661392240 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c000cbb0 con 0x7f09cc101f20 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.660+0000 7f09d2ded700 1 -- 172.21.15.129:0/661392240 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09cc1a1260 con 0x7f09cc101f20 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.660+0000 7f09c9ffb700 1 -- 172.21.15.129:0/661392240 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0018680 con 0x7f09cc101f20 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.660+0000 7f09d2ded700 1 -- 172.21.15.129:0/661392240 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09cc1a8b80 con 0x7f09cc101f20 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.661+0000 7f09d2ded700 1 -- 172.21.15.129:0/661392240 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09cc050170 con 0x7f09cc101f20 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.662+0000 7f09c9ffb700 1 -- 172.21.15.129:0/661392240 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f09c0010430 con 0x7f09cc101f20 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.663+0000 7f09c9ffb700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09bc0a2cb0 0x7f09bc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.663+0000 7f09cbfff700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09bc0a2cb0 0x7f09bc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.663+0000 7f09c9ffb700 1 -- 172.21.15.129:0/661392240 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f09c00cf260 con 0x7f09cc101f20 2024-05-20T17:57:46.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.664+0000 7f09cbfff700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09bc0a2cb0 0x7f09bc0a51c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f09b800b2e0 tx=0x7f09b8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:46.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.668+0000 7f09c9ffb700 1 -- 172.21.15.129:0/661392240 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f09c009c210 con 0x7f09cc101f20 2024-05-20T17:57:46.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.873+0000 7f09d2ded700 1 -- 172.21.15.129:0/661392240 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f09cc02dc60 con 0x7f09cc101f20 2024-05-20T17:57:46.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.874+0000 7f09c9ffb700 1 -- 172.21.15.129:0/661392240 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f09c009bb10 con 0x7f09cc101f20 2024-05-20T17:57:46.873 INFO:teuthology.orchestra.run.smithi129.stdout:253403070505 2024-05-20T17:57:46.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 -- 172.21.15.129:0/661392240 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09bc0a2cb0 msgr2=0x7f09bc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:46.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09bc0a2cb0 0x7f09bc0a51c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f09b800b2e0 tx=0x7f09b8005fd0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 -- 172.21.15.129:0/661392240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09cc101f20 msgr2=0x7f09cc19c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:46.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09cc101f20 0x7f09cc19c0a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f09c000e9a0 tx=0x7f09c000ecb0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 -- 172.21.15.129:0/661392240 shutdown_connections 2024-05-20T17:57:46.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09bc0a2cb0 0x7f09bc0a51c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09cc10fad0 0x7f09cc1a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09cc101f20 0x7f09cc19c0a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 --2- 172.21.15.129:0/661392240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09cc102870 0x7f09cc19c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:46.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 -- 172.21.15.129:0/661392240 >> 172.21.15.129:0/661392240 conn(0x7f09cc0fddd0 msgr2=0x7f09cc1115e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:46.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 -- 172.21.15.129:0/661392240 shutdown_connections 2024-05-20T17:57:46.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:46.876+0000 7f09d2ded700 1 -- 172.21.15.129:0/661392240 wait complete. 2024-05-20T17:57:47.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:47 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/488864292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-20T17:57:47.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:47 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:47 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/661392240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-20T17:57:47.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:47 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/488864292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-20T17:57:47.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:47 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:47 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/661392240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-20T17:57:47.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:47 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/488864292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-20T17:57:47.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:47 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:57:47.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:47 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/661392240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-20T17:57:47.519 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332136 got 219043332141 for osd.7 2024-05-20T17:57:47.519 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:48.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-05-20T17:57:48.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-20T17:57:48.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-05-20T17:57:48.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6, 5, 9]}]: dispatch 2024-05-20T17:57:48.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-05-20T17:57:48.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-20T17:57:48.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-20T17:57:48.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-05-20T17:57:48.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:48 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-05-20T17:57:48.109 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:48 smithi129 ceph-mon[32029]: pgmap v316: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:48.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-05-20T17:57:48.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-20T17:57:48.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-05-20T17:57:48.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6, 5, 9]}]: dispatch 2024-05-20T17:57:48.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-05-20T17:57:48.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-20T17:57:48.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-20T17:57:48.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-05-20T17:57:48.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:48 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-05-20T17:57:48.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:48 smithi154 ceph-mon[36482]: pgmap v316: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:48.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-05-20T17:57:48.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-20T17:57:48.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-05-20T17:57:48.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6, 5, 9]}]: dispatch 2024-05-20T17:57:48.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-05-20T17:57:48.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-20T17:57:48.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-20T17:57:48.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-05-20T17:57:48.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:48 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-05-20T17:57:48.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:48 smithi204 ceph-mon[37076]: pgmap v316: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:57:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:49 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-05-20T17:57:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:49 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-20T17:57:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:49 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-05-20T17:57:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:49 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6, 5, 9]}]': finished 2024-05-20T17:57:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:49 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-05-20T17:57:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:49 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-20T17:57:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:49 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-20T17:57:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:49 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-05-20T17:57:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:49 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-05-20T17:57:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:49 smithi154 ceph-mon[36482]: osdmap e87: 12 total, 12 up, 12 in 2024-05-20T17:57:49.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:49 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-05-20T17:57:49.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:49 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-20T17:57:49.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:49 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-05-20T17:57:49.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:49 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6, 5, 9]}]': finished 2024-05-20T17:57:49.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:49 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-05-20T17:57:49.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:49 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-20T17:57:49.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:49 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-20T17:57:49.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:49 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-05-20T17:57:49.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:49 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-05-20T17:57:49.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:49 smithi129 ceph-mon[32029]: osdmap e87: 12 total, 12 up, 12 in 2024-05-20T17:57:49.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:49 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-05-20T17:57:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:49 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-20T17:57:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:49 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-05-20T17:57:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:49 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6, 5, 9]}]': finished 2024-05-20T17:57:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:49 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-05-20T17:57:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:49 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-20T17:57:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:49 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-20T17:57:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:49 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-05-20T17:57:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:49 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-05-20T17:57:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:49 smithi204 ceph-mon[37076]: osdmap e87: 12 total, 12 up, 12 in 2024-05-20T17:57:49.955 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738443 got 34359738448 for osd.0 2024-05-20T17:57:49.955 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:50.324 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:50 smithi129 ceph-mon[32029]: osdmap e88: 12 total, 12 up, 12 in 2024-05-20T17:57:50.325 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:50 smithi129 ceph-mon[32029]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-05-20T17:57:50.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:50 smithi154 ceph-mon[36482]: osdmap e88: 12 total, 12 up, 12 in 2024-05-20T17:57:50.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:50 smithi154 ceph-mon[36482]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-05-20T17:57:50.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:50 smithi204 ceph-mon[37076]: osdmap e88: 12 total, 12 up, 12 in 2024-05-20T17:57:50.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:50 smithi204 ceph-mon[37076]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-05-20T17:57:51.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.070+0000 7f8647b51700 1 -- 172.21.15.129:0/3229916417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86401086f0 msgr2=0x7f8640110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:51.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.070+0000 7f8647b51700 1 --2- 172.21.15.129:0/3229916417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86401086f0 0x7f8640110050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f86380099c0 tx=0x7f8638009cd0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.070+0000 7f8647b51700 1 -- 172.21.15.129:0/3229916417 shutdown_connections 2024-05-20T17:57:51.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.070+0000 7f8647b51700 1 --2- 172.21.15.129:0/3229916417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86401086f0 0x7f8640110050 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.070+0000 7f8647b51700 1 --2- 172.21.15.129:0/3229916417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8640103470 0x7f8640103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.070+0000 7f8647b51700 1 --2- 172.21.15.129:0/3229916417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8640107da0 0x7f86401081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.070+0000 7f8647b51700 1 -- 172.21.15.129:0/3229916417 >> 172.21.15.129:0/3229916417 conn(0x7f86400fee60 msgr2=0x7f8640101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.071+0000 7f8647b51700 1 -- 172.21.15.129:0/3229916417 shutdown_connections 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.071+0000 7f8647b51700 1 -- 172.21.15.129:0/3229916417 wait complete. 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.071+0000 7f8647b51700 1 Processor -- start 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.071+0000 7f8647b51700 1 -- start start 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.071+0000 7f8647b51700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8640103470 0x7f8640074940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.071+0000 7f8647b51700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8640107da0 0x7f8640074e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f8647b51700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86401086f0 0x7f86400753c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f8647b51700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8640113470 con 0x7f8640103470 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f8647b51700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f86401132f0 con 0x7f86401086f0 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f8647b51700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f86401135f0 con 0x7f8640107da0 2024-05-20T17:57:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f86458ed700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8640103470 0x7f8640074940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f86458ed700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8640103470 0x7f8640074940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43112/0 (socket says 172.21.15.129:43112) 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f86458ed700 1 -- 172.21.15.129:0/915328433 learned_addr learned my addr 172.21.15.129:0/915328433 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f86450ec700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8640107da0 0x7f8640074e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f86460ee700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86401086f0 0x7f86400753c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f86458ed700 1 -- 172.21.15.129:0/915328433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8640107da0 msgr2=0x7f8640074e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f86458ed700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8640107da0 0x7f8640074e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f86458ed700 1 -- 172.21.15.129:0/915328433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86401086f0 msgr2=0x7f86400753c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f86458ed700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86401086f0 0x7f86400753c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f86458ed700 1 -- 172.21.15.129:0/915328433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86401a80a0 con 0x7f8640103470 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f86458ed700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8640103470 0x7f8640074940 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f863c0060b0 tx=0x7f863c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f8636ffd700 1 -- 172.21.15.129:0/915328433 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c01a070 con 0x7f8640103470 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f8636ffd700 1 -- 172.21.15.129:0/915328433 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f863c00dd20 con 0x7f8640103470 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f8647b51700 1 -- 172.21.15.129:0/915328433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86401a8390 con 0x7f8640103470 2024-05-20T17:57:51.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f8636ffd700 1 -- 172.21.15.129:0/915328433 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c014720 con 0x7f8640103470 2024-05-20T17:57:51.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.072+0000 7f8647b51700 1 -- 172.21.15.129:0/915328433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86401a88d0 con 0x7f8640103470 2024-05-20T17:57:51.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.073+0000 7f8647b51700 1 -- 172.21.15.129:0/915328433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8640050170 con 0x7f8640103470 2024-05-20T17:57:51.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.075+0000 7f8636ffd700 1 -- 172.21.15.129:0/915328433 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f863c01d070 con 0x7f8640103470 2024-05-20T17:57:51.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.075+0000 7f8636ffd700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f862c0a2cb0 0x7f862c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:51.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.076+0000 7f86450ec700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f862c0a2cb0 0x7f862c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:51.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.076+0000 7f8636ffd700 1 -- 172.21.15.129:0/915328433 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f863c0cee40 con 0x7f8640103470 2024-05-20T17:57:51.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.076+0000 7f86450ec700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f862c0a2cb0 0x7f862c0a51c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8640107740 tx=0x7f863000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:51.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.079+0000 7f8636ffd700 1 -- 172.21.15.129:0/915328433 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f863c010070 con 0x7f8640103470 2024-05-20T17:57:51.257 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:51 smithi129 ceph-mon[32029]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-05-20T17:57:51.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.272+0000 7fbef8170700 1 -- 172.21.15.129:0/1023305858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef0104c30 msgr2=0x7fbef010d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:51.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.272+0000 7fbef8170700 1 --2- 172.21.15.129:0/1023305858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef0104c30 0x7fbef010d180 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009990 tx=0x7fbee0009ca0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.272+0000 7fbef8170700 1 -- 172.21.15.129:0/1023305858 shutdown_connections 2024-05-20T17:57:51.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.272+0000 7fbef8170700 1 --2- 172.21.15.129:0/1023305858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbef010d750 0x7fbef010fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.272+0000 7fbef8170700 1 --2- 172.21.15.129:0/1023305858 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbef01042e0 0x7fbef01046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.273+0000 7fbef8170700 1 --2- 172.21.15.129:0/1023305858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef0104c30 0x7fbef010d180 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.273+0000 7fbef8170700 1 -- 172.21.15.129:0/1023305858 >> 172.21.15.129:0/1023305858 conn(0x7fbef00fdc40 msgr2=0x7fbef0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:51.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.273+0000 7fbef8170700 1 -- 172.21.15.129:0/1023305858 shutdown_connections 2024-05-20T17:57:51.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.273+0000 7fbef8170700 1 -- 172.21.15.129:0/1023305858 wait complete. 2024-05-20T17:57:51.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.273+0000 7fbef8170700 1 Processor -- start 2024-05-20T17:57:51.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.273+0000 7fbef8170700 1 -- start start 2024-05-20T17:57:51.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef8170700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbef01042e0 0x7fbef0199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:51.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef8170700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbef0104c30 0x7fbef019a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:51.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef8170700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef010d750 0x7fbef019e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:51.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef8170700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbef0113400 con 0x7fbef010d750 2024-05-20T17:57:51.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef8170700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbef0113280 con 0x7fbef01042e0 2024-05-20T17:57:51.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef8170700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbef0113580 con 0x7fbef0104c30 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef670d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef010d750 0x7fbef019e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef670d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef010d750 0x7fbef019e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43116/0 (socket says 172.21.15.129:43116) 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef670d700 1 -- 172.21.15.129:0/2157196702 learned_addr learned my addr 172.21.15.129:0/2157196702 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef5f0c700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbef01042e0 0x7fbef0199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef570b700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbef0104c30 0x7fbef019a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef670d700 1 -- 172.21.15.129:0/2157196702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbef0104c30 msgr2=0x7fbef019a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef670d700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbef0104c30 0x7fbef019a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef670d700 1 -- 172.21.15.129:0/2157196702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbef01042e0 msgr2=0x7fbef0199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef670d700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbef01042e0 0x7fbef0199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef670d700 1 -- 172.21.15.129:0/2157196702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef019ec40 con 0x7fbef010d750 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef5f0c700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbef01042e0 0x7fbef0199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef670d700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef010d750 0x7fbef019e5a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbee800ada0 tx=0x7fbee800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbee6ffd700 1 -- 172.21.15.129:0/2157196702 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8017020 con 0x7fbef010d750 2024-05-20T17:57:51.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbee6ffd700 1 -- 172.21.15.129:0/2157196702 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbee8004bb0 con 0x7fbef010d750 2024-05-20T17:57:51.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.274+0000 7fbef8170700 1 -- 172.21.15.129:0/2157196702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbef019ef30 con 0x7fbef010d750 2024-05-20T17:57:51.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.275+0000 7fbee6ffd700 1 -- 172.21.15.129:0/2157196702 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8005520 con 0x7fbef010d750 2024-05-20T17:57:51.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.275+0000 7fbef8170700 1 -- 172.21.15.129:0/2157196702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbef01a6850 con 0x7fbef010d750 2024-05-20T17:57:51.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.275+0000 7fbef8170700 1 -- 172.21.15.129:0/2157196702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbef0050170 con 0x7fbef010d750 2024-05-20T17:57:51.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.278+0000 7fbee6ffd700 1 -- 172.21.15.129:0/2157196702 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fbee80056c0 con 0x7fbef010d750 2024-05-20T17:57:51.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.278+0000 7fbee6ffd700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbedc0a2d30 0x7fbedc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:51.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.279+0000 7fbef5f0c700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbedc0a2d30 0x7fbedc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:51.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.279+0000 7fbee6ffd700 1 -- 172.21.15.129:0/2157196702 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fbee80cee20 con 0x7fbef010d750 2024-05-20T17:57:51.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.279+0000 7fbef5f0c700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbedc0a2d30 0x7fbedc0a5240 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbeec005950 tx=0x7fbeec0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.281+0000 7fbee6ffd700 1 -- 172.21.15.129:0/2157196702 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbee8021020 con 0x7fbef010d750 2024-05-20T17:57:51.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.286+0000 7f8647b51700 1 -- 172.21.15.129:0/915328433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f864002dc60 con 0x7f8640103470 2024-05-20T17:57:51.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.286+0000 7f8636ffd700 1 -- 172.21.15.129:0/915328433 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f863c09bc30 con 0x7f8640103470 2024-05-20T17:57:51.285 INFO:teuthology.orchestra.run.smithi129.stdout:347892351002 2024-05-20T17:57:51.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.288+0000 7f8647b51700 1 -- 172.21.15.129:0/915328433 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f862c0a2cb0 msgr2=0x7f862c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:51.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.288+0000 7f8647b51700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f862c0a2cb0 0x7f862c0a51c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8640107740 tx=0x7f863000b410 comp rx=0 tx=0).stop 2024-05-20T17:57:51.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.289+0000 7f8647b51700 1 -- 172.21.15.129:0/915328433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8640103470 msgr2=0x7f8640074940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.289+0000 7f8647b51700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8640103470 0x7f8640074940 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f863c0060b0 tx=0x7f863c00d3b0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.289+0000 7f8647b51700 1 -- 172.21.15.129:0/915328433 shutdown_connections 2024-05-20T17:57:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.289+0000 7f8647b51700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f862c0a2cb0 0x7f862c0a51c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.289+0000 7f8647b51700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86401086f0 0x7f86400753c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.289+0000 7f8647b51700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8640103470 0x7f8640074940 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.289+0000 7f8647b51700 1 --2- 172.21.15.129:0/915328433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8640107da0 0x7f8640074e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.289+0000 7f8647b51700 1 -- 172.21.15.129:0/915328433 >> 172.21.15.129:0/915328433 conn(0x7f86400fee60 msgr2=0x7f8640105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.289+0000 7f8647b51700 1 -- 172.21.15.129:0/915328433 shutdown_connections 2024-05-20T17:57:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.289+0000 7f8647b51700 1 -- 172.21.15.129:0/915328433 wait complete. 2024-05-20T17:57:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:51 smithi154 ceph-mon[36482]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-05-20T17:57:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:51 smithi204 ceph-mon[37076]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-05-20T17:57:51.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.487+0000 7fbef8170700 1 -- 172.21.15.129:0/2157196702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fbef002dc60 con 0x7fbef010d750 2024-05-20T17:57:51.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.488+0000 7fbee6ffd700 1 -- 172.21.15.129:0/2157196702 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbee809bca0 con 0x7fbef010d750 2024-05-20T17:57:51.487 INFO:teuthology.orchestra.run.smithi129.stdout:77309411400 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 -- 172.21.15.129:0/2157196702 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbedc0a2d30 msgr2=0x7fbedc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbedc0a2d30 0x7fbedc0a5240 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbeec005950 tx=0x7fbeec0058e0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 -- 172.21.15.129:0/2157196702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef010d750 msgr2=0x7fbef019e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef010d750 0x7fbef019e5a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbee800ada0 tx=0x7fbee800c330 comp rx=0 tx=0).stop 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 -- 172.21.15.129:0/2157196702 shutdown_connections 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbedc0a2d30 0x7fbedc0a5240 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbef01042e0 0x7fbef0199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef010d750 0x7fbef019e5a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 --2- 172.21.15.129:0/2157196702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbef0104c30 0x7fbef019a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 -- 172.21.15.129:0/2157196702 >> 172.21.15.129:0/2157196702 conn(0x7fbef00fdc40 msgr2=0x7fbef00fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 -- 172.21.15.129:0/2157196702 shutdown_connections 2024-05-20T17:57:51.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:51.490+0000 7fbef8170700 1 -- 172.21.15.129:0/2157196702 wait complete. 2024-05-20T17:57:51.680 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612634 got 313532612638 for osd.10 2024-05-20T17:57:51.680 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:52.292 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:52 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/915328433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-20T17:57:52.292 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:52 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2157196702' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-20T17:57:52.292 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:52 smithi129 ceph-mon[32029]: pgmap v320: 97 pgs: 4 peering, 93 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-05-20T17:57:52.293 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070502 got 253403070505 for osd.8 2024-05-20T17:57:52.293 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:52.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:52 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/915328433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-20T17:57:52.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:52 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2157196702' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-20T17:57:52.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:52 smithi154 ceph-mon[36482]: pgmap v320: 97 pgs: 4 peering, 93 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-05-20T17:57:52.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:52 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/915328433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-20T17:57:52.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:52 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2157196702' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-20T17:57:52.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:52 smithi204 ceph-mon[37076]: pgmap v320: 97 pgs: 4 peering, 93 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-05-20T17:57:52.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.517+0000 7f59ad7f3700 1 -- 172.21.15.129:0/1733613522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a81075c0 msgr2=0x7f59a810ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:52.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.517+0000 7f59ad7f3700 1 --2- 172.21.15.129:0/1733613522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a81075c0 0x7f59a810ef20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f59a000b0d0 tx=0x7f59a000b3e0 comp rx=0 tx=0).stop 2024-05-20T17:57:52.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.517+0000 7f59ad7f3700 1 -- 172.21.15.129:0/1733613522 shutdown_connections 2024-05-20T17:57:52.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.517+0000 7f59ad7f3700 1 --2- 172.21.15.129:0/1733613522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a81075c0 0x7f59a810ef20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:52.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.517+0000 7f59ad7f3700 1 --2- 172.21.15.129:0/1733613522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59a8102340 0x7f59a8102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:52.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.517+0000 7f59ad7f3700 1 --2- 172.21.15.129:0/1733613522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a8106c70 0x7f59a8107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:52.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.517+0000 7f59ad7f3700 1 -- 172.21.15.129:0/1733613522 >> 172.21.15.129:0/1733613522 conn(0x7f59a80fdc40 msgr2=0x7f59a8100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:52.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.517+0000 7f59ad7f3700 1 -- 172.21.15.129:0/1733613522 shutdown_connections 2024-05-20T17:57:52.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.517+0000 7f59ad7f3700 1 -- 172.21.15.129:0/1733613522 wait complete. 2024-05-20T17:57:52.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.518+0000 7f59ad7f3700 1 Processor -- start 2024-05-20T17:57:52.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.518+0000 7f59ad7f3700 1 -- start start 2024-05-20T17:57:52.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.518+0000 7f59ad7f3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a8102340 0x7f59a819abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:52.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.518+0000 7f59ad7f3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a8106c70 0x7f59a819b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:52.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.518+0000 7f59ad7f3700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59a81075c0 0x7f59a819b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:52.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59ad7f3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59a81123b0 con 0x7f59a8106c70 2024-05-20T17:57:52.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59ad7f3700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59a8112230 con 0x7f59a8102340 2024-05-20T17:57:52.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59ad7f3700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f59a8112530 con 0x7f59a81075c0 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a67fc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a8106c70 0x7f59a819b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a67fc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a8106c70 0x7f59a819b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43138/0 (socket says 172.21.15.129:43138) 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a67fc700 1 -- 172.21.15.129:0/2880346128 learned_addr learned my addr 172.21.15.129:0/2880346128 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a77fe700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59a81075c0 0x7f59a819b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a6ffd700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a8102340 0x7f59a819abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a67fc700 1 -- 172.21.15.129:0/2880346128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59a81075c0 msgr2=0x7f59a819b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a67fc700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59a81075c0 0x7f59a819b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a67fc700 1 -- 172.21.15.129:0/2880346128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a8102340 msgr2=0x7f59a819abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a67fc700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a8102340 0x7f59a819abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a67fc700 1 -- 172.21.15.129:0/2880346128 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a81a5f40 con 0x7f59a8106c70 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a6ffd700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a8102340 0x7f59a819abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.519+0000 7f59a67fc700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a8106c70 0x7f59a819b130 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f599c00b500 tx=0x7f599c00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.520+0000 7f598ffff700 1 -- 172.21.15.129:0/2880346128 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c015020 con 0x7f59a8106c70 2024-05-20T17:57:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.520+0000 7f59ad7f3700 1 -- 172.21.15.129:0/2880346128 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59a81a6230 con 0x7f59a8106c70 2024-05-20T17:57:52.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.520+0000 7f598ffff700 1 -- 172.21.15.129:0/2880346128 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f599c011d10 con 0x7f59a8106c70 2024-05-20T17:57:52.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.520+0000 7f598ffff700 1 -- 172.21.15.129:0/2880346128 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c00f2e0 con 0x7f59a8106c70 2024-05-20T17:57:52.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.520+0000 7f59ad7f3700 1 -- 172.21.15.129:0/2880346128 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59a81a67c0 con 0x7f59a8106c70 2024-05-20T17:57:52.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.521+0000 7f59ad7f3700 1 -- 172.21.15.129:0/2880346128 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59a8050170 con 0x7f59a8106c70 2024-05-20T17:57:52.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.522+0000 7f598ffff700 1 -- 172.21.15.129:0/2880346128 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f599c018020 con 0x7f59a8106c70 2024-05-20T17:57:52.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.523+0000 7f598ffff700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f59900a2cb0 0x7f59900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:52.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.523+0000 7f59a6ffd700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f59900a2cb0 0x7f59900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:52.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.523+0000 7f598ffff700 1 -- 172.21.15.129:0/2880346128 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f599c0ce870 con 0x7f59a8106c70 2024-05-20T17:57:52.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.523+0000 7f59a6ffd700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f59900a2cb0 0x7f59900a51c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5998005fd0 tx=0x7f5998005dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:52.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.528+0000 7f598ffff700 1 -- 172.21.15.129:0/2880346128 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f599c01f020 con 0x7f59a8106c70 2024-05-20T17:57:52.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.729+0000 7f59ad7f3700 1 -- 172.21.15.129:0/2880346128 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f59a806aa80 con 0x7f59a8106c70 2024-05-20T17:57:52.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.729+0000 7f598ffff700 1 -- 172.21.15.129:0/2880346128 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f599c09b710 con 0x7f59a8106c70 2024-05-20T17:57:52.728 INFO:teuthology.orchestra.run.smithi129.stdout:287762808869 2024-05-20T17:57:52.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.731+0000 7f59ad7f3700 1 -- 172.21.15.129:0/2880346128 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f59900a2cb0 msgr2=0x7f59900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:52.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.731+0000 7f59ad7f3700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f59900a2cb0 0x7f59900a51c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5998005fd0 tx=0x7f5998005dc0 comp rx=0 tx=0).stop 2024-05-20T17:57:52.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.731+0000 7f59ad7f3700 1 -- 172.21.15.129:0/2880346128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a8106c70 msgr2=0x7f59a819b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.731+0000 7f59ad7f3700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a8106c70 0x7f59a819b130 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f599c00b500 tx=0x7f599c00b810 comp rx=0 tx=0).stop 2024-05-20T17:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.731+0000 7f59ad7f3700 1 -- 172.21.15.129:0/2880346128 shutdown_connections 2024-05-20T17:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.731+0000 7f59ad7f3700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f59900a2cb0 0x7f59900a51c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.731+0000 7f59ad7f3700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a8102340 0x7f59a819abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.731+0000 7f59ad7f3700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f59a81075c0 0x7f59a819b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.731+0000 7f59ad7f3700 1 --2- 172.21.15.129:0/2880346128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a8106c70 0x7f59a819b130 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.731+0000 7f59ad7f3700 1 -- 172.21.15.129:0/2880346128 >> 172.21.15.129:0/2880346128 conn(0x7f59a80fdc40 msgr2=0x7f59a81063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.732+0000 7f59ad7f3700 1 -- 172.21.15.129:0/2880346128 shutdown_connections 2024-05-20T17:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:52.732+0000 7f59ad7f3700 1 -- 172.21.15.129:0/2880346128 wait complete. 2024-05-20T17:57:53.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:53 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2880346128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-20T17:57:53.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:53 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2880346128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-20T17:57:53.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:53 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2880346128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-20T17:57:53.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.608+0000 7f6b1803e700 1 -- 172.21.15.129:0/3659546648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b101042e0 msgr2=0x7f6b101046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:53.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.608+0000 7f6b1803e700 1 --2- 172.21.15.129:0/3659546648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b101042e0 0x7f6b101046f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6b00009990 tx=0x7f6b00009ca0 comp rx=0 tx=0).stop 2024-05-20T17:57:53.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.609+0000 7f6b1803e700 1 -- 172.21.15.129:0/3659546648 shutdown_connections 2024-05-20T17:57:53.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.609+0000 7f6b1803e700 1 --2- 172.21.15.129:0/3659546648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b1010d750 0x7f6b1010fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:53.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.609+0000 7f6b1803e700 1 --2- 172.21.15.129:0/3659546648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b101042e0 0x7f6b101046f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:53.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.609+0000 7f6b1803e700 1 --2- 172.21.15.129:0/3659546648 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b10104c30 0x7f6b1010d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:53.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.609+0000 7f6b1803e700 1 -- 172.21.15.129:0/3659546648 >> 172.21.15.129:0/3659546648 conn(0x7f6b100fdc40 msgr2=0x7f6b10100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:53.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.609+0000 7f6b1803e700 1 -- 172.21.15.129:0/3659546648 shutdown_connections 2024-05-20T17:57:53.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.609+0000 7f6b1803e700 1 -- 172.21.15.129:0/3659546648 wait complete. 2024-05-20T17:57:53.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b1803e700 1 Processor -- start 2024-05-20T17:57:53.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b1803e700 1 -- start start 2024-05-20T17:57:53.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b1803e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b101042e0 0x7f6b1019f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:53.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b1803e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b10104c30 0x7f6b1019fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:53.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b1803e700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b1010d750 0x7f6b10199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:53.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b1803e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b101133f0 con 0x7f6b101042e0 2024-05-20T17:57:53.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b1803e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b10113270 con 0x7f6b10104c30 2024-05-20T17:57:53.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b1803e700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6b10113570 con 0x7f6b1010d750 2024-05-20T17:57:53.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b15dda700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b101042e0 0x7f6b1019f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:53.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b15dda700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b101042e0 0x7f6b1019f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43166/0 (socket says 172.21.15.129:43166) 2024-05-20T17:57:53.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b15dda700 1 -- 172.21.15.129:0/4257450729 learned_addr learned my addr 172.21.15.129:0/4257450729 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:53.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b165db700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b1010d750 0x7f6b10199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:53.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b155d9700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b10104c30 0x7f6b1019fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:53.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b15dda700 1 -- 172.21.15.129:0/4257450729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b1010d750 msgr2=0x7f6b10199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:53.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b15dda700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b1010d750 0x7f6b10199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:53.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b15dda700 1 -- 172.21.15.129:0/4257450729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b10104c30 msgr2=0x7f6b1019fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:53.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.610+0000 7f6b15dda700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b10104c30 0x7f6b1019fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:53.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.611+0000 7f6b15dda700 1 -- 172.21.15.129:0/4257450729 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1019a1d0 con 0x7f6b101042e0 2024-05-20T17:57:53.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.611+0000 7f6b15dda700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b101042e0 0x7f6b1019f9a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6b00000c00 tx=0x7f6b000047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:53.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.611+0000 7f6b06ffd700 1 -- 172.21.15.129:0/4257450729 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00027070 con 0x7f6b101042e0 2024-05-20T17:57:53.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.611+0000 7f6b06ffd700 1 -- 172.21.15.129:0/4257450729 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b00003e40 con 0x7f6b101042e0 2024-05-20T17:57:53.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.611+0000 7f6b1803e700 1 -- 172.21.15.129:0/4257450729 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b1019a460 con 0x7f6b101042e0 2024-05-20T17:57:53.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.611+0000 7f6b06ffd700 1 -- 172.21.15.129:0/4257450729 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b00022d10 con 0x7f6b101042e0 2024-05-20T17:57:53.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.612+0000 7f6b1803e700 1 -- 172.21.15.129:0/4257450729 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b1019a940 con 0x7f6b101042e0 2024-05-20T17:57:53.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.613+0000 7f6b1803e700 1 -- 172.21.15.129:0/4257450729 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b1006ad70 con 0x7f6b101042e0 2024-05-20T17:57:53.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.614+0000 7f6b06ffd700 1 -- 172.21.15.129:0/4257450729 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f6b0002a070 con 0x7f6b101042e0 2024-05-20T17:57:53.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.615+0000 7f6b06ffd700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6afc0a2cb0 0x7f6afc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:53.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.615+0000 7f6b155d9700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6afc0a2cb0 0x7f6afc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:53.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.616+0000 7f6b06ffd700 1 -- 172.21.15.129:0/4257450729 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f6b000cff80 con 0x7f6b101042e0 2024-05-20T17:57:53.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.616+0000 7f6b155d9700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6afc0a2cb0 0x7f6afc0a51c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6b1019b260 tx=0x7f6b0c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:53.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.620+0000 7f6b06ffd700 1 -- 172.21.15.129:0/4257450729 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6b0005e850 con 0x7f6b101042e0 2024-05-20T17:57:53.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.820+0000 7f6b1803e700 1 -- 172.21.15.129:0/4257450729 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f6b101a6750 con 0x7f6b101042e0 2024-05-20T17:57:53.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.820+0000 7f6b06ffd700 1 -- 172.21.15.129:0/4257450729 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6b00013070 con 0x7f6b101042e0 2024-05-20T17:57:53.819 INFO:teuthology.orchestra.run.smithi129.stdout:107374182467 2024-05-20T17:57:53.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.822+0000 7f6b1803e700 1 -- 172.21.15.129:0/4257450729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6afc0a2cb0 msgr2=0x7f6afc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:53.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.822+0000 7f6b1803e700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6afc0a2cb0 0x7f6afc0a51c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6b1019b260 tx=0x7f6b0c00a380 comp rx=0 tx=0).stop 2024-05-20T17:57:53.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.822+0000 7f6b1803e700 1 -- 172.21.15.129:0/4257450729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b101042e0 msgr2=0x7f6b1019f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.822+0000 7f6b1803e700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b101042e0 0x7f6b1019f9a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6b00000c00 tx=0x7f6b000047b0 comp rx=0 tx=0).stop 2024-05-20T17:57:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.823+0000 7f6b1803e700 1 -- 172.21.15.129:0/4257450729 shutdown_connections 2024-05-20T17:57:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.823+0000 7f6b1803e700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f6afc0a2cb0 0x7f6afc0a51c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.823+0000 7f6b1803e700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6b1010d750 0x7f6b10199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.823+0000 7f6b1803e700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b101042e0 0x7f6b1019f9a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.823+0000 7f6b1803e700 1 --2- 172.21.15.129:0/4257450729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b10104c30 0x7f6b1019fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.823+0000 7f6b1803e700 1 -- 172.21.15.129:0/4257450729 >> 172.21.15.129:0/4257450729 conn(0x7f6b100fdc40 msgr2=0x7f6b100fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.823+0000 7f6b1803e700 1 -- 172.21.15.129:0/4257450729 shutdown_connections 2024-05-20T17:57:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:53.823+0000 7f6b1803e700 1 -- 172.21.15.129:0/4257450729 wait complete. 2024-05-20T17:57:54.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:54 smithi129 ceph-mon[32029]: pgmap v321: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-05-20T17:57:54.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:54 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4257450729' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-20T17:57:54.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:54 smithi154 ceph-mon[36482]: pgmap v321: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-05-20T17:57:54.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:54 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/4257450729' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-20T17:57:54.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:54 smithi204 ceph-mon[37076]: pgmap v321: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-05-20T17:57:54.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:54 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/4257450729' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-20T17:57:54.698 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808864 got 287762808869 for osd.9 2024-05-20T17:57:54.698 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:55.109 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411394 got 77309411400 for osd.2 2024-05-20T17:57:55.110 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:55.322 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:55 smithi129 ceph-mon[32029]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-05-20T17:57:55.323 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:55 smithi129 ceph-mon[32029]: Cluster is now healthy 2024-05-20T17:57:55.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.323+0000 7f2ddddf4700 1 -- 172.21.15.129:0/92933106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2dd8103450 msgr2=0x7f2dd8103860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.323+0000 7f2ddddf4700 1 --2- 172.21.15.129:0/92933106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2dd8103450 0x7f2dd8103860 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80099c0 tx=0x7f2dc8009cd0 comp rx=0 tx=0).stop 2024-05-20T17:57:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.323+0000 7f2ddddf4700 1 -- 172.21.15.129:0/92933106 shutdown_connections 2024-05-20T17:57:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.323+0000 7f2ddddf4700 1 --2- 172.21.15.129:0/92933106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2dd81086d0 0x7f2dd8110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.323+0000 7f2ddddf4700 1 --2- 172.21.15.129:0/92933106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2dd8103450 0x7f2dd8103860 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.323+0000 7f2ddddf4700 1 --2- 172.21.15.129:0/92933106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd8107d80 0x7f2dd8108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.323+0000 7f2ddddf4700 1 -- 172.21.15.129:0/92933106 >> 172.21.15.129:0/92933106 conn(0x7f2dd80fed20 msgr2=0x7f2dd8101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.324+0000 7f2ddddf4700 1 -- 172.21.15.129:0/92933106 shutdown_connections 2024-05-20T17:57:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.324+0000 7f2ddddf4700 1 -- 172.21.15.129:0/92933106 wait complete. 2024-05-20T17:57:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.324+0000 7f2ddddf4700 1 Processor -- start 2024-05-20T17:57:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2ddddf4700 1 -- start start 2024-05-20T17:57:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2ddddf4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd8103450 0x7f2dd8199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2ddddf4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2dd8107d80 0x7f2dd819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2ddddf4700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2dd81086d0 0x7f2dd819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2ddddf4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd8113450 con 0x7f2dd8107d80 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2ddddf4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd81132d0 con 0x7f2dd8103450 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2ddddf4700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd81135d0 con 0x7f2dd81086d0 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2dcffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2dd8107d80 0x7f2dd819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2dcffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2dd8107d80 0x7f2dd819a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43182/0 (socket says 172.21.15.129:43182) 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2dcffff700 1 -- 172.21.15.129:0/1457810661 learned_addr learned my addr 172.21.15.129:0/1457810661 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2dd77fe700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd8103450 0x7f2dd8199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2dd7fff700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2dd81086d0 0x7f2dd819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2dcffff700 1 -- 172.21.15.129:0/1457810661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2dd81086d0 msgr2=0x7f2dd819e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:55.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2dcffff700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2dd81086d0 0x7f2dd819e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2dcffff700 1 -- 172.21.15.129:0/1457810661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd8103450 msgr2=0x7f2dd8199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2dcffff700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd8103450 0x7f2dd8199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.325+0000 7f2dcffff700 1 -- 172.21.15.129:0/1457810661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd819ec10 con 0x7f2dd8107d80 2024-05-20T17:57:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.326+0000 7f2dcffff700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2dd8107d80 0x7f2dd819a2e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000e970 tx=0x7f2dc000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.326+0000 7f2dd57fa700 1 -- 172.21.15.129:0/1457810661 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc000cbd0 con 0x7f2dd8107d80 2024-05-20T17:57:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.326+0000 7f2dd57fa700 1 -- 172.21.15.129:0/1457810661 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dc000cd70 con 0x7f2dd8107d80 2024-05-20T17:57:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.326+0000 7f2ddddf4700 1 -- 172.21.15.129:0/1457810661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd819ef00 con 0x7f2dd8107d80 2024-05-20T17:57:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.326+0000 7f2dd57fa700 1 -- 172.21.15.129:0/1457810661 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc00056b0 con 0x7f2dd8107d80 2024-05-20T17:57:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.326+0000 7f2ddddf4700 1 -- 172.21.15.129:0/1457810661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd81a6820 con 0x7f2dd8107d80 2024-05-20T17:57:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.327+0000 7f2ddddf4700 1 -- 172.21.15.129:0/1457810661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd8050170 con 0x7f2dd8107d80 2024-05-20T17:57:55.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.328+0000 7f2dd57fa700 1 -- 172.21.15.129:0/1457810661 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2dc0004750 con 0x7f2dd8107d80 2024-05-20T17:57:55.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.329+0000 7f2dd57fa700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2dc40a2cb0 0x7f2dc40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:55.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.329+0000 7f2dd77fe700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2dc40a2cb0 0x7f2dc40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:55.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.329+0000 7f2dd57fa700 1 -- 172.21.15.129:0/1457810661 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f2dc00cf980 con 0x7f2dd8107d80 2024-05-20T17:57:55.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.330+0000 7f2dd77fe700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2dc40a2cb0 0x7f2dc40a51c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8003800 tx=0x7f2dc8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:55.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.333+0000 7f2dd57fa700 1 -- 172.21.15.129:0/1457810661 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2dc009c770 con 0x7f2dd8107d80 2024-05-20T17:57:55.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:55 smithi154 ceph-mon[36482]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-05-20T17:57:55.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:55 smithi154 ceph-mon[36482]: Cluster is now healthy 2024-05-20T17:57:55.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:55 smithi204 ceph-mon[37076]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-05-20T17:57:55.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:55 smithi204 ceph-mon[37076]: Cluster is now healthy 2024-05-20T17:57:55.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.535+0000 7f2ddddf4700 1 -- 172.21.15.129:0/1457810661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f2dd802dc60 con 0x7f2dd8107d80 2024-05-20T17:57:55.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.535+0000 7f2dd57fa700 1 -- 172.21.15.129:0/1457810661 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2dc009c070 con 0x7f2dd8107d80 2024-05-20T17:57:55.534 INFO:teuthology.orchestra.run.smithi129.stdout:55834574925 2024-05-20T17:57:55.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.537+0000 7f2ddddf4700 1 -- 172.21.15.129:0/1457810661 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2dc40a2cb0 msgr2=0x7f2dc40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:55.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.537+0000 7f2ddddf4700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2dc40a2cb0 0x7f2dc40a51c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8003800 tx=0x7f2dc8005fb0 comp rx=0 tx=0).stop 2024-05-20T17:57:55.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.537+0000 7f2ddddf4700 1 -- 172.21.15.129:0/1457810661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2dd8107d80 msgr2=0x7f2dd819a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:55.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.537+0000 7f2ddddf4700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2dd8107d80 0x7f2dd819a2e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000e970 tx=0x7f2dc000ec80 comp rx=0 tx=0).stop 2024-05-20T17:57:55.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.537+0000 7f2ddddf4700 1 -- 172.21.15.129:0/1457810661 shutdown_connections 2024-05-20T17:57:55.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.537+0000 7f2ddddf4700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2dc40a2cb0 0x7f2dc40a51c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:55.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.537+0000 7f2ddddf4700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd8103450 0x7f2dd8199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:55.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.537+0000 7f2ddddf4700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2dd81086d0 0x7f2dd819e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:55.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.537+0000 7f2ddddf4700 1 --2- 172.21.15.129:0/1457810661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2dd8107d80 0x7f2dd819a2e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:55.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.537+0000 7f2ddddf4700 1 -- 172.21.15.129:0/1457810661 >> 172.21.15.129:0/1457810661 conn(0x7f2dd80fed20 msgr2=0x7f2dd8107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:55.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.537+0000 7f2ddddf4700 1 -- 172.21.15.129:0/1457810661 shutdown_connections 2024-05-20T17:57:55.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:55.538+0000 7f2ddddf4700 1 -- 172.21.15.129:0/1457810661 wait complete. 2024-05-20T17:57:56.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.043+0000 7f79f78db700 1 -- 172.21.15.129:0/954001670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f01086e0 msgr2=0x7f79f0110040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:56.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.043+0000 7f79f78db700 1 --2- 172.21.15.129:0/954001670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f01086e0 0x7f79f0110040 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f79e800b0d0 tx=0x7f79e800b3e0 comp rx=0 tx=0).stop 2024-05-20T17:57:56.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.043+0000 7f79f78db700 1 -- 172.21.15.129:0/954001670 shutdown_connections 2024-05-20T17:57:56.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.043+0000 7f79f78db700 1 --2- 172.21.15.129:0/954001670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f01086e0 0x7f79f0110040 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:56.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.043+0000 7f79f78db700 1 --2- 172.21.15.129:0/954001670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79f0103460 0x7f79f0103870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.043+0000 7f79f78db700 1 --2- 172.21.15.129:0/954001670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0107d90 0x7f79f01081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.043+0000 7f79f78db700 1 -- 172.21.15.129:0/954001670 >> 172.21.15.129:0/954001670 conn(0x7f79f00fed50 msgr2=0x7f79f01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.044+0000 7f79f78db700 1 -- 172.21.15.129:0/954001670 shutdown_connections 2024-05-20T17:57:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.044+0000 7f79f78db700 1 -- 172.21.15.129:0/954001670 wait complete. 2024-05-20T17:57:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.044+0000 7f79f78db700 1 Processor -- start 2024-05-20T17:57:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f78db700 1 -- start start 2024-05-20T17:57:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f78db700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79f0103460 0x7f79f0199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f78db700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0107d90 0x7f79f019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f78db700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f01086e0 0x7f79f019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f78db700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79f0113480 con 0x7f79f01086e0 2024-05-20T17:57:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f78db700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f79f0113300 con 0x7f79f0107d90 2024-05-20T17:57:56.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f78db700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f79f0113600 con 0x7f79f0103460 2024-05-20T17:57:56.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f5e78700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f01086e0 0x7f79f019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:56.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f5e78700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f01086e0 0x7f79f019e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43206/0 (socket says 172.21.15.129:43206) 2024-05-20T17:57:56.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f5e78700 1 -- 172.21.15.129:0/1506068585 learned_addr learned my addr 172.21.15.129:0/1506068585 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:57:56.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f4e76700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0107d90 0x7f79f019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:56.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f5677700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79f0103460 0x7f79f0199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:56.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f5e78700 1 -- 172.21.15.129:0/1506068585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79f0103460 msgr2=0x7f79f0199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:56.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f5e78700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79f0103460 0x7f79f0199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:56.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f5e78700 1 -- 172.21.15.129:0/1506068585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0107d90 msgr2=0x7f79f019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:56.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f5e78700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0107d90 0x7f79f019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:56.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f5e78700 1 -- 172.21.15.129:0/1506068585 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f019ebe0 con 0x7f79f01086e0 2024-05-20T17:57:56.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.045+0000 7f79f5e78700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f01086e0 0x7f79f019e540 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f79e8006070 tx=0x7f79e8004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.046+0000 7f79e67fc700 1 -- 172.21.15.129:0/1506068585 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8021070 con 0x7f79f01086e0 2024-05-20T17:57:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.046+0000 7f79e67fc700 1 -- 172.21.15.129:0/1506068585 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e8007b80 con 0x7f79f01086e0 2024-05-20T17:57:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.046+0000 7f79f78db700 1 -- 172.21.15.129:0/1506068585 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f019ee70 con 0x7f79f01086e0 2024-05-20T17:57:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.046+0000 7f79e67fc700 1 -- 172.21.15.129:0/1506068585 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8005330 con 0x7f79f01086e0 2024-05-20T17:57:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.046+0000 7f79f78db700 1 -- 172.21.15.129:0/1506068585 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f01a67f0 con 0x7f79f01086e0 2024-05-20T17:57:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.047+0000 7f79f78db700 1 -- 172.21.15.129:0/1506068585 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f0104910 con 0x7f79f01086e0 2024-05-20T17:57:56.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.048+0000 7f79e67fc700 1 -- 172.21.15.129:0/1506068585 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f79e8024070 con 0x7f79f01086e0 2024-05-20T17:57:56.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.048+0000 7f79e67fc700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79dc0a2cb0 0x7f79dc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:57:56.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.049+0000 7f79f5677700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79dc0a2cb0 0x7f79dc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:57:56.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.049+0000 7f79e67fc700 1 -- 172.21.15.129:0/1506068585 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f79e800f020 con 0x7f79f01086e0 2024-05-20T17:57:56.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.049+0000 7f79f5677700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79dc0a2cb0 0x7f79dc0a51c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f79ec005950 tx=0x7f79ec0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:57:56.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.052+0000 7f79e67fc700 1 -- 172.21.15.129:0/1506068585 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f79e802b020 con 0x7f79f01086e0 2024-05-20T17:57:56.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.255+0000 7f79f78db700 1 -- 172.21.15.129:0/1506068585 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f79f019b070 con 0x7f79f01086e0 2024-05-20T17:57:56.254 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:56 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1457810661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-20T17:57:56.254 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:56 smithi129 ceph-mon[32029]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 72 B/s, 0 objects/s recovering 2024-05-20T17:57:56.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.256+0000 7f79e67fc700 1 -- 172.21.15.129:0/1506068585 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f79e809da60 con 0x7f79f01086e0 2024-05-20T17:57:56.255 INFO:teuthology.orchestra.run.smithi129.stdout:197568495669 2024-05-20T17:57:56.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 -- 172.21.15.129:0/1506068585 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79dc0a2cb0 msgr2=0x7f79dc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:56.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79dc0a2cb0 0x7f79dc0a51c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f79ec005950 tx=0x7f79ec0058e0 comp rx=0 tx=0).stop 2024-05-20T17:57:56.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 -- 172.21.15.129:0/1506068585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f01086e0 msgr2=0x7f79f019e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:57:56.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f01086e0 0x7f79f019e540 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f79e8006070 tx=0x7f79e8004d00 comp rx=0 tx=0).stop 2024-05-20T17:57:56.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 -- 172.21.15.129:0/1506068585 shutdown_connections 2024-05-20T17:57:56.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f79dc0a2cb0 0x7f79dc0a51c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:56.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f01086e0 0x7f79f019e540 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:56.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f79f0103460 0x7f79f0199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:56.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 --2- 172.21.15.129:0/1506068585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79f0107d90 0x7f79f019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:57:56.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 -- 172.21.15.129:0/1506068585 >> 172.21.15.129:0/1506068585 conn(0x7f79f00fed50 msgr2=0x7f79f0107470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:57:56.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 -- 172.21.15.129:0/1506068585 shutdown_connections 2024-05-20T17:57:56.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:57:56.258+0000 7f79f78db700 1 -- 172.21.15.129:0/1506068585 wait complete. 2024-05-20T17:57:56.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:56 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1457810661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-20T17:57:56.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:56 smithi154 ceph-mon[36482]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 72 B/s, 0 objects/s recovering 2024-05-20T17:57:56.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:56 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1457810661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-20T17:57:56.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:56 smithi204 ceph-mon[37076]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 72 B/s, 0 objects/s recovering 2024-05-20T17:57:56.610 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182464 got 107374182467 for osd.3 2024-05-20T17:57:56.610 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:57.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:57 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1506068585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-20T17:57:57.230 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495662 got 197568495669 for osd.6 2024-05-20T17:57:57.231 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:57.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:57 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1506068585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-20T17:57:57.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:57 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1506068585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-20T17:57:57.594 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574920 got 55834574925 for osd.1 2024-05-20T17:57:57.595 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:58.082 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350996 got 347892351002 for osd.11 2024-05-20T17:57:58.082 DEBUG:teuthology.parallel:result is None 2024-05-20T17:57:58.083 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-20T17:57:58.083 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph pg dump --format=json 2024-05-20T17:57:58.373 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:58 smithi129 ceph-mon[32029]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 60 B/s, 0 objects/s recovering 2024-05-20T17:57:58.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:58 smithi154 ceph-mon[36482]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 60 B/s, 0 objects/s recovering 2024-05-20T17:57:58.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:58 smithi204 ceph-mon[37076]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 60 B/s, 0 objects/s recovering 2024-05-20T17:57:58.846 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:58:00.033 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:57:59 smithi129 ceph-mon[32029]: pgmap v324: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 1 op/s; 54 B/s, 0 objects/s recovering 2024-05-20T17:58:00.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:57:59 smithi154 ceph-mon[36482]: pgmap v324: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 1 op/s; 54 B/s, 0 objects/s recovering 2024-05-20T17:58:00.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:57:59 smithi204 ceph-mon[37076]: pgmap v324: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 1 op/s; 54 B/s, 0 objects/s recovering 2024-05-20T17:58:00.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.567+0000 7feff3168700 1 -- 172.21.15.129:0/3818841591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec107600 msgr2=0x7fefec10ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:00.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.567+0000 7feff3168700 1 --2- 172.21.15.129:0/3818841591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec107600 0x7fefec10ef60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fefe400b0d0 tx=0x7fefe400b3e0 comp rx=0 tx=0).stop 2024-05-20T17:58:00.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.567+0000 7feff3168700 1 -- 172.21.15.129:0/3818841591 shutdown_connections 2024-05-20T17:58:00.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.567+0000 7feff3168700 1 --2- 172.21.15.129:0/3818841591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec107600 0x7fefec10ef60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:00.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.567+0000 7feff3168700 1 --2- 172.21.15.129:0/3818841591 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefec1023d0 0x7fefec1027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:00.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.567+0000 7feff3168700 1 --2- 172.21.15.129:0/3818841591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefec106cb0 0x7fefec1070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:00.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.567+0000 7feff3168700 1 -- 172.21.15.129:0/3818841591 >> 172.21.15.129:0/3818841591 conn(0x7fefec077c80 msgr2=0x7fefec0780c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:00.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.567+0000 7feff3168700 1 -- 172.21.15.129:0/3818841591 shutdown_connections 2024-05-20T17:58:00.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.567+0000 7feff3168700 1 -- 172.21.15.129:0/3818841591 wait complete. 2024-05-20T17:58:00.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff3168700 1 Processor -- start 2024-05-20T17:58:00.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff3168700 1 -- start start 2024-05-20T17:58:00.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff3168700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec1023d0 0x7fefec19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:00.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff3168700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefec106cb0 0x7fefec19b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:00.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff3168700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefec107600 0x7fefec19b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:00.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff3168700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fefec112380 con 0x7fefec1023d0 2024-05-20T17:58:00.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff3168700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefec112200 con 0x7fefec107600 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff3168700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fefec112500 con 0x7fefec106cb0 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff0f04700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec1023d0 0x7fefec19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff0f04700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec1023d0 0x7fefec19ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49830/0 (socket says 172.21.15.129:49830) 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff0f04700 1 -- 172.21.15.129:0/1529189683 learned_addr learned my addr 172.21.15.129:0/1529189683 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7fefebfff700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefec106cb0 0x7fefec19b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.568+0000 7feff1705700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefec107600 0x7fefec19b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7feff0f04700 1 -- 172.21.15.129:0/1529189683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefec106cb0 msgr2=0x7fefec19b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7feff0f04700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefec106cb0 0x7fefec19b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7feff0f04700 1 -- 172.21.15.129:0/1529189683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefec107600 msgr2=0x7fefec19b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7feff0f04700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefec107600 0x7fefec19b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7feff0f04700 1 -- 172.21.15.129:0/1529189683 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefec1a5ed0 con 0x7fefec1023d0 2024-05-20T17:58:00.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7fefebfff700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefec106cb0 0x7fefec19b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:00.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7feff0f04700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec1023d0 0x7fefec19ab60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fefe000e9a0 tx=0x7fefe000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:00.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7fefe9ffb700 1 -- 172.21.15.129:0/1529189683 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe000ca10 con 0x7fefec1023d0 2024-05-20T17:58:00.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7fefe9ffb700 1 -- 172.21.15.129:0/1529189683 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefe000cbb0 con 0x7fefec1023d0 2024-05-20T17:58:00.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7feff3168700 1 -- 172.21.15.129:0/1529189683 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefec1989b0 con 0x7fefec1023d0 2024-05-20T17:58:00.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7fefe9ffb700 1 -- 172.21.15.129:0/1529189683 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe0018680 con 0x7fefec1023d0 2024-05-20T17:58:00.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.569+0000 7feff3168700 1 -- 172.21.15.129:0/1529189683 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefec198f40 con 0x7fefec1023d0 2024-05-20T17:58:00.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.570+0000 7feff3168700 1 -- 172.21.15.129:0/1529189683 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefec050170 con 0x7fefec1023d0 2024-05-20T17:58:00.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.572+0000 7fefe9ffb700 1 -- 172.21.15.129:0/1529189683 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fefe0010430 con 0x7fefec1023d0 2024-05-20T17:58:00.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.572+0000 7fefe9ffb700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fefdc0a2d30 0x7fefdc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:00.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.572+0000 7fefebfff700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fefdc0a2d30 0x7fefdc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:00.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.572+0000 7fefe9ffb700 1 -- 172.21.15.129:0/1529189683 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fefe00cf360 con 0x7fefec1023d0 2024-05-20T17:58:00.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.573+0000 7fefebfff700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fefdc0a2d30 0x7fefdc0a5240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fefec06a5c0 tx=0x7fefd8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:00.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.577+0000 7fefe9ffb700 1 -- 172.21.15.129:0/1529189683 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fefe009c150 con 0x7fefec1023d0 2024-05-20T17:58:00.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.791+0000 7feff3168700 1 -- 172.21.15.129:0/1529189683 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fefec101340 con 0x7fefdc0a2d30 2024-05-20T17:58:00.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.798+0000 7fefe9ffb700 1 -- 172.21.15.129:0/1529189683 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258934 (secure 0 0 0) 0x7fefec101340 con 0x7fefdc0a2d30 2024-05-20T17:58:00.800 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:00.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.801+0000 7feff3168700 1 -- 172.21.15.129:0/1529189683 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fefdc0a2d30 msgr2=0x7fefdc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:00.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.801+0000 7feff3168700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fefdc0a2d30 0x7fefdc0a5240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fefec06a5c0 tx=0x7fefd8009260 comp rx=0 tx=0).stop 2024-05-20T17:58:00.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.801+0000 7feff3168700 1 -- 172.21.15.129:0/1529189683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec1023d0 msgr2=0x7fefec19ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:00.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.801+0000 7feff3168700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec1023d0 0x7fefec19ab60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fefe000e9a0 tx=0x7fefe000ecb0 comp rx=0 tx=0).stop 2024-05-20T17:58:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.801+0000 7feff3168700 1 -- 172.21.15.129:0/1529189683 shutdown_connections 2024-05-20T17:58:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.802+0000 7feff3168700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fefdc0a2d30 0x7fefdc0a5240 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.802+0000 7feff3168700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefec107600 0x7fefec19b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.802+0000 7feff3168700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec1023d0 0x7fefec19ab60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.802+0000 7feff3168700 1 --2- 172.21.15.129:0/1529189683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefec106cb0 0x7fefec19b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.802+0000 7feff3168700 1 -- 172.21.15.129:0/1529189683 >> 172.21.15.129:0/1529189683 conn(0x7fefec077c80 msgr2=0x7fefec100c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.802+0000 7feff3168700 1 -- 172.21.15.129:0/1529189683 shutdown_connections 2024-05-20T17:58:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:00.802+0000 7feff3168700 1 -- 172.21.15.129:0/1529189683 wait complete. 2024-05-20T17:58:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-05-20T17:58:01.504 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-05-20T17:57:59.726153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652446,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":285,"num_read_kb":255,"num_write":258,"num_write_kb":3023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":203,"ondisk_log_size":203,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364948,"kb_used_data":11556,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576876,"statfs":{"total":1151940427776,"available":1151566721024,"internally_reserved":0,"allocated":11833344,"data_stored":7505814,"data_compressed":182331,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.682518"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043221+0000","last_change":"2024-05-20T17:56:45.746793+0000","last_active":"2024-05-20T17:57:49.043221+0000","last_peered":"2024-05-20T17:57:49.043221+0000","last_clean":"2024-05-20T17:57:49.043221+0000","last_became_active":"2024-05-20T17:56:45.746244+0000","last_became_peered":"2024-05-20T17:56:45.746244+0000","last_unstale":"2024-05-20T17:57:49.043221+0000","last_undegraded":"2024-05-20T17:57:49.043221+0000","last_fullsized":"2024-05-20T17:57:49.043221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:37:03.703599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048122+0000","last_change":"2024-05-20T17:56:45.744538+0000","last_active":"2024-05-20T17:57:49.048122+0000","last_peered":"2024-05-20T17:57:49.048122+0000","last_clean":"2024-05-20T17:57:49.048122+0000","last_became_active":"2024-05-20T17:56:45.744213+0000","last_became_peered":"2024-05-20T17:56:45.744213+0000","last_unstale":"2024-05-20T17:57:49.048122+0000","last_undegraded":"2024-05-20T17:57:49.048122+0000","last_fullsized":"2024-05-20T17:57:49.048122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:19:26.163003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047534+0000","last_change":"2024-05-20T17:56:45.749725+0000","last_active":"2024-05-20T17:57:49.047534+0000","last_peered":"2024-05-20T17:57:49.047534+0000","last_clean":"2024-05-20T17:57:49.047534+0000","last_became_active":"2024-05-20T17:56:45.749392+0000","last_became_peered":"2024-05-20T17:56:45.749392+0000","last_unstale":"2024-05-20T17:57:49.047534+0000","last_undegraded":"2024-05-20T17:57:49.047534+0000","last_fullsized":"2024-05-20T17:57:49.047534+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:43:33.184192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.045895+0000","last_change":"2024-05-20T17:56:45.755191+0000","last_active":"2024-05-20T17:57:49.045895+0000","last_peered":"2024-05-20T17:57:49.045895+0000","last_clean":"2024-05-20T17:57:49.045895+0000","last_became_active":"2024-05-20T17:56:45.753524+0000","last_became_peered":"2024-05-20T17:56:45.753524+0000","last_unstale":"2024-05-20T17:57:49.045895+0000","last_undegraded":"2024-05-20T17:57:49.045895+0000","last_fullsized":"2024-05-20T17:57:49.045895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:11:23.099919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047705+0000","last_change":"2024-05-20T17:56:45.756291+0000","last_active":"2024-05-20T17:57:49.047705+0000","last_peered":"2024-05-20T17:57:49.047705+0000","last_clean":"2024-05-20T17:57:49.047705+0000","last_became_active":"2024-05-20T17:56:45.755910+0000","last_became_peered":"2024-05-20T17:56:45.755910+0000","last_unstale":"2024-05-20T17:57:49.047705+0000","last_undegraded":"2024-05-20T17:57:49.047705+0000","last_fullsized":"2024-05-20T17:57:49.047705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:37:36.941012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043930+0000","last_change":"2024-05-20T17:56:45.748756+0000","last_active":"2024-05-20T17:57:49.043930+0000","last_peered":"2024-05-20T17:57:49.043930+0000","last_clean":"2024-05-20T17:57:49.043930+0000","last_became_active":"2024-05-20T17:56:45.748406+0000","last_became_peered":"2024-05-20T17:56:45.748406+0000","last_unstale":"2024-05-20T17:57:49.043930+0000","last_undegraded":"2024-05-20T17:57:49.043930+0000","last_fullsized":"2024-05-20T17:57:49.043930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:16:36.527832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.051459+0000","last_change":"2024-05-20T17:57:49.051459+0000","last_active":"2024-05-20T17:57:49.051459+0000","last_peered":"2024-05-20T17:57:49.051459+0000","last_clean":"2024-05-20T17:57:49.051459+0000","last_became_active":"2024-05-20T17:57:49.050933+0000","last_became_peered":"2024-05-20T17:57:49.050933+0000","last_unstale":"2024-05-20T17:57:49.051459+0000","last_undegraded":"2024-05-20T17:57:49.051459+0000","last_fullsized":"2024-05-20T17:57:49.051459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:18.214102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047727+0000","last_change":"2024-05-20T17:56:45.757387+0000","last_active":"2024-05-20T17:57:49.047727+0000","last_peered":"2024-05-20T17:57:49.047727+0000","last_clean":"2024-05-20T17:57:49.047727+0000","last_became_active":"2024-05-20T17:56:45.756374+0000","last_became_peered":"2024-05-20T17:56:45.756374+0000","last_unstale":"2024-05-20T17:57:49.047727+0000","last_undegraded":"2024-05-20T17:57:49.047727+0000","last_fullsized":"2024-05-20T17:57:49.047727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:53:52.751019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.077064+0000","last_change":"2024-05-20T17:56:45.751933+0000","last_active":"2024-05-20T17:57:49.077064+0000","last_peered":"2024-05-20T17:57:49.077064+0000","last_clean":"2024-05-20T17:57:49.077064+0000","last_became_active":"2024-05-20T17:56:45.751414+0000","last_became_peered":"2024-05-20T17:56:45.751414+0000","last_unstale":"2024-05-20T17:57:49.077064+0000","last_undegraded":"2024-05-20T17:57:49.077064+0000","last_fullsized":"2024-05-20T17:57:49.077064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:56:52.974830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044617+0000","last_change":"2024-05-20T17:56:45.749511+0000","last_active":"2024-05-20T17:57:49.044617+0000","last_peered":"2024-05-20T17:57:49.044617+0000","last_clean":"2024-05-20T17:57:49.044617+0000","last_became_active":"2024-05-20T17:56:45.749211+0000","last_became_peered":"2024-05-20T17:56:45.749211+0000","last_unstale":"2024-05-20T17:57:49.044617+0000","last_undegraded":"2024-05-20T17:57:49.044617+0000","last_fullsized":"2024-05-20T17:57:49.044617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:15.633296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048160+0000","last_change":"2024-05-20T17:56:45.753743+0000","last_active":"2024-05-20T17:57:49.048160+0000","last_peered":"2024-05-20T17:57:49.048160+0000","last_clean":"2024-05-20T17:57:49.048160+0000","last_became_active":"2024-05-20T17:56:45.753378+0000","last_became_peered":"2024-05-20T17:56:45.753378+0000","last_unstale":"2024-05-20T17:57:49.048160+0000","last_undegraded":"2024-05-20T17:57:49.048160+0000","last_fullsized":"2024-05-20T17:57:49.048160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:54.563895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.045731+0000","last_change":"2024-05-20T17:56:45.754890+0000","last_active":"2024-05-20T17:57:49.045731+0000","last_peered":"2024-05-20T17:57:49.045731+0000","last_clean":"2024-05-20T17:57:49.045731+0000","last_became_active":"2024-05-20T17:56:45.754333+0000","last_became_peered":"2024-05-20T17:56:45.754333+0000","last_unstale":"2024-05-20T17:57:49.045731+0000","last_undegraded":"2024-05-20T17:57:49.045731+0000","last_fullsized":"2024-05-20T17:57:49.045731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:51.025516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048832+0000","last_change":"2024-05-20T17:56:45.745662+0000","last_active":"2024-05-20T17:57:49.048832+0000","last_peered":"2024-05-20T17:57:49.048832+0000","last_clean":"2024-05-20T17:57:49.048832+0000","last_became_active":"2024-05-20T17:56:45.745281+0000","last_became_peered":"2024-05-20T17:56:45.745281+0000","last_unstale":"2024-05-20T17:57:49.048832+0000","last_undegraded":"2024-05-20T17:57:49.048832+0000","last_fullsized":"2024-05-20T17:57:49.048832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:38:57.057301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047567+0000","last_change":"2024-05-20T17:56:45.749164+0000","last_active":"2024-05-20T17:57:49.047567+0000","last_peered":"2024-05-20T17:57:49.047567+0000","last_clean":"2024-05-20T17:57:49.047567+0000","last_became_active":"2024-05-20T17:56:45.748822+0000","last_became_peered":"2024-05-20T17:56:45.748822+0000","last_unstale":"2024-05-20T17:57:49.047567+0000","last_undegraded":"2024-05-20T17:57:49.047567+0000","last_fullsized":"2024-05-20T17:57:49.047567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:45:47.487116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047331+0000","last_change":"2024-05-20T17:56:45.751185+0000","last_active":"2024-05-20T17:57:49.047331+0000","last_peered":"2024-05-20T17:57:49.047331+0000","last_clean":"2024-05-20T17:57:49.047331+0000","last_became_active":"2024-05-20T17:56:45.750851+0000","last_became_peered":"2024-05-20T17:56:45.750851+0000","last_unstale":"2024-05-20T17:57:49.047331+0000","last_undegraded":"2024-05-20T17:57:49.047331+0000","last_fullsized":"2024-05-20T17:57:49.047331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:31:07.792717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076152+0000","last_change":"2024-05-20T17:56:45.748101+0000","last_active":"2024-05-20T17:57:49.076152+0000","last_peered":"2024-05-20T17:57:49.076152+0000","last_clean":"2024-05-20T17:57:49.076152+0000","last_became_active":"2024-05-20T17:56:45.747754+0000","last_became_peered":"2024-05-20T17:56:45.747754+0000","last_unstale":"2024-05-20T17:57:49.076152+0000","last_undegraded":"2024-05-20T17:57:49.076152+0000","last_fullsized":"2024-05-20T17:57:49.076152+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:24:54.702601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043420+0000","last_change":"2024-05-20T17:56:45.747497+0000","last_active":"2024-05-20T17:57:49.043420+0000","last_peered":"2024-05-20T17:57:49.043420+0000","last_clean":"2024-05-20T17:57:49.043420+0000","last_became_active":"2024-05-20T17:56:45.747139+0000","last_became_peered":"2024-05-20T17:56:45.747139+0000","last_unstale":"2024-05-20T17:57:49.043420+0000","last_undegraded":"2024-05-20T17:57:49.043420+0000","last_fullsized":"2024-05-20T17:57:49.043420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:39:13.102413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.045005+0000","last_change":"2024-05-20T17:56:45.746194+0000","last_active":"2024-05-20T17:57:49.045005+0000","last_peered":"2024-05-20T17:57:49.045005+0000","last_clean":"2024-05-20T17:57:49.045005+0000","last_became_active":"2024-05-20T17:56:45.745809+0000","last_became_peered":"2024-05-20T17:56:45.745809+0000","last_unstale":"2024-05-20T17:57:49.045005+0000","last_undegraded":"2024-05-20T17:57:49.045005+0000","last_fullsized":"2024-05-20T17:57:49.045005+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:26:29.133790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049613+0000","last_change":"2024-05-20T17:56:45.749918+0000","last_active":"2024-05-20T17:57:49.049613+0000","last_peered":"2024-05-20T17:57:49.049613+0000","last_clean":"2024-05-20T17:57:49.049613+0000","last_became_active":"2024-05-20T17:56:45.749560+0000","last_became_peered":"2024-05-20T17:56:45.749560+0000","last_unstale":"2024-05-20T17:57:49.049613+0000","last_undegraded":"2024-05-20T17:57:49.049613+0000","last_fullsized":"2024-05-20T17:57:49.049613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:33:12.737206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.082268+0000","last_change":"2024-05-20T17:57:49.082268+0000","last_active":"2024-05-20T17:57:49.082268+0000","last_peered":"2024-05-20T17:57:49.082268+0000","last_clean":"2024-05-20T17:57:49.082268+0000","last_became_active":"2024-05-20T17:57:49.078471+0000","last_became_peered":"2024-05-20T17:57:49.078471+0000","last_unstale":"2024-05-20T17:57:49.082268+0000","last_undegraded":"2024-05-20T17:57:49.082268+0000","last_fullsized":"2024-05-20T17:57:49.082268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:50:14.617477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048569+0000","last_change":"2024-05-20T17:56:45.758330+0000","last_active":"2024-05-20T17:57:49.048569+0000","last_peered":"2024-05-20T17:57:49.048569+0000","last_clean":"2024-05-20T17:57:49.048569+0000","last_became_active":"2024-05-20T17:56:45.755873+0000","last_became_peered":"2024-05-20T17:56:45.755873+0000","last_unstale":"2024-05-20T17:57:49.048569+0000","last_undegraded":"2024-05-20T17:57:49.048569+0000","last_fullsized":"2024-05-20T17:57:49.048569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:53:31.653497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048896+0000","last_change":"2024-05-20T17:56:44.728508+0000","last_active":"2024-05-20T17:57:49.048896+0000","last_peered":"2024-05-20T17:57:49.048896+0000","last_clean":"2024-05-20T17:57:49.048896+0000","last_became_active":"2024-05-20T17:56:44.727875+0000","last_became_peered":"2024-05-20T17:56:44.727875+0000","last_unstale":"2024-05-20T17:57:49.048896+0000","last_undegraded":"2024-05-20T17:57:49.048896+0000","last_fullsized":"2024-05-20T17:57:49.048896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:12:14.349039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048041+0000","last_change":"2024-05-20T17:56:45.750000+0000","last_active":"2024-05-20T17:57:49.048041+0000","last_peered":"2024-05-20T17:57:49.048041+0000","last_clean":"2024-05-20T17:57:49.048041+0000","last_became_active":"2024-05-20T17:56:45.749644+0000","last_became_peered":"2024-05-20T17:56:45.749644+0000","last_unstale":"2024-05-20T17:57:49.048041+0000","last_undegraded":"2024-05-20T17:57:49.048041+0000","last_fullsized":"2024-05-20T17:57:49.048041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:32:46.085081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048536+0000","last_change":"2024-05-20T17:56:44.722692+0000","last_active":"2024-05-20T17:57:49.048536+0000","last_peered":"2024-05-20T17:57:49.048536+0000","last_clean":"2024-05-20T17:57:49.048536+0000","last_became_active":"2024-05-20T17:56:44.722170+0000","last_became_peered":"2024-05-20T17:56:44.722170+0000","last_unstale":"2024-05-20T17:57:49.048536+0000","last_undegraded":"2024-05-20T17:57:49.048536+0000","last_fullsized":"2024-05-20T17:57:49.048536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:38:03.679562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076329+0000","last_change":"2024-05-20T17:56:45.751666+0000","last_active":"2024-05-20T17:57:49.076329+0000","last_peered":"2024-05-20T17:57:49.076329+0000","last_clean":"2024-05-20T17:57:49.076329+0000","last_became_active":"2024-05-20T17:56:45.751291+0000","last_became_peered":"2024-05-20T17:56:45.751291+0000","last_unstale":"2024-05-20T17:57:49.076329+0000","last_undegraded":"2024-05-20T17:57:49.076329+0000","last_fullsized":"2024-05-20T17:57:49.076329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:29:51.517363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048411+0000","last_change":"2024-05-20T17:56:44.721121+0000","last_active":"2024-05-20T17:57:49.048411+0000","last_peered":"2024-05-20T17:57:49.048411+0000","last_clean":"2024-05-20T17:57:49.048411+0000","last_became_active":"2024-05-20T17:56:44.720693+0000","last_became_peered":"2024-05-20T17:56:44.720693+0000","last_unstale":"2024-05-20T17:57:49.048411+0000","last_undegraded":"2024-05-20T17:57:49.048411+0000","last_fullsized":"2024-05-20T17:57:49.048411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:44.223828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047699+0000","last_change":"2024-05-20T17:56:45.745925+0000","last_active":"2024-05-20T17:57:49.047699+0000","last_peered":"2024-05-20T17:57:49.047699+0000","last_clean":"2024-05-20T17:57:49.047699+0000","last_became_active":"2024-05-20T17:56:45.745303+0000","last_became_peered":"2024-05-20T17:56:45.745303+0000","last_unstale":"2024-05-20T17:57:49.047699+0000","last_undegraded":"2024-05-20T17:57:49.047699+0000","last_fullsized":"2024-05-20T17:57:49.047699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:41:00.230681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047287+0000","last_change":"2024-05-20T17:56:44.719048+0000","last_active":"2024-05-20T17:57:49.047287+0000","last_peered":"2024-05-20T17:57:49.047287+0000","last_clean":"2024-05-20T17:57:49.047287+0000","last_became_active":"2024-05-20T17:56:44.718397+0000","last_became_peered":"2024-05-20T17:56:44.718397+0000","last_unstale":"2024-05-20T17:57:49.047287+0000","last_undegraded":"2024-05-20T17:57:49.047287+0000","last_fullsized":"2024-05-20T17:57:49.047287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:11:53.898226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047924+0000","last_change":"2024-05-20T17:56:45.752395+0000","last_active":"2024-05-20T17:57:49.047924+0000","last_peered":"2024-05-20T17:57:49.047924+0000","last_clean":"2024-05-20T17:57:49.047924+0000","last_became_active":"2024-05-20T17:56:45.751940+0000","last_became_peered":"2024-05-20T17:56:45.751940+0000","last_unstale":"2024-05-20T17:57:49.047924+0000","last_undegraded":"2024-05-20T17:57:49.047924+0000","last_fullsized":"2024-05-20T17:57:49.047924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:33:57.780042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043811+0000","last_change":"2024-05-20T17:56:44.714712+0000","last_active":"2024-05-20T17:57:49.043811+0000","last_peered":"2024-05-20T17:57:49.043811+0000","last_clean":"2024-05-20T17:57:49.043811+0000","last_became_active":"2024-05-20T17:56:44.714396+0000","last_became_peered":"2024-05-20T17:56:44.714396+0000","last_unstale":"2024-05-20T17:57:49.043811+0000","last_undegraded":"2024-05-20T17:57:49.043811+0000","last_fullsized":"2024-05-20T17:57:49.043811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:21:04.831289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047434+0000","last_change":"2024-05-20T17:56:45.758160+0000","last_active":"2024-05-20T17:57:49.047434+0000","last_peered":"2024-05-20T17:57:49.047434+0000","last_clean":"2024-05-20T17:57:49.047434+0000","last_became_active":"2024-05-20T17:56:45.757405+0000","last_became_peered":"2024-05-20T17:56:45.757405+0000","last_unstale":"2024-05-20T17:57:49.047434+0000","last_undegraded":"2024-05-20T17:57:49.047434+0000","last_fullsized":"2024-05-20T17:57:49.047434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:17.625246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048566+0000","last_change":"2024-05-20T17:56:44.716342+0000","last_active":"2024-05-20T17:57:49.048566+0000","last_peered":"2024-05-20T17:57:49.048566+0000","last_clean":"2024-05-20T17:57:49.048566+0000","last_became_active":"2024-05-20T17:56:44.716010+0000","last_became_peered":"2024-05-20T17:56:44.716010+0000","last_unstale":"2024-05-20T17:57:49.048566+0000","last_undegraded":"2024-05-20T17:57:49.048566+0000","last_fullsized":"2024-05-20T17:57:49.048566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:00:25.073312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047826+0000","last_change":"2024-05-20T17:56:45.755288+0000","last_active":"2024-05-20T17:57:49.047826+0000","last_peered":"2024-05-20T17:57:49.047826+0000","last_clean":"2024-05-20T17:57:49.047826+0000","last_became_active":"2024-05-20T17:56:45.754952+0000","last_became_peered":"2024-05-20T17:56:45.754952+0000","last_unstale":"2024-05-20T17:57:49.047826+0000","last_undegraded":"2024-05-20T17:57:49.047826+0000","last_fullsized":"2024-05-20T17:57:49.047826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:44:27.631722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.046896+0000","last_change":"2024-05-20T17:56:44.717887+0000","last_active":"2024-05-20T17:57:49.046896+0000","last_peered":"2024-05-20T17:57:49.046896+0000","last_clean":"2024-05-20T17:57:49.046896+0000","last_became_active":"2024-05-20T17:56:44.717581+0000","last_became_peered":"2024-05-20T17:56:44.717581+0000","last_unstale":"2024-05-20T17:57:49.046896+0000","last_undegraded":"2024-05-20T17:57:49.046896+0000","last_fullsized":"2024-05-20T17:57:49.046896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:42:54.703112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044734+0000","last_change":"2024-05-20T17:56:45.748448+0000","last_active":"2024-05-20T17:57:49.044734+0000","last_peered":"2024-05-20T17:57:49.044734+0000","last_clean":"2024-05-20T17:57:49.044734+0000","last_became_active":"2024-05-20T17:56:45.748016+0000","last_became_peered":"2024-05-20T17:56:45.748016+0000","last_unstale":"2024-05-20T17:57:49.044734+0000","last_undegraded":"2024-05-20T17:57:49.044734+0000","last_fullsized":"2024-05-20T17:57:49.044734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:31:39.350477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044413+0000","last_change":"2024-05-20T17:56:44.709483+0000","last_active":"2024-05-20T17:57:49.044413+0000","last_peered":"2024-05-20T17:57:49.044413+0000","last_clean":"2024-05-20T17:57:49.044413+0000","last_became_active":"2024-05-20T17:56:44.708649+0000","last_became_peered":"2024-05-20T17:56:44.708649+0000","last_unstale":"2024-05-20T17:57:49.044413+0000","last_undegraded":"2024-05-20T17:57:49.044413+0000","last_fullsized":"2024-05-20T17:57:49.044413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:31:46.580256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076645+0000","last_change":"2024-05-20T17:56:45.752279+0000","last_active":"2024-05-20T17:57:49.076645+0000","last_peered":"2024-05-20T17:57:49.076645+0000","last_clean":"2024-05-20T17:57:49.076645+0000","last_became_active":"2024-05-20T17:56:45.751790+0000","last_became_peered":"2024-05-20T17:56:45.751790+0000","last_unstale":"2024-05-20T17:57:49.076645+0000","last_undegraded":"2024-05-20T17:57:49.076645+0000","last_fullsized":"2024-05-20T17:57:49.076645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:06:38.113559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.077752+0000","last_change":"2024-05-20T17:57:49.077752+0000","last_active":"2024-05-20T17:57:49.077752+0000","last_peered":"2024-05-20T17:57:49.077752+0000","last_clean":"2024-05-20T17:57:49.077752+0000","last_became_active":"2024-05-20T17:57:49.077277+0000","last_became_peered":"2024-05-20T17:57:49.077277+0000","last_unstale":"2024-05-20T17:57:49.077752+0000","last_undegraded":"2024-05-20T17:57:49.077752+0000","last_fullsized":"2024-05-20T17:57:49.077752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:24.954772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.051693+0000","last_change":"2024-05-20T17:57:49.051693+0000","last_active":"2024-05-20T17:57:49.051693+0000","last_peered":"2024-05-20T17:57:49.051693+0000","last_clean":"2024-05-20T17:57:49.051693+0000","last_became_active":"2024-05-20T17:57:49.051215+0000","last_became_peered":"2024-05-20T17:57:49.051215+0000","last_unstale":"2024-05-20T17:57:49.051693+0000","last_undegraded":"2024-05-20T17:57:49.051693+0000","last_fullsized":"2024-05-20T17:57:49.051693+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:52:35.406908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047744+0000","last_change":"2024-05-20T17:56:44.709548+0000","last_active":"2024-05-20T17:57:49.047744+0000","last_peered":"2024-05-20T17:57:49.047744+0000","last_clean":"2024-05-20T17:57:49.047744+0000","last_became_active":"2024-05-20T17:56:44.709209+0000","last_became_peered":"2024-05-20T17:56:44.709209+0000","last_unstale":"2024-05-20T17:57:49.047744+0000","last_undegraded":"2024-05-20T17:57:49.047744+0000","last_fullsized":"2024-05-20T17:57:49.047744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:28:13.118336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076469+0000","last_change":"2024-05-20T17:56:45.755141+0000","last_active":"2024-05-20T17:57:49.076469+0000","last_peered":"2024-05-20T17:57:49.076469+0000","last_clean":"2024-05-20T17:57:49.076469+0000","last_became_active":"2024-05-20T17:56:45.754786+0000","last_became_peered":"2024-05-20T17:56:45.754786+0000","last_unstale":"2024-05-20T17:57:49.076469+0000","last_undegraded":"2024-05-20T17:57:49.076469+0000","last_fullsized":"2024-05-20T17:57:49.076469+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:45:21.044602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'26","reported_seq":188,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:56.097368+0000","last_change":"2024-05-20T17:56:44.713578+0000","last_active":"2024-05-20T17:57:56.097368+0000","last_peered":"2024-05-20T17:57:56.097368+0000","last_clean":"2024-05-20T17:57:56.097368+0000","last_became_active":"2024-05-20T17:56:44.713203+0000","last_became_peered":"2024-05-20T17:56:44.713203+0000","last_unstale":"2024-05-20T17:57:56.097368+0000","last_undegraded":"2024-05-20T17:57:56.097368+0000","last_fullsized":"2024-05-20T17:57:56.097368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:36:54.386036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":143,"num_read_kb":130,"num_write":26,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048157+0000","last_change":"2024-05-20T17:56:45.752023+0000","last_active":"2024-05-20T17:57:49.048157+0000","last_peered":"2024-05-20T17:57:49.048157+0000","last_clean":"2024-05-20T17:57:49.048157+0000","last_became_active":"2024-05-20T17:56:45.751672+0000","last_became_peered":"2024-05-20T17:56:45.751672+0000","last_unstale":"2024-05-20T17:57:49.048157+0000","last_undegraded":"2024-05-20T17:57:49.048157+0000","last_fullsized":"2024-05-20T17:57:49.048157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:23:49.127016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049088+0000","last_change":"2024-05-20T17:56:44.717336+0000","last_active":"2024-05-20T17:57:49.049088+0000","last_peered":"2024-05-20T17:57:49.049088+0000","last_clean":"2024-05-20T17:57:49.049088+0000","last_became_active":"2024-05-20T17:56:44.717016+0000","last_became_peered":"2024-05-20T17:56:44.717016+0000","last_unstale":"2024-05-20T17:57:49.049088+0000","last_undegraded":"2024-05-20T17:57:49.049088+0000","last_fullsized":"2024-05-20T17:57:49.049088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:50:46.433461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049004+0000","last_change":"2024-05-20T17:56:45.752411+0000","last_active":"2024-05-20T17:57:49.049004+0000","last_peered":"2024-05-20T17:57:49.049004+0000","last_clean":"2024-05-20T17:57:49.049004+0000","last_became_active":"2024-05-20T17:56:45.752078+0000","last_became_peered":"2024-05-20T17:56:45.752078+0000","last_unstale":"2024-05-20T17:57:49.049004+0000","last_undegraded":"2024-05-20T17:57:49.049004+0000","last_fullsized":"2024-05-20T17:57:49.049004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:22:15.560666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047164+0000","last_change":"2024-05-20T17:56:44.723262+0000","last_active":"2024-05-20T17:57:49.047164+0000","last_peered":"2024-05-20T17:57:49.047164+0000","last_clean":"2024-05-20T17:57:49.047164+0000","last_became_active":"2024-05-20T17:56:44.722865+0000","last_became_peered":"2024-05-20T17:56:44.722865+0000","last_unstale":"2024-05-20T17:57:49.047164+0000","last_undegraded":"2024-05-20T17:57:49.047164+0000","last_fullsized":"2024-05-20T17:57:49.047164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:15:02.820718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.052110+0000","last_change":"2024-05-20T17:57:49.052110+0000","last_active":"2024-05-20T17:57:49.052110+0000","last_peered":"2024-05-20T17:57:49.052110+0000","last_clean":"2024-05-20T17:57:49.052110+0000","last_became_active":"2024-05-20T17:57:49.051356+0000","last_became_peered":"2024-05-20T17:57:49.051356+0000","last_unstale":"2024-05-20T17:57:49.052110+0000","last_undegraded":"2024-05-20T17:57:49.052110+0000","last_fullsized":"2024-05-20T17:57:49.052110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:32.899615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.046141+0000","last_change":"2024-05-20T17:56:44.718396+0000","last_active":"2024-05-20T17:57:49.046141+0000","last_peered":"2024-05-20T17:57:49.046141+0000","last_clean":"2024-05-20T17:57:49.046141+0000","last_became_active":"2024-05-20T17:56:44.718031+0000","last_became_peered":"2024-05-20T17:56:44.718031+0000","last_unstale":"2024-05-20T17:57:49.046141+0000","last_undegraded":"2024-05-20T17:57:49.046141+0000","last_fullsized":"2024-05-20T17:57:49.046141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:42:40.955520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047471+0000","last_change":"2024-05-20T17:56:45.759302+0000","last_active":"2024-05-20T17:57:49.047471+0000","last_peered":"2024-05-20T17:57:49.047471+0000","last_clean":"2024-05-20T17:57:49.047471+0000","last_became_active":"2024-05-20T17:56:45.758889+0000","last_became_peered":"2024-05-20T17:56:45.758889+0000","last_unstale":"2024-05-20T17:57:49.047471+0000","last_undegraded":"2024-05-20T17:57:49.047471+0000","last_fullsized":"2024-05-20T17:57:49.047471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:40.685471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047716+0000","last_change":"2024-05-20T17:56:44.726502+0000","last_active":"2024-05-20T17:57:49.047716+0000","last_peered":"2024-05-20T17:57:49.047716+0000","last_clean":"2024-05-20T17:57:49.047716+0000","last_became_active":"2024-05-20T17:56:44.726168+0000","last_became_peered":"2024-05-20T17:56:44.726168+0000","last_unstale":"2024-05-20T17:57:49.047716+0000","last_undegraded":"2024-05-20T17:57:49.047716+0000","last_fullsized":"2024-05-20T17:57:49.047716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:45:14.523968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047696+0000","last_change":"2024-05-20T17:56:45.759355+0000","last_active":"2024-05-20T17:57:49.047696+0000","last_peered":"2024-05-20T17:57:49.047696+0000","last_clean":"2024-05-20T17:57:49.047696+0000","last_became_active":"2024-05-20T17:56:45.758920+0000","last_became_peered":"2024-05-20T17:56:45.758920+0000","last_unstale":"2024-05-20T17:57:49.047696+0000","last_undegraded":"2024-05-20T17:57:49.047696+0000","last_fullsized":"2024-05-20T17:57:49.047696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:46:02.372250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048617+0000","last_change":"2024-05-20T17:56:44.718507+0000","last_active":"2024-05-20T17:57:49.048617+0000","last_peered":"2024-05-20T17:57:49.048617+0000","last_clean":"2024-05-20T17:57:49.048617+0000","last_became_active":"2024-05-20T17:56:44.718188+0000","last_became_peered":"2024-05-20T17:56:44.718188+0000","last_unstale":"2024-05-20T17:57:49.048617+0000","last_undegraded":"2024-05-20T17:57:49.048617+0000","last_fullsized":"2024-05-20T17:57:49.048617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:43:59.792942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043201+0000","last_change":"2024-05-20T17:56:45.751361+0000","last_active":"2024-05-20T17:57:49.043201+0000","last_peered":"2024-05-20T17:57:49.043201+0000","last_clean":"2024-05-20T17:57:49.043201+0000","last_became_active":"2024-05-20T17:56:45.750032+0000","last_became_peered":"2024-05-20T17:56:45.750032+0000","last_unstale":"2024-05-20T17:57:49.043201+0000","last_undegraded":"2024-05-20T17:57:49.043201+0000","last_fullsized":"2024-05-20T17:57:49.043201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:01:42.729156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044973+0000","last_change":"2024-05-20T17:56:44.714404+0000","last_active":"2024-05-20T17:57:49.044973+0000","last_peered":"2024-05-20T17:57:49.044973+0000","last_clean":"2024-05-20T17:57:49.044973+0000","last_became_active":"2024-05-20T17:56:44.714072+0000","last_became_peered":"2024-05-20T17:56:44.714072+0000","last_unstale":"2024-05-20T17:57:49.044973+0000","last_undegraded":"2024-05-20T17:57:49.044973+0000","last_fullsized":"2024-05-20T17:57:49.044973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:39:53.983513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044438+0000","last_change":"2024-05-20T17:56:45.749785+0000","last_active":"2024-05-20T17:57:49.044438+0000","last_peered":"2024-05-20T17:57:49.044438+0000","last_clean":"2024-05-20T17:57:49.044438+0000","last_became_active":"2024-05-20T17:56:45.749457+0000","last_became_peered":"2024-05-20T17:56:45.749457+0000","last_unstale":"2024-05-20T17:57:49.044438+0000","last_undegraded":"2024-05-20T17:57:49.044438+0000","last_fullsized":"2024-05-20T17:57:49.044438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:53:05.136882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047562+0000","last_change":"2024-05-20T17:56:44.722787+0000","last_active":"2024-05-20T17:57:49.047562+0000","last_peered":"2024-05-20T17:57:49.047562+0000","last_clean":"2024-05-20T17:57:49.047562+0000","last_became_active":"2024-05-20T17:56:44.722214+0000","last_became_peered":"2024-05-20T17:56:44.722214+0000","last_unstale":"2024-05-20T17:57:49.047562+0000","last_undegraded":"2024-05-20T17:57:49.047562+0000","last_fullsized":"2024-05-20T17:57:49.047562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:40:16.705718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044275+0000","last_change":"2024-05-20T17:56:45.757515+0000","last_active":"2024-05-20T17:57:49.044275+0000","last_peered":"2024-05-20T17:57:49.044275+0000","last_clean":"2024-05-20T17:57:49.044275+0000","last_became_active":"2024-05-20T17:56:45.757174+0000","last_became_peered":"2024-05-20T17:56:45.757174+0000","last_unstale":"2024-05-20T17:57:49.044275+0000","last_undegraded":"2024-05-20T17:57:49.044275+0000","last_fullsized":"2024-05-20T17:57:49.044275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:18:14.154827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.052366+0000","last_change":"2024-05-20T17:57:49.051489+0000","last_active":"2024-05-20T17:57:49.052366+0000","last_peered":"2024-05-20T17:57:49.052366+0000","last_clean":"2024-05-20T17:57:49.052366+0000","last_became_active":"2024-05-20T17:57:49.051070+0000","last_became_peered":"2024-05-20T17:57:49.051070+0000","last_unstale":"2024-05-20T17:57:49.052366+0000","last_undegraded":"2024-05-20T17:57:49.052366+0000","last_fullsized":"2024-05-20T17:57:49.052366+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:18.745185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048453+0000","last_change":"2024-05-20T17:56:44.724642+0000","last_active":"2024-05-20T17:57:49.048453+0000","last_peered":"2024-05-20T17:57:49.048453+0000","last_clean":"2024-05-20T17:57:49.048453+0000","last_became_active":"2024-05-20T17:56:44.724311+0000","last_became_peered":"2024-05-20T17:56:44.724311+0000","last_unstale":"2024-05-20T17:57:49.048453+0000","last_undegraded":"2024-05-20T17:57:49.048453+0000","last_fullsized":"2024-05-20T17:57:49.048453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:47:35.336236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047978+0000","last_change":"2024-05-20T17:56:45.758466+0000","last_active":"2024-05-20T17:57:49.047978+0000","last_peered":"2024-05-20T17:57:49.047978+0000","last_clean":"2024-05-20T17:57:49.047978+0000","last_became_active":"2024-05-20T17:56:45.758126+0000","last_became_peered":"2024-05-20T17:56:45.758126+0000","last_unstale":"2024-05-20T17:57:49.047978+0000","last_undegraded":"2024-05-20T17:57:49.047978+0000","last_fullsized":"2024-05-20T17:57:49.047978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:36.830828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.045642+0000","last_change":"2024-05-20T17:56:44.711173+0000","last_active":"2024-05-20T17:57:49.045642+0000","last_peered":"2024-05-20T17:57:49.045642+0000","last_clean":"2024-05-20T17:57:49.045642+0000","last_became_active":"2024-05-20T17:56:44.710778+0000","last_became_peered":"2024-05-20T17:56:44.710778+0000","last_unstale":"2024-05-20T17:57:49.045642+0000","last_undegraded":"2024-05-20T17:57:49.045642+0000","last_fullsized":"2024-05-20T17:57:49.045642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:20:14.150346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047587+0000","last_change":"2024-05-20T17:55:34.871483+0000","last_active":"2024-05-20T17:57:49.047587+0000","last_peered":"2024-05-20T17:57:49.047587+0000","last_clean":"2024-05-20T17:57:49.047587+0000","last_became_active":"2024-05-20T17:55:34.871028+0000","last_became_peered":"2024-05-20T17:55:34.871028+0000","last_unstale":"2024-05-20T17:57:49.047587+0000","last_undegraded":"2024-05-20T17:57:49.047587+0000","last_fullsized":"2024-05-20T17:57:49.047587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_clean_scrub_stamp":"2024-05-20T17:52:08.599715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:29:40.159577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048877+0000","last_change":"2024-05-20T17:56:45.747725+0000","last_active":"2024-05-20T17:57:49.048877+0000","last_peered":"2024-05-20T17:57:49.048877+0000","last_clean":"2024-05-20T17:57:49.048877+0000","last_became_active":"2024-05-20T17:56:45.747232+0000","last_became_peered":"2024-05-20T17:56:45.747232+0000","last_unstale":"2024-05-20T17:57:49.048877+0000","last_undegraded":"2024-05-20T17:57:49.048877+0000","last_fullsized":"2024-05-20T17:57:49.048877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:09:26.589905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047923+0000","last_change":"2024-05-20T17:56:44.721004+0000","last_active":"2024-05-20T17:57:49.047923+0000","last_peered":"2024-05-20T17:57:49.047923+0000","last_clean":"2024-05-20T17:57:49.047923+0000","last_became_active":"2024-05-20T17:56:44.720620+0000","last_became_peered":"2024-05-20T17:56:44.720620+0000","last_unstale":"2024-05-20T17:57:49.047923+0000","last_undegraded":"2024-05-20T17:57:49.047923+0000","last_fullsized":"2024-05-20T17:57:49.047923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:42:14.710663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048325+0000","last_change":"2024-05-20T17:56:44.714754+0000","last_active":"2024-05-20T17:57:49.048325+0000","last_peered":"2024-05-20T17:57:49.048325+0000","last_clean":"2024-05-20T17:57:49.048325+0000","last_became_active":"2024-05-20T17:56:44.714378+0000","last_became_peered":"2024-05-20T17:56:44.714378+0000","last_unstale":"2024-05-20T17:57:49.048325+0000","last_undegraded":"2024-05-20T17:57:49.048325+0000","last_fullsized":"2024-05-20T17:57:49.048325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:46:53.969960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048822+0000","last_change":"2024-05-20T17:56:45.742415+0000","last_active":"2024-05-20T17:57:49.048822+0000","last_peered":"2024-05-20T17:57:49.048822+0000","last_clean":"2024-05-20T17:57:49.048822+0000","last_became_active":"2024-05-20T17:56:45.741941+0000","last_became_peered":"2024-05-20T17:56:45.741941+0000","last_unstale":"2024-05-20T17:57:49.048822+0000","last_undegraded":"2024-05-20T17:57:49.048822+0000","last_fullsized":"2024-05-20T17:57:49.048822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:20:03.191021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.045123+0000","last_change":"2024-05-20T17:56:44.724361+0000","last_active":"2024-05-20T17:57:49.045123+0000","last_peered":"2024-05-20T17:57:49.045123+0000","last_clean":"2024-05-20T17:57:49.045123+0000","last_became_active":"2024-05-20T17:56:44.724033+0000","last_became_peered":"2024-05-20T17:56:44.724033+0000","last_unstale":"2024-05-20T17:57:49.045123+0000","last_undegraded":"2024-05-20T17:57:49.045123+0000","last_fullsized":"2024-05-20T17:57:49.045123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:10:13.756606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.052049+0000","last_change":"2024-05-20T17:57:49.051338+0000","last_active":"2024-05-20T17:57:49.052049+0000","last_peered":"2024-05-20T17:57:49.052049+0000","last_clean":"2024-05-20T17:57:49.052049+0000","last_became_active":"2024-05-20T17:57:49.050618+0000","last_became_peered":"2024-05-20T17:57:49.050618+0000","last_unstale":"2024-05-20T17:57:49.052049+0000","last_undegraded":"2024-05-20T17:57:49.052049+0000","last_fullsized":"2024-05-20T17:57:49.052049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:02:09.846175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.046306+0000","last_change":"2024-05-20T17:56:44.716296+0000","last_active":"2024-05-20T17:57:49.046306+0000","last_peered":"2024-05-20T17:57:49.046306+0000","last_clean":"2024-05-20T17:57:49.046306+0000","last_became_active":"2024-05-20T17:56:44.715952+0000","last_became_peered":"2024-05-20T17:56:44.715952+0000","last_unstale":"2024-05-20T17:57:49.046306+0000","last_undegraded":"2024-05-20T17:57:49.046306+0000","last_fullsized":"2024-05-20T17:57:49.046306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:52:37.986399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048194+0000","last_change":"2024-05-20T17:56:45.751874+0000","last_active":"2024-05-20T17:57:49.048194+0000","last_peered":"2024-05-20T17:57:49.048194+0000","last_clean":"2024-05-20T17:57:49.048194+0000","last_became_active":"2024-05-20T17:56:45.751534+0000","last_became_peered":"2024-05-20T17:56:45.751534+0000","last_unstale":"2024-05-20T17:57:49.048194+0000","last_undegraded":"2024-05-20T17:57:49.048194+0000","last_fullsized":"2024-05-20T17:57:49.048194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:05:27.248115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076231+0000","last_change":"2024-05-20T17:56:44.724271+0000","last_active":"2024-05-20T17:57:49.076231+0000","last_peered":"2024-05-20T17:57:49.076231+0000","last_clean":"2024-05-20T17:57:49.076231+0000","last_became_active":"2024-05-20T17:56:44.723947+0000","last_became_peered":"2024-05-20T17:56:44.723947+0000","last_unstale":"2024-05-20T17:57:49.076231+0000","last_undegraded":"2024-05-20T17:57:49.076231+0000","last_fullsized":"2024-05-20T17:57:49.076231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:32:13.482059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047658+0000","last_change":"2024-05-20T17:56:45.751296+0000","last_active":"2024-05-20T17:57:49.047658+0000","last_peered":"2024-05-20T17:57:49.047658+0000","last_clean":"2024-05-20T17:57:49.047658+0000","last_became_active":"2024-05-20T17:56:45.748035+0000","last_became_peered":"2024-05-20T17:56:45.748035+0000","last_unstale":"2024-05-20T17:57:49.047658+0000","last_undegraded":"2024-05-20T17:57:49.047658+0000","last_fullsized":"2024-05-20T17:57:49.047658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:37:43.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047741+0000","last_change":"2024-05-20T17:56:44.720461+0000","last_active":"2024-05-20T17:57:49.047741+0000","last_peered":"2024-05-20T17:57:49.047741+0000","last_clean":"2024-05-20T17:57:49.047741+0000","last_became_active":"2024-05-20T17:56:44.720115+0000","last_became_peered":"2024-05-20T17:56:44.720115+0000","last_unstale":"2024-05-20T17:57:49.047741+0000","last_undegraded":"2024-05-20T17:57:49.047741+0000","last_fullsized":"2024-05-20T17:57:49.047741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:14:47.330193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.042771+0000","last_change":"2024-05-20T17:56:45.745025+0000","last_active":"2024-05-20T17:57:49.042771+0000","last_peered":"2024-05-20T17:57:49.042771+0000","last_clean":"2024-05-20T17:57:49.042771+0000","last_became_active":"2024-05-20T17:56:45.744626+0000","last_became_peered":"2024-05-20T17:56:45.744626+0000","last_unstale":"2024-05-20T17:57:49.042771+0000","last_undegraded":"2024-05-20T17:57:49.042771+0000","last_fullsized":"2024-05-20T17:57:49.042771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:08:51.052423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049081+0000","last_change":"2024-05-20T17:56:44.726099+0000","last_active":"2024-05-20T17:57:49.049081+0000","last_peered":"2024-05-20T17:57:49.049081+0000","last_clean":"2024-05-20T17:57:49.049081+0000","last_became_active":"2024-05-20T17:56:44.725714+0000","last_became_peered":"2024-05-20T17:56:44.725714+0000","last_unstale":"2024-05-20T17:57:49.049081+0000","last_undegraded":"2024-05-20T17:57:49.049081+0000","last_fullsized":"2024-05-20T17:57:49.049081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:51.326907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048179+0000","last_change":"2024-05-20T17:56:45.757376+0000","last_active":"2024-05-20T17:57:49.048179+0000","last_peered":"2024-05-20T17:57:49.048179+0000","last_clean":"2024-05-20T17:57:49.048179+0000","last_became_active":"2024-05-20T17:56:45.757053+0000","last_became_peered":"2024-05-20T17:56:45.757053+0000","last_unstale":"2024-05-20T17:57:49.048179+0000","last_undegraded":"2024-05-20T17:57:49.048179+0000","last_fullsized":"2024-05-20T17:57:49.048179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:50:27.727055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047850+0000","last_change":"2024-05-20T17:56:44.726341+0000","last_active":"2024-05-20T17:57:49.047850+0000","last_peered":"2024-05-20T17:57:49.047850+0000","last_clean":"2024-05-20T17:57:49.047850+0000","last_became_active":"2024-05-20T17:56:44.726008+0000","last_became_peered":"2024-05-20T17:56:44.726008+0000","last_unstale":"2024-05-20T17:57:49.047850+0000","last_undegraded":"2024-05-20T17:57:49.047850+0000","last_fullsized":"2024-05-20T17:57:49.047850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:25:07.520179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047791+0000","last_change":"2024-05-20T17:56:45.760469+0000","last_active":"2024-05-20T17:57:49.047791+0000","last_peered":"2024-05-20T17:57:49.047791+0000","last_clean":"2024-05-20T17:57:49.047791+0000","last_became_active":"2024-05-20T17:56:45.760100+0000","last_became_peered":"2024-05-20T17:56:45.760100+0000","last_unstale":"2024-05-20T17:57:49.047791+0000","last_undegraded":"2024-05-20T17:57:49.047791+0000","last_fullsized":"2024-05-20T17:57:49.047791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:54:06.309355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.051674+0000","last_change":"2024-05-20T17:57:49.051674+0000","last_active":"2024-05-20T17:57:49.051674+0000","last_peered":"2024-05-20T17:57:49.051674+0000","last_clean":"2024-05-20T17:57:49.051674+0000","last_became_active":"2024-05-20T17:57:49.051182+0000","last_became_peered":"2024-05-20T17:57:49.051182+0000","last_unstale":"2024-05-20T17:57:49.051674+0000","last_undegraded":"2024-05-20T17:57:49.051674+0000","last_fullsized":"2024-05-20T17:57:49.051674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:29:35.458339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048360+0000","last_change":"2024-05-20T17:56:45.757302+0000","last_active":"2024-05-20T17:57:49.048360+0000","last_peered":"2024-05-20T17:57:49.048360+0000","last_clean":"2024-05-20T17:57:49.048360+0000","last_became_active":"2024-05-20T17:56:45.754169+0000","last_became_peered":"2024-05-20T17:56:45.754169+0000","last_unstale":"2024-05-20T17:57:49.048360+0000","last_undegraded":"2024-05-20T17:57:49.048360+0000","last_fullsized":"2024-05-20T17:57:49.048360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:03:10.794342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047306+0000","last_change":"2024-05-20T17:56:44.724478+0000","last_active":"2024-05-20T17:57:49.047306+0000","last_peered":"2024-05-20T17:57:49.047306+0000","last_clean":"2024-05-20T17:57:49.047306+0000","last_became_active":"2024-05-20T17:56:44.723927+0000","last_became_peered":"2024-05-20T17:56:44.723927+0000","last_unstale":"2024-05-20T17:57:49.047306+0000","last_undegraded":"2024-05-20T17:57:49.047306+0000","last_fullsized":"2024-05-20T17:57:49.047306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:04:12.003957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048094+0000","last_change":"2024-05-20T17:56:45.758135+0000","last_active":"2024-05-20T17:57:49.048094+0000","last_peered":"2024-05-20T17:57:49.048094+0000","last_clean":"2024-05-20T17:57:49.048094+0000","last_became_active":"2024-05-20T17:56:45.757358+0000","last_became_peered":"2024-05-20T17:56:45.757358+0000","last_unstale":"2024-05-20T17:57:49.048094+0000","last_undegraded":"2024-05-20T17:57:49.048094+0000","last_fullsized":"2024-05-20T17:57:49.048094+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:13:06.435055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048817+0000","last_change":"2024-05-20T17:56:44.719822+0000","last_active":"2024-05-20T17:57:49.048817+0000","last_peered":"2024-05-20T17:57:49.048817+0000","last_clean":"2024-05-20T17:57:49.048817+0000","last_became_active":"2024-05-20T17:56:44.719454+0000","last_became_peered":"2024-05-20T17:56:44.719454+0000","last_unstale":"2024-05-20T17:57:49.048817+0000","last_undegraded":"2024-05-20T17:57:49.048817+0000","last_fullsized":"2024-05-20T17:57:49.048817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:03:21.048330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048426+0000","last_change":"2024-05-20T17:56:45.753510+0000","last_active":"2024-05-20T17:57:49.048426+0000","last_peered":"2024-05-20T17:57:49.048426+0000","last_clean":"2024-05-20T17:57:49.048426+0000","last_became_active":"2024-05-20T17:56:45.752718+0000","last_became_peered":"2024-05-20T17:56:45.752718+0000","last_unstale":"2024-05-20T17:57:49.048426+0000","last_undegraded":"2024-05-20T17:57:49.048426+0000","last_fullsized":"2024-05-20T17:57:49.048426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:05:44.677138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.042839+0000","last_change":"2024-05-20T17:56:45.752528+0000","last_active":"2024-05-20T17:57:49.042839+0000","last_peered":"2024-05-20T17:57:49.042839+0000","last_clean":"2024-05-20T17:57:49.042839+0000","last_became_active":"2024-05-20T17:56:45.752171+0000","last_became_peered":"2024-05-20T17:56:45.752171+0000","last_unstale":"2024-05-20T17:57:49.042839+0000","last_undegraded":"2024-05-20T17:57:49.042839+0000","last_fullsized":"2024-05-20T17:57:49.042839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:12:55.826122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048988+0000","last_change":"2024-05-20T17:56:45.749228+0000","last_active":"2024-05-20T17:57:49.048988+0000","last_peered":"2024-05-20T17:57:49.048988+0000","last_clean":"2024-05-20T17:57:49.048988+0000","last_became_active":"2024-05-20T17:56:45.748886+0000","last_became_peered":"2024-05-20T17:56:45.748886+0000","last_unstale":"2024-05-20T17:57:49.048988+0000","last_undegraded":"2024-05-20T17:57:49.048988+0000","last_fullsized":"2024-05-20T17:57:49.048988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:23:10.569720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048826+0000","last_change":"2024-05-20T17:56:45.751105+0000","last_active":"2024-05-20T17:57:49.048826+0000","last_peered":"2024-05-20T17:57:49.048826+0000","last_clean":"2024-05-20T17:57:49.048826+0000","last_became_active":"2024-05-20T17:56:45.750754+0000","last_became_peered":"2024-05-20T17:56:45.750754+0000","last_unstale":"2024-05-20T17:57:49.048826+0000","last_undegraded":"2024-05-20T17:57:49.048826+0000","last_fullsized":"2024-05-20T17:57:49.048826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:57:48.327428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.050574+0000","last_change":"2024-05-20T17:57:49.050574+0000","last_active":"2024-05-20T17:57:49.050574+0000","last_peered":"2024-05-20T17:57:49.050574+0000","last_clean":"2024-05-20T17:57:49.050574+0000","last_became_active":"2024-05-20T17:57:49.050037+0000","last_became_peered":"2024-05-20T17:57:49.050037+0000","last_unstale":"2024-05-20T17:57:49.050574+0000","last_undegraded":"2024-05-20T17:57:49.050574+0000","last_fullsized":"2024-05-20T17:57:49.050574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:01:56.453842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048581+0000","last_change":"2024-05-20T17:56:45.746122+0000","last_active":"2024-05-20T17:57:49.048581+0000","last_peered":"2024-05-20T17:57:49.048581+0000","last_clean":"2024-05-20T17:57:49.048581+0000","last_became_active":"2024-05-20T17:56:45.745773+0000","last_became_peered":"2024-05-20T17:56:45.745773+0000","last_unstale":"2024-05-20T17:57:49.048581+0000","last_undegraded":"2024-05-20T17:57:49.048581+0000","last_fullsized":"2024-05-20T17:57:49.048581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:21:27.909974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043155+0000","last_change":"2024-05-20T17:56:45.755018+0000","last_active":"2024-05-20T17:57:49.043155+0000","last_peered":"2024-05-20T17:57:49.043155+0000","last_clean":"2024-05-20T17:57:49.043155+0000","last_became_active":"2024-05-20T17:56:45.754615+0000","last_became_peered":"2024-05-20T17:56:45.754615+0000","last_unstale":"2024-05-20T17:57:49.043155+0000","last_undegraded":"2024-05-20T17:57:49.043155+0000","last_fullsized":"2024-05-20T17:57:49.043155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:36:39.100566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049568+0000","last_change":"2024-05-20T17:56:45.751091+0000","last_active":"2024-05-20T17:57:49.049568+0000","last_peered":"2024-05-20T17:57:49.049568+0000","last_clean":"2024-05-20T17:57:49.049568+0000","last_became_active":"2024-05-20T17:56:45.748573+0000","last_became_peered":"2024-05-20T17:56:45.748573+0000","last_unstale":"2024-05-20T17:57:49.049568+0000","last_undegraded":"2024-05-20T17:57:49.049568+0000","last_fullsized":"2024-05-20T17:57:49.049568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:11:16.140967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049006+0000","last_change":"2024-05-20T17:56:45.756373+0000","last_active":"2024-05-20T17:57:49.049006+0000","last_peered":"2024-05-20T17:57:49.049006+0000","last_clean":"2024-05-20T17:57:49.049006+0000","last_became_active":"2024-05-20T17:56:45.755993+0000","last_became_peered":"2024-05-20T17:56:45.755993+0000","last_unstale":"2024-05-20T17:57:49.049006+0000","last_undegraded":"2024-05-20T17:57:49.049006+0000","last_fullsized":"2024-05-20T17:57:49.049006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:34:44.491341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.046975+0000","last_change":"2024-05-20T17:56:45.751866+0000","last_active":"2024-05-20T17:57:49.046975+0000","last_peered":"2024-05-20T17:57:49.046975+0000","last_clean":"2024-05-20T17:57:49.046975+0000","last_became_active":"2024-05-20T17:56:45.751518+0000","last_became_peered":"2024-05-20T17:56:45.751518+0000","last_unstale":"2024-05-20T17:57:49.046975+0000","last_undegraded":"2024-05-20T17:57:49.046975+0000","last_fullsized":"2024-05-20T17:57:49.046975+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:38:18.736177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047633+0000","last_change":"2024-05-20T17:56:45.749903+0000","last_active":"2024-05-20T17:57:49.047633+0000","last_peered":"2024-05-20T17:57:49.047633+0000","last_clean":"2024-05-20T17:57:49.047633+0000","last_became_active":"2024-05-20T17:56:45.749500+0000","last_became_peered":"2024-05-20T17:56:45.749500+0000","last_unstale":"2024-05-20T17:57:49.047633+0000","last_undegraded":"2024-05-20T17:57:49.047633+0000","last_fullsized":"2024-05-20T17:57:49.047633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:39:34.875111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076415+0000","last_change":"2024-05-20T17:56:45.747370+0000","last_active":"2024-05-20T17:57:49.076415+0000","last_peered":"2024-05-20T17:57:49.076415+0000","last_clean":"2024-05-20T17:57:49.076415+0000","last_became_active":"2024-05-20T17:56:45.747036+0000","last_became_peered":"2024-05-20T17:56:45.747036+0000","last_unstale":"2024-05-20T17:57:49.076415+0000","last_undegraded":"2024-05-20T17:57:49.076415+0000","last_fullsized":"2024-05-20T17:57:49.076415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:02:02.220522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048964+0000","last_change":"2024-05-20T17:56:45.752762+0000","last_active":"2024-05-20T17:57:49.048964+0000","last_peered":"2024-05-20T17:57:49.048964+0000","last_clean":"2024-05-20T17:57:49.048964+0000","last_became_active":"2024-05-20T17:56:45.752345+0000","last_became_peered":"2024-05-20T17:56:45.752345+0000","last_unstale":"2024-05-20T17:57:49.048964+0000","last_undegraded":"2024-05-20T17:57:49.048964+0000","last_fullsized":"2024-05-20T17:57:49.048964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:14:49.861456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.042837+0000","last_change":"2024-05-20T17:56:45.750229+0000","last_active":"2024-05-20T17:57:49.042837+0000","last_peered":"2024-05-20T17:57:49.042837+0000","last_clean":"2024-05-20T17:57:49.042837+0000","last_became_active":"2024-05-20T17:56:45.749872+0000","last_became_peered":"2024-05-20T17:56:45.749872+0000","last_unstale":"2024-05-20T17:57:49.042837+0000","last_undegraded":"2024-05-20T17:57:49.042837+0000","last_fullsized":"2024-05-20T17:57:49.042837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:56:58.015339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280272,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61844,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":149,"num_read_kb":136,"num_write":75,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":185532,"data_compressed":5580,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":950272,"data_stored":462373,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612641,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":1114098,"data_compressed":43326,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808870,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":462955,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070508,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":462395,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332144,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":1053440,"data_compressed":41466,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495670,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":462490,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":524001,"data_compressed":7677,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":462463,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":523453,"data_compressed":7677,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":462711,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920832,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":462395,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757306,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":1053040,"data_compressed":41466,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61628,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61080,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61240,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-20T17:58:01.508 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph pg dump --format=json 2024-05-20T17:58:01.756 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:58:02.137 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:01 smithi129 ceph-mon[32029]: from='client.14979 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:02.138 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:01 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:58:02.138 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:01 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:58:02.138 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:01 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:02.138 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:01 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:02.138 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:01 smithi129 ceph-mon[32029]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:58:02.138 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:01 smithi129 ceph-mon[32029]: pgmap v325: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 48 B/s, 0 objects/s recovering 2024-05-20T17:58:02.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:01 smithi154 ceph-mon[36482]: from='client.14979 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:02.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:01 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:58:02.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:01 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:58:02.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:01 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:02.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:01 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:02.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:01 smithi154 ceph-mon[36482]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:58:02.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:01 smithi154 ceph-mon[36482]: pgmap v325: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 48 B/s, 0 objects/s recovering 2024-05-20T17:58:02.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:01 smithi204 ceph-mon[37076]: from='client.14979 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:02.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:01 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:58:02.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:01 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:58:02.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:01 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:02.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:01 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:02.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:01 smithi204 ceph-mon[37076]: from='mgr.14150 172.21.15.129:0/1920557668' entity='mgr.x' 2024-05-20T17:58:02.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:01 smithi204 ceph-mon[37076]: pgmap v325: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 48 B/s, 0 objects/s recovering 2024-05-20T17:58:03.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.442+0000 7f5c27815700 1 -- 172.21.15.129:0/1741886074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c201075c0 msgr2=0x7f5c2010ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:03.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.442+0000 7f5c27815700 1 --2- 172.21.15.129:0/1741886074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c201075c0 0x7f5c2010ef20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5c1800b0d0 tx=0x7f5c1800b3e0 comp rx=0 tx=0).stop 2024-05-20T17:58:03.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.442+0000 7f5c27815700 1 -- 172.21.15.129:0/1741886074 shutdown_connections 2024-05-20T17:58:03.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.442+0000 7f5c27815700 1 --2- 172.21.15.129:0/1741886074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c20102340 0x7f5c20102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:03.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.442+0000 7f5c27815700 1 --2- 172.21.15.129:0/1741886074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c201075c0 0x7f5c2010ef20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:03.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.442+0000 7f5c27815700 1 --2- 172.21.15.129:0/1741886074 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c20106c70 0x7f5c20107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:03.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.442+0000 7f5c27815700 1 -- 172.21.15.129:0/1741886074 >> 172.21.15.129:0/1741886074 conn(0x7f5c200fdc70 msgr2=0x7f5c20100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:03.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.442+0000 7f5c27815700 1 -- 172.21.15.129:0/1741886074 shutdown_connections 2024-05-20T17:58:03.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.442+0000 7f5c27815700 1 -- 172.21.15.129:0/1741886074 wait complete. 2024-05-20T17:58:03.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c27815700 1 Processor -- start 2024-05-20T17:58:03.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c27815700 1 -- start start 2024-05-20T17:58:03.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c27815700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c20102340 0x7f5c2019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:03.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c27815700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c20106c70 0x7f5c2019b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:03.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c27815700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c201075c0 0x7f5c201a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:03.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c27815700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5c20112360 con 0x7f5c20102340 2024-05-20T17:58:03.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c27815700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5c201121e0 con 0x7f5c20106c70 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c27815700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5c201124e0 con 0x7f5c201075c0 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c255b1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c20102340 0x7f5c2019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c255b1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c20102340 0x7f5c2019ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49854/0 (socket says 172.21.15.129:49854) 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c255b1700 1 -- 172.21.15.129:0/3148586611 learned_addr learned my addr 172.21.15.129:0/3148586611 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c25db2700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c201075c0 0x7f5c201a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c24db0700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c20106c70 0x7f5c2019b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c255b1700 1 -- 172.21.15.129:0/3148586611 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c201075c0 msgr2=0x7f5c201a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c255b1700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c201075c0 0x7f5c201a5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c255b1700 1 -- 172.21.15.129:0/3148586611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c20106c70 msgr2=0x7f5c2019b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c255b1700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c20106c70 0x7f5c2019b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c255b1700 1 -- 172.21.15.129:0/3148586611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2019b910 con 0x7f5c20102340 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.443+0000 7f5c24db0700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c20106c70 0x7f5c2019b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:03.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.444+0000 7f5c255b1700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c20102340 0x7f5c2019ab20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5c1000d950 tx=0x7f5c1000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:03.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.444+0000 7f5c167fc700 1 -- 172.21.15.129:0/3148586611 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c10016070 con 0x7f5c20102340 2024-05-20T17:58:03.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.444+0000 7f5c167fc700 1 -- 172.21.15.129:0/3148586611 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c10009e20 con 0x7f5c20102340 2024-05-20T17:58:03.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.444+0000 7f5c27815700 1 -- 172.21.15.129:0/3148586611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c201a6200 con 0x7f5c20102340 2024-05-20T17:58:03.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.444+0000 7f5c167fc700 1 -- 172.21.15.129:0/3148586611 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1000f620 con 0x7f5c20102340 2024-05-20T17:58:03.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.444+0000 7f5c27815700 1 -- 172.21.15.129:0/3148586611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c201a6710 con 0x7f5c20102340 2024-05-20T17:58:03.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.445+0000 7f5c27815700 1 -- 172.21.15.129:0/3148586611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c201037f0 con 0x7f5c20102340 2024-05-20T17:58:03.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.446+0000 7f5c167fc700 1 -- 172.21.15.129:0/3148586611 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f5c10019070 con 0x7f5c20102340 2024-05-20T17:58:03.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.447+0000 7f5c167fc700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5c0c0a2cb0 0x7f5c0c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:03.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.447+0000 7f5c24db0700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5c0c0a2cb0 0x7f5c0c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:03.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.447+0000 7f5c167fc700 1 -- 172.21.15.129:0/3148586611 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f5c100cee10 con 0x7f5c20102340 2024-05-20T17:58:03.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.447+0000 7f5c24db0700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5c0c0a2cb0 0x7f5c0c0a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5c201017e0 tx=0x7f5c1c009550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:03.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.450+0000 7f5c167fc700 1 -- 172.21.15.129:0/3148586611 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5c1009bc00 con 0x7f5c20102340 2024-05-20T17:58:03.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.642+0000 7f5c27815700 1 -- 172.21.15.129:0/3148586611 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5c20074ec0 con 0x7f5c0c0a2cb0 2024-05-20T17:58:03.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.649+0000 7f5c167fc700 1 -- 172.21.15.129:0/3148586611 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258934 (secure 0 0 0) 0x7f5c20074ec0 con 0x7f5c0c0a2cb0 2024-05-20T17:58:03.654 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:03.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 -- 172.21.15.129:0/3148586611 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5c0c0a2cb0 msgr2=0x7f5c0c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:03.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5c0c0a2cb0 0x7f5c0c0a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5c201017e0 tx=0x7f5c1c009550 comp rx=0 tx=0).stop 2024-05-20T17:58:03.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 -- 172.21.15.129:0/3148586611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c20102340 msgr2=0x7f5c2019ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:03.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c20102340 0x7f5c2019ab20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5c1000d950 tx=0x7f5c1000dc60 comp rx=0 tx=0).stop 2024-05-20T17:58:03.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 -- 172.21.15.129:0/3148586611 shutdown_connections 2024-05-20T17:58:03.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5c0c0a2cb0 0x7f5c0c0a51c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:03.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c201075c0 0x7f5c201a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:03.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c20102340 0x7f5c2019ab20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:03.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 --2- 172.21.15.129:0/3148586611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c20106c70 0x7f5c2019b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:03.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 -- 172.21.15.129:0/3148586611 >> 172.21.15.129:0/3148586611 conn(0x7f5c200fdc70 msgr2=0x7f5c20106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:03.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 -- 172.21.15.129:0/3148586611 shutdown_connections 2024-05-20T17:58:03.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:03.652+0000 7f5c27815700 1 -- 172.21.15.129:0/3148586611 wait complete. 2024-05-20T17:58:03.657 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-05-20T17:58:03.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:03 smithi129 ceph-mon[32029]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:03.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:03 smithi129 ceph-mon[32029]: pgmap v326: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:04.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:03 smithi154 ceph-mon[36482]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:04.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:03 smithi154 ceph-mon[36482]: pgmap v326: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:04.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:03 smithi204 ceph-mon[37076]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:04.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:03 smithi204 ceph-mon[37076]: pgmap v326: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:04.312 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-05-20T17:58:01.726843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652446,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":295,"num_read_kb":260,"num_write":258,"num_write_kb":3023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":203,"ondisk_log_size":203,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":364948,"kb_used_data":11556,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576876,"statfs":{"total":1151940427776,"available":1151566721024,"internally_reserved":0,"allocated":11833344,"data_stored":7505814,"data_compressed":182331,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003450"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043221+0000","last_change":"2024-05-20T17:56:45.746793+0000","last_active":"2024-05-20T17:57:49.043221+0000","last_peered":"2024-05-20T17:57:49.043221+0000","last_clean":"2024-05-20T17:57:49.043221+0000","last_became_active":"2024-05-20T17:56:45.746244+0000","last_became_peered":"2024-05-20T17:56:45.746244+0000","last_unstale":"2024-05-20T17:57:49.043221+0000","last_undegraded":"2024-05-20T17:57:49.043221+0000","last_fullsized":"2024-05-20T17:57:49.043221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:37:03.703599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048122+0000","last_change":"2024-05-20T17:56:45.744538+0000","last_active":"2024-05-20T17:57:49.048122+0000","last_peered":"2024-05-20T17:57:49.048122+0000","last_clean":"2024-05-20T17:57:49.048122+0000","last_became_active":"2024-05-20T17:56:45.744213+0000","last_became_peered":"2024-05-20T17:56:45.744213+0000","last_unstale":"2024-05-20T17:57:49.048122+0000","last_undegraded":"2024-05-20T17:57:49.048122+0000","last_fullsized":"2024-05-20T17:57:49.048122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:19:26.163003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047534+0000","last_change":"2024-05-20T17:56:45.749725+0000","last_active":"2024-05-20T17:57:49.047534+0000","last_peered":"2024-05-20T17:57:49.047534+0000","last_clean":"2024-05-20T17:57:49.047534+0000","last_became_active":"2024-05-20T17:56:45.749392+0000","last_became_peered":"2024-05-20T17:56:45.749392+0000","last_unstale":"2024-05-20T17:57:49.047534+0000","last_undegraded":"2024-05-20T17:57:49.047534+0000","last_fullsized":"2024-05-20T17:57:49.047534+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:43:33.184192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.045895+0000","last_change":"2024-05-20T17:56:45.755191+0000","last_active":"2024-05-20T17:57:49.045895+0000","last_peered":"2024-05-20T17:57:49.045895+0000","last_clean":"2024-05-20T17:57:49.045895+0000","last_became_active":"2024-05-20T17:56:45.753524+0000","last_became_peered":"2024-05-20T17:56:45.753524+0000","last_unstale":"2024-05-20T17:57:49.045895+0000","last_undegraded":"2024-05-20T17:57:49.045895+0000","last_fullsized":"2024-05-20T17:57:49.045895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:11:23.099919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047705+0000","last_change":"2024-05-20T17:56:45.756291+0000","last_active":"2024-05-20T17:57:49.047705+0000","last_peered":"2024-05-20T17:57:49.047705+0000","last_clean":"2024-05-20T17:57:49.047705+0000","last_became_active":"2024-05-20T17:56:45.755910+0000","last_became_peered":"2024-05-20T17:56:45.755910+0000","last_unstale":"2024-05-20T17:57:49.047705+0000","last_undegraded":"2024-05-20T17:57:49.047705+0000","last_fullsized":"2024-05-20T17:57:49.047705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:37:36.941012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043930+0000","last_change":"2024-05-20T17:56:45.748756+0000","last_active":"2024-05-20T17:57:49.043930+0000","last_peered":"2024-05-20T17:57:49.043930+0000","last_clean":"2024-05-20T17:57:49.043930+0000","last_became_active":"2024-05-20T17:56:45.748406+0000","last_became_peered":"2024-05-20T17:56:45.748406+0000","last_unstale":"2024-05-20T17:57:49.043930+0000","last_undegraded":"2024-05-20T17:57:49.043930+0000","last_fullsized":"2024-05-20T17:57:49.043930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:16:36.527832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.051459+0000","last_change":"2024-05-20T17:57:49.051459+0000","last_active":"2024-05-20T17:57:49.051459+0000","last_peered":"2024-05-20T17:57:49.051459+0000","last_clean":"2024-05-20T17:57:49.051459+0000","last_became_active":"2024-05-20T17:57:49.050933+0000","last_became_peered":"2024-05-20T17:57:49.050933+0000","last_unstale":"2024-05-20T17:57:49.051459+0000","last_undegraded":"2024-05-20T17:57:49.051459+0000","last_fullsized":"2024-05-20T17:57:49.051459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:18.214102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047727+0000","last_change":"2024-05-20T17:56:45.757387+0000","last_active":"2024-05-20T17:57:49.047727+0000","last_peered":"2024-05-20T17:57:49.047727+0000","last_clean":"2024-05-20T17:57:49.047727+0000","last_became_active":"2024-05-20T17:56:45.756374+0000","last_became_peered":"2024-05-20T17:56:45.756374+0000","last_unstale":"2024-05-20T17:57:49.047727+0000","last_undegraded":"2024-05-20T17:57:49.047727+0000","last_fullsized":"2024-05-20T17:57:49.047727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:53:52.751019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.077064+0000","last_change":"2024-05-20T17:56:45.751933+0000","last_active":"2024-05-20T17:57:49.077064+0000","last_peered":"2024-05-20T17:57:49.077064+0000","last_clean":"2024-05-20T17:57:49.077064+0000","last_became_active":"2024-05-20T17:56:45.751414+0000","last_became_peered":"2024-05-20T17:56:45.751414+0000","last_unstale":"2024-05-20T17:57:49.077064+0000","last_undegraded":"2024-05-20T17:57:49.077064+0000","last_fullsized":"2024-05-20T17:57:49.077064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:56:52.974830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044617+0000","last_change":"2024-05-20T17:56:45.749511+0000","last_active":"2024-05-20T17:57:49.044617+0000","last_peered":"2024-05-20T17:57:49.044617+0000","last_clean":"2024-05-20T17:57:49.044617+0000","last_became_active":"2024-05-20T17:56:45.749211+0000","last_became_peered":"2024-05-20T17:56:45.749211+0000","last_unstale":"2024-05-20T17:57:49.044617+0000","last_undegraded":"2024-05-20T17:57:49.044617+0000","last_fullsized":"2024-05-20T17:57:49.044617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:15.633296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048160+0000","last_change":"2024-05-20T17:56:45.753743+0000","last_active":"2024-05-20T17:57:49.048160+0000","last_peered":"2024-05-20T17:57:49.048160+0000","last_clean":"2024-05-20T17:57:49.048160+0000","last_became_active":"2024-05-20T17:56:45.753378+0000","last_became_peered":"2024-05-20T17:56:45.753378+0000","last_unstale":"2024-05-20T17:57:49.048160+0000","last_undegraded":"2024-05-20T17:57:49.048160+0000","last_fullsized":"2024-05-20T17:57:49.048160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:54.563895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.045731+0000","last_change":"2024-05-20T17:56:45.754890+0000","last_active":"2024-05-20T17:57:49.045731+0000","last_peered":"2024-05-20T17:57:49.045731+0000","last_clean":"2024-05-20T17:57:49.045731+0000","last_became_active":"2024-05-20T17:56:45.754333+0000","last_became_peered":"2024-05-20T17:56:45.754333+0000","last_unstale":"2024-05-20T17:57:49.045731+0000","last_undegraded":"2024-05-20T17:57:49.045731+0000","last_fullsized":"2024-05-20T17:57:49.045731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:51.025516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048832+0000","last_change":"2024-05-20T17:56:45.745662+0000","last_active":"2024-05-20T17:57:49.048832+0000","last_peered":"2024-05-20T17:57:49.048832+0000","last_clean":"2024-05-20T17:57:49.048832+0000","last_became_active":"2024-05-20T17:56:45.745281+0000","last_became_peered":"2024-05-20T17:56:45.745281+0000","last_unstale":"2024-05-20T17:57:49.048832+0000","last_undegraded":"2024-05-20T17:57:49.048832+0000","last_fullsized":"2024-05-20T17:57:49.048832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:38:57.057301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047567+0000","last_change":"2024-05-20T17:56:45.749164+0000","last_active":"2024-05-20T17:57:49.047567+0000","last_peered":"2024-05-20T17:57:49.047567+0000","last_clean":"2024-05-20T17:57:49.047567+0000","last_became_active":"2024-05-20T17:56:45.748822+0000","last_became_peered":"2024-05-20T17:56:45.748822+0000","last_unstale":"2024-05-20T17:57:49.047567+0000","last_undegraded":"2024-05-20T17:57:49.047567+0000","last_fullsized":"2024-05-20T17:57:49.047567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:45:47.487116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047331+0000","last_change":"2024-05-20T17:56:45.751185+0000","last_active":"2024-05-20T17:57:49.047331+0000","last_peered":"2024-05-20T17:57:49.047331+0000","last_clean":"2024-05-20T17:57:49.047331+0000","last_became_active":"2024-05-20T17:56:45.750851+0000","last_became_peered":"2024-05-20T17:56:45.750851+0000","last_unstale":"2024-05-20T17:57:49.047331+0000","last_undegraded":"2024-05-20T17:57:49.047331+0000","last_fullsized":"2024-05-20T17:57:49.047331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:31:07.792717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076152+0000","last_change":"2024-05-20T17:56:45.748101+0000","last_active":"2024-05-20T17:57:49.076152+0000","last_peered":"2024-05-20T17:57:49.076152+0000","last_clean":"2024-05-20T17:57:49.076152+0000","last_became_active":"2024-05-20T17:56:45.747754+0000","last_became_peered":"2024-05-20T17:56:45.747754+0000","last_unstale":"2024-05-20T17:57:49.076152+0000","last_undegraded":"2024-05-20T17:57:49.076152+0000","last_fullsized":"2024-05-20T17:57:49.076152+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:24:54.702601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043420+0000","last_change":"2024-05-20T17:56:45.747497+0000","last_active":"2024-05-20T17:57:49.043420+0000","last_peered":"2024-05-20T17:57:49.043420+0000","last_clean":"2024-05-20T17:57:49.043420+0000","last_became_active":"2024-05-20T17:56:45.747139+0000","last_became_peered":"2024-05-20T17:56:45.747139+0000","last_unstale":"2024-05-20T17:57:49.043420+0000","last_undegraded":"2024-05-20T17:57:49.043420+0000","last_fullsized":"2024-05-20T17:57:49.043420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:39:13.102413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.045005+0000","last_change":"2024-05-20T17:56:45.746194+0000","last_active":"2024-05-20T17:57:49.045005+0000","last_peered":"2024-05-20T17:57:49.045005+0000","last_clean":"2024-05-20T17:57:49.045005+0000","last_became_active":"2024-05-20T17:56:45.745809+0000","last_became_peered":"2024-05-20T17:56:45.745809+0000","last_unstale":"2024-05-20T17:57:49.045005+0000","last_undegraded":"2024-05-20T17:57:49.045005+0000","last_fullsized":"2024-05-20T17:57:49.045005+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:26:29.133790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049613+0000","last_change":"2024-05-20T17:56:45.749918+0000","last_active":"2024-05-20T17:57:49.049613+0000","last_peered":"2024-05-20T17:57:49.049613+0000","last_clean":"2024-05-20T17:57:49.049613+0000","last_became_active":"2024-05-20T17:56:45.749560+0000","last_became_peered":"2024-05-20T17:56:45.749560+0000","last_unstale":"2024-05-20T17:57:49.049613+0000","last_undegraded":"2024-05-20T17:57:49.049613+0000","last_fullsized":"2024-05-20T17:57:49.049613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:33:12.737206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.082268+0000","last_change":"2024-05-20T17:57:49.082268+0000","last_active":"2024-05-20T17:57:49.082268+0000","last_peered":"2024-05-20T17:57:49.082268+0000","last_clean":"2024-05-20T17:57:49.082268+0000","last_became_active":"2024-05-20T17:57:49.078471+0000","last_became_peered":"2024-05-20T17:57:49.078471+0000","last_unstale":"2024-05-20T17:57:49.082268+0000","last_undegraded":"2024-05-20T17:57:49.082268+0000","last_fullsized":"2024-05-20T17:57:49.082268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:50:14.617477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048569+0000","last_change":"2024-05-20T17:56:45.758330+0000","last_active":"2024-05-20T17:57:49.048569+0000","last_peered":"2024-05-20T17:57:49.048569+0000","last_clean":"2024-05-20T17:57:49.048569+0000","last_became_active":"2024-05-20T17:56:45.755873+0000","last_became_peered":"2024-05-20T17:56:45.755873+0000","last_unstale":"2024-05-20T17:57:49.048569+0000","last_undegraded":"2024-05-20T17:57:49.048569+0000","last_fullsized":"2024-05-20T17:57:49.048569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:53:31.653497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048896+0000","last_change":"2024-05-20T17:56:44.728508+0000","last_active":"2024-05-20T17:57:49.048896+0000","last_peered":"2024-05-20T17:57:49.048896+0000","last_clean":"2024-05-20T17:57:49.048896+0000","last_became_active":"2024-05-20T17:56:44.727875+0000","last_became_peered":"2024-05-20T17:56:44.727875+0000","last_unstale":"2024-05-20T17:57:49.048896+0000","last_undegraded":"2024-05-20T17:57:49.048896+0000","last_fullsized":"2024-05-20T17:57:49.048896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:12:14.349039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048041+0000","last_change":"2024-05-20T17:56:45.750000+0000","last_active":"2024-05-20T17:57:49.048041+0000","last_peered":"2024-05-20T17:57:49.048041+0000","last_clean":"2024-05-20T17:57:49.048041+0000","last_became_active":"2024-05-20T17:56:45.749644+0000","last_became_peered":"2024-05-20T17:56:45.749644+0000","last_unstale":"2024-05-20T17:57:49.048041+0000","last_undegraded":"2024-05-20T17:57:49.048041+0000","last_fullsized":"2024-05-20T17:57:49.048041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:32:46.085081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048536+0000","last_change":"2024-05-20T17:56:44.722692+0000","last_active":"2024-05-20T17:57:49.048536+0000","last_peered":"2024-05-20T17:57:49.048536+0000","last_clean":"2024-05-20T17:57:49.048536+0000","last_became_active":"2024-05-20T17:56:44.722170+0000","last_became_peered":"2024-05-20T17:56:44.722170+0000","last_unstale":"2024-05-20T17:57:49.048536+0000","last_undegraded":"2024-05-20T17:57:49.048536+0000","last_fullsized":"2024-05-20T17:57:49.048536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:38:03.679562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076329+0000","last_change":"2024-05-20T17:56:45.751666+0000","last_active":"2024-05-20T17:57:49.076329+0000","last_peered":"2024-05-20T17:57:49.076329+0000","last_clean":"2024-05-20T17:57:49.076329+0000","last_became_active":"2024-05-20T17:56:45.751291+0000","last_became_peered":"2024-05-20T17:56:45.751291+0000","last_unstale":"2024-05-20T17:57:49.076329+0000","last_undegraded":"2024-05-20T17:57:49.076329+0000","last_fullsized":"2024-05-20T17:57:49.076329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:29:51.517363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048411+0000","last_change":"2024-05-20T17:56:44.721121+0000","last_active":"2024-05-20T17:57:49.048411+0000","last_peered":"2024-05-20T17:57:49.048411+0000","last_clean":"2024-05-20T17:57:49.048411+0000","last_became_active":"2024-05-20T17:56:44.720693+0000","last_became_peered":"2024-05-20T17:56:44.720693+0000","last_unstale":"2024-05-20T17:57:49.048411+0000","last_undegraded":"2024-05-20T17:57:49.048411+0000","last_fullsized":"2024-05-20T17:57:49.048411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:44.223828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047699+0000","last_change":"2024-05-20T17:56:45.745925+0000","last_active":"2024-05-20T17:57:49.047699+0000","last_peered":"2024-05-20T17:57:49.047699+0000","last_clean":"2024-05-20T17:57:49.047699+0000","last_became_active":"2024-05-20T17:56:45.745303+0000","last_became_peered":"2024-05-20T17:56:45.745303+0000","last_unstale":"2024-05-20T17:57:49.047699+0000","last_undegraded":"2024-05-20T17:57:49.047699+0000","last_fullsized":"2024-05-20T17:57:49.047699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:41:00.230681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047287+0000","last_change":"2024-05-20T17:56:44.719048+0000","last_active":"2024-05-20T17:57:49.047287+0000","last_peered":"2024-05-20T17:57:49.047287+0000","last_clean":"2024-05-20T17:57:49.047287+0000","last_became_active":"2024-05-20T17:56:44.718397+0000","last_became_peered":"2024-05-20T17:56:44.718397+0000","last_unstale":"2024-05-20T17:57:49.047287+0000","last_undegraded":"2024-05-20T17:57:49.047287+0000","last_fullsized":"2024-05-20T17:57:49.047287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:11:53.898226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047924+0000","last_change":"2024-05-20T17:56:45.752395+0000","last_active":"2024-05-20T17:57:49.047924+0000","last_peered":"2024-05-20T17:57:49.047924+0000","last_clean":"2024-05-20T17:57:49.047924+0000","last_became_active":"2024-05-20T17:56:45.751940+0000","last_became_peered":"2024-05-20T17:56:45.751940+0000","last_unstale":"2024-05-20T17:57:49.047924+0000","last_undegraded":"2024-05-20T17:57:49.047924+0000","last_fullsized":"2024-05-20T17:57:49.047924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:33:57.780042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043811+0000","last_change":"2024-05-20T17:56:44.714712+0000","last_active":"2024-05-20T17:57:49.043811+0000","last_peered":"2024-05-20T17:57:49.043811+0000","last_clean":"2024-05-20T17:57:49.043811+0000","last_became_active":"2024-05-20T17:56:44.714396+0000","last_became_peered":"2024-05-20T17:56:44.714396+0000","last_unstale":"2024-05-20T17:57:49.043811+0000","last_undegraded":"2024-05-20T17:57:49.043811+0000","last_fullsized":"2024-05-20T17:57:49.043811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:21:04.831289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047434+0000","last_change":"2024-05-20T17:56:45.758160+0000","last_active":"2024-05-20T17:57:49.047434+0000","last_peered":"2024-05-20T17:57:49.047434+0000","last_clean":"2024-05-20T17:57:49.047434+0000","last_became_active":"2024-05-20T17:56:45.757405+0000","last_became_peered":"2024-05-20T17:56:45.757405+0000","last_unstale":"2024-05-20T17:57:49.047434+0000","last_undegraded":"2024-05-20T17:57:49.047434+0000","last_fullsized":"2024-05-20T17:57:49.047434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:17.625246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048566+0000","last_change":"2024-05-20T17:56:44.716342+0000","last_active":"2024-05-20T17:57:49.048566+0000","last_peered":"2024-05-20T17:57:49.048566+0000","last_clean":"2024-05-20T17:57:49.048566+0000","last_became_active":"2024-05-20T17:56:44.716010+0000","last_became_peered":"2024-05-20T17:56:44.716010+0000","last_unstale":"2024-05-20T17:57:49.048566+0000","last_undegraded":"2024-05-20T17:57:49.048566+0000","last_fullsized":"2024-05-20T17:57:49.048566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:00:25.073312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047826+0000","last_change":"2024-05-20T17:56:45.755288+0000","last_active":"2024-05-20T17:57:49.047826+0000","last_peered":"2024-05-20T17:57:49.047826+0000","last_clean":"2024-05-20T17:57:49.047826+0000","last_became_active":"2024-05-20T17:56:45.754952+0000","last_became_peered":"2024-05-20T17:56:45.754952+0000","last_unstale":"2024-05-20T17:57:49.047826+0000","last_undegraded":"2024-05-20T17:57:49.047826+0000","last_fullsized":"2024-05-20T17:57:49.047826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:44:27.631722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.046896+0000","last_change":"2024-05-20T17:56:44.717887+0000","last_active":"2024-05-20T17:57:49.046896+0000","last_peered":"2024-05-20T17:57:49.046896+0000","last_clean":"2024-05-20T17:57:49.046896+0000","last_became_active":"2024-05-20T17:56:44.717581+0000","last_became_peered":"2024-05-20T17:56:44.717581+0000","last_unstale":"2024-05-20T17:57:49.046896+0000","last_undegraded":"2024-05-20T17:57:49.046896+0000","last_fullsized":"2024-05-20T17:57:49.046896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:42:54.703112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044734+0000","last_change":"2024-05-20T17:56:45.748448+0000","last_active":"2024-05-20T17:57:49.044734+0000","last_peered":"2024-05-20T17:57:49.044734+0000","last_clean":"2024-05-20T17:57:49.044734+0000","last_became_active":"2024-05-20T17:56:45.748016+0000","last_became_peered":"2024-05-20T17:56:45.748016+0000","last_unstale":"2024-05-20T17:57:49.044734+0000","last_undegraded":"2024-05-20T17:57:49.044734+0000","last_fullsized":"2024-05-20T17:57:49.044734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:31:39.350477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044413+0000","last_change":"2024-05-20T17:56:44.709483+0000","last_active":"2024-05-20T17:57:49.044413+0000","last_peered":"2024-05-20T17:57:49.044413+0000","last_clean":"2024-05-20T17:57:49.044413+0000","last_became_active":"2024-05-20T17:56:44.708649+0000","last_became_peered":"2024-05-20T17:56:44.708649+0000","last_unstale":"2024-05-20T17:57:49.044413+0000","last_undegraded":"2024-05-20T17:57:49.044413+0000","last_fullsized":"2024-05-20T17:57:49.044413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:31:46.580256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076645+0000","last_change":"2024-05-20T17:56:45.752279+0000","last_active":"2024-05-20T17:57:49.076645+0000","last_peered":"2024-05-20T17:57:49.076645+0000","last_clean":"2024-05-20T17:57:49.076645+0000","last_became_active":"2024-05-20T17:56:45.751790+0000","last_became_peered":"2024-05-20T17:56:45.751790+0000","last_unstale":"2024-05-20T17:57:49.076645+0000","last_undegraded":"2024-05-20T17:57:49.076645+0000","last_fullsized":"2024-05-20T17:57:49.076645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:06:38.113559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.077752+0000","last_change":"2024-05-20T17:57:49.077752+0000","last_active":"2024-05-20T17:57:49.077752+0000","last_peered":"2024-05-20T17:57:49.077752+0000","last_clean":"2024-05-20T17:57:49.077752+0000","last_became_active":"2024-05-20T17:57:49.077277+0000","last_became_peered":"2024-05-20T17:57:49.077277+0000","last_unstale":"2024-05-20T17:57:49.077752+0000","last_undegraded":"2024-05-20T17:57:49.077752+0000","last_fullsized":"2024-05-20T17:57:49.077752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:24.954772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.051693+0000","last_change":"2024-05-20T17:57:49.051693+0000","last_active":"2024-05-20T17:57:49.051693+0000","last_peered":"2024-05-20T17:57:49.051693+0000","last_clean":"2024-05-20T17:57:49.051693+0000","last_became_active":"2024-05-20T17:57:49.051215+0000","last_became_peered":"2024-05-20T17:57:49.051215+0000","last_unstale":"2024-05-20T17:57:49.051693+0000","last_undegraded":"2024-05-20T17:57:49.051693+0000","last_fullsized":"2024-05-20T17:57:49.051693+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:52:35.406908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047744+0000","last_change":"2024-05-20T17:56:44.709548+0000","last_active":"2024-05-20T17:57:49.047744+0000","last_peered":"2024-05-20T17:57:49.047744+0000","last_clean":"2024-05-20T17:57:49.047744+0000","last_became_active":"2024-05-20T17:56:44.709209+0000","last_became_peered":"2024-05-20T17:56:44.709209+0000","last_unstale":"2024-05-20T17:57:49.047744+0000","last_undegraded":"2024-05-20T17:57:49.047744+0000","last_fullsized":"2024-05-20T17:57:49.047744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:28:13.118336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076469+0000","last_change":"2024-05-20T17:56:45.755141+0000","last_active":"2024-05-20T17:57:49.076469+0000","last_peered":"2024-05-20T17:57:49.076469+0000","last_clean":"2024-05-20T17:57:49.076469+0000","last_became_active":"2024-05-20T17:56:45.754786+0000","last_became_peered":"2024-05-20T17:56:45.754786+0000","last_unstale":"2024-05-20T17:57:49.076469+0000","last_undegraded":"2024-05-20T17:57:49.076469+0000","last_fullsized":"2024-05-20T17:57:49.076469+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:45:21.044602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'26","reported_seq":198,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:58:01.105077+0000","last_change":"2024-05-20T17:56:44.713578+0000","last_active":"2024-05-20T17:58:01.105077+0000","last_peered":"2024-05-20T17:58:01.105077+0000","last_clean":"2024-05-20T17:58:01.105077+0000","last_became_active":"2024-05-20T17:56:44.713203+0000","last_became_peered":"2024-05-20T17:56:44.713203+0000","last_unstale":"2024-05-20T17:58:01.105077+0000","last_undegraded":"2024-05-20T17:58:01.105077+0000","last_fullsized":"2024-05-20T17:58:01.105077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:36:54.386036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":153,"num_read_kb":135,"num_write":26,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048157+0000","last_change":"2024-05-20T17:56:45.752023+0000","last_active":"2024-05-20T17:57:49.048157+0000","last_peered":"2024-05-20T17:57:49.048157+0000","last_clean":"2024-05-20T17:57:49.048157+0000","last_became_active":"2024-05-20T17:56:45.751672+0000","last_became_peered":"2024-05-20T17:56:45.751672+0000","last_unstale":"2024-05-20T17:57:49.048157+0000","last_undegraded":"2024-05-20T17:57:49.048157+0000","last_fullsized":"2024-05-20T17:57:49.048157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:23:49.127016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049088+0000","last_change":"2024-05-20T17:56:44.717336+0000","last_active":"2024-05-20T17:57:49.049088+0000","last_peered":"2024-05-20T17:57:49.049088+0000","last_clean":"2024-05-20T17:57:49.049088+0000","last_became_active":"2024-05-20T17:56:44.717016+0000","last_became_peered":"2024-05-20T17:56:44.717016+0000","last_unstale":"2024-05-20T17:57:49.049088+0000","last_undegraded":"2024-05-20T17:57:49.049088+0000","last_fullsized":"2024-05-20T17:57:49.049088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:50:46.433461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049004+0000","last_change":"2024-05-20T17:56:45.752411+0000","last_active":"2024-05-20T17:57:49.049004+0000","last_peered":"2024-05-20T17:57:49.049004+0000","last_clean":"2024-05-20T17:57:49.049004+0000","last_became_active":"2024-05-20T17:56:45.752078+0000","last_became_peered":"2024-05-20T17:56:45.752078+0000","last_unstale":"2024-05-20T17:57:49.049004+0000","last_undegraded":"2024-05-20T17:57:49.049004+0000","last_fullsized":"2024-05-20T17:57:49.049004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:22:15.560666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047164+0000","last_change":"2024-05-20T17:56:44.723262+0000","last_active":"2024-05-20T17:57:49.047164+0000","last_peered":"2024-05-20T17:57:49.047164+0000","last_clean":"2024-05-20T17:57:49.047164+0000","last_became_active":"2024-05-20T17:56:44.722865+0000","last_became_peered":"2024-05-20T17:56:44.722865+0000","last_unstale":"2024-05-20T17:57:49.047164+0000","last_undegraded":"2024-05-20T17:57:49.047164+0000","last_fullsized":"2024-05-20T17:57:49.047164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:15:02.820718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.052110+0000","last_change":"2024-05-20T17:57:49.052110+0000","last_active":"2024-05-20T17:57:49.052110+0000","last_peered":"2024-05-20T17:57:49.052110+0000","last_clean":"2024-05-20T17:57:49.052110+0000","last_became_active":"2024-05-20T17:57:49.051356+0000","last_became_peered":"2024-05-20T17:57:49.051356+0000","last_unstale":"2024-05-20T17:57:49.052110+0000","last_undegraded":"2024-05-20T17:57:49.052110+0000","last_fullsized":"2024-05-20T17:57:49.052110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:32.899615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.046141+0000","last_change":"2024-05-20T17:56:44.718396+0000","last_active":"2024-05-20T17:57:49.046141+0000","last_peered":"2024-05-20T17:57:49.046141+0000","last_clean":"2024-05-20T17:57:49.046141+0000","last_became_active":"2024-05-20T17:56:44.718031+0000","last_became_peered":"2024-05-20T17:56:44.718031+0000","last_unstale":"2024-05-20T17:57:49.046141+0000","last_undegraded":"2024-05-20T17:57:49.046141+0000","last_fullsized":"2024-05-20T17:57:49.046141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:42:40.955520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047471+0000","last_change":"2024-05-20T17:56:45.759302+0000","last_active":"2024-05-20T17:57:49.047471+0000","last_peered":"2024-05-20T17:57:49.047471+0000","last_clean":"2024-05-20T17:57:49.047471+0000","last_became_active":"2024-05-20T17:56:45.758889+0000","last_became_peered":"2024-05-20T17:56:45.758889+0000","last_unstale":"2024-05-20T17:57:49.047471+0000","last_undegraded":"2024-05-20T17:57:49.047471+0000","last_fullsized":"2024-05-20T17:57:49.047471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:40.685471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047716+0000","last_change":"2024-05-20T17:56:44.726502+0000","last_active":"2024-05-20T17:57:49.047716+0000","last_peered":"2024-05-20T17:57:49.047716+0000","last_clean":"2024-05-20T17:57:49.047716+0000","last_became_active":"2024-05-20T17:56:44.726168+0000","last_became_peered":"2024-05-20T17:56:44.726168+0000","last_unstale":"2024-05-20T17:57:49.047716+0000","last_undegraded":"2024-05-20T17:57:49.047716+0000","last_fullsized":"2024-05-20T17:57:49.047716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:45:14.523968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047696+0000","last_change":"2024-05-20T17:56:45.759355+0000","last_active":"2024-05-20T17:57:49.047696+0000","last_peered":"2024-05-20T17:57:49.047696+0000","last_clean":"2024-05-20T17:57:49.047696+0000","last_became_active":"2024-05-20T17:56:45.758920+0000","last_became_peered":"2024-05-20T17:56:45.758920+0000","last_unstale":"2024-05-20T17:57:49.047696+0000","last_undegraded":"2024-05-20T17:57:49.047696+0000","last_fullsized":"2024-05-20T17:57:49.047696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:46:02.372250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048617+0000","last_change":"2024-05-20T17:56:44.718507+0000","last_active":"2024-05-20T17:57:49.048617+0000","last_peered":"2024-05-20T17:57:49.048617+0000","last_clean":"2024-05-20T17:57:49.048617+0000","last_became_active":"2024-05-20T17:56:44.718188+0000","last_became_peered":"2024-05-20T17:56:44.718188+0000","last_unstale":"2024-05-20T17:57:49.048617+0000","last_undegraded":"2024-05-20T17:57:49.048617+0000","last_fullsized":"2024-05-20T17:57:49.048617+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:43:59.792942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043201+0000","last_change":"2024-05-20T17:56:45.751361+0000","last_active":"2024-05-20T17:57:49.043201+0000","last_peered":"2024-05-20T17:57:49.043201+0000","last_clean":"2024-05-20T17:57:49.043201+0000","last_became_active":"2024-05-20T17:56:45.750032+0000","last_became_peered":"2024-05-20T17:56:45.750032+0000","last_unstale":"2024-05-20T17:57:49.043201+0000","last_undegraded":"2024-05-20T17:57:49.043201+0000","last_fullsized":"2024-05-20T17:57:49.043201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:01:42.729156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044973+0000","last_change":"2024-05-20T17:56:44.714404+0000","last_active":"2024-05-20T17:57:49.044973+0000","last_peered":"2024-05-20T17:57:49.044973+0000","last_clean":"2024-05-20T17:57:49.044973+0000","last_became_active":"2024-05-20T17:56:44.714072+0000","last_became_peered":"2024-05-20T17:56:44.714072+0000","last_unstale":"2024-05-20T17:57:49.044973+0000","last_undegraded":"2024-05-20T17:57:49.044973+0000","last_fullsized":"2024-05-20T17:57:49.044973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:39:53.983513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044438+0000","last_change":"2024-05-20T17:56:45.749785+0000","last_active":"2024-05-20T17:57:49.044438+0000","last_peered":"2024-05-20T17:57:49.044438+0000","last_clean":"2024-05-20T17:57:49.044438+0000","last_became_active":"2024-05-20T17:56:45.749457+0000","last_became_peered":"2024-05-20T17:56:45.749457+0000","last_unstale":"2024-05-20T17:57:49.044438+0000","last_undegraded":"2024-05-20T17:57:49.044438+0000","last_fullsized":"2024-05-20T17:57:49.044438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:53:05.136882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047562+0000","last_change":"2024-05-20T17:56:44.722787+0000","last_active":"2024-05-20T17:57:49.047562+0000","last_peered":"2024-05-20T17:57:49.047562+0000","last_clean":"2024-05-20T17:57:49.047562+0000","last_became_active":"2024-05-20T17:56:44.722214+0000","last_became_peered":"2024-05-20T17:56:44.722214+0000","last_unstale":"2024-05-20T17:57:49.047562+0000","last_undegraded":"2024-05-20T17:57:49.047562+0000","last_fullsized":"2024-05-20T17:57:49.047562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:40:16.705718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.044275+0000","last_change":"2024-05-20T17:56:45.757515+0000","last_active":"2024-05-20T17:57:49.044275+0000","last_peered":"2024-05-20T17:57:49.044275+0000","last_clean":"2024-05-20T17:57:49.044275+0000","last_became_active":"2024-05-20T17:56:45.757174+0000","last_became_peered":"2024-05-20T17:56:45.757174+0000","last_unstale":"2024-05-20T17:57:49.044275+0000","last_undegraded":"2024-05-20T17:57:49.044275+0000","last_fullsized":"2024-05-20T17:57:49.044275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:18:14.154827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.052366+0000","last_change":"2024-05-20T17:57:49.051489+0000","last_active":"2024-05-20T17:57:49.052366+0000","last_peered":"2024-05-20T17:57:49.052366+0000","last_clean":"2024-05-20T17:57:49.052366+0000","last_became_active":"2024-05-20T17:57:49.051070+0000","last_became_peered":"2024-05-20T17:57:49.051070+0000","last_unstale":"2024-05-20T17:57:49.052366+0000","last_undegraded":"2024-05-20T17:57:49.052366+0000","last_fullsized":"2024-05-20T17:57:49.052366+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:18.745185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048453+0000","last_change":"2024-05-20T17:56:44.724642+0000","last_active":"2024-05-20T17:57:49.048453+0000","last_peered":"2024-05-20T17:57:49.048453+0000","last_clean":"2024-05-20T17:57:49.048453+0000","last_became_active":"2024-05-20T17:56:44.724311+0000","last_became_peered":"2024-05-20T17:56:44.724311+0000","last_unstale":"2024-05-20T17:57:49.048453+0000","last_undegraded":"2024-05-20T17:57:49.048453+0000","last_fullsized":"2024-05-20T17:57:49.048453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:47:35.336236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047978+0000","last_change":"2024-05-20T17:56:45.758466+0000","last_active":"2024-05-20T17:57:49.047978+0000","last_peered":"2024-05-20T17:57:49.047978+0000","last_clean":"2024-05-20T17:57:49.047978+0000","last_became_active":"2024-05-20T17:56:45.758126+0000","last_became_peered":"2024-05-20T17:56:45.758126+0000","last_unstale":"2024-05-20T17:57:49.047978+0000","last_undegraded":"2024-05-20T17:57:49.047978+0000","last_fullsized":"2024-05-20T17:57:49.047978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:36.830828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.045642+0000","last_change":"2024-05-20T17:56:44.711173+0000","last_active":"2024-05-20T17:57:49.045642+0000","last_peered":"2024-05-20T17:57:49.045642+0000","last_clean":"2024-05-20T17:57:49.045642+0000","last_became_active":"2024-05-20T17:56:44.710778+0000","last_became_peered":"2024-05-20T17:56:44.710778+0000","last_unstale":"2024-05-20T17:57:49.045642+0000","last_undegraded":"2024-05-20T17:57:49.045642+0000","last_fullsized":"2024-05-20T17:57:49.045642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:20:14.150346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047587+0000","last_change":"2024-05-20T17:55:34.871483+0000","last_active":"2024-05-20T17:57:49.047587+0000","last_peered":"2024-05-20T17:57:49.047587+0000","last_clean":"2024-05-20T17:57:49.047587+0000","last_became_active":"2024-05-20T17:55:34.871028+0000","last_became_peered":"2024-05-20T17:55:34.871028+0000","last_unstale":"2024-05-20T17:57:49.047587+0000","last_undegraded":"2024-05-20T17:57:49.047587+0000","last_fullsized":"2024-05-20T17:57:49.047587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_clean_scrub_stamp":"2024-05-20T17:52:08.599715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:29:40.159577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048877+0000","last_change":"2024-05-20T17:56:45.747725+0000","last_active":"2024-05-20T17:57:49.048877+0000","last_peered":"2024-05-20T17:57:49.048877+0000","last_clean":"2024-05-20T17:57:49.048877+0000","last_became_active":"2024-05-20T17:56:45.747232+0000","last_became_peered":"2024-05-20T17:56:45.747232+0000","last_unstale":"2024-05-20T17:57:49.048877+0000","last_undegraded":"2024-05-20T17:57:49.048877+0000","last_fullsized":"2024-05-20T17:57:49.048877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:09:26.589905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047923+0000","last_change":"2024-05-20T17:56:44.721004+0000","last_active":"2024-05-20T17:57:49.047923+0000","last_peered":"2024-05-20T17:57:49.047923+0000","last_clean":"2024-05-20T17:57:49.047923+0000","last_became_active":"2024-05-20T17:56:44.720620+0000","last_became_peered":"2024-05-20T17:56:44.720620+0000","last_unstale":"2024-05-20T17:57:49.047923+0000","last_undegraded":"2024-05-20T17:57:49.047923+0000","last_fullsized":"2024-05-20T17:57:49.047923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:42:14.710663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048325+0000","last_change":"2024-05-20T17:56:44.714754+0000","last_active":"2024-05-20T17:57:49.048325+0000","last_peered":"2024-05-20T17:57:49.048325+0000","last_clean":"2024-05-20T17:57:49.048325+0000","last_became_active":"2024-05-20T17:56:44.714378+0000","last_became_peered":"2024-05-20T17:56:44.714378+0000","last_unstale":"2024-05-20T17:57:49.048325+0000","last_undegraded":"2024-05-20T17:57:49.048325+0000","last_fullsized":"2024-05-20T17:57:49.048325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:46:53.969960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048822+0000","last_change":"2024-05-20T17:56:45.742415+0000","last_active":"2024-05-20T17:57:49.048822+0000","last_peered":"2024-05-20T17:57:49.048822+0000","last_clean":"2024-05-20T17:57:49.048822+0000","last_became_active":"2024-05-20T17:56:45.741941+0000","last_became_peered":"2024-05-20T17:56:45.741941+0000","last_unstale":"2024-05-20T17:57:49.048822+0000","last_undegraded":"2024-05-20T17:57:49.048822+0000","last_fullsized":"2024-05-20T17:57:49.048822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:20:03.191021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.045123+0000","last_change":"2024-05-20T17:56:44.724361+0000","last_active":"2024-05-20T17:57:49.045123+0000","last_peered":"2024-05-20T17:57:49.045123+0000","last_clean":"2024-05-20T17:57:49.045123+0000","last_became_active":"2024-05-20T17:56:44.724033+0000","last_became_peered":"2024-05-20T17:56:44.724033+0000","last_unstale":"2024-05-20T17:57:49.045123+0000","last_undegraded":"2024-05-20T17:57:49.045123+0000","last_fullsized":"2024-05-20T17:57:49.045123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:10:13.756606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.052049+0000","last_change":"2024-05-20T17:57:49.051338+0000","last_active":"2024-05-20T17:57:49.052049+0000","last_peered":"2024-05-20T17:57:49.052049+0000","last_clean":"2024-05-20T17:57:49.052049+0000","last_became_active":"2024-05-20T17:57:49.050618+0000","last_became_peered":"2024-05-20T17:57:49.050618+0000","last_unstale":"2024-05-20T17:57:49.052049+0000","last_undegraded":"2024-05-20T17:57:49.052049+0000","last_fullsized":"2024-05-20T17:57:49.052049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:02:09.846175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.046306+0000","last_change":"2024-05-20T17:56:44.716296+0000","last_active":"2024-05-20T17:57:49.046306+0000","last_peered":"2024-05-20T17:57:49.046306+0000","last_clean":"2024-05-20T17:57:49.046306+0000","last_became_active":"2024-05-20T17:56:44.715952+0000","last_became_peered":"2024-05-20T17:56:44.715952+0000","last_unstale":"2024-05-20T17:57:49.046306+0000","last_undegraded":"2024-05-20T17:57:49.046306+0000","last_fullsized":"2024-05-20T17:57:49.046306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:52:37.986399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048194+0000","last_change":"2024-05-20T17:56:45.751874+0000","last_active":"2024-05-20T17:57:49.048194+0000","last_peered":"2024-05-20T17:57:49.048194+0000","last_clean":"2024-05-20T17:57:49.048194+0000","last_became_active":"2024-05-20T17:56:45.751534+0000","last_became_peered":"2024-05-20T17:56:45.751534+0000","last_unstale":"2024-05-20T17:57:49.048194+0000","last_undegraded":"2024-05-20T17:57:49.048194+0000","last_fullsized":"2024-05-20T17:57:49.048194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:05:27.248115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076231+0000","last_change":"2024-05-20T17:56:44.724271+0000","last_active":"2024-05-20T17:57:49.076231+0000","last_peered":"2024-05-20T17:57:49.076231+0000","last_clean":"2024-05-20T17:57:49.076231+0000","last_became_active":"2024-05-20T17:56:44.723947+0000","last_became_peered":"2024-05-20T17:56:44.723947+0000","last_unstale":"2024-05-20T17:57:49.076231+0000","last_undegraded":"2024-05-20T17:57:49.076231+0000","last_fullsized":"2024-05-20T17:57:49.076231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:32:13.482059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047658+0000","last_change":"2024-05-20T17:56:45.751296+0000","last_active":"2024-05-20T17:57:49.047658+0000","last_peered":"2024-05-20T17:57:49.047658+0000","last_clean":"2024-05-20T17:57:49.047658+0000","last_became_active":"2024-05-20T17:56:45.748035+0000","last_became_peered":"2024-05-20T17:56:45.748035+0000","last_unstale":"2024-05-20T17:57:49.047658+0000","last_undegraded":"2024-05-20T17:57:49.047658+0000","last_fullsized":"2024-05-20T17:57:49.047658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:37:43.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047741+0000","last_change":"2024-05-20T17:56:44.720461+0000","last_active":"2024-05-20T17:57:49.047741+0000","last_peered":"2024-05-20T17:57:49.047741+0000","last_clean":"2024-05-20T17:57:49.047741+0000","last_became_active":"2024-05-20T17:56:44.720115+0000","last_became_peered":"2024-05-20T17:56:44.720115+0000","last_unstale":"2024-05-20T17:57:49.047741+0000","last_undegraded":"2024-05-20T17:57:49.047741+0000","last_fullsized":"2024-05-20T17:57:49.047741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:14:47.330193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.042771+0000","last_change":"2024-05-20T17:56:45.745025+0000","last_active":"2024-05-20T17:57:49.042771+0000","last_peered":"2024-05-20T17:57:49.042771+0000","last_clean":"2024-05-20T17:57:49.042771+0000","last_became_active":"2024-05-20T17:56:45.744626+0000","last_became_peered":"2024-05-20T17:56:45.744626+0000","last_unstale":"2024-05-20T17:57:49.042771+0000","last_undegraded":"2024-05-20T17:57:49.042771+0000","last_fullsized":"2024-05-20T17:57:49.042771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:08:51.052423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049081+0000","last_change":"2024-05-20T17:56:44.726099+0000","last_active":"2024-05-20T17:57:49.049081+0000","last_peered":"2024-05-20T17:57:49.049081+0000","last_clean":"2024-05-20T17:57:49.049081+0000","last_became_active":"2024-05-20T17:56:44.725714+0000","last_became_peered":"2024-05-20T17:56:44.725714+0000","last_unstale":"2024-05-20T17:57:49.049081+0000","last_undegraded":"2024-05-20T17:57:49.049081+0000","last_fullsized":"2024-05-20T17:57:49.049081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:51.326907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048179+0000","last_change":"2024-05-20T17:56:45.757376+0000","last_active":"2024-05-20T17:57:49.048179+0000","last_peered":"2024-05-20T17:57:49.048179+0000","last_clean":"2024-05-20T17:57:49.048179+0000","last_became_active":"2024-05-20T17:56:45.757053+0000","last_became_peered":"2024-05-20T17:56:45.757053+0000","last_unstale":"2024-05-20T17:57:49.048179+0000","last_undegraded":"2024-05-20T17:57:49.048179+0000","last_fullsized":"2024-05-20T17:57:49.048179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:50:27.727055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047850+0000","last_change":"2024-05-20T17:56:44.726341+0000","last_active":"2024-05-20T17:57:49.047850+0000","last_peered":"2024-05-20T17:57:49.047850+0000","last_clean":"2024-05-20T17:57:49.047850+0000","last_became_active":"2024-05-20T17:56:44.726008+0000","last_became_peered":"2024-05-20T17:56:44.726008+0000","last_unstale":"2024-05-20T17:57:49.047850+0000","last_undegraded":"2024-05-20T17:57:49.047850+0000","last_fullsized":"2024-05-20T17:57:49.047850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:25:07.520179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047791+0000","last_change":"2024-05-20T17:56:45.760469+0000","last_active":"2024-05-20T17:57:49.047791+0000","last_peered":"2024-05-20T17:57:49.047791+0000","last_clean":"2024-05-20T17:57:49.047791+0000","last_became_active":"2024-05-20T17:56:45.760100+0000","last_became_peered":"2024-05-20T17:56:45.760100+0000","last_unstale":"2024-05-20T17:57:49.047791+0000","last_undegraded":"2024-05-20T17:57:49.047791+0000","last_fullsized":"2024-05-20T17:57:49.047791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:54:06.309355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.051674+0000","last_change":"2024-05-20T17:57:49.051674+0000","last_active":"2024-05-20T17:57:49.051674+0000","last_peered":"2024-05-20T17:57:49.051674+0000","last_clean":"2024-05-20T17:57:49.051674+0000","last_became_active":"2024-05-20T17:57:49.051182+0000","last_became_peered":"2024-05-20T17:57:49.051182+0000","last_unstale":"2024-05-20T17:57:49.051674+0000","last_undegraded":"2024-05-20T17:57:49.051674+0000","last_fullsized":"2024-05-20T17:57:49.051674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:29:35.458339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048360+0000","last_change":"2024-05-20T17:56:45.757302+0000","last_active":"2024-05-20T17:57:49.048360+0000","last_peered":"2024-05-20T17:57:49.048360+0000","last_clean":"2024-05-20T17:57:49.048360+0000","last_became_active":"2024-05-20T17:56:45.754169+0000","last_became_peered":"2024-05-20T17:56:45.754169+0000","last_unstale":"2024-05-20T17:57:49.048360+0000","last_undegraded":"2024-05-20T17:57:49.048360+0000","last_fullsized":"2024-05-20T17:57:49.048360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:03:10.794342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047306+0000","last_change":"2024-05-20T17:56:44.724478+0000","last_active":"2024-05-20T17:57:49.047306+0000","last_peered":"2024-05-20T17:57:49.047306+0000","last_clean":"2024-05-20T17:57:49.047306+0000","last_became_active":"2024-05-20T17:56:44.723927+0000","last_became_peered":"2024-05-20T17:56:44.723927+0000","last_unstale":"2024-05-20T17:57:49.047306+0000","last_undegraded":"2024-05-20T17:57:49.047306+0000","last_fullsized":"2024-05-20T17:57:49.047306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:04:12.003957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048094+0000","last_change":"2024-05-20T17:56:45.758135+0000","last_active":"2024-05-20T17:57:49.048094+0000","last_peered":"2024-05-20T17:57:49.048094+0000","last_clean":"2024-05-20T17:57:49.048094+0000","last_became_active":"2024-05-20T17:56:45.757358+0000","last_became_peered":"2024-05-20T17:56:45.757358+0000","last_unstale":"2024-05-20T17:57:49.048094+0000","last_undegraded":"2024-05-20T17:57:49.048094+0000","last_fullsized":"2024-05-20T17:57:49.048094+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:13:06.435055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048817+0000","last_change":"2024-05-20T17:56:44.719822+0000","last_active":"2024-05-20T17:57:49.048817+0000","last_peered":"2024-05-20T17:57:49.048817+0000","last_clean":"2024-05-20T17:57:49.048817+0000","last_became_active":"2024-05-20T17:56:44.719454+0000","last_became_peered":"2024-05-20T17:56:44.719454+0000","last_unstale":"2024-05-20T17:57:49.048817+0000","last_undegraded":"2024-05-20T17:57:49.048817+0000","last_fullsized":"2024-05-20T17:57:49.048817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:03:21.048330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048426+0000","last_change":"2024-05-20T17:56:45.753510+0000","last_active":"2024-05-20T17:57:49.048426+0000","last_peered":"2024-05-20T17:57:49.048426+0000","last_clean":"2024-05-20T17:57:49.048426+0000","last_became_active":"2024-05-20T17:56:45.752718+0000","last_became_peered":"2024-05-20T17:56:45.752718+0000","last_unstale":"2024-05-20T17:57:49.048426+0000","last_undegraded":"2024-05-20T17:57:49.048426+0000","last_fullsized":"2024-05-20T17:57:49.048426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:05:44.677138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.042839+0000","last_change":"2024-05-20T17:56:45.752528+0000","last_active":"2024-05-20T17:57:49.042839+0000","last_peered":"2024-05-20T17:57:49.042839+0000","last_clean":"2024-05-20T17:57:49.042839+0000","last_became_active":"2024-05-20T17:56:45.752171+0000","last_became_peered":"2024-05-20T17:56:45.752171+0000","last_unstale":"2024-05-20T17:57:49.042839+0000","last_undegraded":"2024-05-20T17:57:49.042839+0000","last_fullsized":"2024-05-20T17:57:49.042839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:12:55.826122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048988+0000","last_change":"2024-05-20T17:56:45.749228+0000","last_active":"2024-05-20T17:57:49.048988+0000","last_peered":"2024-05-20T17:57:49.048988+0000","last_clean":"2024-05-20T17:57:49.048988+0000","last_became_active":"2024-05-20T17:56:45.748886+0000","last_became_peered":"2024-05-20T17:56:45.748886+0000","last_unstale":"2024-05-20T17:57:49.048988+0000","last_undegraded":"2024-05-20T17:57:49.048988+0000","last_fullsized":"2024-05-20T17:57:49.048988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:23:10.569720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048826+0000","last_change":"2024-05-20T17:56:45.751105+0000","last_active":"2024-05-20T17:57:49.048826+0000","last_peered":"2024-05-20T17:57:49.048826+0000","last_clean":"2024-05-20T17:57:49.048826+0000","last_became_active":"2024-05-20T17:56:45.750754+0000","last_became_peered":"2024-05-20T17:56:45.750754+0000","last_unstale":"2024-05-20T17:57:49.048826+0000","last_undegraded":"2024-05-20T17:57:49.048826+0000","last_fullsized":"2024-05-20T17:57:49.048826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:57:48.327428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.050574+0000","last_change":"2024-05-20T17:57:49.050574+0000","last_active":"2024-05-20T17:57:49.050574+0000","last_peered":"2024-05-20T17:57:49.050574+0000","last_clean":"2024-05-20T17:57:49.050574+0000","last_became_active":"2024-05-20T17:57:49.050037+0000","last_became_peered":"2024-05-20T17:57:49.050037+0000","last_unstale":"2024-05-20T17:57:49.050574+0000","last_undegraded":"2024-05-20T17:57:49.050574+0000","last_fullsized":"2024-05-20T17:57:49.050574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:01:56.453842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048581+0000","last_change":"2024-05-20T17:56:45.746122+0000","last_active":"2024-05-20T17:57:49.048581+0000","last_peered":"2024-05-20T17:57:49.048581+0000","last_clean":"2024-05-20T17:57:49.048581+0000","last_became_active":"2024-05-20T17:56:45.745773+0000","last_became_peered":"2024-05-20T17:56:45.745773+0000","last_unstale":"2024-05-20T17:57:49.048581+0000","last_undegraded":"2024-05-20T17:57:49.048581+0000","last_fullsized":"2024-05-20T17:57:49.048581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:21:27.909974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.043155+0000","last_change":"2024-05-20T17:56:45.755018+0000","last_active":"2024-05-20T17:57:49.043155+0000","last_peered":"2024-05-20T17:57:49.043155+0000","last_clean":"2024-05-20T17:57:49.043155+0000","last_became_active":"2024-05-20T17:56:45.754615+0000","last_became_peered":"2024-05-20T17:56:45.754615+0000","last_unstale":"2024-05-20T17:57:49.043155+0000","last_undegraded":"2024-05-20T17:57:49.043155+0000","last_fullsized":"2024-05-20T17:57:49.043155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:36:39.100566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049568+0000","last_change":"2024-05-20T17:56:45.751091+0000","last_active":"2024-05-20T17:57:49.049568+0000","last_peered":"2024-05-20T17:57:49.049568+0000","last_clean":"2024-05-20T17:57:49.049568+0000","last_became_active":"2024-05-20T17:56:45.748573+0000","last_became_peered":"2024-05-20T17:56:45.748573+0000","last_unstale":"2024-05-20T17:57:49.049568+0000","last_undegraded":"2024-05-20T17:57:49.049568+0000","last_fullsized":"2024-05-20T17:57:49.049568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:11:16.140967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.049006+0000","last_change":"2024-05-20T17:56:45.756373+0000","last_active":"2024-05-20T17:57:49.049006+0000","last_peered":"2024-05-20T17:57:49.049006+0000","last_clean":"2024-05-20T17:57:49.049006+0000","last_became_active":"2024-05-20T17:56:45.755993+0000","last_became_peered":"2024-05-20T17:56:45.755993+0000","last_unstale":"2024-05-20T17:57:49.049006+0000","last_undegraded":"2024-05-20T17:57:49.049006+0000","last_fullsized":"2024-05-20T17:57:49.049006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:34:44.491341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.046975+0000","last_change":"2024-05-20T17:56:45.751866+0000","last_active":"2024-05-20T17:57:49.046975+0000","last_peered":"2024-05-20T17:57:49.046975+0000","last_clean":"2024-05-20T17:57:49.046975+0000","last_became_active":"2024-05-20T17:56:45.751518+0000","last_became_peered":"2024-05-20T17:56:45.751518+0000","last_unstale":"2024-05-20T17:57:49.046975+0000","last_undegraded":"2024-05-20T17:57:49.046975+0000","last_fullsized":"2024-05-20T17:57:49.046975+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:38:18.736177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.047633+0000","last_change":"2024-05-20T17:56:45.749903+0000","last_active":"2024-05-20T17:57:49.047633+0000","last_peered":"2024-05-20T17:57:49.047633+0000","last_clean":"2024-05-20T17:57:49.047633+0000","last_became_active":"2024-05-20T17:56:45.749500+0000","last_became_peered":"2024-05-20T17:56:45.749500+0000","last_unstale":"2024-05-20T17:57:49.047633+0000","last_undegraded":"2024-05-20T17:57:49.047633+0000","last_fullsized":"2024-05-20T17:57:49.047633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:39:34.875111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.076415+0000","last_change":"2024-05-20T17:56:45.747370+0000","last_active":"2024-05-20T17:57:49.076415+0000","last_peered":"2024-05-20T17:57:49.076415+0000","last_clean":"2024-05-20T17:57:49.076415+0000","last_became_active":"2024-05-20T17:56:45.747036+0000","last_became_peered":"2024-05-20T17:56:45.747036+0000","last_unstale":"2024-05-20T17:57:49.076415+0000","last_undegraded":"2024-05-20T17:57:49.076415+0000","last_fullsized":"2024-05-20T17:57:49.076415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:02:02.220522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.048964+0000","last_change":"2024-05-20T17:56:45.752762+0000","last_active":"2024-05-20T17:57:49.048964+0000","last_peered":"2024-05-20T17:57:49.048964+0000","last_clean":"2024-05-20T17:57:49.048964+0000","last_became_active":"2024-05-20T17:56:45.752345+0000","last_became_peered":"2024-05-20T17:56:45.752345+0000","last_unstale":"2024-05-20T17:57:49.048964+0000","last_undegraded":"2024-05-20T17:57:49.048964+0000","last_fullsized":"2024-05-20T17:57:49.048964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:14:49.861456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-20T17:57:49.042837+0000","last_change":"2024-05-20T17:56:45.750229+0000","last_active":"2024-05-20T17:57:49.042837+0000","last_peered":"2024-05-20T17:57:49.042837+0000","last_clean":"2024-05-20T17:57:49.042837+0000","last_became_active":"2024-05-20T17:56:45.749872+0000","last_became_peered":"2024-05-20T17:56:45.749872+0000","last_unstale":"2024-05-20T17:57:49.042837+0000","last_undegraded":"2024-05-20T17:57:49.042837+0000","last_fullsized":"2024-05-20T17:57:49.042837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:56:58.015339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280272,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61844,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":159,"num_read_kb":141,"num_write":75,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":185532,"data_compressed":5580,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":950272,"data_stored":462373,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612642,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1081344,"data_stored":1114098,"data_compressed":43326,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808871,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":462955,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070508,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":462395,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332144,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":1053440,"data_compressed":41466,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495670,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":462490,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":524001,"data_compressed":7677,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":462463,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":523453,"data_compressed":7677,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":462711,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920832,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":462395,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757306,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":1053040,"data_compressed":41466,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34009,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61628,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61080,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":61240,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-20T17:58:04.316 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-20T17:58:04.316 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-20T17:58:04.316 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-20T17:58:04.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph health --format=json 2024-05-20T17:58:04.554 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:58:06.006 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:05 smithi129 ceph-mon[32029]: pgmap v327: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:06.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:05 smithi154 ceph-mon[36482]: pgmap v327: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:06.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:05 smithi204 ceph-mon[37076]: pgmap v327: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:06.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.276+0000 7f07b8812700 1 -- 172.21.15.129:0/2092177221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07b4101960 msgr2=0x7f07b4101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:06.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.276+0000 7f07b8812700 1 --2- 172.21.15.129:0/2092177221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07b4101960 0x7f07b4101d70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f07a40099c0 tx=0x7f07a4009cd0 comp rx=0 tx=0).stop 2024-05-20T17:58:06.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.276+0000 7f07b8812700 1 -- 172.21.15.129:0/2092177221 shutdown_connections 2024-05-20T17:58:06.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.276+0000 7f07b8812700 1 --2- 172.21.15.129:0/2092177221 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07b41086d0 0x7f07b4110030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.276+0000 7f07b8812700 1 --2- 172.21.15.129:0/2092177221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07b4101960 0x7f07b4101d70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.276+0000 7f07b8812700 1 --2- 172.21.15.129:0/2092177221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07b4107d80 0x7f07b4108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.276+0000 7f07b8812700 1 -- 172.21.15.129:0/2092177221 >> 172.21.15.129:0/2092177221 conn(0x7f07b40fd360 msgr2=0x7f07b40ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.276+0000 7f07b8812700 1 -- 172.21.15.129:0/2092177221 shutdown_connections 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.276+0000 7f07b8812700 1 -- 172.21.15.129:0/2092177221 wait complete. 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.277+0000 7f07b8812700 1 Processor -- start 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.277+0000 7f07b8812700 1 -- start start 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.277+0000 7f07b8812700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07b4101960 0x7f07b4199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.277+0000 7f07b8812700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07b4107d80 0x7f07b419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.277+0000 7f07b8812700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07b41086d0 0x7f07b419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.277+0000 7f07b8812700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07b41134d0 con 0x7f07b4107d80 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.277+0000 7f07b8812700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07b4113350 con 0x7f07b4101960 2024-05-20T17:58:06.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.277+0000 7f07b8812700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f07b4113650 con 0x7f07b41086d0 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b1d9b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07b4107d80 0x7f07b419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b1d9b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07b4107d80 0x7f07b419a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49880/0 (socket says 172.21.15.129:49880) 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b1d9b700 1 -- 172.21.15.129:0/3402343657 learned_addr learned my addr 172.21.15.129:0/3402343657 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b259c700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07b4101960 0x7f07b4199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b2d9d700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07b41086d0 0x7f07b419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b1d9b700 1 -- 172.21.15.129:0/3402343657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07b41086d0 msgr2=0x7f07b419e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b1d9b700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07b41086d0 0x7f07b419e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b1d9b700 1 -- 172.21.15.129:0/3402343657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07b4101960 msgr2=0x7f07b4199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b1d9b700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07b4101960 0x7f07b4199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b1d9b700 1 -- 172.21.15.129:0/3402343657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b419ec50 con 0x7f07b4107d80 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b1d9b700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07b4107d80 0x7f07b419a320 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d980 tx=0x7f07a800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:06.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07a37fe700 1 -- 172.21.15.129:0/3402343657 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8017070 con 0x7f07b4107d80 2024-05-20T17:58:06.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b8812700 1 -- 172.21.15.129:0/3402343657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b419ef40 con 0x7f07b4107d80 2024-05-20T17:58:06.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07a37fe700 1 -- 172.21.15.129:0/3402343657 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07a8009e10 con 0x7f07b4107d80 2024-05-20T17:58:06.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07a37fe700 1 -- 172.21.15.129:0/3402343657 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a800f620 con 0x7f07b4107d80 2024-05-20T17:58:06.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.278+0000 7f07b8812700 1 -- 172.21.15.129:0/3402343657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b41a6860 con 0x7f07b4107d80 2024-05-20T17:58:06.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.279+0000 7f07b8812700 1 -- 172.21.15.129:0/3402343657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b4050170 con 0x7f07b4107d80 2024-05-20T17:58:06.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.280+0000 7f07a37fe700 1 -- 172.21.15.129:0/3402343657 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f07a8010460 con 0x7f07b4107d80 2024-05-20T17:58:06.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.281+0000 7f07a37fe700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f079c0a2cb0 0x7f079c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:06.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.281+0000 7f07b259c700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f079c0a2cb0 0x7f079c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:06.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.281+0000 7f07a37fe700 1 -- 172.21.15.129:0/3402343657 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f07a80ced00 con 0x7f07b4107d80 2024-05-20T17:58:06.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.281+0000 7f07b259c700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f079c0a2cb0 0x7f079c0a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f07a4000c00 tx=0x7f07a401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:06.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.285+0000 7f07a37fe700 1 -- 172.21.15.129:0/3402343657 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f07a809baf0 con 0x7f07b4107d80 2024-05-20T17:58:06.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.527+0000 7f07b8812700 1 -- 172.21.15.129:0/3402343657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f07b419b0e0 con 0x7f07b4107d80 2024-05-20T17:58:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.527+0000 7f07a37fe700 1 -- 172.21.15.129:0/3402343657 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f07a809b3f0 con 0x7f07b4107d80 2024-05-20T17:58:06.526 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:06.526 INFO:teuthology.orchestra.run.smithi129.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-20T17:58:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.529+0000 7f07b8812700 1 -- 172.21.15.129:0/3402343657 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f079c0a2cb0 msgr2=0x7f079c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.529+0000 7f07b8812700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f079c0a2cb0 0x7f079c0a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f07a4000c00 tx=0x7f07a401a040 comp rx=0 tx=0).stop 2024-05-20T17:58:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.529+0000 7f07b8812700 1 -- 172.21.15.129:0/3402343657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07b4107d80 msgr2=0x7f07b419a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.529+0000 7f07b8812700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07b4107d80 0x7f07b419a320 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d980 tx=0x7f07a800dc90 comp rx=0 tx=0).stop 2024-05-20T17:58:06.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.530+0000 7f07b8812700 1 -- 172.21.15.129:0/3402343657 shutdown_connections 2024-05-20T17:58:06.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.530+0000 7f07b8812700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f079c0a2cb0 0x7f079c0a51c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:06.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.530+0000 7f07b8812700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07b4101960 0x7f07b4199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:06.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.530+0000 7f07b8812700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07b41086d0 0x7f07b419e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:06.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.530+0000 7f07b8812700 1 --2- 172.21.15.129:0/3402343657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07b4107d80 0x7f07b419a320 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:06.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.530+0000 7f07b8812700 1 -- 172.21.15.129:0/3402343657 >> 172.21.15.129:0/3402343657 conn(0x7f07b40fd360 msgr2=0x7f07b41074c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:06.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.530+0000 7f07b8812700 1 -- 172.21.15.129:0/3402343657 shutdown_connections 2024-05-20T17:58:06.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:06.530+0000 7f07b8812700 1 -- 172.21.15.129:0/3402343657 wait complete. 2024-05-20T17:58:07.135 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:06 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3402343657' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-20T17:58:07.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:06 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3402343657' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-20T17:58:07.164 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-20T17:58:07.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:58:07.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:06 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3402343657' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-20T17:58:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.279+0000 7fc9e6c37640 1 -- 172.21.15.129:0/649915846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e0102ac0 msgr2=0x7fc9e01037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.279+0000 7fc9e6c37640 1 --2- 172.21.15.129:0/649915846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e0102ac0 0x7fc9e01037c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009900 tx=0x7fc9d802a410 comp rx=0 tx=0).stop 2024-05-20T17:58:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.279+0000 7fc9e6c37640 1 -- 172.21.15.129:0/649915846 shutdown_connections 2024-05-20T17:58:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.279+0000 7fc9e6c37640 1 --2- 172.21.15.129:0/649915846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0108a00 0x7fc9e010ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.279+0000 7fc9e6c37640 1 --2- 172.21.15.129:0/649915846 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9e0103d00 0x7fc9e0104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.279+0000 7fc9e6c37640 1 --2- 172.21.15.129:0/649915846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e0102ac0 0x7fc9e01037c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.279+0000 7fc9e6c37640 1 -- 172.21.15.129:0/649915846 >> 172.21.15.129:0/649915846 conn(0x7fc9e00fdcf0 msgr2=0x7fc9e0100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.279+0000 7fc9e6c37640 1 -- 172.21.15.129:0/649915846 shutdown_connections 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.279+0000 7fc9e6c37640 1 -- 172.21.15.129:0/649915846 wait complete. 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e6c37640 1 Processor -- start 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e6c37640 1 -- start start 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e6c37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0102ac0 0x7fc9e0199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e6c37640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9e0103d00 0x7fc9e01998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e6c37640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e0108a00 0x7fc9e01a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e6c37640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e010e760 con 0x7fc9e0108a00 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e6c37640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e010e5e0 con 0x7fc9e0102ac0 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e6c37640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e010e8e0 con 0x7fc9e0103d00 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e51d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e0108a00 0x7fc9e01a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e51d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e0108a00 0x7fc9e01a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49908/0 (socket says 172.21.15.129:49908) 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e51d4640 1 -- 172.21.15.129:0/2350296618 learned_addr learned my addr 172.21.15.129:0/2350296618 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e49d3640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0102ac0 0x7fc9e0199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9dffff640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9e0103d00 0x7fc9e01998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e51d4640 1 -- 172.21.15.129:0/2350296618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9e0103d00 msgr2=0x7fc9e01998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e51d4640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9e0103d00 0x7fc9e01998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e51d4640 1 -- 172.21.15.129:0/2350296618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0102ac0 msgr2=0x7fc9e0199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e51d4640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0102ac0 0x7fc9e0199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e51d4640 1 -- 172.21.15.129:0/2350296618 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e01a0e60 con 0x7fc9e0108a00 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.280+0000 7fc9e49d3640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0102ac0 0x7fc9e0199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.281+0000 7fc9e51d4640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e0108a00 0x7fc9e01a07f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc9c800c970 tx=0x7fc9c800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.281+0000 7fc9ddffb640 1 -- 172.21.15.129:0/2350296618 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8007bb0 con 0x7fc9e0108a00 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.281+0000 7fc9ddffb640 1 -- 172.21.15.129:0/2350296618 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c8007d50 con 0x7fc9e0108a00 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.281+0000 7fc9e6c37640 1 -- 172.21.15.129:0/2350296618 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e01a1150 con 0x7fc9e0108a00 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.281+0000 7fc9ddffb640 1 -- 172.21.15.129:0/2350296618 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c800f450 con 0x7fc9e0108a00 2024-05-20T17:58:07.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.281+0000 7fc9e6c37640 1 -- 172.21.15.129:0/2350296618 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e01a1650 con 0x7fc9e0108a00 2024-05-20T17:58:07.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.282+0000 7fc9e6c37640 1 -- 172.21.15.129:0/2350296618 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9e0107810 con 0x7fc9e0108a00 2024-05-20T17:58:07.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.283+0000 7fc9ddffb640 1 -- 172.21.15.129:0/2350296618 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc9c8004f60 con 0x7fc9e0108a00 2024-05-20T17:58:07.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.284+0000 7fc9ddffb640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc9cc0a2c20 0x7fc9cc0a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.284+0000 7fc9e49d3640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc9cc0a2c20 0x7fc9cc0a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:07.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.284+0000 7fc9ddffb640 1 -- 172.21.15.129:0/2350296618 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fc9c80cf000 con 0x7fc9e0108a00 2024-05-20T17:58:07.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.284+0000 7fc9e49d3640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc9cc0a2c20 0x7fc9cc0a5130 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc9d802aa00 tx=0x7fc9d8031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:07.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.289+0000 7fc9ddffb640 1 -- 172.21.15.129:0/2350296618 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc9c8014030 con 0x7fc9e0108a00 2024-05-20T17:58:07.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.469+0000 7fc9e6c37640 1 -- 172.21.15.129:0/2350296618 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc9e0062750 con 0x7fc9e0108a00 2024-05-20T17:58:07.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.471+0000 7fc9ddffb640 1 -- 172.21.15.129:0/2350296618 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22680 (secure 0 0 0) 0x7fc9c809bdf0 con 0x7fc9e0108a00 2024-05-20T17:58:07.470 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:07.470 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":88,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:57:49.033961+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:58:07.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 -- 172.21.15.129:0/2350296618 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc9cc0a2c20 msgr2=0x7fc9cc0a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc9cc0a2c20 0x7fc9cc0a5130 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc9d802aa00 tx=0x7fc9d8031400 comp rx=0 tx=0).stop 2024-05-20T17:58:07.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 -- 172.21.15.129:0/2350296618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e0108a00 msgr2=0x7fc9e01a07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e0108a00 0x7fc9e01a07f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc9c800c970 tx=0x7fc9c800ce30 comp rx=0 tx=0).stop 2024-05-20T17:58:07.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 -- 172.21.15.129:0/2350296618 shutdown_connections 2024-05-20T17:58:07.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc9cc0a2c20 0x7fc9cc0a5130 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e0108a00 0x7fc9e01a07f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9e0103d00 0x7fc9e01998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 --2- 172.21.15.129:0/2350296618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e0102ac0 0x7fc9e0199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 -- 172.21.15.129:0/2350296618 >> 172.21.15.129:0/2350296618 conn(0x7fc9e00fdcf0 msgr2=0x7fc9e01049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:07.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 -- 172.21.15.129:0/2350296618 shutdown_connections 2024-05-20T17:58:07.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.473+0000 7fc9e6c37640 1 -- 172.21.15.129:0/2350296618 wait complete. 2024-05-20T17:58:07.484 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-20T17:56:42.965073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-20T17:56:44.277499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T17:58:07.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-20T17:58:07.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.600+0000 7f095e856640 1 -- 172.21.15.129:0/2912704111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958101140 msgr2=0x7f0958101e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.600+0000 7f095e856640 1 --2- 172.21.15.129:0/2912704111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958101140 0x7f0958101e40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0948009900 tx=0x7f094802a410 comp rx=0 tx=0).stop 2024-05-20T17:58:07.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.600+0000 7f095e856640 1 -- 172.21.15.129:0/2912704111 shutdown_connections 2024-05-20T17:58:07.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.600+0000 7f095e856640 1 --2- 172.21.15.129:0/2912704111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0958102380 0x7f0958102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.600+0000 7f095e856640 1 --2- 172.21.15.129:0/2912704111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0958107080 0x7f09581094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.600+0000 7f095e856640 1 --2- 172.21.15.129:0/2912704111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958101140 0x7f0958101e40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.600+0000 7f095e856640 1 -- 172.21.15.129:0/2912704111 >> 172.21.15.129:0/2912704111 conn(0x7f09580fc330 msgr2=0x7f09580fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:07.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.600+0000 7f095e856640 1 -- 172.21.15.129:0/2912704111 shutdown_connections 2024-05-20T17:58:07.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.600+0000 7f095e856640 1 -- 172.21.15.129:0/2912704111 wait complete. 2024-05-20T17:58:07.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f095e856640 1 Processor -- start 2024-05-20T17:58:07.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f095e856640 1 -- start start 2024-05-20T17:58:07.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f095e856640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958101140 0x7f095810a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f095e856640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0958102380 0x7f095819e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f095e856640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0958107080 0x7f09581a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f095e856640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f095810d620 con 0x7f0958101140 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f095e856640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f095810d4a0 con 0x7f0958102380 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f095e856640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f095810d7a0 con 0x7f0958107080 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f0957fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958101140 0x7f095810a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f0957fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958101140 0x7f095810a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49924/0 (socket says 172.21.15.129:49924) 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f0957fff640 1 -- 172.21.15.129:0/442618394 learned_addr learned my addr 172.21.15.129:0/442618394 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f095cdf3640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0958107080 0x7f09581a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f0957fff640 1 -- 172.21.15.129:0/442618394 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0958107080 msgr2=0x7f09581a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f0957fff640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0958107080 0x7f09581a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f0957fff640 1 -- 172.21.15.129:0/442618394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0958102380 msgr2=0x7f095819e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f0957fff640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0958102380 0x7f095819e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.601+0000 7f0957fff640 1 -- 172.21.15.129:0/442618394 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09581a0e40 con 0x7f0958101140 2024-05-20T17:58:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.602+0000 7f0957fff640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958101140 0x7f095810a5f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0948006010 tx=0x7f09480043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.602+0000 7f09557fa640 1 -- 172.21.15.129:0/442618394 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094803f030 con 0x7f0958101140 2024-05-20T17:58:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.602+0000 7f09557fa640 1 -- 172.21.15.129:0/442618394 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0948003ce0 con 0x7f0958101140 2024-05-20T17:58:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.602+0000 7f095e856640 1 -- 172.21.15.129:0/442618394 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09581a10d0 con 0x7f0958101140 2024-05-20T17:58:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.602+0000 7f09557fa640 1 -- 172.21.15.129:0/442618394 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0948035070 con 0x7f0958101140 2024-05-20T17:58:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.602+0000 7f095e856640 1 -- 172.21.15.129:0/442618394 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09581a15c0 con 0x7f0958101140 2024-05-20T17:58:07.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.603+0000 7f095e856640 1 -- 172.21.15.129:0/442618394 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0958105e90 con 0x7f0958101140 2024-05-20T17:58:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.604+0000 7f09557fa640 1 -- 172.21.15.129:0/442618394 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f0948042030 con 0x7f0958101140 2024-05-20T17:58:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.605+0000 7f09557fa640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09400a2c20 0x7f09400a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.605+0000 7f09577fe640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09400a2c20 0x7f09400a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.605+0000 7f09557fa640 1 -- 172.21.15.129:0/442618394 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f09480efad0 con 0x7f0958101140 2024-05-20T17:58:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.605+0000 7f09577fe640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09400a2c20 0x7f09400a5130 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f095810a000 tx=0x7f094c005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.609+0000 7f09557fa640 1 -- 172.21.15.129:0/442618394 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f09480bc8c0 con 0x7f0958101140 2024-05-20T17:58:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.788+0000 7f095e856640 1 -- 172.21.15.129:0/442618394 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f095802d2a0 con 0x7f0958101140 2024-05-20T17:58:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.788+0000 7f09557fa640 1 -- 172.21.15.129:0/442618394 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f09480bc1c0 con 0x7f0958101140 2024-05-20T17:58:07.787 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T17:58:07.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 -- 172.21.15.129:0/442618394 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09400a2c20 msgr2=0x7f09400a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09400a2c20 0x7f09400a5130 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f095810a000 tx=0x7f094c005ed0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 -- 172.21.15.129:0/442618394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958101140 msgr2=0x7f095810a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958101140 0x7f095810a5f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0948006010 tx=0x7f09480043b0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 -- 172.21.15.129:0/442618394 shutdown_connections 2024-05-20T17:58:07.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f09400a2c20 0x7f09400a5130 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0958102380 0x7f095819e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0958107080 0x7f09581a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 --2- 172.21.15.129:0/442618394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958101140 0x7f095810a5f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 -- 172.21.15.129:0/442618394 >> 172.21.15.129:0/442618394 conn(0x7f09580fc330 msgr2=0x7f0958103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 -- 172.21.15.129:0/442618394 shutdown_connections 2024-05-20T17:58:07.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.790+0000 7f095e856640 1 -- 172.21.15.129:0/442618394 wait complete. 2024-05-20T17:58:07.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-20T17:58:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.917+0000 7fc3a279c640 1 -- 172.21.15.129:0/3692678863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c101990 msgr2=0x7fc39c102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.917+0000 7fc3a279c640 1 --2- 172.21.15.129:0/3692678863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c101990 0x7fc39c102690 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc38c009900 tx=0x7fc38c02a410 comp rx=0 tx=0).stop 2024-05-20T17:58:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.917+0000 7fc3a279c640 1 -- 172.21.15.129:0/3692678863 shutdown_connections 2024-05-20T17:58:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.917+0000 7fc3a279c640 1 --2- 172.21.15.129:0/3692678863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c102bd0 0x7fc39c102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.917+0000 7fc3a279c640 1 --2- 172.21.15.129:0/3692678863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc39c1078d0 0x7fc39c109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.917+0000 7fc3a279c640 1 --2- 172.21.15.129:0/3692678863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c101990 0x7fc39c102690 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.917+0000 7fc3a279c640 1 -- 172.21.15.129:0/3692678863 >> 172.21.15.129:0/3692678863 conn(0x7fc39c0fcb80 msgr2=0x7fc39c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.917+0000 7fc3a279c640 1 -- 172.21.15.129:0/3692678863 shutdown_connections 2024-05-20T17:58:07.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.917+0000 7fc3a279c640 1 -- 172.21.15.129:0/3692678863 wait complete. 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc3a279c640 1 Processor -- start 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc3a279c640 1 -- start start 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc3a279c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc39c101990 0x7fc39c1995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc3a279c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c102bd0 0x7fc39c199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc3a279c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c1078d0 0x7fc39c1a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc3a279c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c10d630 con 0x7fc39c1078d0 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc3a279c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c10d4b0 con 0x7fc39c102bd0 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc3a279c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c10d7b0 con 0x7fc39c101990 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc3a0d39640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c1078d0 0x7fc39c1a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc3a0d39640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c1078d0 0x7fc39c1a0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49940/0 (socket says 172.21.15.129:49940) 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc3a0d39640 1 -- 172.21.15.129:0/1660193834 learned_addr learned my addr 172.21.15.129:0/1660193834 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:07.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.918+0000 7fc39bfff640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc39c101990 0x7fc39c1995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc393fff640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c102bd0 0x7fc39c199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc3a0d39640 1 -- 172.21.15.129:0/1660193834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc39c101990 msgr2=0x7fc39c1995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc3a0d39640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc39c101990 0x7fc39c1995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc3a0d39640 1 -- 172.21.15.129:0/1660193834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c102bd0 msgr2=0x7fc39c199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc3a0d39640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c102bd0 0x7fc39c199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc3a0d39640 1 -- 172.21.15.129:0/1660193834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39c1a1100 con 0x7fc39c1078d0 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc3a0d39640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c1078d0 0x7fc39c1a0a90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc39400eaa0 tx=0x7fc39400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc399ffb640 1 -- 172.21.15.129:0/1660193834 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39400cdb0 con 0x7fc39c1078d0 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc399ffb640 1 -- 172.21.15.129:0/1660193834 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc394005020 con 0x7fc39c1078d0 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc3a279c640 1 -- 172.21.15.129:0/1660193834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39c1a13f0 con 0x7fc39c1078d0 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc399ffb640 1 -- 172.21.15.129:0/1660193834 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3940116b0 con 0x7fc39c1078d0 2024-05-20T17:58:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.919+0000 7fc3a279c640 1 -- 172.21.15.129:0/1660193834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc39c1a1940 con 0x7fc39c1078d0 2024-05-20T17:58:07.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.920+0000 7fc3a279c640 1 -- 172.21.15.129:0/1660193834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc39c06b250 con 0x7fc39c1078d0 2024-05-20T17:58:07.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.921+0000 7fc399ffb640 1 -- 172.21.15.129:0/1660193834 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc394005560 con 0x7fc39c1078d0 2024-05-20T17:58:07.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.922+0000 7fc399ffb640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3880a2c20 0x7fc3880a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:07.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.922+0000 7fc39bfff640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3880a2c20 0x7fc3880a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:07.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.922+0000 7fc399ffb640 1 -- 172.21.15.129:0/1660193834 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fc3940cf7b0 con 0x7fc39c1078d0 2024-05-20T17:58:07.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.923+0000 7fc39bfff640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3880a2c20 0x7fc3880a5130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc38c02aa00 tx=0x7fc38c0315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:07.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:07.926+0000 7fc399ffb640 1 -- 172.21.15.129:0/1660193834 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc394016030 con 0x7fc39c1078d0 2024-05-20T17:58:08.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.103+0000 7fc3a279c640 1 -- 172.21.15.129:0/1660193834 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc39c1a1c00 con 0x7fc39c1078d0 2024-05-20T17:58:08.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.104+0000 7fc399ffb640 1 -- 172.21.15.129:0/1660193834 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fc39409c5a0 con 0x7fc39c1078d0 2024-05-20T17:58:08.103 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-05-20T17:58:08.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 -- 172.21.15.129:0/1660193834 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3880a2c20 msgr2=0x7fc3880a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3880a2c20 0x7fc3880a5130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc38c02aa00 tx=0x7fc38c0315b0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 -- 172.21.15.129:0/1660193834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c1078d0 msgr2=0x7fc39c1a0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c1078d0 0x7fc39c1a0a90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc39400eaa0 tx=0x7fc39400ef60 comp rx=0 tx=0).stop 2024-05-20T17:58:08.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 -- 172.21.15.129:0/1660193834 shutdown_connections 2024-05-20T17:58:08.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3880a2c20 0x7fc3880a5130 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c102bd0 0x7fc39c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c1078d0 0x7fc39c1a0a90 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 --2- 172.21.15.129:0/1660193834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc39c101990 0x7fc39c1995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 -- 172.21.15.129:0/1660193834 >> 172.21.15.129:0/1660193834 conn(0x7fc39c0fcb80 msgr2=0x7fc39c068e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:08.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 -- 172.21.15.129:0/1660193834 shutdown_connections 2024-05-20T17:58:08.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.106+0000 7fc3a279c640 1 -- 172.21.15.129:0/1660193834 wait complete. 2024-05-20T17:58:08.115 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:07 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2350296618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:08.115 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:07 smithi129 ceph-mon[32029]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:58:08.116 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-20T17:58:08.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:07 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2350296618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:08.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:07 smithi154 ceph-mon[36482]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:58:08.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:07 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2350296618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:08.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:07 smithi204 ceph-mon[37076]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:58:08.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.232+0000 7fb4af1aa640 1 -- 172.21.15.129:0/2955651584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a8101990 msgr2=0x7fb4a8102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.232+0000 7fb4af1aa640 1 --2- 172.21.15.129:0/2955651584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a8101990 0x7fb4a8102690 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb49c009900 tx=0x7fb49c02a410 comp rx=0 tx=0).stop 2024-05-20T17:58:08.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.232+0000 7fb4af1aa640 1 -- 172.21.15.129:0/2955651584 shutdown_connections 2024-05-20T17:58:08.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.232+0000 7fb4af1aa640 1 --2- 172.21.15.129:0/2955651584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a81078d0 0x7fb4a8109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.232+0000 7fb4af1aa640 1 --2- 172.21.15.129:0/2955651584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4a8102bd0 0x7fb4a8102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.232+0000 7fb4af1aa640 1 --2- 172.21.15.129:0/2955651584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a8101990 0x7fb4a8102690 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.232+0000 7fb4af1aa640 1 -- 172.21.15.129:0/2955651584 >> 172.21.15.129:0/2955651584 conn(0x7fb4a80fcb80 msgr2=0x7fb4a80ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:08.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.232+0000 7fb4af1aa640 1 -- 172.21.15.129:0/2955651584 shutdown_connections 2024-05-20T17:58:08.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.232+0000 7fb4af1aa640 1 -- 172.21.15.129:0/2955651584 wait complete. 2024-05-20T17:58:08.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.232+0000 7fb4af1aa640 1 Processor -- start 2024-05-20T17:58:08.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.232+0000 7fb4af1aa640 1 -- start start 2024-05-20T17:58:08.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4af1aa640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4a8101990 0x7fb4a8198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4af1aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a8102bd0 0x7fb4a8198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4af1aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a81078d0 0x7fb4a819f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4af1aa640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a810d5e0 con 0x7fb4a8102bd0 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4af1aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a810d460 con 0x7fb4a81078d0 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4af1aa640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a810d760 con 0x7fb4a8101990 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a8102bd0 0x7fb4a8198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a8102bd0 0x7fb4a8198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49954/0 (socket says 172.21.15.129:49954) 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4a7fff640 1 -- 172.21.15.129:0/242497670 learned_addr learned my addr 172.21.15.129:0/242497670 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4acf46640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4a8101990 0x7fb4a8198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4ad747640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a81078d0 0x7fb4a819f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4a7fff640 1 -- 172.21.15.129:0/242497670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4a8101990 msgr2=0x7fb4a8198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4a7fff640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4a8101990 0x7fb4a8198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4a7fff640 1 -- 172.21.15.129:0/242497670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a81078d0 msgr2=0x7fb4a819f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4a7fff640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a81078d0 0x7fb4a819f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4a7fff640 1 -- 172.21.15.129:0/242497670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a8068f40 con 0x7fb4a8102bd0 2024-05-20T17:58:08.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4acf46640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4a8101990 0x7fb4a8198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:08.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4ad747640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a81078d0 0x7fb4a819f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:08.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4a7fff640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a8102bd0 0x7fb4a8198a40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb49400c970 tx=0x7fb49400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:08.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.233+0000 7fb4a5ffb640 1 -- 172.21.15.129:0/242497670 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb494007bb0 con 0x7fb4a8102bd0 2024-05-20T17:58:08.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.234+0000 7fb4a5ffb640 1 -- 172.21.15.129:0/242497670 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb494007d50 con 0x7fb4a8102bd0 2024-05-20T17:58:08.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.234+0000 7fb4af1aa640 1 -- 172.21.15.129:0/242497670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a8069230 con 0x7fb4a8102bd0 2024-05-20T17:58:08.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.234+0000 7fb4a5ffb640 1 -- 172.21.15.129:0/242497670 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49400f450 con 0x7fb4a8102bd0 2024-05-20T17:58:08.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.234+0000 7fb4af1aa640 1 -- 172.21.15.129:0/242497670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a8069730 con 0x7fb4a8102bd0 2024-05-20T17:58:08.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.235+0000 7fb4af1aa640 1 -- 172.21.15.129:0/242497670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4a806b370 con 0x7fb4a8102bd0 2024-05-20T17:58:08.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.236+0000 7fb4a5ffb640 1 -- 172.21.15.129:0/242497670 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb494005020 con 0x7fb4a8102bd0 2024-05-20T17:58:08.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.236+0000 7fb4a5ffb640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb4980a2c20 0x7fb4980a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.237+0000 7fb4acf46640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb4980a2c20 0x7fb4980a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.237+0000 7fb4a5ffb640 1 -- 172.21.15.129:0/242497670 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fb4940cefd0 con 0x7fb4a8102bd0 2024-05-20T17:58:08.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.237+0000 7fb4acf46640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb4980a2c20 0x7fb4980a5130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb49c009230 tx=0x7fb49c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:08.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.241+0000 7fb4a5ffb640 1 -- 172.21.15.129:0/242497670 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb494014030 con 0x7fb4a8102bd0 2024-05-20T17:58:08.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.416+0000 7fb4af1aa640 1 -- 172.21.15.129:0/242497670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb4a80699f0 con 0x7fb4a8102bd0 2024-05-20T17:58:08.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.416+0000 7fb4a5ffb640 1 -- 172.21.15.129:0/242497670 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fb49409be60 con 0x7fb4a8102bd0 2024-05-20T17:58:08.415 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-05-20T17:58:08.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.418+0000 7fb4af1aa640 1 -- 172.21.15.129:0/242497670 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb4980a2c20 msgr2=0x7fb4980a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.418+0000 7fb4af1aa640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb4980a2c20 0x7fb4980a5130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb49c009230 tx=0x7fb49c000bc0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.418+0000 7fb4af1aa640 1 -- 172.21.15.129:0/242497670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a8102bd0 msgr2=0x7fb4a8198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.418+0000 7fb4af1aa640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a8102bd0 0x7fb4a8198a40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb49400c970 tx=0x7fb49400ce30 comp rx=0 tx=0).stop 2024-05-20T17:58:08.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.419+0000 7fb4af1aa640 1 -- 172.21.15.129:0/242497670 shutdown_connections 2024-05-20T17:58:08.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.419+0000 7fb4af1aa640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb4980a2c20 0x7fb4980a5130 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.419+0000 7fb4af1aa640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a81078d0 0x7fb4a819f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.419+0000 7fb4af1aa640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4a8102bd0 0x7fb4a8198a40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.419+0000 7fb4af1aa640 1 --2- 172.21.15.129:0/242497670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb4a8101990 0x7fb4a8198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.419+0000 7fb4af1aa640 1 -- 172.21.15.129:0/242497670 >> 172.21.15.129:0/242497670 conn(0x7fb4a80fcb80 msgr2=0x7fb4a81038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:08.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.419+0000 7fb4af1aa640 1 -- 172.21.15.129:0/242497670 shutdown_connections 2024-05-20T17:58:08.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.419+0000 7fb4af1aa640 1 -- 172.21.15.129:0/242497670 wait complete. 2024-05-20T17:58:08.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:58:08.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.542+0000 7f16a96eb640 1 -- 172.21.15.129:0/172978563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16a4102960 msgr2=0x7f16a4102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.542+0000 7f16a96eb640 1 --2- 172.21.15.129:0/172978563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16a4102960 0x7f16a4102df0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f168c009900 tx=0x7f168c02a410 comp rx=0 tx=0).stop 2024-05-20T17:58:08.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.542+0000 7f16a96eb640 1 -- 172.21.15.129:0/172978563 shutdown_connections 2024-05-20T17:58:08.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.542+0000 7f16a96eb640 1 --2- 172.21.15.129:0/172978563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16a4103480 0x7f16a4109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.542+0000 7f16a96eb640 1 --2- 172.21.15.129:0/172978563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16a4102960 0x7f16a4102df0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.542+0000 7f16a96eb640 1 --2- 172.21.15.129:0/172978563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a4101f80 0x7f16a4102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.542+0000 7f16a96eb640 1 -- 172.21.15.129:0/172978563 >> 172.21.15.129:0/172978563 conn(0x7f16a40fc360 msgr2=0x7f16a40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:08.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.542+0000 7f16a96eb640 1 -- 172.21.15.129:0/172978563 shutdown_connections 2024-05-20T17:58:08.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.542+0000 7f16a96eb640 1 -- 172.21.15.129:0/172978563 wait complete. 2024-05-20T17:58:08.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.542+0000 7f16a96eb640 1 Processor -- start 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.542+0000 7f16a96eb640 1 -- start start 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a96eb640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16a4101f80 0x7f16a4198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a96eb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16a4102960 0x7f16a41986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a96eb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a4103480 0x7f16a419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a96eb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f16a410d610 con 0x7f16a4102960 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a96eb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16a410d490 con 0x7f16a4103480 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a96eb640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f16a410d790 con 0x7f16a4101f80 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f169bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16a4102960 0x7f16a41986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f169bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16a4102960 0x7f16a41986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49982/0 (socket says 172.21.15.129:49982) 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f169bfff640 1 -- 172.21.15.129:0/2732448814 learned_addr learned my addr 172.21.15.129:0/2732448814 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a2ffd640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16a4101f80 0x7f16a4198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a37fe640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a4103480 0x7f16a419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f169bfff640 1 -- 172.21.15.129:0/2732448814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16a4101f80 msgr2=0x7f16a4198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f169bfff640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16a4101f80 0x7f16a4198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f169bfff640 1 -- 172.21.15.129:0/2732448814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a4103480 msgr2=0x7f16a419f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f169bfff640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a4103480 0x7f16a419f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f169bfff640 1 -- 172.21.15.129:0/2732448814 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a419fca0 con 0x7f16a4102960 2024-05-20T17:58:08.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f169bfff640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16a4102960 0x7f16a41986a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f168c006010 tx=0x7f168c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:08.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a0ff9640 1 -- 172.21.15.129:0/2732448814 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c03f030 con 0x7f16a4102960 2024-05-20T17:58:08.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a0ff9640 1 -- 172.21.15.129:0/2732448814 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f168c003ce0 con 0x7f16a4102960 2024-05-20T17:58:08.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a96eb640 1 -- 172.21.15.129:0/2732448814 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a419ff30 con 0x7f16a4102960 2024-05-20T17:58:08.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a0ff9640 1 -- 172.21.15.129:0/2732448814 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c035070 con 0x7f16a4102960 2024-05-20T17:58:08.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.543+0000 7f16a96eb640 1 -- 172.21.15.129:0/2732448814 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a41a0340 con 0x7f16a4102960 2024-05-20T17:58:08.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.544+0000 7f16a96eb640 1 -- 172.21.15.129:0/2732448814 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a4106920 con 0x7f16a4102960 2024-05-20T17:58:08.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.546+0000 7f16a0ff9640 1 -- 172.21.15.129:0/2732448814 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f168c042030 con 0x7f16a4102960 2024-05-20T17:58:08.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.547+0000 7f16a0ff9640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f16900a2c20 0x7f16900a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.547+0000 7f16a2ffd640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f16900a2c20 0x7f16900a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.547+0000 7f16a0ff9640 1 -- 172.21.15.129:0/2732448814 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f168c0efdf0 con 0x7f16a4102960 2024-05-20T17:58:08.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.548+0000 7f16a2ffd640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f16900a2c20 0x7f16900a5130 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f16a406a230 tx=0x7f169400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:08.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.551+0000 7f16a0ff9640 1 -- 172.21.15.129:0/2732448814 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f168c0bcbe0 con 0x7f16a4102960 2024-05-20T17:58:08.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.764+0000 7f16a96eb640 1 -- 172.21.15.129:0/2732448814 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16a402d460 con 0x7f16a4102960 2024-05-20T17:58:08.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.764+0000 7f16a0ff9640 1 -- 172.21.15.129:0/2732448814 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11829 (secure 0 0 0) 0x7f168c0bc4e0 con 0x7f16a4102960 2024-05-20T17:58:08.763 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:08.763 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:56:49.336282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T17:58:08.764 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 20 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 -- 172.21.15.129:0/2732448814 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f16900a2c20 msgr2=0x7f16900a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f16900a2c20 0x7f16900a5130 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f16a406a230 tx=0x7f169400b390 comp rx=0 tx=0).stop 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 -- 172.21.15.129:0/2732448814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16a4102960 msgr2=0x7f16a41986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16a4102960 0x7f16a41986a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f168c006010 tx=0x7f168c004360 comp rx=0 tx=0).stop 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 -- 172.21.15.129:0/2732448814 shutdown_connections 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f16900a2c20 0x7f16900a5130 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a4103480 0x7f16a419f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16a4102960 0x7f16a41986a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 --2- 172.21.15.129:0/2732448814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16a4101f80 0x7f16a4198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 -- 172.21.15.129:0/2732448814 >> 172.21.15.129:0/2732448814 conn(0x7f16a40fc360 msgr2=0x7f16a4103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 -- 172.21.15.129:0/2732448814 shutdown_connections 2024-05-20T17:58:08.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.767+0000 7f16a96eb640 1 -- 172.21.15.129:0/2732448814 wait complete. 2024-05-20T17:58:08.778 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:58:08.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.892+0000 7f158a65a640 1 -- 172.21.15.129:0/3637019053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15841089f0 msgr2=0x7f158410ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.892+0000 7f158a65a640 1 --2- 172.21.15.129:0/3637019053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15841089f0 0x7f158410ae10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f157c009ca0 tx=0x7f157c02a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.892+0000 7f158a65a640 1 -- 172.21.15.129:0/3637019053 shutdown_connections 2024-05-20T17:58:08.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.892+0000 7f158a65a640 1 --2- 172.21.15.129:0/3637019053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15841089f0 0x7f158410ae10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.892+0000 7f158a65a640 1 --2- 172.21.15.129:0/3637019053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1584103cf0 0x7f1584104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 --2- 172.21.15.129:0/3637019053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1584102ab0 0x7f15841037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 -- 172.21.15.129:0/3637019053 >> 172.21.15.129:0/3637019053 conn(0x7f15840fdcc0 msgr2=0x7f1584100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:08.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 -- 172.21.15.129:0/3637019053 shutdown_connections 2024-05-20T17:58:08.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 -- 172.21.15.129:0/3637019053 wait complete. 2024-05-20T17:58:08.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 Processor -- start 2024-05-20T17:58:08.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 -- start start 2024-05-20T17:58:08.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1584102ab0 0x7f1584199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1584103cf0 0x7f1584199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15841089f0 0x7f15841a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f158410e730 con 0x7f1584103cf0 2024-05-20T17:58:08.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f158410e5b0 con 0x7f15841089f0 2024-05-20T17:58:08.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.893+0000 7f158a65a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f158410e8b0 con 0x7f1584102ab0 2024-05-20T17:58:08.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f157bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1584103cf0 0x7f1584199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f157bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1584103cf0 0x7f1584199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50000/0 (socket says 172.21.15.129:50000) 2024-05-20T17:58:08.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f157bfff640 1 -- 172.21.15.129:0/605277876 learned_addr learned my addr 172.21.15.129:0/605277876 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:08.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f1583fff640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1584102ab0 0x7f1584199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f1588bf7640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15841089f0 0x7f15841a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f157bfff640 1 -- 172.21.15.129:0/605277876 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1584102ab0 msgr2=0x7f1584199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f157bfff640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1584102ab0 0x7f1584199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f157bfff640 1 -- 172.21.15.129:0/605277876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15841089f0 msgr2=0x7f15841a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:08.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f157bfff640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15841089f0 0x7f15841a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:08.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f157bfff640 1 -- 172.21.15.129:0/605277876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15841a0d50 con 0x7f1584103cf0 2024-05-20T17:58:08.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f157bfff640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1584103cf0 0x7f1584199750 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f156c00d770 tx=0x7f156c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:08.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f1581ffb640 1 -- 172.21.15.129:0/605277876 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c017070 con 0x7f1584103cf0 2024-05-20T17:58:08.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f1581ffb640 1 -- 172.21.15.129:0/605277876 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f156c005500 con 0x7f1584103cf0 2024-05-20T17:58:08.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f158a65a640 1 -- 172.21.15.129:0/605277876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15841a0fe0 con 0x7f1584103cf0 2024-05-20T17:58:08.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f1581ffb640 1 -- 172.21.15.129:0/605277876 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c004e00 con 0x7f1584103cf0 2024-05-20T17:58:08.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.894+0000 7f158a65a640 1 -- 172.21.15.129:0/605277876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15841a14a0 con 0x7f1584103cf0 2024-05-20T17:58:08.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.895+0000 7f158a65a640 1 -- 172.21.15.129:0/605277876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1584107800 con 0x7f1584103cf0 2024-05-20T17:58:08.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.897+0000 7f1581ffb640 1 -- 172.21.15.129:0/605277876 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f156c004fa0 con 0x7f1584103cf0 2024-05-20T17:58:08.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.898+0000 7f1581ffb640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f15700a2c20 0x7f15700a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:08.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.898+0000 7f1583fff640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f15700a2c20 0x7f15700a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:08.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.898+0000 7f1581ffb640 1 -- 172.21.15.129:0/605277876 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f156c0cf050 con 0x7f1584103cf0 2024-05-20T17:58:08.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.898+0000 7f1583fff640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f15700a2c20 0x7f15700a5130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f15841036b0 tx=0x7f157400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:08.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:08.902+0000 7f1581ffb640 1 -- 172.21.15.129:0/605277876 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f156c09be80 con 0x7f1584103cf0 2024-05-20T17:58:09.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.081+0000 7f158a65a640 1 -- 172.21.15.129:0/605277876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1584062750 con 0x7f1584103cf0 2024-05-20T17:58:09.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.082+0000 7f1581ffb640 1 -- 172.21.15.129:0/605277876 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22680 (secure 0 0 0) 0x7f156c09b780 con 0x7f1584103cf0 2024-05-20T17:58:09.081 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:09.081 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":88,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:57:49.033961+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:58:09.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.084+0000 7f158a65a640 1 -- 172.21.15.129:0/605277876 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f15700a2c20 msgr2=0x7f15700a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.084+0000 7f158a65a640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f15700a2c20 0x7f15700a5130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f15841036b0 tx=0x7f157400b390 comp rx=0 tx=0).stop 2024-05-20T17:58:09.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.084+0000 7f158a65a640 1 -- 172.21.15.129:0/605277876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1584103cf0 msgr2=0x7f1584199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.084+0000 7f158a65a640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1584103cf0 0x7f1584199750 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f156c00d770 tx=0x7f156c00dc30 comp rx=0 tx=0).stop 2024-05-20T17:58:09.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.084+0000 7f158a65a640 1 -- 172.21.15.129:0/605277876 shutdown_connections 2024-05-20T17:58:09.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.084+0000 7f158a65a640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f15700a2c20 0x7f15700a5130 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.084+0000 7f158a65a640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15841089f0 0x7f15841a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.084+0000 7f158a65a640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1584103cf0 0x7f1584199750 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.084+0000 7f158a65a640 1 --2- 172.21.15.129:0/605277876 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1584102ab0 0x7f1584199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.084+0000 7f158a65a640 1 -- 172.21.15.129:0/605277876 >> 172.21.15.129:0/605277876 conn(0x7f15840fdcc0 msgr2=0x7f15841049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:09.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.085+0000 7f158a65a640 1 -- 172.21.15.129:0/605277876 shutdown_connections 2024-05-20T17:58:09.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.085+0000 7f158a65a640 1 -- 172.21.15.129:0/605277876 wait complete. 2024-05-20T17:58:09.094 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:08 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/442618394' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:09.094 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:08 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1660193834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:09.094 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:08 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/242497670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:09.094 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:08 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2732448814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:09.095 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-20T17:56:42.965073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-20T17:56:44.277499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T17:58:09.096 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-20T17:58:09.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:08 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/442618394' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:09.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:08 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1660193834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:09.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:08 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/242497670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:09.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:08 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2732448814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:09.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:08 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/442618394' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:09.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:08 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1660193834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:09.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:08 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/242497670' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:09.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:08 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2732448814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.212+0000 7fbb5de06640 1 -- 172.21.15.129:0/4133199745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb58102bf0 msgr2=0x7fbb58103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.212+0000 7fbb5de06640 1 --2- 172.21.15.129:0/4133199745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb58102bf0 0x7fbb58103000 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009900 tx=0x7fbb4002a410 comp rx=0 tx=0).stop 2024-05-20T17:58:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.212+0000 7fbb5de06640 1 -- 172.21.15.129:0/4133199745 shutdown_connections 2024-05-20T17:58:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.212+0000 7fbb5de06640 1 --2- 172.21.15.129:0/4133199745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb581078f0 0x7fbb58109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.212+0000 7fbb5de06640 1 --2- 172.21.15.129:0/4133199745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb58102bf0 0x7fbb58103000 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.212+0000 7fbb5de06640 1 --2- 172.21.15.129:0/4133199745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb581019b0 0x7fbb581026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.212+0000 7fbb5de06640 1 -- 172.21.15.129:0/4133199745 >> 172.21.15.129:0/4133199745 conn(0x7fbb580fcba0 msgr2=0x7fbb580ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.212+0000 7fbb5de06640 1 -- 172.21.15.129:0/4133199745 shutdown_connections 2024-05-20T17:58:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.212+0000 7fbb5de06640 1 -- 172.21.15.129:0/4133199745 wait complete. 2024-05-20T17:58:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.212+0000 7fbb5de06640 1 Processor -- start 2024-05-20T17:58:09.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.212+0000 7fbb5de06640 1 -- start start 2024-05-20T17:58:09.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb5de06640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb581019b0 0x7fbb58074810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb5de06640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb58102bf0 0x7fbb5819e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb5de06640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb581078f0 0x7fbb581a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb5de06640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5810d620 con 0x7fbb581019b0 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb5de06640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5810d4a0 con 0x7fbb58102bf0 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb5de06640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5810d7a0 con 0x7fbb581078f0 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb577fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb581019b0 0x7fbb58074810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb577fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb581019b0 0x7fbb58074810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50018/0 (socket says 172.21.15.129:50018) 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb577fe640 1 -- 172.21.15.129:0/4283656256 learned_addr learned my addr 172.21.15.129:0/4283656256 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb4effd640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb58102bf0 0x7fbb5819e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb57fff640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb581078f0 0x7fbb581a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb577fe640 1 -- 172.21.15.129:0/4283656256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb581078f0 msgr2=0x7fbb581a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb577fe640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb581078f0 0x7fbb581a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb577fe640 1 -- 172.21.15.129:0/4283656256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb58102bf0 msgr2=0x7fbb5819e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb577fe640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb58102bf0 0x7fbb5819e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb577fe640 1 -- 172.21.15.129:0/4283656256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb581a0df0 con 0x7fbb581019b0 2024-05-20T17:58:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb577fe640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb581019b0 0x7fbb58074810 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800c970 tx=0x7fbb4800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.213+0000 7fbb557fa640 1 -- 172.21.15.129:0/4283656256 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48004220 con 0x7fbb581019b0 2024-05-20T17:58:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.214+0000 7fbb5de06640 1 -- 172.21.15.129:0/4283656256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb581a10e0 con 0x7fbb581019b0 2024-05-20T17:58:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.214+0000 7fbb557fa640 1 -- 172.21.15.129:0/4283656256 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb480043c0 con 0x7fbb581019b0 2024-05-20T17:58:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.214+0000 7fbb557fa640 1 -- 172.21.15.129:0/4283656256 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48004a00 con 0x7fbb581019b0 2024-05-20T17:58:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.214+0000 7fbb5de06640 1 -- 172.21.15.129:0/4283656256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb581a1630 con 0x7fbb581019b0 2024-05-20T17:58:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.215+0000 7fbb5de06640 1 -- 172.21.15.129:0/4283656256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb58106700 con 0x7fbb581019b0 2024-05-20T17:58:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.216+0000 7fbb557fa640 1 -- 172.21.15.129:0/4283656256 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fbb480079b0 con 0x7fbb581019b0 2024-05-20T17:58:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.217+0000 7fbb557fa640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbb440a2c20 0x7fbb440a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.217+0000 7fbb4effd640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbb440a2c20 0x7fbb440a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.217+0000 7fbb557fa640 1 -- 172.21.15.129:0/4283656256 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fbb480cf340 con 0x7fbb581019b0 2024-05-20T17:58:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.217+0000 7fbb4effd640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbb440a2c20 0x7fbb440a5130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009230 tx=0x7fbb40031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:09.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.220+0000 7fbb557fa640 1 -- 172.21.15.129:0/4283656256 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbb48014030 con 0x7fbb581019b0 2024-05-20T17:58:09.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.397+0000 7fbb5de06640 1 -- 172.21.15.129:0/4283656256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbb5802d4f0 con 0x7fbb581019b0 2024-05-20T17:58:09.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.398+0000 7fbb557fa640 1 -- 172.21.15.129:0/4283656256 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fbb4809bae0 con 0x7fbb581019b0 2024-05-20T17:58:09.397 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T17:58:09.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 -- 172.21.15.129:0/4283656256 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbb440a2c20 msgr2=0x7fbb440a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbb440a2c20 0x7fbb440a5130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009230 tx=0x7fbb40031400 comp rx=0 tx=0).stop 2024-05-20T17:58:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 -- 172.21.15.129:0/4283656256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb581019b0 msgr2=0x7fbb58074810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb581019b0 0x7fbb58074810 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800c970 tx=0x7fbb4800ce30 comp rx=0 tx=0).stop 2024-05-20T17:58:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 -- 172.21.15.129:0/4283656256 shutdown_connections 2024-05-20T17:58:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fbb440a2c20 0x7fbb440a5130 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb58102bf0 0x7fbb5819e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbb581078f0 0x7fbb581a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 --2- 172.21.15.129:0/4283656256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb581019b0 0x7fbb58074810 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 -- 172.21.15.129:0/4283656256 >> 172.21.15.129:0/4283656256 conn(0x7fbb580fcba0 msgr2=0x7fbb581038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 -- 172.21.15.129:0/4283656256 shutdown_connections 2024-05-20T17:58:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.400+0000 7fbb5de06640 1 -- 172.21.15.129:0/4283656256 wait complete. 2024-05-20T17:58:09.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-20T17:58:09.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.524+0000 7f55014d3640 1 -- 172.21.15.129:0/2773243072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc101990 msgr2=0x7f54fc102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.524+0000 7f55014d3640 1 --2- 172.21.15.129:0/2773243072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc101990 0x7f54fc102690 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f54ec009ca0 tx=0x7f54ec02a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.525+0000 7f55014d3640 1 -- 172.21.15.129:0/2773243072 shutdown_connections 2024-05-20T17:58:09.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.525+0000 7f55014d3640 1 --2- 172.21.15.129:0/2773243072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc102bd0 0x7f54fc102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.525+0000 7f55014d3640 1 --2- 172.21.15.129:0/2773243072 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54fc1078d0 0x7f54fc109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.525+0000 7f55014d3640 1 --2- 172.21.15.129:0/2773243072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc101990 0x7f54fc102690 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.525+0000 7f55014d3640 1 -- 172.21.15.129:0/2773243072 >> 172.21.15.129:0/2773243072 conn(0x7f54fc0fcb80 msgr2=0x7f54fc0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.525+0000 7f55014d3640 1 -- 172.21.15.129:0/2773243072 shutdown_connections 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.525+0000 7f55014d3640 1 -- 172.21.15.129:0/2773243072 wait complete. 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.525+0000 7f55014d3640 1 Processor -- start 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.525+0000 7f55014d3640 1 -- start start 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f55014d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc101990 0x7f54fc0698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f55014d3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc102bd0 0x7f54fc19e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f55014d3640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54fc1078d0 0x7f54fc1a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f55014d3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54fc10d5e0 con 0x7f54fc102bd0 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f55014d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54fc10d460 con 0x7f54fc101990 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f55014d3640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f54fc10d760 con 0x7f54fc1078d0 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54fa7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc102bd0 0x7f54fc19e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54fa7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc102bd0 0x7f54fc19e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50024/0 (socket says 172.21.15.129:50024) 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54fa7fc640 1 -- 172.21.15.129:0/1405456065 learned_addr learned my addr 172.21.15.129:0/1405456065 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54faffd640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc101990 0x7f54fc0698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54fb7fe640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54fc1078d0 0x7f54fc1a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54fa7fc640 1 -- 172.21.15.129:0/1405456065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54fc1078d0 msgr2=0x7f54fc1a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54fa7fc640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54fc1078d0 0x7f54fc1a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54fa7fc640 1 -- 172.21.15.129:0/1405456065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc101990 msgr2=0x7f54fc0698b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54fa7fc640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc101990 0x7f54fc0698b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54fa7fc640 1 -- 172.21.15.129:0/1405456065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54fc1a10f0 con 0x7f54fc102bd0 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54fa7fc640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc102bd0 0x7f54fc19e380 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f54e400c890 tx=0x7f54e400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54f3fff640 1 -- 172.21.15.129:0/1405456065 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e4004220 con 0x7f54fc102bd0 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54f3fff640 1 -- 172.21.15.129:0/1405456065 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54e40043c0 con 0x7f54fc102bd0 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f54f3fff640 1 -- 172.21.15.129:0/1405456065 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e40049e0 con 0x7f54fc102bd0 2024-05-20T17:58:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f55014d3640 1 -- 172.21.15.129:0/1405456065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54fc1a13e0 con 0x7f54fc102bd0 2024-05-20T17:58:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.526+0000 7f55014d3640 1 -- 172.21.15.129:0/1405456065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54fc1a1930 con 0x7f54fc102bd0 2024-05-20T17:58:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.527+0000 7f55014d3640 1 -- 172.21.15.129:0/1405456065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54fc1066e0 con 0x7f54fc102bd0 2024-05-20T17:58:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.529+0000 7f54f3fff640 1 -- 172.21.15.129:0/1405456065 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f54e40079b0 con 0x7f54fc102bd0 2024-05-20T17:58:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.529+0000 7f54f3fff640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f54e80a2c20 0x7f54e80a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.529+0000 7f54faffd640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f54e80a2c20 0x7f54e80a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.529+0000 7f54f3fff640 1 -- 172.21.15.129:0/1405456065 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f54e40cf270 con 0x7f54fc102bd0 2024-05-20T17:58:09.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.530+0000 7f54faffd640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f54e80a2c20 0x7f54e80a5130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f54ec000f80 tx=0x7f54ec005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:09.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.533+0000 7f54f3fff640 1 -- 172.21.15.129:0/1405456065 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f54e4014030 con 0x7f54fc102bd0 2024-05-20T17:58:09.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.712+0000 7f55014d3640 1 -- 172.21.15.129:0/1405456065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f54fc02d380 con 0x7f54fc102bd0 2024-05-20T17:58:09.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.713+0000 7f54f3fff640 1 -- 172.21.15.129:0/1405456065 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f54e409c060 con 0x7f54fc102bd0 2024-05-20T17:58:09.712 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-05-20T17:58:09.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 -- 172.21.15.129:0/1405456065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f54e80a2c20 msgr2=0x7f54e80a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f54e80a2c20 0x7f54e80a5130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f54ec000f80 tx=0x7f54ec005fd0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 -- 172.21.15.129:0/1405456065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc102bd0 msgr2=0x7f54fc19e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc102bd0 0x7f54fc19e380 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f54e400c890 tx=0x7f54e400cd50 comp rx=0 tx=0).stop 2024-05-20T17:58:09.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 -- 172.21.15.129:0/1405456065 shutdown_connections 2024-05-20T17:58:09.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f54e80a2c20 0x7f54e80a5130 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54fc1078d0 0x7f54fc1a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc102bd0 0x7f54fc19e380 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 --2- 172.21.15.129:0/1405456065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc101990 0x7f54fc0698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 -- 172.21.15.129:0/1405456065 >> 172.21.15.129:0/1405456065 conn(0x7f54fc0fcb80 msgr2=0x7f54fc1038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:09.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 -- 172.21.15.129:0/1405456065 shutdown_connections 2024-05-20T17:58:09.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.715+0000 7f55014d3640 1 -- 172.21.15.129:0/1405456065 wait complete. 2024-05-20T17:58:09.726 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-20T17:58:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.843+0000 7fdd63a40640 1 -- 172.21.15.129:0/4072261261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd5c101990 msgr2=0x7fdd5c102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.843+0000 7fdd63a40640 1 --2- 172.21.15.129:0/4072261261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd5c101990 0x7fdd5c102690 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009900 tx=0x7fdd5802a410 comp rx=0 tx=0).stop 2024-05-20T17:58:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.843+0000 7fdd63a40640 1 -- 172.21.15.129:0/4072261261 shutdown_connections 2024-05-20T17:58:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.843+0000 7fdd63a40640 1 --2- 172.21.15.129:0/4072261261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd5c1078d0 0x7fdd5c109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.843+0000 7fdd63a40640 1 --2- 172.21.15.129:0/4072261261 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd5c102bd0 0x7fdd5c102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.843+0000 7fdd63a40640 1 --2- 172.21.15.129:0/4072261261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd5c101990 0x7fdd5c102690 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.843+0000 7fdd63a40640 1 -- 172.21.15.129:0/4072261261 >> 172.21.15.129:0/4072261261 conn(0x7fdd5c0fcb80 msgr2=0x7fdd5c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.843+0000 7fdd63a40640 1 -- 172.21.15.129:0/4072261261 shutdown_connections 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.843+0000 7fdd63a40640 1 -- 172.21.15.129:0/4072261261 wait complete. 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.844+0000 7fdd63a40640 1 Processor -- start 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.844+0000 7fdd63a40640 1 -- start start 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.844+0000 7fdd63a40640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd5c101990 0x7fdd5c199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.844+0000 7fdd63a40640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd5c102bd0 0x7fdd5c199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.844+0000 7fdd63a40640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd5c1078d0 0x7fdd5c1a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.844+0000 7fdd63a40640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5c069500 con 0x7fdd5c102bd0 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.844+0000 7fdd63a40640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5c069380 con 0x7fdd5c1078d0 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.844+0000 7fdd63a40640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5c069680 con 0x7fdd5c101990 2024-05-20T17:58:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.844+0000 7fdd60fdb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd5c102bd0 0x7fdd5c199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.844+0000 7fdd60fdb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd5c102bd0 0x7fdd5c199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51654/0 (socket says 172.21.15.129:51654) 2024-05-20T17:58:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.844+0000 7fdd60fdb640 1 -- 172.21.15.129:0/1436766750 learned_addr learned my addr 172.21.15.129:0/1436766750 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd617dc640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd5c101990 0x7fdd5c199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd61fdd640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd5c1078d0 0x7fdd5c1a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd60fdb640 1 -- 172.21.15.129:0/1436766750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd5c101990 msgr2=0x7fdd5c199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd60fdb640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd5c101990 0x7fdd5c199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd60fdb640 1 -- 172.21.15.129:0/1436766750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd5c1078d0 msgr2=0x7fdd5c1a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd60fdb640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd5c1078d0 0x7fdd5c1a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd60fdb640 1 -- 172.21.15.129:0/1436766750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd5c1a1160 con 0x7fdd5c102bd0 2024-05-20T17:58:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd617dc640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd5c101990 0x7fdd5c199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd61fdd640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd5c1078d0 0x7fdd5c1a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd60fdb640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd5c102bd0 0x7fdd5c199b60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00d770 tx=0x7fdd4c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd527fc640 1 -- 172.21.15.129:0/1436766750 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c017070 con 0x7fdd5c102bd0 2024-05-20T17:58:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd63a40640 1 -- 172.21.15.129:0/1436766750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd5c1a1450 con 0x7fdd5c102bd0 2024-05-20T17:58:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd63a40640 1 -- 172.21.15.129:0/1436766750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd5c1a19a0 con 0x7fdd5c102bd0 2024-05-20T17:58:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd527fc640 1 -- 172.21.15.129:0/1436766750 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd4c005500 con 0x7fdd5c102bd0 2024-05-20T17:58:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.845+0000 7fdd527fc640 1 -- 172.21.15.129:0/1436766750 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c004e00 con 0x7fdd5c102bd0 2024-05-20T17:58:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.846+0000 7fdd63a40640 1 -- 172.21.15.129:0/1436766750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd5c1066e0 con 0x7fdd5c102bd0 2024-05-20T17:58:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.848+0000 7fdd527fc640 1 -- 172.21.15.129:0/1436766750 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fdd4c004fa0 con 0x7fdd5c102bd0 2024-05-20T17:58:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.848+0000 7fdd527fc640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdd480a2c20 0x7fdd480a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.848+0000 7fdd617dc640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdd480a2c20 0x7fdd480a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.848+0000 7fdd527fc640 1 -- 172.21.15.129:0/1436766750 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fdd4c0cf140 con 0x7fdd5c102bd0 2024-05-20T17:58:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.849+0000 7fdd617dc640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdd480a2c20 0x7fdd480a5130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd58000c00 tx=0x7fdd58031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:09.852+0000 7fdd527fc640 1 -- 172.21.15.129:0/1436766750 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdd4c09bf30 con 0x7fdd5c102bd0 2024-05-20T17:58:10.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.030+0000 7fdd63a40640 1 -- 172.21.15.129:0/1436766750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdd5c02d3a0 con 0x7fdd5c102bd0 2024-05-20T17:58:10.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.031+0000 7fdd527fc640 1 -- 172.21.15.129:0/1436766750 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fdd4c09b830 con 0x7fdd5c102bd0 2024-05-20T17:58:10.029 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-05-20T17:58:10.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 -- 172.21.15.129:0/1436766750 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdd480a2c20 msgr2=0x7fdd480a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdd480a2c20 0x7fdd480a5130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd58000c00 tx=0x7fdd58031400 comp rx=0 tx=0).stop 2024-05-20T17:58:10.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 -- 172.21.15.129:0/1436766750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd5c102bd0 msgr2=0x7fdd5c199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd5c102bd0 0x7fdd5c199b60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00d770 tx=0x7fdd4c00dc30 comp rx=0 tx=0).stop 2024-05-20T17:58:10.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 -- 172.21.15.129:0/1436766750 shutdown_connections 2024-05-20T17:58:10.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fdd480a2c20 0x7fdd480a5130 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd5c1078d0 0x7fdd5c1a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd5c102bd0 0x7fdd5c199b60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 --2- 172.21.15.129:0/1436766750 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdd5c101990 0x7fdd5c199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 -- 172.21.15.129:0/1436766750 >> 172.21.15.129:0/1436766750 conn(0x7fdd5c0fcb80 msgr2=0x7fdd5c1038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:10.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 -- 172.21.15.129:0/1436766750 shutdown_connections 2024-05-20T17:58:10.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.033+0000 7fdd63a40640 1 -- 172.21.15.129:0/1436766750 wait complete. 2024-05-20T17:58:10.042 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:09 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/605277876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:10.042 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:09 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4283656256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:10.042 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:09 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1405456065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:10.042 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:09 smithi129 ceph-mon[32029]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:10.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:58:10.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:09 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/605277876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:10.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:09 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/4283656256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:10.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:09 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1405456065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:10.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:09 smithi154 ceph-mon[36482]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:10.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.155+0000 7f78f6577640 1 -- 172.21.15.129:0/197976371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0103cf0 msgr2=0x7f78f0104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.155+0000 7f78f6577640 1 --2- 172.21.15.129:0/197976371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0103cf0 0x7f78f0104100 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f78e0009900 tx=0x7f78e002a410 comp rx=0 tx=0).stop 2024-05-20T17:58:10.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.155+0000 7f78f6577640 1 -- 172.21.15.129:0/197976371 shutdown_connections 2024-05-20T17:58:10.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.155+0000 7f78f6577640 1 --2- 172.21.15.129:0/197976371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78f01089f0 0x7f78f010ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.155+0000 7f78f6577640 1 --2- 172.21.15.129:0/197976371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0103cf0 0x7f78f0104100 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.155+0000 7f78f6577640 1 --2- 172.21.15.129:0/197976371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f0102ab0 0x7f78f01037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.155+0000 7f78f6577640 1 -- 172.21.15.129:0/197976371 >> 172.21.15.129:0/197976371 conn(0x7f78f00fdcc0 msgr2=0x7f78f0100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.155+0000 7f78f6577640 1 -- 172.21.15.129:0/197976371 shutdown_connections 2024-05-20T17:58:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.155+0000 7f78f6577640 1 -- 172.21.15.129:0/197976371 wait complete. 2024-05-20T17:58:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.156+0000 7f78f6577640 1 Processor -- start 2024-05-20T17:58:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.156+0000 7f78f6577640 1 -- start start 2024-05-20T17:58:10.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.156+0000 7f78f6577640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78f0102ab0 0x7f78f0199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.156+0000 7f78f6577640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0103cf0 0x7f78f01997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.156+0000 7f78f6577640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f01089f0 0x7f78f01a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.156+0000 7f78f6577640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78f010e780 con 0x7f78f0103cf0 2024-05-20T17:58:10.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.156+0000 7f78f6577640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78f010e600 con 0x7f78f01089f0 2024-05-20T17:58:10.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.156+0000 7f78f6577640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78f010e900 con 0x7f78f0102ab0 2024-05-20T17:58:10.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.156+0000 7f78ef7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0103cf0 0x7f78f01997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.156+0000 7f78ef7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0103cf0 0x7f78f01997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51672/0 (socket says 172.21.15.129:51672) 2024-05-20T17:58:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.156+0000 7f78ef7fe640 1 -- 172.21.15.129:0/38620179 learned_addr learned my addr 172.21.15.129:0/38620179 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78effff640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78f0102ab0 0x7f78f0199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78f4b14640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f01089f0 0x7f78f01a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78ef7fe640 1 -- 172.21.15.129:0/38620179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78f0102ab0 msgr2=0x7f78f0199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78ef7fe640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78f0102ab0 0x7f78f0199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78ef7fe640 1 -- 172.21.15.129:0/38620179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f01089f0 msgr2=0x7f78f01a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78ef7fe640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f01089f0 0x7f78f01a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78ef7fe640 1 -- 172.21.15.129:0/38620179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f01a0dd0 con 0x7f78f0103cf0 2024-05-20T17:58:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78f4b14640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f01089f0 0x7f78f01a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:10.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78ef7fe640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0103cf0 0x7f78f01997d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f78e0031440 tx=0x7f78e0032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:10.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78ed7fa640 1 -- 172.21.15.129:0/38620179 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e003f030 con 0x7f78f0103cf0 2024-05-20T17:58:10.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78f6577640 1 -- 172.21.15.129:0/38620179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f01a1060 con 0x7f78f0103cf0 2024-05-20T17:58:10.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78ed7fa640 1 -- 172.21.15.129:0/38620179 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e0003ce0 con 0x7f78f0103cf0 2024-05-20T17:58:10.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78ed7fa640 1 -- 172.21.15.129:0/38620179 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0035070 con 0x7f78f0103cf0 2024-05-20T17:58:10.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.157+0000 7f78f6577640 1 -- 172.21.15.129:0/38620179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f01a1550 con 0x7f78f0103cf0 2024-05-20T17:58:10.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.158+0000 7f78f6577640 1 -- 172.21.15.129:0/38620179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f0107800 con 0x7f78f0103cf0 2024-05-20T17:58:10.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.159+0000 7f78ed7fa640 1 -- 172.21.15.129:0/38620179 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f78e0042030 con 0x7f78f0103cf0 2024-05-20T17:58:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.160+0000 7f78ed7fa640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78dc0a2c20 0x7f78dc0a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.160+0000 7f78effff640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78dc0a2c20 0x7f78dc0a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.160+0000 7f78ed7fa640 1 -- 172.21.15.129:0/38620179 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f78e00efc70 con 0x7f78f0103cf0 2024-05-20T17:58:10.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.160+0000 7f78effff640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78dc0a2c20 0x7f78dc0a5130 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f78d8005d00 tx=0x7f78d8005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:10.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.164+0000 7f78ed7fa640 1 -- 172.21.15.129:0/38620179 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f78e00bca60 con 0x7f78f0103cf0 2024-05-20T17:58:10.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:09 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/605277876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:10.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:09 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/4283656256' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:10.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:09 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1405456065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:10.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:09 smithi204 ceph-mon[37076]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.376+0000 7f78f6577640 1 -- 172.21.15.129:0/38620179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78f002d2a0 con 0x7f78f0103cf0 2024-05-20T17:58:10.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.377+0000 7f78ed7fa640 1 -- 172.21.15.129:0/38620179 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11829 (secure 0 0 0) 0x7f78e00bc360 con 0x7f78f0103cf0 2024-05-20T17:58:10.376 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:10.376 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:56:49.336282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T17:58:10.376 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 20 2024-05-20T17:58:10.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 -- 172.21.15.129:0/38620179 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78dc0a2c20 msgr2=0x7f78dc0a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78dc0a2c20 0x7f78dc0a5130 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f78d8005d00 tx=0x7f78d8005c50 comp rx=0 tx=0).stop 2024-05-20T17:58:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 -- 172.21.15.129:0/38620179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0103cf0 msgr2=0x7f78f01997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0103cf0 0x7f78f01997d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f78e0031440 tx=0x7f78e0032ce0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 -- 172.21.15.129:0/38620179 shutdown_connections 2024-05-20T17:58:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78dc0a2c20 0x7f78dc0a5130 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f01089f0 0x7f78f01a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78f0103cf0 0x7f78f01997d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 --2- 172.21.15.129:0/38620179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78f0102ab0 0x7f78f0199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 -- 172.21.15.129:0/38620179 >> 172.21.15.129:0/38620179 conn(0x7f78f00fdcc0 msgr2=0x7f78f01049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 -- 172.21.15.129:0/38620179 shutdown_connections 2024-05-20T17:58:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.379+0000 7f78f6577640 1 -- 172.21.15.129:0/38620179 wait complete. 2024-05-20T17:58:10.389 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:58:10.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.502+0000 7f0303b21640 1 -- 172.21.15.129:0/1395043535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02fc101130 msgr2=0x7f02fc101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.502+0000 7f0303b21640 1 --2- 172.21.15.129:0/1395043535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02fc101130 0x7f02fc101e30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009900 tx=0x7f02f802a410 comp rx=0 tx=0).stop 2024-05-20T17:58:10.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.502+0000 7f0303b21640 1 -- 172.21.15.129:0/1395043535 shutdown_connections 2024-05-20T17:58:10.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.502+0000 7f0303b21640 1 --2- 172.21.15.129:0/1395043535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02fc102370 0x7f02fc102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.502+0000 7f0303b21640 1 --2- 172.21.15.129:0/1395043535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02fc107070 0x7f02fc109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.502+0000 7f0303b21640 1 --2- 172.21.15.129:0/1395043535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02fc101130 0x7f02fc101e30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.502+0000 7f0303b21640 1 -- 172.21.15.129:0/1395043535 >> 172.21.15.129:0/1395043535 conn(0x7f02fc0fc320 msgr2=0x7f02fc0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:10.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.502+0000 7f0303b21640 1 -- 172.21.15.129:0/1395043535 shutdown_connections 2024-05-20T17:58:10.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.502+0000 7f0303b21640 1 -- 172.21.15.129:0/1395043535 wait complete. 2024-05-20T17:58:10.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.502+0000 7f0303b21640 1 Processor -- start 2024-05-20T17:58:10.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.502+0000 7f0303b21640 1 -- start start 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f0303b21640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02fc101130 0x7f02fc1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f0303b21640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02fc102370 0x7f02fc1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f0303b21640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02fc107070 0x7f02fc19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f0303b21640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc10d610 con 0x7f02fc101130 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f0303b21640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc10d490 con 0x7f02fc107070 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f0303b21640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc10d790 con 0x7f02fc102370 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03018bd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02fc101130 0x7f02fc1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03018bd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02fc101130 0x7f02fc1981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51682/0 (socket says 172.21.15.129:51682) 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03018bd640 1 -- 172.21.15.129:0/2590168469 learned_addr learned my addr 172.21.15.129:0/2590168469 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03020be640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02fc107070 0x7f02fc19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03010bc640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02fc102370 0x7f02fc1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03018bd640 1 -- 172.21.15.129:0/2590168469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02fc102370 msgr2=0x7f02fc1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03018bd640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02fc102370 0x7f02fc1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03018bd640 1 -- 172.21.15.129:0/2590168469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02fc107070 msgr2=0x7f02fc19f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03018bd640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02fc107070 0x7f02fc19f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03018bd640 1 -- 172.21.15.129:0/2590168469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc19fcf0 con 0x7f02fc101130 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03020be640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02fc107070 0x7f02fc19f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03010bc640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02fc102370 0x7f02fc1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f03018bd640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02fc101130 0x7f02fc1981b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f02f8006010 tx=0x7f02f8031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f02f2ffd640 1 -- 172.21.15.129:0/2590168469 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f803f030 con 0x7f02fc101130 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f02f2ffd640 1 -- 172.21.15.129:0/2590168469 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f8003ad0 con 0x7f02fc101130 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f0303b21640 1 -- 172.21.15.129:0/2590168469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02fc109b80 con 0x7f02fc101130 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f02f2ffd640 1 -- 172.21.15.129:0/2590168469 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8035070 con 0x7f02fc101130 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.503+0000 7f0303b21640 1 -- 172.21.15.129:0/2590168469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02fc10a070 con 0x7f02fc101130 2024-05-20T17:58:10.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.504+0000 7f0303b21640 1 -- 172.21.15.129:0/2590168469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02fc069140 con 0x7f02fc101130 2024-05-20T17:58:10.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.506+0000 7f02f2ffd640 1 -- 172.21.15.129:0/2590168469 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f02f8042030 con 0x7f02fc101130 2024-05-20T17:58:10.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.506+0000 7f02f2ffd640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f02e80a2c20 0x7f02e80a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.506+0000 7f03010bc640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f02e80a2c20 0x7f02e80a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.506+0000 7f02f2ffd640 1 -- 172.21.15.129:0/2590168469 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f02f80efa50 con 0x7f02fc101130 2024-05-20T17:58:10.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.507+0000 7f03010bc640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f02e80a2c20 0x7f02e80a5130 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f02fc199250 tx=0x7f02ec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:10.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.510+0000 7f02f2ffd640 1 -- 172.21.15.129:0/2590168469 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f02f80bc840 con 0x7f02fc101130 2024-05-20T17:58:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.689+0000 7f0303b21640 1 -- 172.21.15.129:0/2590168469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02fc19ff80 con 0x7f02fc101130 2024-05-20T17:58:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.691+0000 7f02f2ffd640 1 -- 172.21.15.129:0/2590168469 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22680 (secure 0 0 0) 0x7f02f80bc140 con 0x7f02fc101130 2024-05-20T17:58:10.690 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:10.690 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":88,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:57:49.033961+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:58:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 -- 172.21.15.129:0/2590168469 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f02e80a2c20 msgr2=0x7f02e80a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f02e80a2c20 0x7f02e80a5130 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f02fc199250 tx=0x7f02ec005f90 comp rx=0 tx=0).stop 2024-05-20T17:58:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 -- 172.21.15.129:0/2590168469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02fc101130 msgr2=0x7f02fc1981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02fc101130 0x7f02fc1981b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f02f8006010 tx=0x7f02f8031b60 comp rx=0 tx=0).stop 2024-05-20T17:58:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 -- 172.21.15.129:0/2590168469 shutdown_connections 2024-05-20T17:58:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f02e80a2c20 0x7f02e80a5130 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02fc107070 0x7f02fc19f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02fc102370 0x7f02fc1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 --2- 172.21.15.129:0/2590168469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02fc101130 0x7f02fc1981b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 -- 172.21.15.129:0/2590168469 >> 172.21.15.129:0/2590168469 conn(0x7f02fc0fc320 msgr2=0x7f02fc103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 -- 172.21.15.129:0/2590168469 shutdown_connections 2024-05-20T17:58:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.693+0000 7f0303b21640 1 -- 172.21.15.129:0/2590168469 wait complete. 2024-05-20T17:58:10.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:58:10.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.817+0000 7f9ab35b5640 1 -- 172.21.15.129:0/855716526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac101970 msgr2=0x7f9aac102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.817+0000 7f9ab35b5640 1 --2- 172.21.15.129:0/855716526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac101970 0x7f9aac102670 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009900 tx=0x7f9aa802a410 comp rx=0 tx=0).stop 2024-05-20T17:58:10.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.817+0000 7f9ab35b5640 1 -- 172.21.15.129:0/855716526 shutdown_connections 2024-05-20T17:58:10.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.817+0000 7f9ab35b5640 1 --2- 172.21.15.129:0/855716526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aac1078b0 0x7f9aac109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.817+0000 7f9ab35b5640 1 --2- 172.21.15.129:0/855716526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aac102bb0 0x7f9aac102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.817+0000 7f9ab35b5640 1 --2- 172.21.15.129:0/855716526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac101970 0x7f9aac102670 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.817+0000 7f9ab35b5640 1 -- 172.21.15.129:0/855716526 >> 172.21.15.129:0/855716526 conn(0x7f9aac0fcb80 msgr2=0x7f9aac0ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.817+0000 7f9ab35b5640 1 -- 172.21.15.129:0/855716526 shutdown_connections 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.817+0000 7f9ab35b5640 1 -- 172.21.15.129:0/855716526 wait complete. 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.818+0000 7f9ab35b5640 1 Processor -- start 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.818+0000 7f9ab35b5640 1 -- start start 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.818+0000 7f9ab35b5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aac101970 0x7f9aac1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.818+0000 7f9ab35b5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac102bb0 0x7f9aac198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.818+0000 7f9ab35b5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aac1078b0 0x7f9aac19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.818+0000 7f9ab35b5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac10d5c0 con 0x7f9aac102bb0 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.818+0000 7f9ab35b5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac10d440 con 0x7f9aac101970 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.818+0000 7f9ab35b5640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac10d740 con 0x7f9aac1078b0 2024-05-20T17:58:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.818+0000 7f9ab0b50640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac102bb0 0x7f9aac198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.818+0000 7f9ab0b50640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac102bb0 0x7f9aac198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51698/0 (socket says 172.21.15.129:51698) 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.818+0000 7f9ab0b50640 1 -- 172.21.15.129:0/3801161045 learned_addr learned my addr 172.21.15.129:0/3801161045 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab1351640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aac101970 0x7f9aac1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab1b52640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aac1078b0 0x7f9aac19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab0b50640 1 -- 172.21.15.129:0/3801161045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aac1078b0 msgr2=0x7f9aac19f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab0b50640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aac1078b0 0x7f9aac19f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab0b50640 1 -- 172.21.15.129:0/3801161045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aac101970 msgr2=0x7f9aac1981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab0b50640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aac101970 0x7f9aac1981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab0b50640 1 -- 172.21.15.129:0/3801161045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aac19fc50 con 0x7f9aac102bb0 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab1351640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aac101970 0x7f9aac1981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab1b52640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aac1078b0 0x7f9aac19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab0b50640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac102bb0 0x7f9aac198700 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00c970 tx=0x7f9a9c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9aa27fc640 1 -- 172.21.15.129:0/3801161045 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c007bb0 con 0x7f9aac102bb0 2024-05-20T17:58:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9aa27fc640 1 -- 172.21.15.129:0/3801161045 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a9c007d50 con 0x7f9aac102bb0 2024-05-20T17:58:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab35b5640 1 -- 172.21.15.129:0/3801161045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aac19ff40 con 0x7f9aac102bb0 2024-05-20T17:58:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9aa27fc640 1 -- 172.21.15.129:0/3801161045 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c00f450 con 0x7f9aac102bb0 2024-05-20T17:58:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.819+0000 7f9ab35b5640 1 -- 172.21.15.129:0/3801161045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aac1a0350 con 0x7f9aac102bb0 2024-05-20T17:58:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.820+0000 7f9ab35b5640 1 -- 172.21.15.129:0/3801161045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aac1066c0 con 0x7f9aac102bb0 2024-05-20T17:58:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.822+0000 7f9aa27fc640 1 -- 172.21.15.129:0/3801161045 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f9a9c004fe0 con 0x7f9aac102bb0 2024-05-20T17:58:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.823+0000 7f9aa27fc640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9a980a2c20 0x7f9a980a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.823+0000 7f9ab1351640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9a980a2c20 0x7f9a980a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.823+0000 7f9aa27fc640 1 -- 172.21.15.129:0/3801161045 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f9a9c0cf410 con 0x7f9aac102bb0 2024-05-20T17:58:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.823+0000 7f9ab1351640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9a980a2c20 0x7f9a980a5130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009230 tx=0x7f9aa8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:10.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:10.827+0000 7f9aa27fc640 1 -- 172.21.15.129:0/3801161045 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9a9c014030 con 0x7f9aac102bb0 2024-05-20T17:58:11.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.042+0000 7f9ab35b5640 1 -- 172.21.15.129:0/3801161045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9aac02d2a0 con 0x7f9aac102bb0 2024-05-20T17:58:11.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.043+0000 7f9aa27fc640 1 -- 172.21.15.129:0/3801161045 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11829 (secure 0 0 0) 0x7f9a9c09c200 con 0x7f9aac102bb0 2024-05-20T17:58:11.042 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:11.042 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:56:49.336282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T17:58:11.042 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 20 2024-05-20T17:58:11.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 -- 172.21.15.129:0/3801161045 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9a980a2c20 msgr2=0x7f9a980a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:11.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9a980a2c20 0x7f9a980a5130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009230 tx=0x7f9aa8000bc0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 -- 172.21.15.129:0/3801161045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac102bb0 msgr2=0x7f9aac198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:11.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac102bb0 0x7f9aac198700 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00c970 tx=0x7f9a9c00ce30 comp rx=0 tx=0).stop 2024-05-20T17:58:11.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 -- 172.21.15.129:0/3801161045 shutdown_connections 2024-05-20T17:58:11.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f9a980a2c20 0x7f9a980a5130 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9aac1078b0 0x7f9aac19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac102bb0 0x7f9aac198700 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 --2- 172.21.15.129:0/3801161045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aac101970 0x7f9aac1981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 -- 172.21.15.129:0/3801161045 >> 172.21.15.129:0/3801161045 conn(0x7f9aac0fcb80 msgr2=0x7f9aac103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:11.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 -- 172.21.15.129:0/3801161045 shutdown_connections 2024-05-20T17:58:11.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.045+0000 7f9ab35b5640 1 -- 172.21.15.129:0/3801161045 wait complete. 2024-05-20T17:58:11.054 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:10 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1436766750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:11.054 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:10 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/38620179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:11.055 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:10 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2590168469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:11.056 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-20T17:56:45.096192+0000', 'modified': '2024-05-20T17:56:49.336282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.154:6837/1478533749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 1478533749}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 1478533749}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.154:6839/793468142', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 793468142}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 793468142}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24560, 'qdb_cluster': [24560]} 2024-05-20T17:58:11.056 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-20T17:58:11.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-20T17:58:11.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:10 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1436766750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:11.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:10 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/38620179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:11.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:10 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2590168469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:11.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.171+0000 7fb14098f640 1 -- 172.21.15.129:0/2736160704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13c105ee0 msgr2=0x7fb13c108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:11.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.171+0000 7fb14098f640 1 --2- 172.21.15.129:0/2736160704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13c105ee0 0x7fb13c108300 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb124009900 tx=0x7fb12402a410 comp rx=0 tx=0).stop 2024-05-20T17:58:11.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.171+0000 7fb14098f640 1 -- 172.21.15.129:0/2736160704 shutdown_connections 2024-05-20T17:58:11.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.171+0000 7fb14098f640 1 --2- 172.21.15.129:0/2736160704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb13c108840 0x7fb13c10ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.171+0000 7fb14098f640 1 --2- 172.21.15.129:0/2736160704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13c105ee0 0x7fb13c108300 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.171+0000 7fb14098f640 1 --2- 172.21.15.129:0/2736160704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c103580 0x7fb13c1059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.171+0000 7fb14098f640 1 -- 172.21.15.129:0/2736160704 >> 172.21.15.129:0/2736160704 conn(0x7fb13c0fcb80 msgr2=0x7fb13c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:11.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.171+0000 7fb14098f640 1 -- 172.21.15.129:0/2736160704 shutdown_connections 2024-05-20T17:58:11.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.171+0000 7fb14098f640 1 -- 172.21.15.129:0/2736160704 wait complete. 2024-05-20T17:58:11.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb14098f640 1 Processor -- start 2024-05-20T17:58:11.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb14098f640 1 -- start start 2024-05-20T17:58:11.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb14098f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb13c103580 0x7fb13c199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb14098f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c105ee0 0x7fb13c199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb14098f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13c108840 0x7fb13c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb14098f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10e850 con 0x7fb13c108840 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb14098f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10e6d0 con 0x7fb13c105ee0 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb14098f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10e9d0 con 0x7fb13c103580 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb13ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13c108840 0x7fb13c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb13ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13c108840 0x7fb13c1a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51708/0 (socket says 172.21.15.129:51708) 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb13ad9d640 1 -- 172.21.15.129:0/484494688 learned_addr learned my addr 172.21.15.129:0/484494688 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb139d9b640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c105ee0 0x7fb13c199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb13a59c640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb13c103580 0x7fb13c199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb13ad9d640 1 -- 172.21.15.129:0/484494688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb13c103580 msgr2=0x7fb13c199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb13ad9d640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb13c103580 0x7fb13c199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb13ad9d640 1 -- 172.21.15.129:0/484494688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c105ee0 msgr2=0x7fb13c199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb13ad9d640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c105ee0 0x7fb13c199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb13ad9d640 1 -- 172.21.15.129:0/484494688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c1a1260 con 0x7fb13c108840 2024-05-20T17:58:11.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb139d9b640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c105ee0 0x7fb13c199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:11.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.172+0000 7fb13ad9d640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13c108840 0x7fb13c1a0bf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb13400b3f0 tx=0x7fb13400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:11.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.173+0000 7fb1337fe640 1 -- 172.21.15.129:0/484494688 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134015020 con 0x7fb13c108840 2024-05-20T17:58:11.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.173+0000 7fb14098f640 1 -- 172.21.15.129:0/484494688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13c1a1550 con 0x7fb13c108840 2024-05-20T17:58:11.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.173+0000 7fb1337fe640 1 -- 172.21.15.129:0/484494688 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb134011d10 con 0x7fb13c108840 2024-05-20T17:58:11.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.173+0000 7fb1337fe640 1 -- 172.21.15.129:0/484494688 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13400f2e0 con 0x7fb13c108840 2024-05-20T17:58:11.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.173+0000 7fb14098f640 1 -- 172.21.15.129:0/484494688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb13c1a1b20 con 0x7fb13c108840 2024-05-20T17:58:11.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.175+0000 7fb1337fe640 1 -- 172.21.15.129:0/484494688 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb134018020 con 0x7fb13c108840 2024-05-20T17:58:11.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.175+0000 7fb147839740 1 Processor -- start 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb147839740 1 -- start start 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb147839740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10945470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb147839740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd109459f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb147839740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb147839740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fd108a94d0 con 0x55fd10584590 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb147839740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fd108513e0 con 0x55fd1057c3b0 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb147839740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fd108a5490 con 0x55fd105a5270 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb139d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb139d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51724/0 (socket says 172.21.15.129:51724) 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb139d9b640 1 -- 172.21.15.129:0/1772471745 learned_addr learned my addr 172.21.15.129:0/1772471745 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb13a59c640 1 --2- 172.21.15.129:0/1772471745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd109459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb1337fe640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb1280a2c20 0x7fb1280a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb139d9b640 1 --2- 172.21.15.129:0/1772471745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10945470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.176+0000 7fb13ad9d640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb1280a2c20 0x7fb1280a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.177+0000 7fb139d9b640 1 -- 172.21.15.129:0/1772471745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 msgr2=0x55fd10945470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.177+0000 7fb139d9b640 1 --2- 172.21.15.129:0/1772471745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10945470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.177+0000 7fb139d9b640 1 -- 172.21.15.129:0/1772471745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 msgr2=0x55fd109459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.177+0000 7fb139d9b640 1 --2- 172.21.15.129:0/1772471745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd109459f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.177+0000 7fb139d9b640 1 -- 172.21.15.129:0/1772471745 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd105bd2d0 con 0x55fd10584590 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.177+0000 7fb1337fe640 1 -- 172.21.15.129:0/484494688 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fb1340cea70 con 0x7fb13c108840 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.177+0000 7fb139d9b640 1 --2- 172.21.15.129:0/1772471745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094c8f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb124005f70 tx=0x7fb124000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437b8ff5bc638948 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.177+0000 7fb13ad9d640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb1280a2c20 0x7fb1280a5130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb1340a3950 tx=0x7fb13406e2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.177+0000 7fb130ff9640 1 -- 172.21.15.129:0/1772471745 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124043030 con 0x55fd10584590 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.177+0000 7fb130ff9640 1 -- 172.21.15.129:0/1772471745 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb124005500 con 0x55fd10584590 2024-05-20T17:58:11.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.177+0000 7fb130ff9640 1 -- 172.21.15.129:0/1772471745 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124035040 con 0x55fd10584590 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.178+0000 7fb147839740 1 -- 172.21.15.129:0/1772471745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 msgr2=0x55fd1094c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.178+0000 7fb147839740 1 --2- 172.21.15.129:0/1772471745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094c8f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb124005f70 tx=0x7fb124000bc0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.178+0000 7fb147839740 1 -- 172.21.15.129:0/1772471745 shutdown_connections 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.178+0000 7fb147839740 1 --2- 172.21.15.129:0/1772471745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd109459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.178+0000 7fb147839740 1 --2- 172.21.15.129:0/1772471745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094c8f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.178+0000 7fb147839740 1 --2- 172.21.15.129:0/1772471745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10945470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.178+0000 7fb147839740 1 -- 172.21.15.129:0/1772471745 >> 172.21.15.129:0/1772471745 conn(0x55fd105993c0 msgr2=0x55fd10940e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.178+0000 7fb147839740 1 -- 172.21.15.129:0/1772471745 shutdown_connections 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.178+0000 7fb147839740 1 -- 172.21.15.129:0/1772471745 wait complete. 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.178+0000 7fb147839740 1 -- start start 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.178+0000 7fb147839740 1 Processor -- start 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb147839740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd1094f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb147839740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb147839740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10956720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb147839740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fd1094d2e0 con 0x55fd10584590 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb147839740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fd108648e0 con 0x55fd1057c3b0 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb147839740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fd1094d460 con 0x55fd105a5270 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb13a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb147839740 10 client.0 _ll_register_callbacks cb 0x55fd108319c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb13a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51728/0 (socket says 172.21.15.129:51728) 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb13a59c640 1 -- 172.21.15.129:0/872446498 learned_addr learned my addr 172.21.15.129:0/872446498 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb139d9b640 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10956720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb139d9b640 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd1094f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb13a59c640 1 -- 172.21.15.129:0/872446498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 msgr2=0x55fd10956720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb13a59c640 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10956720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb13a59c640 1 -- 172.21.15.129:0/872446498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 msgr2=0x55fd1094f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:11.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb13a59c640 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd1094f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb13a59c640 1 -- 172.21.15.129:0/872446498 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd106876b0 con 0x55fd10584590 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb139d9b640 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10956720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb13a59c640 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094f7c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00be50 tx=0x7fb12c007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f699d0d9fe59bc1a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb121ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb121ffb640 1 -- 172.21.15.129:0/872446498 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c007e60 con 0x55fd10584590 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb147839740 1 -- 172.21.15.129:0/872446498 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb124005500 con 0x55fd10584590 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.179+0000 7fb121ffb640 1 -- 172.21.15.129:0/872446498 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fb12c018600 con 0x55fd10584590 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb121ffb640 1 -- 172.21.15.129:0/872446498 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb12c018800 con 0x55fd10584590 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb147839740 10 client.15075 fetch_fsmap learned FSMap version 20 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb147839740 1 -- 172.21.15.129:0/872446498 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fd10959750 con 0x55fd10584590 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb121ffb640 1 -- 172.21.15.129:0/872446498 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb124005500 con 0x55fd10584590 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb121ffb640 1 -- 172.21.15.129:0/872446498 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fb12c020030 con 0x55fd10584590 2024-05-20T17:58:11.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb147839740 10 client.15075 fetch_fsmap finished waiting for FSMap version 20 2024-05-20T17:58:11.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb147839740 10 client.15075 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-20T17:58:11.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb147839740 20 client.15075 populate_metadata read hostname 'smithi129' 2024-05-20T17:58:11.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb147839740 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fd10563e40 0x55fd1095d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb147839740 4 client.15075 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:58:11.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb147839740 1 -- 172.21.15.129:0/872446498 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd108b6c00 con 0x55fd10563e40 2024-05-20T17:58:11.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.180+0000 7fb139d9b640 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fd10563e40 0x55fd1095d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.181+0000 7fb121ffb640 10 client.15075 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:11.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.181+0000 7fb139d9b640 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fd10563e40 0x55fd1095d230 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a56154889c66ffc9 server_cookie=1aa65806d3216403 in_seq=0 out_seq=0 2024-05-20T17:58:11.186 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:10 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1436766750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:11.186 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:10 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/38620179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:11.186 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:10 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2590168469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb121ffb640 1 -- 172.21.15.129:0/872446498 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fd108b6c00 con 0x55fd10563e40 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb121ffb640 10 client.15075 handle_command_reply: tid=0 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 1 client.15075 shutdown 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 1 -- 172.21.15.129:0/872446498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 msgr2=0x55fd1094f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094f7c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00be50 tx=0x7fb12c007680 comp rx=0 tx=0).stop 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 1 -- 172.21.15.129:0/872446498 shutdown_connections 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fd10563e40 0x55fd1095d230 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10956720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd1094f7c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb121ffb640 0 client.15075 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 1 --2- 172.21.15.129:0/872446498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd1094f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 1 -- 172.21.15.129:0/872446498 >> 172.21.15.129:0/872446498 conn(0x55fd105993c0 msgr2=0x55fd1093fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:11.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 1 -- 172.21.15.129:0/872446498 shutdown_connections 2024-05-20T17:58:11.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 1 -- 172.21.15.129:0/872446498 wait complete. 2024-05-20T17:58:11.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.182+0000 7fb147839740 20 client.15075 trim_cache size 0 max 16384 2024-05-20T17:58:11.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.200+0000 7fb147839740 1 Processor -- start 2024-05-20T17:58:11.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.200+0000 7fb147839740 1 -- start start 2024-05-20T17:58:11.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.200+0000 7fb147839740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd10949740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.200+0000 7fb147839740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10956640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb147839740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10949c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb147839740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fd1095b570 con 0x55fd10584590 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb147839740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fd1095bc40 con 0x55fd1057c3b0 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb147839740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fd1095c310 con 0x55fd105a5270 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb13a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10956640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb13a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10956640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51742/0 (socket says 172.21.15.129:51742) 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb13a59c640 1 -- 172.21.15.129:0/1900820630 learned_addr learned my addr 172.21.15.129:0/1900820630 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb139d9b640 1 --2- 172.21.15.129:0/1900820630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10949c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb139d9b640 1 --2- 172.21.15.129:0/1900820630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd10949740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb13a59c640 1 -- 172.21.15.129:0/1900820630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 msgr2=0x55fd10949c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb13a59c640 1 --2- 172.21.15.129:0/1900820630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10949c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb13a59c640 1 -- 172.21.15.129:0/1900820630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 msgr2=0x55fd10949740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb13a59c640 1 --2- 172.21.15.129:0/1900820630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd10949740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb13a59c640 1 -- 172.21.15.129:0/1900820630 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd109417b0 con 0x55fd10584590 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb139d9b640 1 --2- 172.21.15.129:0/1900820630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd10949740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:11.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb13a59c640 1 --2- 172.21.15.129:0/1900820630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10956640 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb12c004a80 tx=0x7fb12c009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294b3406eb5d4cae server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb123fff640 1 -- 172.21.15.129:0/1900820630 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c018400 con 0x55fd10584590 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb123fff640 1 -- 172.21.15.129:0/1900820630 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb12c0185a0 con 0x55fd10584590 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.201+0000 7fb123fff640 1 -- 172.21.15.129:0/1900820630 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c017850 con 0x55fd10584590 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.202+0000 7fb147839740 1 -- 172.21.15.129:0/1900820630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 msgr2=0x55fd10956640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.202+0000 7fb147839740 1 --2- 172.21.15.129:0/1900820630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10956640 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb12c004a80 tx=0x7fb12c009e80 comp rx=0 tx=0).stop 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.202+0000 7fb147839740 1 -- 172.21.15.129:0/1900820630 shutdown_connections 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.202+0000 7fb147839740 1 --2- 172.21.15.129:0/1900820630 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd105a5270 0x55fd10949c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.202+0000 7fb147839740 1 --2- 172.21.15.129:0/1900820630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10956640 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.202+0000 7fb147839740 1 --2- 172.21.15.129:0/1900820630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd1057c3b0 0x55fd10949740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.202+0000 7fb147839740 1 -- 172.21.15.129:0/1900820630 >> 172.21.15.129:0/1900820630 conn(0x55fd105993c0 msgr2=0x55fd109407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.202+0000 7fb147839740 1 -- 172.21.15.129:0/1900820630 shutdown_connections 2024-05-20T17:58:11.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.202+0000 7fb147839740 1 -- 172.21.15.129:0/1900820630 wait complete. 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.202+0000 7fb147839740 1 -- start start 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.202+0000 7fb147839740 1 Processor -- start 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb147839740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd1057c3b0 0x55fd109519b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb147839740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10951ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb147839740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd105a5270 0x55fd10954d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb147839740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fd1095d0b0 con 0x55fd10584590 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb147839740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fd1095c9e0 con 0x55fd105a5270 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb147839740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fd1095d780 con 0x55fd1057c3b0 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb147839740 10 client.0 _ll_register_callbacks cb 0x7fb12c009b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb13a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10951ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb13a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10951ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51748/0 (socket says 172.21.15.129:51748) 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb13a59c640 1 -- 172.21.15.129:0/69759097 learned_addr learned my addr 172.21.15.129:0/69759097 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:11.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb139d9b640 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd1057c3b0 0x55fd109519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb139d9b640 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd105a5270 0x55fd10954d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb13a59c640 1 -- 172.21.15.129:0/69759097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd1057c3b0 msgr2=0x55fd109519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb13a59c640 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd1057c3b0 0x55fd109519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb13a59c640 1 -- 172.21.15.129:0/69759097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd105a5270 msgr2=0x55fd10954d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb13a59c640 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd105a5270 0x55fd10954d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb13a59c640 1 -- 172.21.15.129:0/69759097 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd10955350 con 0x55fd10584590 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb139d9b640 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd105a5270 0x55fd10954d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb13a59c640 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10951ef0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb12c018bd0 tx=0x7fb12c018c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4a1fb74faad183 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb121ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb121ffb640 1 -- 172.21.15.129:0/69759097 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c039060 con 0x55fd10584590 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb121ffb640 1 -- 172.21.15.129:0/69759097 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fb12c0173a0 con 0x55fd10584590 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb147839740 1 -- 172.21.15.129:0/69759097 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb12c0185a0 con 0x55fd10584590 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb121ffb640 1 -- 172.21.15.129:0/69759097 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb12c018400 con 0x55fd10584590 2024-05-20T17:58:11.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb147839740 10 client.15081 fetch_fsmap learned FSMap version 20 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.203+0000 7fb147839740 1 -- 172.21.15.129:0/69759097 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fd109565d0 con 0x55fd10584590 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.204+0000 7fb121ffb640 1 -- 172.21.15.129:0/69759097 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb12c0185a0 con 0x55fd10584590 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.204+0000 7fb121ffb640 1 -- 172.21.15.129:0/69759097 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fb12c020040 con 0x55fd10584590 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.204+0000 7fb147839740 10 client.15081 fetch_fsmap finished waiting for FSMap version 20 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.204+0000 7fb147839740 10 client.15081 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.204+0000 7fb147839740 20 client.15081 populate_metadata read hostname 'smithi129' 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.204+0000 7fb147839740 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fd10539280 0x55fd109c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.204+0000 7fb147839740 4 client.15081 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.204+0000 7fb147839740 1 -- 172.21.15.129:0/69759097 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "status"}) -- 0x55fd108b0f50 con 0x55fd10539280 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.204+0000 7fb139d9b640 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fd10539280 0x55fd109c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.204+0000 7fb121ffb640 10 client.15081 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.204+0000 7fb139d9b640 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fd10539280 0x55fd109c6d90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb404eb008dfa96 server_cookie=6ba5282ca14e1906 in_seq=0 out_seq=0 2024-05-20T17:58:11.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb121ffb640 1 -- 172.21.15.129:0/69759097 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fd108b0f50 con 0x55fd10539280 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb121ffb640 10 client.15081 handle_command_reply: tid=0 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 1 client.15081 shutdown 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 1 -- 172.21.15.129:0/69759097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 msgr2=0x55fd10951ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10951ef0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb12c018bd0 tx=0x7fb12c018c00 comp rx=0 tx=0).stop 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 1 -- 172.21.15.129:0/69759097 shutdown_connections 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fd10539280 0x55fd109c6d90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd105a5270 0x55fd10954d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fd10584590 0x55fd10951ef0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb121ffb640 0 client.15081 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 1 --2- 172.21.15.129:0/69759097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fd1057c3b0 0x55fd109519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 1 -- 172.21.15.129:0/69759097 >> 172.21.15.129:0/69759097 conn(0x55fd105993c0 msgr2=0x55fd1094d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:11.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 1 -- 172.21.15.129:0/69759097 shutdown_connections 2024-05-20T17:58:11.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 1 -- 172.21.15.129:0/69759097 wait complete. 2024-05-20T17:58:11.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb147839740 20 client.15081 trim_cache size 0 max 16384 2024-05-20T17:58:11.211 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T17:58:11.211 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "0f1f0dd2-16d1-11ef-bc9a-c7b262605968", 2024-05-20T17:58:11.211 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-05-20T17:58:11.211 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 24560, 2024-05-20T17:58:11.211 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-05-20T17:58:11.211 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-05-20T17:58:11.211 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-05-20T17:58:11.211 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 85.246304039999998, 2024-05-20T17:58:11.211 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 20, 2024-05-20T17:58:11.212 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 88, 2024-05-20T17:58:11.212 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 86, 2024-05-20T17:58:11.212 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 91.048099567999998 2024-05-20T17:58:11.212 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T17:58:11.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb14098f640 1 -- 172.21.15.129:0/484494688 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb1280a2c20 msgr2=0x7fb1280a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:11.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb14098f640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb1280a2c20 0x7fb1280a5130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb1340a3950 tx=0x7fb13406e2d0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb14098f640 1 -- 172.21.15.129:0/484494688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13c108840 msgr2=0x7fb13c1a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:11.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb14098f640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13c108840 0x7fb13c1a0bf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb13400b3f0 tx=0x7fb13400b8b0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb14098f640 1 -- 172.21.15.129:0/484494688 shutdown_connections 2024-05-20T17:58:11.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb14098f640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb1280a2c20 0x7fb1280a5130 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb14098f640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c105ee0 0x7fb13c199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb14098f640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb13c108840 0x7fb13c1a0bf0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb14098f640 1 --2- 172.21.15.129:0/484494688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb13c103580 0x7fb13c199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:11.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.205+0000 7fb14098f640 1 -- 172.21.15.129:0/484494688 >> 172.21.15.129:0/484494688 conn(0x7fb13c0fcb80 msgr2=0x7fb13c104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:11.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.206+0000 7fb14098f640 1 -- 172.21.15.129:0/484494688 shutdown_connections 2024-05-20T17:58:11.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:11.206+0000 7fb14098f640 1 -- 172.21.15.129:0/484494688 wait complete. 2024-05-20T17:58:11.216 INFO:tasks.cephadm:Setup complete, yielding 2024-05-20T17:58:11.216 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-20T17:58:11.230 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi129.front.sepia.ceph.com 2024-05-20T17:58:11.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch status' 2024-05-20T17:58:11.469 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:58:11.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:11 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3801161045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:11.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:11 smithi129 ceph-mon[32029]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:58:12.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:11 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3801161045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:12.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:11 smithi154 ceph-mon[36482]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:58:12.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:11 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3801161045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:12.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:11 smithi204 ceph-mon[37076]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.156+0000 7f29d3359700 1 -- 172.21.15.129:0/3858661366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29cc075530 msgr2=0x7f29cc105d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.156+0000 7f29d3359700 1 --2- 172.21.15.129:0/3858661366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29cc075530 0x7f29cc105d40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009990 tx=0x7f29bc009ca0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.156+0000 7f29d3359700 1 -- 172.21.15.129:0/3858661366 shutdown_connections 2024-05-20T17:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.156+0000 7f29d3359700 1 --2- 172.21.15.129:0/3858661366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29cc108e80 0x7f29cc10b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.156+0000 7f29d3359700 1 --2- 172.21.15.129:0/3858661366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29cc075530 0x7f29cc105d40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.156+0000 7f29d3359700 1 --2- 172.21.15.129:0/3858661366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29cc106310 0x7f29cc1087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.156+0000 7f29d3359700 1 -- 172.21.15.129:0/3858661366 >> 172.21.15.129:0/3858661366 conn(0x7f29cc0fd710 msgr2=0x7f29cc0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.156+0000 7f29d3359700 1 -- 172.21.15.129:0/3858661366 shutdown_connections 2024-05-20T17:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.156+0000 7f29d3359700 1 -- 172.21.15.129:0/3858661366 wait complete. 2024-05-20T17:58:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.156+0000 7f29d3359700 1 Processor -- start 2024-05-20T17:58:13.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.156+0000 7f29d3359700 1 -- start start 2024-05-20T17:58:13.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d3359700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29cc075530 0x7f29cc103590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:13.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d3359700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29cc106310 0x7f29cc101c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:13.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d3359700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29cc108e80 0x7f29cc102180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:13.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d3359700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc10ed70 con 0x7f29cc075530 2024-05-20T17:58:13.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d3359700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc10ebf0 con 0x7f29cc106310 2024-05-20T17:58:13.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d3359700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc10eef0 con 0x7f29cc108e80 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d10f5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29cc075530 0x7f29cc103590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d10f5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29cc075530 0x7f29cc103590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51758/0 (socket says 172.21.15.129:51758) 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d10f5700 1 -- 172.21.15.129:0/2404070408 learned_addr learned my addr 172.21.15.129:0/2404070408 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d18f6700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29cc108e80 0x7f29cc102180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d08f4700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29cc106310 0x7f29cc101c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d10f5700 1 -- 172.21.15.129:0/2404070408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29cc108e80 msgr2=0x7f29cc102180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d10f5700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29cc108e80 0x7f29cc102180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d10f5700 1 -- 172.21.15.129:0/2404070408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29cc106310 msgr2=0x7f29cc101c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d10f5700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29cc106310 0x7f29cc101c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d10f5700 1 -- 172.21.15.129:0/2404070408 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29cc1028e0 con 0x7f29cc075530 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d18f6700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29cc108e80 0x7f29cc102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d10f5700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29cc075530 0x7f29cc103590 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0098e0 tx=0x7f29bc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:13.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29c27fc700 1 -- 172.21.15.129:0/2404070408 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc003c70 con 0x7f29cc075530 2024-05-20T17:58:13.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29c27fc700 1 -- 172.21.15.129:0/2404070408 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29bc004830 con 0x7f29cc075530 2024-05-20T17:58:13.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d3359700 1 -- 172.21.15.129:0/2404070408 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29cc1a3910 con 0x7f29cc075530 2024-05-20T17:58:13.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29c27fc700 1 -- 172.21.15.129:0/2404070408 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc0054e0 con 0x7f29cc075530 2024-05-20T17:58:13.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.157+0000 7f29d3359700 1 -- 172.21.15.129:0/2404070408 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29cc1a3cb0 con 0x7f29cc075530 2024-05-20T17:58:13.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.158+0000 7f29d3359700 1 -- 172.21.15.129:0/2404070408 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29cc050170 con 0x7f29cc075530 2024-05-20T17:58:13.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.160+0000 7f29c27fc700 1 -- 172.21.15.129:0/2404070408 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f29bc003e10 con 0x7f29cc075530 2024-05-20T17:58:13.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.161+0000 7f29c27fc700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f29b80a2cb0 0x7f29b80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:13.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.161+0000 7f29d08f4700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f29b80a2cb0 0x7f29b80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:13.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.161+0000 7f29c27fc700 1 -- 172.21.15.129:0/2404070408 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f29bc0d0770 con 0x7f29cc075530 2024-05-20T17:58:13.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.161+0000 7f29d08f4700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f29b80a2cb0 0x7f29b80a51c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29c8005950 tx=0x7f29c8005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:13.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.164+0000 7f29c27fc700 1 -- 172.21.15.129:0/2404070408 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f29bc013070 con 0x7f29cc075530 2024-05-20T17:58:13.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.379+0000 7f29d3359700 1 -- 172.21.15.129:0/2404070408 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f29cc062a80 con 0x7f29b80a2cb0 2024-05-20T17:58:13.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.380+0000 7f29c27fc700 1 -- 172.21.15.129:0/2404070408 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f29cc062a80 con 0x7f29b80a2cb0 2024-05-20T17:58:13.378 INFO:teuthology.orchestra.run.smithi129.stdout:Backend: cephadm 2024-05-20T17:58:13.378 INFO:teuthology.orchestra.run.smithi129.stdout:Available: Yes 2024-05-20T17:58:13.379 INFO:teuthology.orchestra.run.smithi129.stdout:Paused: No 2024-05-20T17:58:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 -- 172.21.15.129:0/2404070408 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f29b80a2cb0 msgr2=0x7f29b80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f29b80a2cb0 0x7f29b80a51c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29c8005950 tx=0x7f29c8005ed0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 -- 172.21.15.129:0/2404070408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29cc075530 msgr2=0x7f29cc103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29cc075530 0x7f29cc103590 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0098e0 tx=0x7f29bc0047b0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 -- 172.21.15.129:0/2404070408 shutdown_connections 2024-05-20T17:58:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f29b80a2cb0 0x7f29b80a51c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29cc108e80 0x7f29cc102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29cc075530 0x7f29cc103590 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 --2- 172.21.15.129:0/2404070408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29cc106310 0x7f29cc101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 -- 172.21.15.129:0/2404070408 >> 172.21.15.129:0/2404070408 conn(0x7f29cc0fd710 msgr2=0x7f29cc104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:13.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 -- 172.21.15.129:0/2404070408 shutdown_connections 2024-05-20T17:58:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:13.382+0000 7f29d3359700 1 -- 172.21.15.129:0/2404070408 wait complete. 2024-05-20T17:58:14.079 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:13 smithi129 ceph-mon[32029]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:14.079 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:13 smithi129 ceph-mon[32029]: pgmap v331: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:14.107 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-20T17:58:14.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:13 smithi154 ceph-mon[36482]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:14.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:13 smithi154 ceph-mon[36482]: pgmap v331: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:14.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:13 smithi204 ceph-mon[37076]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:14.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:13 smithi204 ceph-mon[37076]: pgmap v331: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:14.355 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:58:15.899 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:15 smithi129 ceph-mon[32029]: pgmap v332: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:16.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:15 smithi154 ceph-mon[36482]: pgmap v332: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:16.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:15 smithi204 ceph-mon[37076]: pgmap v332: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.183+0000 7f2ad5944700 1 -- 172.21.15.129:0/2392635419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ad0103cc0 msgr2=0x7f2ad01060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.183+0000 7f2ad5944700 1 --2- 172.21.15.129:0/2392635419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ad0103cc0 0x7f2ad01060e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009990 tx=0x7f2ac0009ca0 comp rx=0 tx=0).stop 2024-05-20T17:58:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.183+0000 7f2ad5944700 1 -- 172.21.15.129:0/2392635419 shutdown_connections 2024-05-20T17:58:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.183+0000 7f2ad5944700 1 --2- 172.21.15.129:0/2392635419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad0108f80 0x7f2ad010b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.183+0000 7f2ad5944700 1 --2- 172.21.15.129:0/2392635419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ad0103cc0 0x7f2ad01060e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.183+0000 7f2ad5944700 1 --2- 172.21.15.129:0/2392635419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ad0106620 0x7f2ad0108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.183+0000 7f2ad5944700 1 -- 172.21.15.129:0/2392635419 >> 172.21.15.129:0/2392635419 conn(0x7f2ad00fd700 msgr2=0x7f2ad00ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.183+0000 7f2ad5944700 1 -- 172.21.15.129:0/2392635419 shutdown_connections 2024-05-20T17:58:16.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.183+0000 7f2ad5944700 1 -- 172.21.15.129:0/2392635419 wait complete. 2024-05-20T17:58:16.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.184+0000 7f2ad5944700 1 Processor -- start 2024-05-20T17:58:16.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.184+0000 7f2ad5944700 1 -- start start 2024-05-20T17:58:16.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.184+0000 7f2ad5944700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ad0103cc0 0x7f2ad0199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.184+0000 7f2ad5944700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ad0106620 0x7f2ad0199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.184+0000 7f2ad5944700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad0108f80 0x7f2ad01a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.184+0000 7f2ad5944700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad010ed50 con 0x7f2ad0103cc0 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.184+0000 7f2ad5944700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad010ebd0 con 0x7f2ad0106620 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.184+0000 7f2ad5944700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad010eed0 con 0x7f2ad0108f80 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.184+0000 7f2aceffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ad0103cc0 0x7f2ad0199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2aceffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ad0103cc0 0x7f2ad0199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51778/0 (socket says 172.21.15.129:51778) 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2aceffd700 1 -- 172.21.15.129:0/2866168034 learned_addr learned my addr 172.21.15.129:0/2866168034 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2ace7fc700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ad0106620 0x7f2ad0199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2acf7fe700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad0108f80 0x7f2ad01a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2aceffd700 1 -- 172.21.15.129:0/2866168034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad0108f80 msgr2=0x7f2ad01a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2aceffd700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad0108f80 0x7f2ad01a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2aceffd700 1 -- 172.21.15.129:0/2866168034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ad0106620 msgr2=0x7f2ad0199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2aceffd700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ad0106620 0x7f2ad0199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2aceffd700 1 -- 172.21.15.129:0/2866168034 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad01a13d0 con 0x7f2ad0103cc0 2024-05-20T17:58:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2aceffd700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ad0103cc0 0x7f2ad0199870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00098e0 tx=0x7f2ac00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2ad4942700 1 -- 172.21.15.129:0/2866168034 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0003c70 con 0x7f2ad0103cc0 2024-05-20T17:58:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2ad5944700 1 -- 172.21.15.129:0/2866168034 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad01a1660 con 0x7f2ad0103cc0 2024-05-20T17:58:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2ad4942700 1 -- 172.21.15.129:0/2866168034 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac0004830 con 0x7f2ad0103cc0 2024-05-20T17:58:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2ad4942700 1 -- 172.21.15.129:0/2866168034 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0005330 con 0x7f2ad0103cc0 2024-05-20T17:58:16.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.185+0000 7f2ad5944700 1 -- 172.21.15.129:0/2866168034 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad01a1ab0 con 0x7f2ad0103cc0 2024-05-20T17:58:16.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.186+0000 7f2ad5944700 1 -- 172.21.15.129:0/2866168034 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad0050170 con 0x7f2ad0103cc0 2024-05-20T17:58:16.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.187+0000 7f2ad4942700 1 -- 172.21.15.129:0/2866168034 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2ac00054d0 con 0x7f2ad0103cc0 2024-05-20T17:58:16.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.188+0000 7f2ad4942700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2abc0a2cb0 0x7f2abc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:16.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.188+0000 7f2ace7fc700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2abc0a2cb0 0x7f2abc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:16.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.188+0000 7f2ad4942700 1 -- 172.21.15.129:0/2866168034 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f2ac00d0600 con 0x7f2ad0103cc0 2024-05-20T17:58:16.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.189+0000 7f2ace7fc700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2abc0a2cb0 0x7f2abc0a51c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2ad019a920 tx=0x7f2ab800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:16.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.192+0000 7f2ad4942700 1 -- 172.21.15.129:0/2866168034 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2ac0013070 con 0x7f2ad0103cc0 2024-05-20T17:58:16.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.396+0000 7f2ad5944700 1 -- 172.21.15.129:0/2866168034 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f2ad0062a80 con 0x7f2abc0a2cb0 2024-05-20T17:58:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.410+0000 7f2ad4942700 1 -- 172.21.15.129:0/2866168034 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f2ad0062a80 con 0x7f2abc0a2cb0 2024-05-20T17:58:16.409 INFO:teuthology.orchestra.run.smithi129.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-20T17:58:16.409 INFO:teuthology.orchestra.run.smithi129.stdout:mds.a smithi129 running (99s) 15s ago 99s 83.6M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 9e62bdb20c43 2024-05-20T17:58:16.409 INFO:teuthology.orchestra.run.smithi129.stdout:mds.b smithi154 running (96s) 29s ago 96s 95.0M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 7b06ee3220d5 2024-05-20T17:58:16.409 INFO:teuthology.orchestra.run.smithi129.stdout:mds.c smithi204 running (110s) 29s ago 110s 80.2M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 3938857b9ea6 2024-05-20T17:58:16.409 INFO:teuthology.orchestra.run.smithi129.stdout:mds.d smithi129 running (119s) 15s ago 118s 84.0M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 9acc43aa3ab0 2024-05-20T17:58:16.409 INFO:teuthology.orchestra.run.smithi129.stdout:mds.e smithi154 running (116s) 29s ago 115s 80.2M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 9cd4a95b1ef8 2024-05-20T17:58:16.409 INFO:teuthology.orchestra.run.smithi129.stdout:mds.f smithi204 running (113s) 29s ago 112s 81.4M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 756cb0b9c666 2024-05-20T17:58:16.409 INFO:teuthology.orchestra.run.smithi129.stdout:mds.g smithi129 running (102s) 15s ago 101s 80.6M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 ebdb95f64391 2024-05-20T17:58:16.409 INFO:teuthology.orchestra.run.smithi129.stdout:mds.h smithi154 running (107s) 29s ago 107s 84.3M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 7367402f2476 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:mds.i smithi204 running (104s) 29s ago 104s 82.7M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 ab802f7dcb52 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:mds.j smithi129 running (2m) 15s ago 2m 83.0M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 9f47465ebe1d 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:mds.k smithi154 running (2m) 29s ago 2m 81.3M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 32548813b8d3 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:mds.l smithi204 running (2m) 29s ago 2m 82.8M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 eab0016ca770 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:mgr.x smithi129 *:9283,8765 running (9m) 15s ago 9m 583M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 6908fde9dc21 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:mgr.y smithi154 *:8443,8765 running (7m) 29s ago 7m 524M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 f3b4f8606f46 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:mgr.z smithi204 *:8443,8765 running (7m) 29s ago 7m 521M - 19.0.0-2744-gd2b507c7 cfcc61b2d476 bd030ed21de2 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:mon.a smithi129 running (9m) 15s ago 9m 117M 2048M 19.0.0-2744-gd2b507c7 cfcc61b2d476 532f464357bd 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:mon.b smithi154 running (7m) 29s ago 7m 109M 2048M 19.0.0-2744-gd2b507c7 cfcc61b2d476 d52372016c01 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:mon.c smithi204 running (7m) 29s ago 7m 105M 2048M 19.0.0-2744-gd2b507c7 cfcc61b2d476 9a5d5ce86e41 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 smithi129 running (7m) 15s ago 7m 122M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 d05547a54800 2024-05-20T17:58:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 smithi154 running (6m) 29s ago 6m 114M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 3370cb90079b 2024-05-20T17:58:16.411 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 smithi204 running (6m) 29s ago 6m 113M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 5edf29a0299f 2024-05-20T17:58:16.411 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 smithi129 running (5m) 15s ago 5m 113M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 718af1f5aca8 2024-05-20T17:58:16.411 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 smithi154 running (5m) 29s ago 5m 118M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 3eed87ec9193 2024-05-20T17:58:16.411 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 smithi204 running (5m) 29s ago 5m 125M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 77ed5e1c6f5a 2024-05-20T17:58:16.411 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 smithi129 running (4m) 15s ago 4m 112M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 dbe7a1fde5a1 2024-05-20T17:58:16.411 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 smithi154 running (4m) 29s ago 4m 112M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 7c61d5d20223 2024-05-20T17:58:16.411 INFO:teuthology.orchestra.run.smithi129.stdout:osd.8 smithi204 running (3m) 29s ago 3m 120M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 ebef3b669d5e 2024-05-20T17:58:16.411 INFO:teuthology.orchestra.run.smithi129.stdout:osd.9 smithi129 running (3m) 15s ago 3m 119M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 40d2e565f682 2024-05-20T17:58:16.411 INFO:teuthology.orchestra.run.smithi129.stdout:osd.10 smithi154 running (2m) 29s ago 2m 121M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 19ad0d44f55d 2024-05-20T17:58:16.411 INFO:teuthology.orchestra.run.smithi129.stdout:osd.11 smithi204 running (2m) 29s ago 2m 122M 4272M 19.0.0-2744-gd2b507c7 cfcc61b2d476 9ca218df3a6b 2024-05-20T17:58:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 -- 172.21.15.129:0/2866168034 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2abc0a2cb0 msgr2=0x7f2abc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2abc0a2cb0 0x7f2abc0a51c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2ad019a920 tx=0x7f2ab800b410 comp rx=0 tx=0).stop 2024-05-20T17:58:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 -- 172.21.15.129:0/2866168034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ad0103cc0 msgr2=0x7f2ad0199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ad0103cc0 0x7f2ad0199870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00098e0 tx=0x7f2ac00047b0 comp rx=0 tx=0).stop 2024-05-20T17:58:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 -- 172.21.15.129:0/2866168034 shutdown_connections 2024-05-20T17:58:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f2abc0a2cb0 0x7f2abc0a51c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad0108f80 0x7f2ad01a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ad0103cc0 0x7f2ad0199870 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 --2- 172.21.15.129:0/2866168034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ad0106620 0x7f2ad0199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 -- 172.21.15.129:0/2866168034 >> 172.21.15.129:0/2866168034 conn(0x7f2ad00fd700 msgr2=0x7f2ad01047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 -- 172.21.15.129:0/2866168034 shutdown_connections 2024-05-20T17:58:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:16.413+0000 7f2ad5944700 1 -- 172.21.15.129:0/2866168034 wait complete. 2024-05-20T17:58:17.040 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:16 smithi129 ceph-mon[32029]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:17.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-20T17:58:17.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:16 smithi154 ceph-mon[36482]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:17.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:16 smithi204 ceph-mon[37076]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:17.283 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:58:17.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:17 smithi129 ceph-mon[32029]: pgmap v333: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:58:18.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:17 smithi154 ceph-mon[36482]: pgmap v333: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:58:18.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:17 smithi204 ceph-mon[37076]: pgmap v333: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:58:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.967+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/2650668159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f441035b0 msgr2=0x7f3f441039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.967+0000 7f3f4a5aa700 1 --2- 172.21.15.129:0/2650668159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f441035b0 0x7f3f441039c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c0099c0 tx=0x7f3f2c009cd0 comp rx=0 tx=0).stop 2024-05-20T17:58:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.967+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/2650668159 shutdown_connections 2024-05-20T17:58:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.967+0000 7f3f4a5aa700 1 --2- 172.21.15.129:0/2650668159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f441026b0 0x7f3f44102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.967+0000 7f3f4a5aa700 1 --2- 172.21.15.129:0/2650668159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3f44103f00 0x7f3f4410a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.967+0000 7f3f4a5aa700 1 --2- 172.21.15.129:0/2650668159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f441035b0 0x7f3f441039c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.967+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/2650668159 >> 172.21.15.129:0/2650668159 conn(0x7f3f440fcea0 msgr2=0x7f3f440ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:18.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.967+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/2650668159 shutdown_connections 2024-05-20T17:58:18.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.967+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/2650668159 wait complete. 2024-05-20T17:58:18.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:18 smithi129 ceph-mon[32029]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-20T17:58:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.967+0000 7f3f4a5aa700 1 Processor -- start 2024-05-20T17:58:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.967+0000 7f3f4a5aa700 1 -- start start 2024-05-20T17:58:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f4a5aa700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f441026b0 0x7f3f441a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f4a5aa700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3f441035b0 0x7f3f441a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f4a5aa700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f44103f00 0x7f3f441a7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f4a5aa700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4410ded0 con 0x7f3f44103f00 2024-05-20T17:58:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f4a5aa700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4410dd50 con 0x7f3f441026b0 2024-05-20T17:58:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f4a5aa700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4410e050 con 0x7f3f441035b0 2024-05-20T17:58:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f48b47700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f44103f00 0x7f3f441a7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f48b47700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f44103f00 0x7f3f441a7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51792/0 (socket says 172.21.15.129:51792) 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f48b47700 1 -- 172.21.15.129:0/805997755 learned_addr learned my addr 172.21.15.129:0/805997755 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f3bfff700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3f441035b0 0x7f3f441a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f43fff700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f441026b0 0x7f3f441a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f48b47700 1 -- 172.21.15.129:0/805997755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3f441035b0 msgr2=0x7f3f441a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f48b47700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3f441035b0 0x7f3f441a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f48b47700 1 -- 172.21.15.129:0/805997755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f441026b0 msgr2=0x7f3f441a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f48b47700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f441026b0 0x7f3f441a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f48b47700 1 -- 172.21.15.129:0/805997755 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f441a7c20 con 0x7f3f44103f00 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.968+0000 7f3f48b47700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f44103f00 0x7f3f441a7580 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c00ada0 tx=0x7f3f3c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.969+0000 7f3f41ffb700 1 -- 172.21.15.129:0/805997755 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c017020 con 0x7f3f44103f00 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.969+0000 7f3f41ffb700 1 -- 172.21.15.129:0/805997755 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f3c00cdb0 con 0x7f3f44103f00 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.969+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/805997755 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f441a7f10 con 0x7f3f44103f00 2024-05-20T17:58:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.969+0000 7f3f41ffb700 1 -- 172.21.15.129:0/805997755 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c0053e0 con 0x7f3f44103f00 2024-05-20T17:58:18.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.969+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/805997755 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f441a83a0 con 0x7f3f44103f00 2024-05-20T17:58:18.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.970+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/805997755 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f44050170 con 0x7f3f44103f00 2024-05-20T17:58:18.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.971+0000 7f3f41ffb700 1 -- 172.21.15.129:0/805997755 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3f3c00c8d0 con 0x7f3f44103f00 2024-05-20T17:58:18.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.972+0000 7f3f41ffb700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3f300a2c20 0x7f3f300a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.972+0000 7f3f43fff700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3f300a2c20 0x7f3f300a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.972+0000 7f3f41ffb700 1 -- 172.21.15.129:0/805997755 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f3f3c0cecc0 con 0x7f3f44103f00 2024-05-20T17:58:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.973+0000 7f3f43fff700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3f300a2c20 0x7f3f300a5130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3f44103540 tx=0x7f3f340073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:18.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:18.976+0000 7f3f41ffb700 1 -- 172.21.15.129:0/805997755 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3f3c021020 con 0x7f3f44103f00 2024-05-20T17:58:19.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:18 smithi154 ceph-mon[36482]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-20T17:58:19.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.179+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/805997755 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f3f440fee70 con 0x7f3f300a2c20 2024-05-20T17:58:19.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:18 smithi204 ceph-mon[37076]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-20T17:58:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.185+0000 7f3f41ffb700 1 -- 172.21.15.129:0/805997755 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f3f440fee70 con 0x7f3f300a2c20 2024-05-20T17:58:19.184 INFO:teuthology.orchestra.run.smithi129.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-20T17:58:19.184 INFO:teuthology.orchestra.run.smithi129.stdout:mds.all 12/12 32s ago 2m smithi129=a;smithi129=d;smithi129=g;smithi129=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-05-20T17:58:19.184 INFO:teuthology.orchestra.run.smithi129.stdout:mgr 3/3 32s ago 7m smithi129=x;smithi154=y;smithi204=z;count:3 2024-05-20T17:58:19.184 INFO:teuthology.orchestra.run.smithi129.stdout:mon 3/3 32s ago 7m smithi129:172.21.15.129=a;smithi154:172.21.15.154=b;smithi204:172.21.15.204=c;count:3 2024-05-20T17:58:19.184 INFO:teuthology.orchestra.run.smithi129.stdout:osd 12 32s ago - 2024-05-20T17:58:19.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.187+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/805997755 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3f300a2c20 msgr2=0x7f3f300a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:19.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.187+0000 7f3f4a5aa700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3f300a2c20 0x7f3f300a5130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3f44103540 tx=0x7f3f340073d0 comp rx=0 tx=0).stop 2024-05-20T17:58:19.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.187+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/805997755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f44103f00 msgr2=0x7f3f441a7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:19.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.187+0000 7f3f4a5aa700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f44103f00 0x7f3f441a7580 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c00ada0 tx=0x7f3f3c00c330 comp rx=0 tx=0).stop 2024-05-20T17:58:19.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.188+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/805997755 shutdown_connections 2024-05-20T17:58:19.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.188+0000 7f3f4a5aa700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f3f300a2c20 0x7f3f300a5130 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:19.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.188+0000 7f3f4a5aa700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f441026b0 0x7f3f441a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:19.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.188+0000 7f3f4a5aa700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f44103f00 0x7f3f441a7580 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:19.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.188+0000 7f3f4a5aa700 1 --2- 172.21.15.129:0/805997755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3f441035b0 0x7f3f441a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:19.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.188+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/805997755 >> 172.21.15.129:0/805997755 conn(0x7f3f440fcea0 msgr2=0x7f3f440fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:19.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.188+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/805997755 shutdown_connections 2024-05-20T17:58:19.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:19.188+0000 7f3f4a5aa700 1 -- 172.21.15.129:0/805997755 wait complete. 2024-05-20T17:58:19.864 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:19 smithi129 ceph-mon[32029]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:19.865 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:19 smithi129 ceph-mon[32029]: pgmap v334: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:19.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-20T17:58:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:58:20.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:19 smithi154 ceph-mon[36482]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:20.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:19 smithi154 ceph-mon[36482]: pgmap v334: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:20.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:19 smithi204 ceph-mon[37076]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:20.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:19 smithi204 ceph-mon[37076]: pgmap v334: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:21.842 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:21 smithi129 ceph-mon[32029]: pgmap v335: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:21.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.957+0000 7f04b0663700 1 -- 172.21.15.129:0/4242943808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103200 msgr2=0x7f04a8103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:21.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.957+0000 7f04b0663700 1 --2- 172.21.15.129:0/4242943808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103200 0x7f04a8103610 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0498009990 tx=0x7f0498009ca0 comp rx=0 tx=0).stop 2024-05-20T17:58:21.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.958+0000 7f04b0663700 1 -- 172.21.15.129:0/4242943808 shutdown_connections 2024-05-20T17:58:21.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.958+0000 7f04b0663700 1 --2- 172.21.15.129:0/4242943808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a8103ba0 0x7f04a810a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:21.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.958+0000 7f04b0663700 1 --2- 172.21.15.129:0/4242943808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04a81020d0 0x7f04a81024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:21.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.958+0000 7f04b0663700 1 --2- 172.21.15.129:0/4242943808 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103200 0x7f04a8103610 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:21.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.958+0000 7f04b0663700 1 -- 172.21.15.129:0/4242943808 >> 172.21.15.129:0/4242943808 conn(0x7f04a80fd710 msgr2=0x7f04a80ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.958+0000 7f04b0663700 1 -- 172.21.15.129:0/4242943808 shutdown_connections 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.958+0000 7f04b0663700 1 -- 172.21.15.129:0/4242943808 wait complete. 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.958+0000 7f04b0663700 1 Processor -- start 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04b0663700 1 -- start start 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04b0663700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04a81020d0 0x7f04a8198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04b0663700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103200 0x7f04a8198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04b0663700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a8103ba0 0x7f04a819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04b0663700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f04a810dca0 con 0x7f04a8103200 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04b0663700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04a810db20 con 0x7f04a8103ba0 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04b0663700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f04a810de20 con 0x7f04a81020d0 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04adbfe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103200 0x7f04a8198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04adbfe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103200 0x7f04a8198cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39830/0 (socket says 172.21.15.129:39830) 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04adbfe700 1 -- 172.21.15.129:0/3226497360 learned_addr learned my addr 172.21.15.129:0/3226497360 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04aec00700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a8103ba0 0x7f04a819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04ae3ff700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04a81020d0 0x7f04a8198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04adbfe700 1 -- 172.21.15.129:0/3226497360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04a81020d0 msgr2=0x7f04a8198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04adbfe700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04a81020d0 0x7f04a8198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04adbfe700 1 -- 172.21.15.129:0/3226497360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a8103ba0 msgr2=0x7f04a819fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04adbfe700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a8103ba0 0x7f04a819fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04adbfe700 1 -- 172.21.15.129:0/3226497360 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a8074870 con 0x7f04a8103200 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04aec00700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a8103ba0 0x7f04a819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f04adbfe700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103200 0x7f04a8198cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f04980098e0 tx=0x7f04980047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.959+0000 7f049f7fe700 1 -- 172.21.15.129:0/3226497360 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0498003c70 con 0x7f04a8103200 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.960+0000 7f04b0663700 1 -- 172.21.15.129:0/3226497360 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a8074b00 con 0x7f04a8103200 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.960+0000 7f049f7fe700 1 -- 172.21.15.129:0/3226497360 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0498004830 con 0x7f04a8103200 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.960+0000 7f049f7fe700 1 -- 172.21.15.129:0/3226497360 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04980054e0 con 0x7f04a8103200 2024-05-20T17:58:21.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.960+0000 7f04b0663700 1 -- 172.21.15.129:0/3226497360 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a8074f80 con 0x7f04a8103200 2024-05-20T17:58:21.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.961+0000 7f04b0663700 1 -- 172.21.15.129:0/3226497360 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a8050170 con 0x7f04a8103200 2024-05-20T17:58:21.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.962+0000 7f049f7fe700 1 -- 172.21.15.129:0/3226497360 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0498003e10 con 0x7f04a8103200 2024-05-20T17:58:21.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.963+0000 7f049f7fe700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04940a2c20 0x7f04940a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:21.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.963+0000 7f04ae3ff700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04940a2c20 0x7f04940a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:21.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.963+0000 7f049f7fe700 1 -- 172.21.15.129:0/3226497360 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f04980d0590 con 0x7f04a8103200 2024-05-20T17:58:21.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.963+0000 7f04ae3ff700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04940a2c20 0x7f04940a5130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f04a4005950 tx=0x7f04a40058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:21.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:21.967+0000 7f049f7fe700 1 -- 172.21.15.129:0/3226497360 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0498013070 con 0x7f04a8103200 2024-05-20T17:58:22.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:21 smithi154 ceph-mon[36482]: pgmap v335: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:22.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.168+0000 7f04b0663700 1 -- 172.21.15.129:0/3226497360 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f04a8075240 con 0x7f04940a2c20 2024-05-20T17:58:22.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.169+0000 7f049f7fe700 1 -- 172.21.15.129:0/3226497360 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f04a8075240 con 0x7f04940a2c20 2024-05-20T17:58:22.168 INFO:teuthology.orchestra.run.smithi129.stdout:HOST ADDR LABELS STATUS 2024-05-20T17:58:22.168 INFO:teuthology.orchestra.run.smithi129.stdout:smithi129 172.21.15.129 2024-05-20T17:58:22.168 INFO:teuthology.orchestra.run.smithi129.stdout:smithi154 172.21.15.154 2024-05-20T17:58:22.168 INFO:teuthology.orchestra.run.smithi129.stdout:smithi204 172.21.15.204 2024-05-20T17:58:22.168 INFO:teuthology.orchestra.run.smithi129.stdout:3 hosts in cluster 2024-05-20T17:58:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.171+0000 7f04b0663700 1 -- 172.21.15.129:0/3226497360 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04940a2c20 msgr2=0x7f04940a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.171+0000 7f04b0663700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04940a2c20 0x7f04940a5130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f04a4005950 tx=0x7f04a40058e0 comp rx=0 tx=0).stop 2024-05-20T17:58:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.171+0000 7f04b0663700 1 -- 172.21.15.129:0/3226497360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103200 msgr2=0x7f04a8198cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.171+0000 7f04b0663700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103200 0x7f04a8198cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f04980098e0 tx=0x7f04980047b0 comp rx=0 tx=0).stop 2024-05-20T17:58:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.171+0000 7f04b0663700 1 -- 172.21.15.129:0/3226497360 shutdown_connections 2024-05-20T17:58:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.171+0000 7f04b0663700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f04940a2c20 0x7f04940a5130 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.171+0000 7f04b0663700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a8103ba0 0x7f04a819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.171+0000 7f04b0663700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04a81020d0 0x7f04a8198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.171+0000 7f04b0663700 1 --2- 172.21.15.129:0/3226497360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103200 0x7f04a8198cc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.171+0000 7f04b0663700 1 -- 172.21.15.129:0/3226497360 >> 172.21.15.129:0/3226497360 conn(0x7f04a80fd710 msgr2=0x7f04a8106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.172+0000 7f04b0663700 1 -- 172.21.15.129:0/3226497360 shutdown_connections 2024-05-20T17:58:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:22.172+0000 7f04b0663700 1 -- 172.21.15.129:0/3226497360 wait complete. 2024-05-20T17:58:22.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:21 smithi204 ceph-mon[37076]: pgmap v335: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:22.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-20T17:58:23.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:22 smithi154 ceph-mon[36482]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:23.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:22 smithi129 ceph-mon[32029]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:23.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:22 smithi204 ceph-mon[37076]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:23.220 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:58:24.050 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:23 smithi129 ceph-mon[32029]: pgmap v336: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:24.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:23 smithi154 ceph-mon[36482]: pgmap v336: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:24.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:23 smithi204 ceph-mon[37076]: pgmap v336: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:24.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.989+0000 7f820b7cc700 1 -- 172.21.15.129:0/645946658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8204106d80 msgr2=0x7f8204107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:24.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.989+0000 7f820b7cc700 1 --2- 172.21.15.129:0/645946658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8204106d80 0x7f8204107190 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00b0a0 tx=0x7f81fc00b3b0 comp rx=0 tx=0).stop 2024-05-20T17:58:24.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.989+0000 7f820b7cc700 1 -- 172.21.15.129:0/645946658 shutdown_connections 2024-05-20T17:58:24.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.989+0000 7f820b7cc700 1 --2- 172.21.15.129:0/645946658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8204101ac0 0x7f8204103ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:24.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.989+0000 7f820b7cc700 1 --2- 172.21.15.129:0/645946658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8204106d80 0x7f8204107190 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:24.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.989+0000 7f820b7cc700 1 --2- 172.21.15.129:0/645946658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8204104420 0x7f8204106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:24.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.989+0000 7f820b7cc700 1 -- 172.21.15.129:0/645946658 >> 172.21.15.129:0/645946658 conn(0x7f82040fd710 msgr2=0x7f82040ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.989+0000 7f820b7cc700 1 -- 172.21.15.129:0/645946658 shutdown_connections 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.989+0000 7f820b7cc700 1 -- 172.21.15.129:0/645946658 wait complete. 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f820b7cc700 1 Processor -- start 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f820b7cc700 1 -- start start 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f820b7cc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8204101ac0 0x7f820419fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f820b7cc700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8204104420 0x7f82041a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f820b7cc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8204106d80 0x7f82041a72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f820b7cc700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f82041153f0 con 0x7f8204101ac0 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f820b7cc700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8204115270 con 0x7f8204106d80 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f820b7cc700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8204115570 con 0x7f8204104420 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f8209568700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8204101ac0 0x7f820419fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f8209568700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8204101ac0 0x7f820419fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39852/0 (socket says 172.21.15.129:39852) 2024-05-20T17:58:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f8209568700 1 -- 172.21.15.129:0/2277235539 learned_addr learned my addr 172.21.15.129:0/2277235539 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:24.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f8209d69700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8204106d80 0x7f82041a72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:24.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.990+0000 7f8208d67700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8204104420 0x7f82041a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:24.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f8209568700 1 -- 172.21.15.129:0/2277235539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8204104420 msgr2=0x7f82041a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:24.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f8209568700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8204104420 0x7f82041a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:24.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f8209568700 1 -- 172.21.15.129:0/2277235539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8204106d80 msgr2=0x7f82041a72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:24.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f8209568700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8204106d80 0x7f82041a72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:24.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f8209568700 1 -- 172.21.15.129:0/2277235539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82041a7950 con 0x7f8204101ac0 2024-05-20T17:58:24.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f8208d67700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8204104420 0x7f82041a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:24.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f8209568700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8204101ac0 0x7f820419fdf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f820000c8f0 tx=0x7f820000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:24.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f81fa7fc700 1 -- 172.21.15.129:0/2277235539 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200007940 con 0x7f8204101ac0 2024-05-20T17:58:24.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f820b7cc700 1 -- 172.21.15.129:0/2277235539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82041a7c40 con 0x7f8204101ac0 2024-05-20T17:58:24.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f81fa7fc700 1 -- 172.21.15.129:0/2277235539 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f820000f450 con 0x7f8204101ac0 2024-05-20T17:58:24.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f81fa7fc700 1 -- 172.21.15.129:0/2277235539 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200018660 con 0x7f8204101ac0 2024-05-20T17:58:24.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.991+0000 7f820b7cc700 1 -- 172.21.15.129:0/2277235539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82041a8180 con 0x7f8204101ac0 2024-05-20T17:58:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.992+0000 7f820b7cc700 1 -- 172.21.15.129:0/2277235539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8204050170 con 0x7f8204101ac0 2024-05-20T17:58:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.993+0000 7f81fa7fc700 1 -- 172.21.15.129:0/2277235539 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f820000f600 con 0x7f8204101ac0 2024-05-20T17:58:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.994+0000 7f81fa7fc700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f81f00a2c20 0x7f81f00a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.994+0000 7f8208d67700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f81f00a2c20 0x7f81f00a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.994+0000 7f81fa7fc700 1 -- 172.21.15.129:0/2277235539 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f82000ceee0 con 0x7f8204101ac0 2024-05-20T17:58:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.994+0000 7f8208d67700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f81f00a2c20 0x7f81f00a5130 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f82041a0ea0 tx=0x7f81f4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:24.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:24.999+0000 7f81fa7fc700 1 -- 172.21.15.129:0/2277235539 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8200014030 con 0x7f8204101ac0 2024-05-20T17:58:25.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.202+0000 7f820b7cc700 1 -- 172.21.15.129:0/2277235539 --> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f8204062a80 con 0x7f81f00a2c20 2024-05-20T17:58:25.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.204+0000 7f81fa7fc700 1 -- 172.21.15.129:0/2277235539 <== mgr.14150 v2:172.21.15.129:6800/2870089245 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f8204062a80 con 0x7f81f00a2c20 2024-05-20T17:58:25.203 INFO:teuthology.orchestra.run.smithi129.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-20T17:58:25.203 INFO:teuthology.orchestra.run.smithi129.stdout:smithi129 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300HY400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-20T17:58:25.203 INFO:teuthology.orchestra.run.smithi129.stdout:smithi154 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233003B400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-05-20T17:58:25.203 INFO:teuthology.orchestra.run.smithi129.stdout:smithi204 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GD400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-05-20T17:58:25.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 -- 172.21.15.129:0/2277235539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f81f00a2c20 msgr2=0x7f81f00a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:25.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f81f00a2c20 0x7f81f00a5130 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f82041a0ea0 tx=0x7f81f4008040 comp rx=0 tx=0).stop 2024-05-20T17:58:25.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 -- 172.21.15.129:0/2277235539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8204101ac0 msgr2=0x7f820419fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:25.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8204101ac0 0x7f820419fdf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f820000c8f0 tx=0x7f820000cc00 comp rx=0 tx=0).stop 2024-05-20T17:58:25.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 -- 172.21.15.129:0/2277235539 shutdown_connections 2024-05-20T17:58:25.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f81f00a2c20 0x7f81f00a5130 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:25.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8204106d80 0x7f82041a72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:25.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8204101ac0 0x7f820419fdf0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:25.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 --2- 172.21.15.129:0/2277235539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8204104420 0x7f82041a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:25.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 -- 172.21.15.129:0/2277235539 >> 172.21.15.129:0/2277235539 conn(0x7f82040fd710 msgr2=0x7f8204102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:25.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 -- 172.21.15.129:0/2277235539 shutdown_connections 2024-05-20T17:58:25.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:25.206+0000 7f820b7cc700 1 -- 172.21.15.129:0/2277235539 wait complete. 2024-05-20T17:58:25.847 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-20T17:58:25.856 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi129.front.sepia.ceph.com 2024-05-20T17:58:25.856 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-20T17:58:25.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:25 smithi129 ceph-mon[32029]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:25.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:25 smithi129 ceph-mon[32029]: pgmap v337: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:26.098 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:58:26.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:25 smithi154 ceph-mon[36482]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:26.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:25 smithi154 ceph-mon[36482]: pgmap v337: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:26.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:25 smithi204 ceph-mon[37076]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:26.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:25 smithi204 ceph-mon[37076]: pgmap v337: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:28.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.024+0000 7ff3ebece700 1 -- 172.21.15.129:0/2051035838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e41031d0 msgr2=0x7ff3e41035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:28.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.024+0000 7ff3ebece700 1 --2- 172.21.15.129:0/2051035838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e41031d0 0x7ff3e41035e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009990 tx=0x7ff3e0009ca0 comp rx=0 tx=0).stop 2024-05-20T17:58:28.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.024+0000 7ff3ebece700 1 -- 172.21.15.129:0/2051035838 shutdown_connections 2024-05-20T17:58:28.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.024+0000 7ff3ebece700 1 --2- 172.21.15.129:0/2051035838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3e4104ca0 0x7ff3e410b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:28.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.024+0000 7ff3ebece700 1 --2- 172.21.15.129:0/2051035838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e41031d0 0x7ff3e41035e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:28.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.024+0000 7ff3ebece700 1 --2- 172.21.15.129:0/2051035838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3e4104300 0x7ff3e4104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:28.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.024+0000 7ff3ebece700 1 -- 172.21.15.129:0/2051035838 >> 172.21.15.129:0/2051035838 conn(0x7ff3e40fe810 msgr2=0x7ff3e4100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:28.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.024+0000 7ff3ebece700 1 -- 172.21.15.129:0/2051035838 shutdown_connections 2024-05-20T17:58:28.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.024+0000 7ff3ebece700 1 -- 172.21.15.129:0/2051035838 wait complete. 2024-05-20T17:58:28.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ebece700 1 Processor -- start 2024-05-20T17:58:28.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ebece700 1 -- start start 2024-05-20T17:58:28.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ebece700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3e41031d0 0x7ff3e40762e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:28.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ebece700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3e4104300 0x7ff3e4074990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:28.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ebece700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e4104ca0 0x7ff3e4074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:28.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ebece700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e410ed70 con 0x7ff3e4104ca0 2024-05-20T17:58:28.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ebece700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e410ebf0 con 0x7ff3e41031d0 2024-05-20T17:58:28.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ebece700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e410eef0 con 0x7ff3e4104300 2024-05-20T17:58:28.025 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:27 smithi129 ceph-mon[32029]: pgmap v338: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:28.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ea46b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e4104ca0 0x7ff3e4074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ea46b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e4104ca0 0x7ff3e4074ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39874/0 (socket says 172.21.15.129:39874) 2024-05-20T17:58:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ea46b700 1 -- 172.21.15.129:0/3638566007 learned_addr learned my addr 172.21.15.129:0/3638566007 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3e9c6a700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3e41031d0 0x7ff3e40762e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3e9469700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3e4104300 0x7ff3e4074990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3e9c6a700 1 -- 172.21.15.129:0/3638566007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3e4104300 msgr2=0x7ff3e4074990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3e9c6a700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3e4104300 0x7ff3e4074990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3e9c6a700 1 -- 172.21.15.129:0/3638566007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e4104ca0 msgr2=0x7ff3e4074ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3e9c6a700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e4104ca0 0x7ff3e4074ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3e9c6a700 1 -- 172.21.15.129:0/3638566007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e4075630 con 0x7ff3e41031d0 2024-05-20T17:58:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.025+0000 7ff3ea46b700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e4104ca0 0x7ff3e4074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.026+0000 7ff3e9c6a700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3e41031d0 0x7ff3e40762e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0000c00 tx=0x7ff3e00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:28.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.026+0000 7ff3daffd700 1 -- 172.21.15.129:0/3638566007 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0027070 con 0x7ff3e41031d0 2024-05-20T17:58:28.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.026+0000 7ff3ebece700 1 -- 172.21.15.129:0/3638566007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e41a3960 con 0x7ff3e41031d0 2024-05-20T17:58:28.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.026+0000 7ff3daffd700 1 -- 172.21.15.129:0/3638566007 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3e0004150 con 0x7ff3e41031d0 2024-05-20T17:58:28.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.026+0000 7ff3daffd700 1 -- 172.21.15.129:0/3638566007 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0022df0 con 0x7ff3e41031d0 2024-05-20T17:58:28.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.026+0000 7ff3ebece700 1 -- 172.21.15.129:0/3638566007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e41a3d50 con 0x7ff3e41031d0 2024-05-20T17:58:28.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.027+0000 7ff3ebece700 1 -- 172.21.15.129:0/3638566007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e4050170 con 0x7ff3e41031d0 2024-05-20T17:58:28.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.028+0000 7ff3daffd700 1 -- 172.21.15.129:0/3638566007 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff3e002a070 con 0x7ff3e41031d0 2024-05-20T17:58:28.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.029+0000 7ff3daffd700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3d00a2c20 0x7ff3d00a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:28.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.029+0000 7ff3e9469700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3d00a2c20 0x7ff3d00a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:28.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.029+0000 7ff3daffd700 1 -- 172.21.15.129:0/3638566007 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7ff3e00d0150 con 0x7ff3e41031d0 2024-05-20T17:58:28.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.029+0000 7ff3e9469700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3d00a2c20 0x7ff3d00a5130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4075c90 tx=0x7ff3d400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:28.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.033+0000 7ff3daffd700 1 -- 172.21.15.129:0/3638566007 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff3e0013070 con 0x7ff3e41031d0 2024-05-20T17:58:28.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:27 smithi154 ceph-mon[36482]: pgmap v338: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:28.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:27 smithi204 ceph-mon[37076]: pgmap v338: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:28.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.271+0000 7ff3ebece700 1 -- 172.21.15.129:0/3638566007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7ff3e40008d0 con 0x7ff3e41031d0 2024-05-20T17:58:28.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.272+0000 7ff3daffd700 1 -- 172.21.15.129:0/3638566007 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) ==== 76+0+3022 (secure 0 0 0) 0x7ff3e009d130 con 0x7ff3e41031d0 2024-05-20T17:58:28.270 INFO:teuthology.orchestra.run.smithi129.stdout:e20 2024-05-20T17:58:28.271 INFO:teuthology.orchestra.run.smithi129.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-20T17:58:28.271 INFO:teuthology.orchestra.run.smithi129.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-20T17:58:28.271 INFO:teuthology.orchestra.run.smithi129.stdout:legacy client fscid: 1 2024-05-20T17:58:28.271 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:28.271 INFO:teuthology.orchestra.run.smithi129.stdout:Filesystem 'cephfs' (1) 2024-05-20T17:58:28.271 INFO:teuthology.orchestra.run.smithi129.stdout:fs_name cephfs 2024-05-20T17:58:28.271 INFO:teuthology.orchestra.run.smithi129.stdout:epoch 20 2024-05-20T17:58:28.271 INFO:teuthology.orchestra.run.smithi129.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-20T17:58:28.271 INFO:teuthology.orchestra.run.smithi129.stdout:created 2024-05-20T17:56:45.096192+0000 2024-05-20T17:58:28.271 INFO:teuthology.orchestra.run.smithi129.stdout:modified 2024-05-20T17:56:49.336282+0000 2024-05-20T17:58:28.271 INFO:teuthology.orchestra.run.smithi129.stdout:tableserver 0 2024-05-20T17:58:28.272 INFO:teuthology.orchestra.run.smithi129.stdout:root 0 2024-05-20T17:58:28.272 INFO:teuthology.orchestra.run.smithi129.stdout:session_timeout 300 2024-05-20T17:58:28.272 INFO:teuthology.orchestra.run.smithi129.stdout:session_autoclose 300 2024-05-20T17:58:28.272 INFO:teuthology.orchestra.run.smithi129.stdout:max_file_size 1099511627776 2024-05-20T17:58:28.272 INFO:teuthology.orchestra.run.smithi129.stdout:max_xattr_size 65536 2024-05-20T17:58:28.272 INFO:teuthology.orchestra.run.smithi129.stdout:required_client_features {} 2024-05-20T17:58:28.272 INFO:teuthology.orchestra.run.smithi129.stdout:last_failure 0 2024-05-20T17:58:28.272 INFO:teuthology.orchestra.run.smithi129.stdout:last_failure_osd_epoch 0 2024-05-20T17:58:28.272 INFO:teuthology.orchestra.run.smithi129.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-20T17:58:28.272 INFO:teuthology.orchestra.run.smithi129.stdout:max_mds 1 2024-05-20T17:58:28.272 INFO:teuthology.orchestra.run.smithi129.stdout:in 0 2024-05-20T17:58:28.273 INFO:teuthology.orchestra.run.smithi129.stdout:up {0=24560} 2024-05-20T17:58:28.273 INFO:teuthology.orchestra.run.smithi129.stdout:failed 2024-05-20T17:58:28.273 INFO:teuthology.orchestra.run.smithi129.stdout:damaged 2024-05-20T17:58:28.273 INFO:teuthology.orchestra.run.smithi129.stdout:stopped 2024-05-20T17:58:28.273 INFO:teuthology.orchestra.run.smithi129.stdout:data_pools [3] 2024-05-20T17:58:28.273 INFO:teuthology.orchestra.run.smithi129.stdout:metadata_pool 2 2024-05-20T17:58:28.273 INFO:teuthology.orchestra.run.smithi129.stdout:inline_data disabled 2024-05-20T17:58:28.273 INFO:teuthology.orchestra.run.smithi129.stdout:balancer 2024-05-20T17:58:28.273 INFO:teuthology.orchestra.run.smithi129.stdout:bal_rank_mask -1 2024-05-20T17:58:28.273 INFO:teuthology.orchestra.run.smithi129.stdout:standby_count_wanted 0 2024-05-20T17:58:28.273 INFO:teuthology.orchestra.run.smithi129.stdout:qdb_cluster leader: 24560 members: 24560 2024-05-20T17:58:28.274 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b{0:24560} state up:active seq 3 addr [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.274 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.h{0:24521} state up:standby-replay seq 1 addr [v2:172.21.15.154:6836/1478533749,v1:172.21.15.154:6837/1478533749] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.274 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:28.274 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:28.274 INFO:teuthology.orchestra.run.smithi129.stdout:Standby daemons: 2024-05-20T17:58:28.274 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:28.274 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.j{-1:14559} state up:standby seq 1 addr [v2:172.21.15.129:6834/2428051280,v1:172.21.15.129:6835/2428051280] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.274 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d{-1:14577} state up:standby seq 1 addr [v2:172.21.15.129:6836/2359452744,v1:172.21.15.129:6837/2359452744] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.274 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.g{-1:14652} state up:standby seq 1 addr [v2:172.21.15.129:6838/2206181560,v1:172.21.15.129:6839/2206181560] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.274 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a{-1:14661} state up:standby seq 1 addr [v2:172.21.15.129:6840/1623696468,v1:172.21.15.129:6841/1623696468] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.274 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.l{-1:24418} state up:standby seq 1 addr [v2:172.21.15.204:6832/3042064152,v1:172.21.15.204:6833/3042064152] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.275 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.k{-1:24455} state up:standby seq 1 addr [v2:172.21.15.154:6832/3269888601,v1:172.21.15.154:6833/3269888601] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.275 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.f{-1:24469} state up:standby seq 1 addr [v2:172.21.15.204:6834/4037294668,v1:172.21.15.204:6835/4037294668] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.275 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c{-1:24481} state up:standby seq 1 addr [v2:172.21.15.204:6836/259458619,v1:172.21.15.204:6837/259458619] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.275 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.e{-1:24488} state up:standby seq 1 addr [v2:172.21.15.154:6834/3512970065,v1:172.21.15.154:6835/3512970065] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.275 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.i{-1:24511} state up:standby seq 1 addr [v2:172.21.15.204:6838/2880425505,v1:172.21.15.204:6839/2880425505] compat {c=[1],r=[1],i=[1fff]}] 2024-05-20T17:58:28.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 -- 172.21.15.129:0/3638566007 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3d00a2c20 msgr2=0x7ff3d00a5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:28.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3d00a2c20 0x7ff3d00a5130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4075c90 tx=0x7ff3d400b410 comp rx=0 tx=0).stop 2024-05-20T17:58:28.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 -- 172.21.15.129:0/3638566007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3e41031d0 msgr2=0x7ff3e40762e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:28.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3e41031d0 0x7ff3e40762e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0000c00 tx=0x7ff3e00047b0 comp rx=0 tx=0).stop 2024-05-20T17:58:28.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 -- 172.21.15.129:0/3638566007 shutdown_connections 2024-05-20T17:58:28.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ff3d00a2c20 0x7ff3d00a5130 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:28.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3e41031d0 0x7ff3e40762e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:28.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e4104ca0 0x7ff3e4074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:28.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 --2- 172.21.15.129:0/3638566007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3e4104300 0x7ff3e4074990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:28.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 -- 172.21.15.129:0/3638566007 >> 172.21.15.129:0/3638566007 conn(0x7ff3e40fe810 msgr2=0x7ff3e4107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:28.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 -- 172.21.15.129:0/3638566007 shutdown_connections 2024-05-20T17:58:28.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:28.274+0000 7ff3ebece700 1 -- 172.21.15.129:0/3638566007 wait complete. 2024-05-20T17:58:28.277 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 20 2024-05-20T17:58:28.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:28 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3638566007' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-20T17:58:28.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-20T17:58:29.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:28 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3638566007' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-20T17:58:29.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:28 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3638566007' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-20T17:58:29.253 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:58:29.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:29 smithi129 ceph-mon[32029]: pgmap v339: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:30.080 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:29 smithi204 ceph-mon[37076]: pgmap v339: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:30.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:29 smithi154 ceph-mon[36482]: pgmap v339: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.949+0000 7f346af26700 1 -- 172.21.15.129:0/3939069392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34641020c0 msgr2=0x7f34641024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.949+0000 7f346af26700 1 --2- 172.21.15.129:0/3939069392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34641020c0 0x7f34641024d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3458009990 tx=0x7f3458009ca0 comp rx=0 tx=0).stop 2024-05-20T17:58:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.949+0000 7f346af26700 1 -- 172.21.15.129:0/3939069392 shutdown_connections 2024-05-20T17:58:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.949+0000 7f346af26700 1 --2- 172.21.15.129:0/3939069392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3464103b90 0x7f346410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.949+0000 7f346af26700 1 --2- 172.21.15.129:0/3939069392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34641020c0 0x7f34641024d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.949+0000 7f346af26700 1 --2- 172.21.15.129:0/3939069392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34641031f0 0x7f3464103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.949+0000 7f346af26700 1 -- 172.21.15.129:0/3939069392 >> 172.21.15.129:0/3939069392 conn(0x7f34640fd700 msgr2=0x7f34640ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.949+0000 7f346af26700 1 -- 172.21.15.129:0/3939069392 shutdown_connections 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.949+0000 7f346af26700 1 -- 172.21.15.129:0/3939069392 wait complete. 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f346af26700 1 Processor -- start 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f346af26700 1 -- start start 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f346af26700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34641020c0 0x7f3464105e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f346af26700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34641031f0 0x7f3464104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f346af26700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3464103b90 0x7f3464104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f346af26700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f346410dcd0 con 0x7f34641031f0 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f346af26700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f346410db50 con 0x7f3464103b90 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f346af26700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f346410de50 con 0x7f34641020c0 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f3463fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34641031f0 0x7f3464104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:30.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f3463fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34641031f0 0x7f3464104530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34412/0 (socket says 172.21.15.129:34412) 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f3463fff700 1 -- 172.21.15.129:0/2197927557 learned_addr learned my addr 172.21.15.129:0/2197927557 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.950+0000 7f3468cc2700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34641020c0 0x7f3464105e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f34694c3700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3464103b90 0x7f3464104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f3463fff700 1 -- 172.21.15.129:0/2197927557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34641020c0 msgr2=0x7f3464105e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f3463fff700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34641020c0 0x7f3464105e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f3463fff700 1 -- 172.21.15.129:0/2197927557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3464103b90 msgr2=0x7f3464104a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f3463fff700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3464103b90 0x7f3464104a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f3463fff700 1 -- 172.21.15.129:0/2197927557 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34641051d0 con 0x7f34641031f0 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f34694c3700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3464103b90 0x7f3464104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f3468cc2700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34641020c0 0x7f3464105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f3463fff700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34641031f0 0x7f3464104530 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f345000b530 tx=0x7f345000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f3461ffb700 1 -- 172.21.15.129:0/2197927557 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3450015020 con 0x7f34641031f0 2024-05-20T17:58:30.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f3461ffb700 1 -- 172.21.15.129:0/2197927557 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3450011d10 con 0x7f34641031f0 2024-05-20T17:58:30.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f346af26700 1 -- 172.21.15.129:0/2197927557 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34641a39d0 con 0x7f34641031f0 2024-05-20T17:58:30.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f3461ffb700 1 -- 172.21.15.129:0/2197927557 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3450010e50 con 0x7f34641031f0 2024-05-20T17:58:30.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.951+0000 7f346af26700 1 -- 172.21.15.129:0/2197927557 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34641a3f80 con 0x7f34641031f0 2024-05-20T17:58:30.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.952+0000 7f346af26700 1 -- 172.21.15.129:0/2197927557 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3464050170 con 0x7f34641031f0 2024-05-20T17:58:30.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.953+0000 7f3461ffb700 1 -- 172.21.15.129:0/2197927557 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3450018020 con 0x7f34641031f0 2024-05-20T17:58:30.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.954+0000 7f3461ffb700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f34540a2e40 0x7f34540a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:30.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.954+0000 7f3468cc2700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f34540a2e40 0x7f34540a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:30.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.954+0000 7f3461ffb700 1 -- 172.21.15.129:0/2197927557 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f34500cecd0 con 0x7f34641031f0 2024-05-20T17:58:30.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.955+0000 7f3468cc2700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f34540a2e40 0x7f34540a5350 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f34580039c0 tx=0x7f345801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:30.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:30.958+0000 7f3461ffb700 1 -- 172.21.15.129:0/2197927557 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f345001f020 con 0x7f34641031f0 2024-05-20T17:58:31.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.155+0000 7f346af26700 1 -- 172.21.15.129:0/2197927557 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f34641a41b0 con 0x7f34641031f0 2024-05-20T17:58:31.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.157+0000 7f3461ffb700 1 -- 172.21.15.129:0/2197927557 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) ==== 56+0+5324 (secure 0 0 0) 0x7f345009bc20 con 0x7f34641031f0 2024-05-20T17:58:31.155 INFO:teuthology.orchestra.run.smithi129.stdout:epoch 88 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:created 2024-05-20T17:48:49.505110+0000 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:modified 2024-05-20T17:57:49.033961+0000 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:crush_version 26 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:full_ratio 0.95 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:backfillfull_ratio 0.9 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:nearfull_ratio 0.85 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:require_min_compat_client luminous 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:min_compat_client luminous 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:require_osd_release squid 2024-05-20T17:58:31.156 INFO:teuthology.orchestra.run.smithi129.stdout:stretch_mode_enabled false 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:max_osd 12 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6802/4065863746,v1:172.21.15.129:6803/4065863746] [v2:172.21.15.129:6804/4065863746,v1:172.21.15.129:6805/4065863746] exists,up 2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6800/792053596,v1:172.21.15.154:6801/792053596] [v2:172.21.15.154:6802/792053596,v1:172.21.15.154:6803/792053596] exists,up 9b41acf7-4a61-4152-9ec6-a0c980cb29c3 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6800/278897720,v1:172.21.15.204:6801/278897720] [v2:172.21.15.204:6802/278897720,v1:172.21.15.204:6803/278897720] exists,up c6f8c70f-8cd2-46ee-93da-e1ac0b73de83 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 up in weight 1 up_from 25 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6810/310906415,v1:172.21.15.129:6811/310906415] [v2:172.21.15.129:6812/310906415,v1:172.21.15.129:6813/310906415] exists,up 3fbfcecf-5f49-429e-a60f-2aae1d18eb16 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 up in weight 1 up_from 33 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6808/3850885593,v1:172.21.15.154:6809/3850885593] [v2:172.21.15.154:6810/3850885593,v1:172.21.15.154:6811/3850885593] exists,up 3393945c-4ccc-4f2c-ad95-5e884497d5d5 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 up in weight 1 up_from 38 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6808/12527578,v1:172.21.15.204:6809/12527578] [v2:172.21.15.204:6810/12527578,v1:172.21.15.204:6811/12527578] exists,up 9dbdb424-101b-488e-826c-fac4b7f35dfa 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 up in weight 1 up_from 46 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6818/2994413067,v1:172.21.15.129:6819/2994413067] [v2:172.21.15.129:6820/2994413067,v1:172.21.15.129:6821/2994413067] exists,up b87d90ab-d39b-464a-b835-0a1cc50d0eb2 2024-05-20T17:58:31.157 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 up in weight 1 up_from 51 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6816/2792653271,v1:172.21.15.154:6817/2792653271] [v2:172.21.15.154:6818/2792653271,v1:172.21.15.154:6819/2792653271] exists,up e850ec9e-e2a9-4d09-86c3-9f8115e516ad 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.8 up in weight 1 up_from 59 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6816/4118068284,v1:172.21.15.204:6817/4118068284] [v2:172.21.15.204:6818/4118068284,v1:172.21.15.204:6819/4118068284] exists,up a2371db0-2b8e-4a7c-86de-7c2036efca54 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.9 up in weight 1 up_from 67 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6826/2231439755,v1:172.21.15.129:6827/2231439755] [v2:172.21.15.129:6828/2231439755,v1:172.21.15.129:6829/2231439755] exists,up e40eb83d-8e9f-4f69-af66-b505b01d2836 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6824/1707430880,v1:172.21.15.154:6825/1707430880] [v2:172.21.15.154:6826/1707430880,v1:172.21.15.154:6827/1707430880] exists,up fe45b366-5c36-454e-ae75-d6170e11aff1 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.11 up in weight 1 up_from 81 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6824/329726241,v1:172.21.15.204:6825/329726241] [v2:172.21.15.204:6826/329726241,v1:172.21.15.204:6827/329726241] exists,up cc59d526-8616-4b3f-913a-c706b4d575c3 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:pg_upmap_items 2.c [2,4] 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:pg_upmap_items 2.16 [2,4] 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:pg_upmap_items 2.1f [8,1] 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:pg_upmap_items 3.0 [4,6,5,9] 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:pg_upmap_items 3.7 [3,9] 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:pg_upmap_items 3.13 [4,9] 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:pg_upmap_items 3.17 [4,8] 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:pg_upmap_items 3.23 [4,7] 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:pg_upmap_items 3.39 [5,8] 2024-05-20T17:58:31.158 INFO:teuthology.orchestra.run.smithi129.stdout:blocklist 172.21.15.129:0/2611729606 expires 2024-05-21T17:49:47.618673+0000 2024-05-20T17:58:31.159 INFO:teuthology.orchestra.run.smithi129.stdout:blocklist 172.21.15.129:0/874085270 expires 2024-05-21T17:49:47.618673+0000 2024-05-20T17:58:31.159 INFO:teuthology.orchestra.run.smithi129.stdout:blocklist 172.21.15.129:6801/1916241944 expires 2024-05-21T17:49:47.618673+0000 2024-05-20T17:58:31.159 INFO:teuthology.orchestra.run.smithi129.stdout:blocklist 172.21.15.129:6800/1916241944 expires 2024-05-21T17:49:47.618673+0000 2024-05-20T17:58:31.159 INFO:teuthology.orchestra.run.smithi129.stdout:blocklist 172.21.15.129:6800/3614898033 expires 2024-05-21T17:49:21.927054+0000 2024-05-20T17:58:31.159 INFO:teuthology.orchestra.run.smithi129.stdout:blocklist 172.21.15.129:0/1985417481 expires 2024-05-21T17:49:21.927054+0000 2024-05-20T17:58:31.159 INFO:teuthology.orchestra.run.smithi129.stdout:blocklist 172.21.15.129:0/1769091279 expires 2024-05-21T17:49:21.927054+0000 2024-05-20T17:58:31.159 INFO:teuthology.orchestra.run.smithi129.stdout:blocklist 172.21.15.129:6801/3614898033 expires 2024-05-21T17:49:21.927054+0000 2024-05-20T17:58:31.159 INFO:teuthology.orchestra.run.smithi129.stdout:blocklist 172.21.15.129:0/3608430741 expires 2024-05-21T17:49:21.927054+0000 2024-05-20T17:58:31.159 INFO:teuthology.orchestra.run.smithi129.stdout:blocklist 172.21.15.129:0/99394831 expires 2024-05-21T17:49:47.618673+0000 2024-05-20T17:58:31.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 -- 172.21.15.129:0/2197927557 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f34540a2e40 msgr2=0x7f34540a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:31.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f34540a2e40 0x7f34540a5350 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f34580039c0 tx=0x7f345801a040 comp rx=0 tx=0).stop 2024-05-20T17:58:31.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 -- 172.21.15.129:0/2197927557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34641031f0 msgr2=0x7f3464104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:31.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34641031f0 0x7f3464104530 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f345000b530 tx=0x7f345000b8f0 comp rx=0 tx=0).stop 2024-05-20T17:58:31.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 -- 172.21.15.129:0/2197927557 shutdown_connections 2024-05-20T17:58:31.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f34540a2e40 0x7f34540a5350 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:31.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3464103b90 0x7f3464104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:31.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34641020c0 0x7f3464105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:31.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 --2- 172.21.15.129:0/2197927557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34641031f0 0x7f3464104530 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:31.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 -- 172.21.15.129:0/2197927557 >> 172.21.15.129:0/2197927557 conn(0x7f34640fd700 msgr2=0x7f34640ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:31.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 -- 172.21.15.129:0/2197927557 shutdown_connections 2024-05-20T17:58:31.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:31.159+0000 7f346af26700 1 -- 172.21.15.129:0/2197927557 wait complete. 2024-05-20T17:58:31.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:31 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2197927557' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-20T17:58:31.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:31 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2197927557' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-20T17:58:31.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:31 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2197927557' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-20T17:58:31.961 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-20T17:58:31.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:58:32.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.094+0000 7ffbed414640 1 -- 172.21.15.129:0/1142796061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbe81089d0 msgr2=0x7ffbe810adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.094+0000 7ffbed414640 1 --2- 172.21.15.129:0/1142796061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbe81089d0 0x7ffbe810adf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009900 tx=0x7ffbe002a410 comp rx=0 tx=0).stop 2024-05-20T17:58:32.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.094+0000 7ffbed414640 1 -- 172.21.15.129:0/1142796061 shutdown_connections 2024-05-20T17:58:32.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.094+0000 7ffbed414640 1 --2- 172.21.15.129:0/1142796061 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbe81089d0 0x7ffbe810adf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.094+0000 7ffbed414640 1 --2- 172.21.15.129:0/1142796061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbe8103cd0 0x7ffbe81040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.094+0000 7ffbed414640 1 --2- 172.21.15.129:0/1142796061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbe8102a90 0x7ffbe8103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.094+0000 7ffbed414640 1 -- 172.21.15.129:0/1142796061 >> 172.21.15.129:0/1142796061 conn(0x7ffbe80fdc80 msgr2=0x7ffbe8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:32.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.094+0000 7ffbed414640 1 -- 172.21.15.129:0/1142796061 shutdown_connections 2024-05-20T17:58:32.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.094+0000 7ffbed414640 1 -- 172.21.15.129:0/1142796061 wait complete. 2024-05-20T17:58:32.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.095+0000 7ffbed414640 1 Processor -- start 2024-05-20T17:58:32.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.095+0000 7ffbed414640 1 -- start start 2024-05-20T17:58:32.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbed414640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbe8102a90 0x7ffbe8199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbed414640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbe8103cd0 0x7ffbe8199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbed414640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbe81089d0 0x7ffbe81a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbed414640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe810e6f0 con 0x7ffbe81089d0 2024-05-20T17:58:32.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbed414640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe810e570 con 0x7ffbe8102a90 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbed414640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe810e870 con 0x7ffbe8103cd0 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbe81089d0 0x7ffbe81a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbe81089d0 0x7ffbe81a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34424/0 (socket says 172.21.15.129:34424) 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe77fe640 1 -- 172.21.15.129:0/2307840444 learned_addr learned my addr 172.21.15.129:0/2307840444 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe67fc640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbe8103cd0 0x7ffbe8199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe6ffd640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbe8102a90 0x7ffbe8199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe77fe640 1 -- 172.21.15.129:0/2307840444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbe8103cd0 msgr2=0x7ffbe8199770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe77fe640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbe8103cd0 0x7ffbe8199770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe77fe640 1 -- 172.21.15.129:0/2307840444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbe8102a90 msgr2=0x7ffbe8199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe77fe640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbe8102a90 0x7ffbe8199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe77fe640 1 -- 172.21.15.129:0/2307840444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe81a0c70 con 0x7ffbe81089d0 2024-05-20T17:58:32.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe6ffd640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbe8102a90 0x7ffbe8199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:32.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbe77fe640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbe81089d0 0x7ffbe81a06b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009230 tx=0x7ffbe0031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:32.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbcffff640 1 -- 172.21.15.129:0/2307840444 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe003f030 con 0x7ffbe81089d0 2024-05-20T17:58:32.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbcffff640 1 -- 172.21.15.129:0/2307840444 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbe0003ad0 con 0x7ffbe81089d0 2024-05-20T17:58:32.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbed414640 1 -- 172.21.15.129:0/2307840444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe81a0f00 con 0x7ffbe81089d0 2024-05-20T17:58:32.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.096+0000 7ffbcffff640 1 -- 172.21.15.129:0/2307840444 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0035070 con 0x7ffbe81089d0 2024-05-20T17:58:32.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.097+0000 7ffbed414640 1 -- 172.21.15.129:0/2307840444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe81a1310 con 0x7ffbe81089d0 2024-05-20T17:58:32.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.097+0000 7ffbed414640 1 -- 172.21.15.129:0/2307840444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbe81077e0 con 0x7ffbe81089d0 2024-05-20T17:58:32.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.099+0000 7ffbcffff640 1 -- 172.21.15.129:0/2307840444 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ffbe0042030 con 0x7ffbe81089d0 2024-05-20T17:58:32.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.099+0000 7ffbcffff640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffbd00a2b90 0x7ffbd00a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.100+0000 7ffbe6ffd640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffbd00a2b90 0x7ffbd00a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.100+0000 7ffbcffff640 1 -- 172.21.15.129:0/2307840444 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7ffbe00efac0 con 0x7ffbe81089d0 2024-05-20T17:58:32.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.100+0000 7ffbe6ffd640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffbd00a2b90 0x7ffbd00a50a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffbe8103690 tx=0x7ffbd8009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:32.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.103+0000 7ffbcffff640 1 -- 172.21.15.129:0/2307840444 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffbe00bca10 con 0x7ffbe81089d0 2024-05-20T17:58:32.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.282+0000 7ffbed414640 1 -- 172.21.15.129:0/2307840444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffbe802d530 con 0x7ffbe81089d0 2024-05-20T17:58:32.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.284+0000 7ffbcffff640 1 -- 172.21.15.129:0/2307840444 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22680 (secure 0 0 0) 0x7ffbe00bc310 con 0x7ffbe81089d0 2024-05-20T17:58:32.283 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:32.283 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":88,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:57:49.033961+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:58:32.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 -- 172.21.15.129:0/2307840444 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffbd00a2b90 msgr2=0x7ffbd00a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffbd00a2b90 0x7ffbd00a50a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffbe8103690 tx=0x7ffbd8009500 comp rx=0 tx=0).stop 2024-05-20T17:58:32.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 -- 172.21.15.129:0/2307840444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbe81089d0 msgr2=0x7ffbe81a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbe81089d0 0x7ffbe81a06b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009230 tx=0x7ffbe0031b60 comp rx=0 tx=0).stop 2024-05-20T17:58:32.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 -- 172.21.15.129:0/2307840444 shutdown_connections 2024-05-20T17:58:32.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7ffbd00a2b90 0x7ffbd00a50a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbe81089d0 0x7ffbe81a06b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbe8103cd0 0x7ffbe8199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 --2- 172.21.15.129:0/2307840444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbe8102a90 0x7ffbe8199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 -- 172.21.15.129:0/2307840444 >> 172.21.15.129:0/2307840444 conn(0x7ffbe80fdc80 msgr2=0x7ffbe81049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:32.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 -- 172.21.15.129:0/2307840444 shutdown_connections 2024-05-20T17:58:32.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.286+0000 7ffbed414640 1 -- 172.21.15.129:0/2307840444 wait complete. 2024-05-20T17:58:32.297 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-20T17:56:42.965073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-20T17:56:44.277499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T17:58:32.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-20T17:58:32.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.413+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/1270857313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e64104060 msgr2=0x7f1e64104470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.413+0000 7f1e6b0a2640 1 --2- 172.21.15.129:0/1270857313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e64104060 0x7f1e64104470 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1e58009900 tx=0x7f1e5802a410 comp rx=0 tx=0).stop 2024-05-20T17:58:32.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.413+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/1270857313 shutdown_connections 2024-05-20T17:58:32.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.413+0000 7f1e6b0a2640 1 --2- 172.21.15.129:0/1270857313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e64101210 0x7f1e6410ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.413+0000 7f1e6b0a2640 1 --2- 172.21.15.129:0/1270857313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e64100710 0x7f1e64100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.413+0000 7f1e6b0a2640 1 --2- 172.21.15.129:0/1270857313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e64104060 0x7f1e64104470 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.413+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/1270857313 >> 172.21.15.129:0/1270857313 conn(0x7f1e640fc330 msgr2=0x7f1e640fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:32.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.413+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/1270857313 shutdown_connections 2024-05-20T17:58:32.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.413+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/1270857313 wait complete. 2024-05-20T17:58:32.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e6b0a2640 1 Processor -- start 2024-05-20T17:58:32.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e6b0a2640 1 -- start start 2024-05-20T17:58:32.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e6b0a2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e64100710 0x7f1e641992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e6b0a2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e64101210 0x7f1e64199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e6b0a2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e64104060 0x7f1e641a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e6b0a2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6410e720 con 0x7f1e64104060 2024-05-20T17:58:32.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e6b0a2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6410e5a0 con 0x7f1e64101210 2024-05-20T17:58:32.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e6b0a2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6410e8a0 con 0x7f1e64100710 2024-05-20T17:58:32.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e6963f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e64104060 0x7f1e641a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e6963f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e64104060 0x7f1e641a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34438/0 (socket says 172.21.15.129:34438) 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e6963f640 1 -- 172.21.15.129:0/2863285325 learned_addr learned my addr 172.21.15.129:0/2863285325 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.414+0000 7f1e68e3e640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e64100710 0x7f1e641992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e63fff640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e64101210 0x7f1e64199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e6963f640 1 -- 172.21.15.129:0/2863285325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e64100710 msgr2=0x7f1e641992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e6963f640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e64100710 0x7f1e641992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e6963f640 1 -- 172.21.15.129:0/2863285325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e64101210 msgr2=0x7f1e64199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e6963f640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e64101210 0x7f1e64199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e6963f640 1 -- 172.21.15.129:0/2863285325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e641a0e20 con 0x7f1e64104060 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e63fff640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e64101210 0x7f1e64199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e68e3e640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e64100710 0x7f1e641992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e6963f640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e64104060 0x7f1e641a07b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c00eaa0 tx=0x7f1e5c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e61ffb640 1 -- 172.21.15.129:0/2863285325 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c00cdb0 con 0x7f1e64104060 2024-05-20T17:58:32.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e61ffb640 1 -- 172.21.15.129:0/2863285325 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e5c005020 con 0x7f1e64104060 2024-05-20T17:58:32.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/2863285325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e641a1110 con 0x7f1e64104060 2024-05-20T17:58:32.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e61ffb640 1 -- 172.21.15.129:0/2863285325 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c0116b0 con 0x7f1e64104060 2024-05-20T17:58:32.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.415+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/2863285325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e641a1610 con 0x7f1e64104060 2024-05-20T17:58:32.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.416+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/2863285325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e641079f0 con 0x7f1e64104060 2024-05-20T17:58:32.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.417+0000 7f1e61ffb640 1 -- 172.21.15.129:0/2863285325 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1e5c005560 con 0x7f1e64104060 2024-05-20T17:58:32.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.418+0000 7f1e61ffb640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1e540a2b90 0x7f1e540a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.418+0000 7f1e68e3e640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1e540a2b90 0x7f1e540a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.418+0000 7f1e61ffb640 1 -- 172.21.15.129:0/2863285325 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f1e5c0cf380 con 0x7f1e64104060 2024-05-20T17:58:32.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.418+0000 7f1e68e3e640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1e540a2b90 0x7f1e540a50a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1e58009230 tx=0x7f1e58000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:32.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.423+0000 7f1e61ffb640 1 -- 172.21.15.129:0/2863285325 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1e5c016030 con 0x7f1e64104060 2024-05-20T17:58:32.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.598+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/2863285325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1e64062750 con 0x7f1e64104060 2024-05-20T17:58:32.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.598+0000 7f1e61ffb640 1 -- 172.21.15.129:0/2863285325 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f1e5c09c2d0 con 0x7f1e64104060 2024-05-20T17:58:32.597 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T17:58:32.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/2863285325 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1e540a2b90 msgr2=0x7f1e540a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1e540a2b90 0x7f1e540a50a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1e58009230 tx=0x7f1e58000bc0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/2863285325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e64104060 msgr2=0x7f1e641a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e64104060 0x7f1e641a07b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c00eaa0 tx=0x7f1e5c00ef60 comp rx=0 tx=0).stop 2024-05-20T17:58:32.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/2863285325 shutdown_connections 2024-05-20T17:58:32.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f1e540a2b90 0x7f1e540a50a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e64101210 0x7f1e64199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e64104060 0x7f1e641a07b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 --2- 172.21.15.129:0/2863285325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1e64100710 0x7f1e641992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/2863285325 >> 172.21.15.129:0/2863285325 conn(0x7f1e640fc330 msgr2=0x7f1e64104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:32.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/2863285325 shutdown_connections 2024-05-20T17:58:32.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.600+0000 7f1e6b0a2640 1 -- 172.21.15.129:0/2863285325 wait complete. 2024-05-20T17:58:32.609 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:32 smithi129 ceph-mon[32029]: pgmap v340: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:32.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-20T17:58:32.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:32 smithi154 ceph-mon[36482]: pgmap v340: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:32.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:32 smithi204 ceph-mon[37076]: pgmap v340: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T17:58:32.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.727+0000 7f57707fe640 1 -- 172.21.15.129:0/430549195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5768102bf0 msgr2=0x7f5768103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.727+0000 7f57707fe640 1 --2- 172.21.15.129:0/430549195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5768102bf0 0x7f5768103000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f575c009900 tx=0x7f575c02a410 comp rx=0 tx=0).stop 2024-05-20T17:58:32.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.727+0000 7f57707fe640 1 -- 172.21.15.129:0/430549195 shutdown_connections 2024-05-20T17:58:32.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.727+0000 7f57707fe640 1 --2- 172.21.15.129:0/430549195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57681078f0 0x7f5768109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.727+0000 7f57707fe640 1 --2- 172.21.15.129:0/430549195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5768102bf0 0x7f5768103000 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.727+0000 7f57707fe640 1 --2- 172.21.15.129:0/430549195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57681019b0 0x7f57681026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.727+0000 7f57707fe640 1 -- 172.21.15.129:0/430549195 >> 172.21.15.129:0/430549195 conn(0x7f57680fcba0 msgr2=0x7f57680ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.727+0000 7f57707fe640 1 -- 172.21.15.129:0/430549195 shutdown_connections 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.727+0000 7f57707fe640 1 -- 172.21.15.129:0/430549195 wait complete. 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f57707fe640 1 Processor -- start 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f57707fe640 1 -- start start 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f57707fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57681019b0 0x7f57680747f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f57707fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5768102bf0 0x7f576819e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f57707fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57681078f0 0x7f57681a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f57707fe640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f576810d680 con 0x7f5768102bf0 2024-05-20T17:58:32.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f57707fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f576810d500 con 0x7f57681078f0 2024-05-20T17:58:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f57707fe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f576810d800 con 0x7f57681019b0 2024-05-20T17:58:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f576dd99640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5768102bf0 0x7f576819e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f576dd99640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5768102bf0 0x7f576819e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34448/0 (socket says 172.21.15.129:34448) 2024-05-20T17:58:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f576dd99640 1 -- 172.21.15.129:0/1401735522 learned_addr learned my addr 172.21.15.129:0/1401735522 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f576ed9b640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57681078f0 0x7f57681a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f576e59a640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57681019b0 0x7f57680747f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f576dd99640 1 -- 172.21.15.129:0/1401735522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57681019b0 msgr2=0x7f57680747f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f576dd99640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57681019b0 0x7f57680747f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f576dd99640 1 -- 172.21.15.129:0/1401735522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57681078f0 msgr2=0x7f57681a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f576dd99640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57681078f0 0x7f57681a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f576dd99640 1 -- 172.21.15.129:0/1401735522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57681a0dc0 con 0x7f5768102bf0 2024-05-20T17:58:32.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.728+0000 7f576e59a640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57681019b0 0x7f57680747f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:32.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.729+0000 7f576dd99640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5768102bf0 0x7f576819e0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f575c006010 tx=0x7f575c0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:32.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.729+0000 7f57677fe640 1 -- 172.21.15.129:0/1401735522 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c03f030 con 0x7f5768102bf0 2024-05-20T17:58:32.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.729+0000 7f57707fe640 1 -- 172.21.15.129:0/1401735522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57681a1050 con 0x7f5768102bf0 2024-05-20T17:58:32.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.729+0000 7f57677fe640 1 -- 172.21.15.129:0/1401735522 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f575c003ce0 con 0x7f5768102bf0 2024-05-20T17:58:32.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.729+0000 7f57677fe640 1 -- 172.21.15.129:0/1401735522 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c035070 con 0x7f5768102bf0 2024-05-20T17:58:32.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.729+0000 7f57707fe640 1 -- 172.21.15.129:0/1401735522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57681a1610 con 0x7f5768102bf0 2024-05-20T17:58:32.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.730+0000 7f57707fe640 1 -- 172.21.15.129:0/1401735522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5768106700 con 0x7f5768102bf0 2024-05-20T17:58:32.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.731+0000 7f57677fe640 1 -- 172.21.15.129:0/1401735522 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f575c042030 con 0x7f5768102bf0 2024-05-20T17:58:32.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.732+0000 7f57677fe640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f57500a2b90 0x7f57500a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:32.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.732+0000 7f576e59a640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f57500a2b90 0x7f57500a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:32.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.732+0000 7f57677fe640 1 -- 172.21.15.129:0/1401735522 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f575c0efb80 con 0x7f5768102bf0 2024-05-20T17:58:32.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.732+0000 7f576e59a640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f57500a2b90 0x7f57500a50a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f57681025b0 tx=0x7f57580073c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:32.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.736+0000 7f57677fe640 1 -- 172.21.15.129:0/1401735522 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f575c0bcad0 con 0x7f5768102bf0 2024-05-20T17:58:32.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.913+0000 7f57707fe640 1 -- 172.21.15.129:0/1401735522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5768062750 con 0x7f5768102bf0 2024-05-20T17:58:32.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.913+0000 7f57677fe640 1 -- 172.21.15.129:0/1401735522 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f575c0bc3d0 con 0x7f5768102bf0 2024-05-20T17:58:32.913 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-05-20T17:58:32.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.915+0000 7f57707fe640 1 -- 172.21.15.129:0/1401735522 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f57500a2b90 msgr2=0x7f57500a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.915+0000 7f57707fe640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f57500a2b90 0x7f57500a50a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f57681025b0 tx=0x7f57580073c0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.915+0000 7f57707fe640 1 -- 172.21.15.129:0/1401735522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5768102bf0 msgr2=0x7f576819e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:32.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.915+0000 7f57707fe640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5768102bf0 0x7f576819e0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f575c006010 tx=0x7f575c0043c0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.915+0000 7f57707fe640 1 -- 172.21.15.129:0/1401735522 shutdown_connections 2024-05-20T17:58:32.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.915+0000 7f57707fe640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f57500a2b90 0x7f57500a50a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.915+0000 7f57707fe640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57681078f0 0x7f57681a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.915+0000 7f57707fe640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5768102bf0 0x7f576819e0e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.915+0000 7f57707fe640 1 --2- 172.21.15.129:0/1401735522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f57681019b0 0x7f57680747f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:32.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.915+0000 7f57707fe640 1 -- 172.21.15.129:0/1401735522 >> 172.21.15.129:0/1401735522 conn(0x7f57680fcba0 msgr2=0x7f57681038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:32.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.916+0000 7f57707fe640 1 -- 172.21.15.129:0/1401735522 shutdown_connections 2024-05-20T17:58:32.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:32.916+0000 7f57707fe640 1 -- 172.21.15.129:0/1401735522 wait complete. 2024-05-20T17:58:32.926 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-20T17:58:33.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.042+0000 7f8efa54b640 1 -- 172.21.15.129:0/731726604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef41089d0 msgr2=0x7f8ef410adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.042+0000 7f8efa54b640 1 --2- 172.21.15.129:0/731726604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef41089d0 0x7f8ef410adf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009ca0 tx=0x7f8eec02a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.042+0000 7f8efa54b640 1 -- 172.21.15.129:0/731726604 shutdown_connections 2024-05-20T17:58:33.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.042+0000 7f8efa54b640 1 --2- 172.21.15.129:0/731726604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef41089d0 0x7f8ef410adf0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.042+0000 7f8efa54b640 1 --2- 172.21.15.129:0/731726604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef4103cd0 0x7f8ef41040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.042+0000 7f8efa54b640 1 --2- 172.21.15.129:0/731726604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef4102a90 0x7f8ef4103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.042+0000 7f8efa54b640 1 -- 172.21.15.129:0/731726604 >> 172.21.15.129:0/731726604 conn(0x7f8ef40fdc80 msgr2=0x7f8ef4100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:33.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.042+0000 7f8efa54b640 1 -- 172.21.15.129:0/731726604 shutdown_connections 2024-05-20T17:58:33.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.042+0000 7f8efa54b640 1 -- 172.21.15.129:0/731726604 wait complete. 2024-05-20T17:58:33.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.042+0000 7f8efa54b640 1 Processor -- start 2024-05-20T17:58:33.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.042+0000 7f8efa54b640 1 -- start start 2024-05-20T17:58:33.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8efa54b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef4102a90 0x7f8ef4075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8efa54b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef4103cd0 0x7f8ef4073ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8efa54b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef41089d0 0x7f8ef40743e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8efa54b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef410e710 con 0x7f8ef4103cd0 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8efa54b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef410e590 con 0x7f8ef4102a90 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8efa54b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef410e890 con 0x7f8ef41089d0 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef4103cd0 0x7f8ef4073ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef4103cd0 0x7f8ef4073ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34470/0 (socket says 172.21.15.129:34470) 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef37fe640 1 -- 172.21.15.129:0/3475685473 learned_addr learned my addr 172.21.15.129:0/3475685473 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef8ae8640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef41089d0 0x7f8ef40743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef3fff640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef4102a90 0x7f8ef4075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef37fe640 1 -- 172.21.15.129:0/3475685473 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef41089d0 msgr2=0x7f8ef40743e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef37fe640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef41089d0 0x7f8ef40743e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef37fe640 1 -- 172.21.15.129:0/3475685473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef4102a90 msgr2=0x7f8ef4075800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef37fe640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef4102a90 0x7f8ef4075800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef37fe640 1 -- 172.21.15.129:0/3475685473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef4074b10 con 0x7f8ef4103cd0 2024-05-20T17:58:33.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef3fff640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef4102a90 0x7f8ef4075800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:33.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.043+0000 7f8ef37fe640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef4103cd0 0x7f8ef4073ea0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800e9f0 tx=0x7f8ee800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:33.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.044+0000 7f8ef17fa640 1 -- 172.21.15.129:0/3475685473 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee800cd80 con 0x7f8ef4103cd0 2024-05-20T17:58:33.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.044+0000 7f8efa54b640 1 -- 172.21.15.129:0/3475685473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef41a3310 con 0x7f8ef4103cd0 2024-05-20T17:58:33.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.044+0000 7f8ef17fa640 1 -- 172.21.15.129:0/3475685473 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee8005500 con 0x7f8ef4103cd0 2024-05-20T17:58:33.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.044+0000 7f8ef17fa640 1 -- 172.21.15.129:0/3475685473 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8010690 con 0x7f8ef4103cd0 2024-05-20T17:58:33.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.044+0000 7f8efa54b640 1 -- 172.21.15.129:0/3475685473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef41a37d0 con 0x7f8ef4103cd0 2024-05-20T17:58:33.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.045+0000 7f8efa54b640 1 -- 172.21.15.129:0/3475685473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ef41077e0 con 0x7f8ef4103cd0 2024-05-20T17:58:33.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.046+0000 7f8ef17fa640 1 -- 172.21.15.129:0/3475685473 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8ee80046c0 con 0x7f8ef4103cd0 2024-05-20T17:58:33.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.047+0000 7f8ef17fa640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8edc0a2b90 0x7f8edc0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.047+0000 7f8ef3fff640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8edc0a2b90 0x7f8edc0a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:33.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.047+0000 7f8ef17fa640 1 -- 172.21.15.129:0/3475685473 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f8ee80cf710 con 0x7f8ef4103cd0 2024-05-20T17:58:33.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.047+0000 7f8ef3fff640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8edc0a2b90 0x7f8edc0a50a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4103690 tx=0x7f8ee4005e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:33.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.050+0000 7f8ef17fa640 1 -- 172.21.15.129:0/3475685473 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8ee8053850 con 0x7f8ef4103cd0 2024-05-20T17:58:33.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.229+0000 7f8efa54b640 1 -- 172.21.15.129:0/3475685473 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8ef4062750 con 0x7f8ef4103cd0 2024-05-20T17:58:33.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.230+0000 7f8ef17fa640 1 -- 172.21.15.129:0/3475685473 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f8ee809c660 con 0x7f8ef4103cd0 2024-05-20T17:58:33.228 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-05-20T17:58:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 -- 172.21.15.129:0/3475685473 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8edc0a2b90 msgr2=0x7f8edc0a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8edc0a2b90 0x7f8edc0a50a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4103690 tx=0x7f8ee4005e70 comp rx=0 tx=0).stop 2024-05-20T17:58:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 -- 172.21.15.129:0/3475685473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef4103cd0 msgr2=0x7f8ef4073ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef4103cd0 0x7f8ef4073ea0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800e9f0 tx=0x7f8ee800edf0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 -- 172.21.15.129:0/3475685473 shutdown_connections 2024-05-20T17:58:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f8edc0a2b90 0x7f8edc0a50a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8ef41089d0 0x7f8ef40743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef4103cd0 0x7f8ef4073ea0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 --2- 172.21.15.129:0/3475685473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ef4102a90 0x7f8ef4075800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 -- 172.21.15.129:0/3475685473 >> 172.21.15.129:0/3475685473 conn(0x7f8ef40fdc80 msgr2=0x7f8ef41049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:33.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 -- 172.21.15.129:0/3475685473 shutdown_connections 2024-05-20T17:58:33.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.232+0000 7f8efa54b640 1 -- 172.21.15.129:0/3475685473 wait complete. 2024-05-20T17:58:33.242 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:58:33.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.355+0000 7f690da06640 1 -- 172.21.15.129:0/2149370025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908102bd0 msgr2=0x7f6908102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.355+0000 7f690da06640 1 --2- 172.21.15.129:0/2149370025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908102bd0 0x7f6908102fe0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009ca0 tx=0x7f68fc02a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.355+0000 7f690da06640 1 -- 172.21.15.129:0/2149370025 shutdown_connections 2024-05-20T17:58:33.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.355+0000 7f690da06640 1 --2- 172.21.15.129:0/2149370025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69081078d0 0x7f6908109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.355+0000 7f690da06640 1 --2- 172.21.15.129:0/2149370025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908102bd0 0x7f6908102fe0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.355+0000 7f690da06640 1 --2- 172.21.15.129:0/2149370025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6908101990 0x7f6908102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.355+0000 7f690da06640 1 -- 172.21.15.129:0/2149370025 >> 172.21.15.129:0/2149370025 conn(0x7f69080fcb80 msgr2=0x7f69080ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:33.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.355+0000 7f690da06640 1 -- 172.21.15.129:0/2149370025 shutdown_connections 2024-05-20T17:58:33.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.355+0000 7f690da06640 1 -- 172.21.15.129:0/2149370025 wait complete. 2024-05-20T17:58:33.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.355+0000 7f690da06640 1 Processor -- start 2024-05-20T17:58:33.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f690da06640 1 -- start start 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f690da06640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6908101990 0x7f69081992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f690da06640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908102bd0 0x7f6908199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f690da06640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69081078d0 0x7f69081a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f690da06640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f690810e710 con 0x7f6908102bd0 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f690da06640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f690810e590 con 0x7f69081078d0 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f690da06640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f690810e890 con 0x7f6908101990 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f69067fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908102bd0 0x7f6908199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f69067fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908102bd0 0x7f6908199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34490/0 (socket says 172.21.15.129:34490) 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f69067fc640 1 -- 172.21.15.129:0/1980821298 learned_addr learned my addr 172.21.15.129:0/1980821298 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f6906ffd640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6908101990 0x7f69081992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f69067fc640 1 -- 172.21.15.129:0/1980821298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6908101990 msgr2=0x7f69081992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f69077fe640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69081078d0 0x7f69081a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f69067fc640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6908101990 0x7f69081992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f69067fc640 1 -- 172.21.15.129:0/1980821298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69081078d0 msgr2=0x7f69081a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f69067fc640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69081078d0 0x7f69081a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f69067fc640 1 -- 172.21.15.129:0/1980821298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69081a0e30 con 0x7f6908102bd0 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f69077fe640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69081078d0 0x7f69081a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.356+0000 7f69067fc640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908102bd0 0x7f6908199830 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f68fc02ade0 tx=0x7f68fc031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.357+0000 7f690ca04640 1 -- 172.21.15.129:0/1980821298 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc02fa20 con 0x7f6908102bd0 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.357+0000 7f690da06640 1 -- 172.21.15.129:0/1980821298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69081a10c0 con 0x7f6908102bd0 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.357+0000 7f690ca04640 1 -- 172.21.15.129:0/1980821298 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68fc02fbc0 con 0x7f6908102bd0 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.357+0000 7f690ca04640 1 -- 172.21.15.129:0/1980821298 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc046750 con 0x7f6908102bd0 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.357+0000 7f690da06640 1 -- 172.21.15.129:0/1980821298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69081a1660 con 0x7f6908102bd0 2024-05-20T17:58:33.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.358+0000 7f690da06640 1 -- 172.21.15.129:0/1980821298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69081066e0 con 0x7f6908102bd0 2024-05-20T17:58:33.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.359+0000 7f690ca04640 1 -- 172.21.15.129:0/1980821298 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f68fc02fd70 con 0x7f6908102bd0 2024-05-20T17:58:33.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.360+0000 7f690ca04640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68f00a2b90 0x7f68f00a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.360+0000 7f6906ffd640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68f00a2b90 0x7f68f00a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:33.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.360+0000 7f690ca04640 1 -- 172.21.15.129:0/1980821298 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f68fc0f07a0 con 0x7f6908102bd0 2024-05-20T17:58:33.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.360+0000 7f6906ffd640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68f00a2b90 0x7f68f00a50a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6900005fd0 tx=0x7f6900005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:33.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.363+0000 7f690ca04640 1 -- 172.21.15.129:0/1980821298 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f68fc039380 con 0x7f6908102bd0 2024-05-20T17:58:33.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.575+0000 7f690da06640 1 -- 172.21.15.129:0/1980821298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f690802d2a0 con 0x7f6908102bd0 2024-05-20T17:58:33.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.576+0000 7f690ca04640 1 -- 172.21.15.129:0/1980821298 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11829 (secure 0 0 0) 0x7f68fc0bd810 con 0x7f6908102bd0 2024-05-20T17:58:33.575 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:33.575 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:56:49.336282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T17:58:33.575 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 20 2024-05-20T17:58:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 -- 172.21.15.129:0/1980821298 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68f00a2b90 msgr2=0x7f68f00a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68f00a2b90 0x7f68f00a50a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6900005fd0 tx=0x7f6900005c70 comp rx=0 tx=0).stop 2024-05-20T17:58:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 -- 172.21.15.129:0/1980821298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908102bd0 msgr2=0x7f6908199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908102bd0 0x7f6908199830 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f68fc02ade0 tx=0x7f68fc031ee0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 -- 172.21.15.129:0/1980821298 shutdown_connections 2024-05-20T17:58:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68f00a2b90 0x7f68f00a50a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69081078d0 0x7f69081a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908102bd0 0x7f6908199830 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 --2- 172.21.15.129:0/1980821298 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6908101990 0x7f69081992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 -- 172.21.15.129:0/1980821298 >> 172.21.15.129:0/1980821298 conn(0x7f69080fcb80 msgr2=0x7f69081038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:33.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 -- 172.21.15.129:0/1980821298 shutdown_connections 2024-05-20T17:58:33.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.578+0000 7f690da06640 1 -- 172.21.15.129:0/1980821298 wait complete. 2024-05-20T17:58:33.587 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:33 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2307840444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:33.587 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:33 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2863285325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:33.587 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:33 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1401735522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:33.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:58:33.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:33 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2307840444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:33.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:33 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2863285325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:33.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:33 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1401735522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:33.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:33 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2307840444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:33.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:33 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2863285325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:33.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:33 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1401735522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.702+0000 7efd3fefe640 1 -- 172.21.15.129:0/1043877852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd38102ab0 msgr2=0x7efd381037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.702+0000 7efd3fefe640 1 --2- 172.21.15.129:0/1043877852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd38102ab0 0x7efd381037b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efd34009ca0 tx=0x7efd3402a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.702+0000 7efd3fefe640 1 -- 172.21.15.129:0/1043877852 shutdown_connections 2024-05-20T17:58:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.702+0000 7efd3fefe640 1 --2- 172.21.15.129:0/1043877852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd381089f0 0x7efd3810ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.702+0000 7efd3fefe640 1 --2- 172.21.15.129:0/1043877852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd38103cf0 0x7efd38104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.702+0000 7efd3fefe640 1 --2- 172.21.15.129:0/1043877852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd38102ab0 0x7efd381037b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.702+0000 7efd3fefe640 1 -- 172.21.15.129:0/1043877852 >> 172.21.15.129:0/1043877852 conn(0x7efd380fdcc0 msgr2=0x7efd38100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.702+0000 7efd3fefe640 1 -- 172.21.15.129:0/1043877852 shutdown_connections 2024-05-20T17:58:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.702+0000 7efd3fefe640 1 -- 172.21.15.129:0/1043877852 wait complete. 2024-05-20T17:58:33.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.702+0000 7efd3fefe640 1 Processor -- start 2024-05-20T17:58:33.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.702+0000 7efd3fefe640 1 -- start start 2024-05-20T17:58:33.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3fefe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd38102ab0 0x7efd381992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3fefe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd38103cf0 0x7efd38199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3fefe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd381089f0 0x7efd381a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3fefe640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efd3810e750 con 0x7efd38102ab0 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3fefe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd3810e5d0 con 0x7efd381089f0 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3fefe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efd3810e8d0 con 0x7efd38103cf0 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3dc9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd38102ab0 0x7efd381992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3dc9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd38102ab0 0x7efd381992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34500/0 (socket says 172.21.15.129:34500) 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3dc9a640 1 -- 172.21.15.129:0/2315238355 learned_addr learned my addr 172.21.15.129:0/2315238355 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3dc9a640 1 -- 172.21.15.129:0/2315238355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd38103cf0 msgr2=0x7efd38199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3dc9a640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd38103cf0 0x7efd38199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3dc9a640 1 -- 172.21.15.129:0/2315238355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd381089f0 msgr2=0x7efd381a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3dc9a640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd381089f0 0x7efd381a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3dc9a640 1 -- 172.21.15.129:0/2315238355 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd381a0e30 con 0x7efd38102ab0 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3dc9a640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd38102ab0 0x7efd381992f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efd34000f80 tx=0x7efd340319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd2effd640 1 -- 172.21.15.129:0/2315238355 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd34003bf0 con 0x7efd38102ab0 2024-05-20T17:58:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd2effd640 1 -- 172.21.15.129:0/2315238355 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd34003d90 con 0x7efd38102ab0 2024-05-20T17:58:33.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3fefe640 1 -- 172.21.15.129:0/2315238355 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd381a10c0 con 0x7efd38102ab0 2024-05-20T17:58:33.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd2effd640 1 -- 172.21.15.129:0/2315238355 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3402fd30 con 0x7efd38102ab0 2024-05-20T17:58:33.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.703+0000 7efd3fefe640 1 -- 172.21.15.129:0/2315238355 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd381a1610 con 0x7efd38102ab0 2024-05-20T17:58:33.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.704+0000 7efd3fefe640 1 -- 172.21.15.129:0/2315238355 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd38107800 con 0x7efd38102ab0 2024-05-20T17:58:33.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.706+0000 7efd2effd640 1 -- 172.21.15.129:0/2315238355 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7efd34004040 con 0x7efd38102ab0 2024-05-20T17:58:33.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.707+0000 7efd2effd640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd240a2b90 0x7efd240a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:33.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.707+0000 7efd3d499640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd240a2b90 0x7efd240a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:33.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.707+0000 7efd2effd640 1 -- 172.21.15.129:0/2315238355 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7efd340f07a0 con 0x7efd38102ab0 2024-05-20T17:58:33.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.707+0000 7efd3d499640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd240a2b90 0x7efd240a50a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efd28005fd0 tx=0x7efd28005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:33.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.711+0000 7efd2effd640 1 -- 172.21.15.129:0/2315238355 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7efd340bd780 con 0x7efd38102ab0 2024-05-20T17:58:33.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.894+0000 7efd3fefe640 1 -- 172.21.15.129:0/2315238355 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd3802d470 con 0x7efd38102ab0 2024-05-20T17:58:33.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.895+0000 7efd2effd640 1 -- 172.21.15.129:0/2315238355 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22680 (secure 0 0 0) 0x7efd340bd080 con 0x7efd38102ab0 2024-05-20T17:58:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:33.894 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":88,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:57:49.033961+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:58:33.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.897+0000 7efd3fefe640 1 -- 172.21.15.129:0/2315238355 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd240a2b90 msgr2=0x7efd240a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.897+0000 7efd3fefe640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd240a2b90 0x7efd240a50a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efd28005fd0 tx=0x7efd28005850 comp rx=0 tx=0).stop 2024-05-20T17:58:33.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.897+0000 7efd3fefe640 1 -- 172.21.15.129:0/2315238355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd38102ab0 msgr2=0x7efd381992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:33.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.897+0000 7efd3fefe640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd38102ab0 0x7efd381992f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efd34000f80 tx=0x7efd340319f0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.898+0000 7efd3fefe640 1 -- 172.21.15.129:0/2315238355 shutdown_connections 2024-05-20T17:58:33.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.898+0000 7efd3fefe640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7efd240a2b90 0x7efd240a50a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.898+0000 7efd3fefe640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd381089f0 0x7efd381a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.898+0000 7efd3fefe640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efd38103cf0 0x7efd38199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.898+0000 7efd3fefe640 1 --2- 172.21.15.129:0/2315238355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd38102ab0 0x7efd381992f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:33.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.898+0000 7efd3fefe640 1 -- 172.21.15.129:0/2315238355 >> 172.21.15.129:0/2315238355 conn(0x7efd380fdcc0 msgr2=0x7efd381049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:33.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.898+0000 7efd3fefe640 1 -- 172.21.15.129:0/2315238355 shutdown_connections 2024-05-20T17:58:33.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:33.898+0000 7efd3fefe640 1 -- 172.21.15.129:0/2315238355 wait complete. 2024-05-20T17:58:33.908 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-20T17:56:42.965073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-20T17:56:44.277499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T17:58:33.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-20T17:58:34.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.023+0000 7f78acc75640 1 -- 172.21.15.129:0/1415728246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78a8101990 msgr2=0x7f78a8102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.023+0000 7f78acc75640 1 --2- 172.21.15.129:0/1415728246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78a8101990 0x7f78a8102690 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7898009900 tx=0x7f789802a410 comp rx=0 tx=0).stop 2024-05-20T17:58:34.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.023+0000 7f78acc75640 1 -- 172.21.15.129:0/1415728246 shutdown_connections 2024-05-20T17:58:34.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.023+0000 7f78acc75640 1 --2- 172.21.15.129:0/1415728246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a81078d0 0x7f78a8109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.023+0000 7f78acc75640 1 --2- 172.21.15.129:0/1415728246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a8102bd0 0x7f78a8102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.023+0000 7f78acc75640 1 --2- 172.21.15.129:0/1415728246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78a8101990 0x7f78a8102690 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.023+0000 7f78acc75640 1 -- 172.21.15.129:0/1415728246 >> 172.21.15.129:0/1415728246 conn(0x7f78a80fcb80 msgr2=0x7f78a80ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:34.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.023+0000 7f78acc75640 1 -- 172.21.15.129:0/1415728246 shutdown_connections 2024-05-20T17:58:34.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.023+0000 7f78acc75640 1 -- 172.21.15.129:0/1415728246 wait complete. 2024-05-20T17:58:34.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.024+0000 7f78acc75640 1 Processor -- start 2024-05-20T17:58:34.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.024+0000 7f78acc75640 1 -- start start 2024-05-20T17:58:34.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.024+0000 7f78acc75640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101990 0x7f78a8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.024+0000 7f78acc75640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78a8102bd0 0x7f78a8198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.024+0000 7f78acc75640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a81078d0 0x7f78a819f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.024+0000 7f78acc75640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78a810d630 con 0x7f78a8102bd0 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.024+0000 7f78acc75640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78a810d4b0 con 0x7f78a8101990 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.024+0000 7f78acc75640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78a810d7b0 con 0x7f78a81078d0 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.024+0000 7f78a5d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78a8102bd0 0x7f78a8198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.024+0000 7f78a5d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78a8102bd0 0x7f78a8198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34512/0 (socket says 172.21.15.129:34512) 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.024+0000 7f78a5d9b640 1 -- 172.21.15.129:0/1415173180 learned_addr learned my addr 172.21.15.129:0/1415173180 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78a6d9d640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a81078d0 0x7f78a819f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78a659c640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101990 0x7f78a8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78a5d9b640 1 -- 172.21.15.129:0/1415173180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a81078d0 msgr2=0x7f78a819f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78a5d9b640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a81078d0 0x7f78a819f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78a5d9b640 1 -- 172.21.15.129:0/1415173180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101990 msgr2=0x7f78a8198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78a5d9b640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101990 0x7f78a8198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78a5d9b640 1 -- 172.21.15.129:0/1415173180 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a81a0060 con 0x7f78a8102bd0 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78a659c640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101990 0x7f78a8198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78a6d9d640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a81078d0 0x7f78a819f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78a5d9b640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78a8102bd0 0x7f78a8198a60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f789c00d770 tx=0x7f789c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78977fe640 1 -- 172.21.15.129:0/1415173180 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c017070 con 0x7f78a8102bd0 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78acc75640 1 -- 172.21.15.129:0/1415173180 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a81a0350 con 0x7f78a8102bd0 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78977fe640 1 -- 172.21.15.129:0/1415173180 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789c005500 con 0x7f78a8102bd0 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78977fe640 1 -- 172.21.15.129:0/1415173180 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c004e00 con 0x7f78a8102bd0 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.025+0000 7f78acc75640 1 -- 172.21.15.129:0/1415173180 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a81a08a0 con 0x7f78a8102bd0 2024-05-20T17:58:34.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.026+0000 7f78acc75640 1 -- 172.21.15.129:0/1415173180 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a81066e0 con 0x7f78a8102bd0 2024-05-20T17:58:34.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.027+0000 7f78977fe640 1 -- 172.21.15.129:0/1415173180 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f789c004fa0 con 0x7f78a8102bd0 2024-05-20T17:58:34.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.028+0000 7f78977fe640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78900a2b90 0x7f78900a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.028+0000 7f78a659c640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78900a2b90 0x7f78900a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.028+0000 7f78977fe640 1 -- 172.21.15.129:0/1415173180 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f789c0cefb0 con 0x7f78a8102bd0 2024-05-20T17:58:34.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.029+0000 7f78a659c640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78900a2b90 0x7f78900a50a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f78980096c0 tx=0x7f7898000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:34.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.032+0000 7f78977fe640 1 -- 172.21.15.129:0/1415173180 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f789c09bf00 con 0x7f78a8102bd0 2024-05-20T17:58:34.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.208+0000 7f78acc75640 1 -- 172.21.15.129:0/1415173180 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f78a802d420 con 0x7f78a8102bd0 2024-05-20T17:58:34.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.208+0000 7f78977fe640 1 -- 172.21.15.129:0/1415173180 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f789c09b800 con 0x7f78a8102bd0 2024-05-20T17:58:34.207 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T17:58:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.210+0000 7f78acc75640 1 -- 172.21.15.129:0/1415173180 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78900a2b90 msgr2=0x7f78900a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.210+0000 7f78acc75640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78900a2b90 0x7f78900a50a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f78980096c0 tx=0x7f7898000bc0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.210+0000 7f78acc75640 1 -- 172.21.15.129:0/1415173180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78a8102bd0 msgr2=0x7f78a8198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.210+0000 7f78acc75640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78a8102bd0 0x7f78a8198a60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f789c00d770 tx=0x7f789c00dc30 comp rx=0 tx=0).stop 2024-05-20T17:58:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.210+0000 7f78acc75640 1 -- 172.21.15.129:0/1415173180 shutdown_connections 2024-05-20T17:58:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.210+0000 7f78acc75640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f78900a2b90 0x7f78900a50a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.210+0000 7f78acc75640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78a81078d0 0x7f78a819f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.210+0000 7f78acc75640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78a8102bd0 0x7f78a8198a60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.210+0000 7f78acc75640 1 --2- 172.21.15.129:0/1415173180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a8101990 0x7f78a8198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.211+0000 7f78acc75640 1 -- 172.21.15.129:0/1415173180 >> 172.21.15.129:0/1415173180 conn(0x7f78a80fcb80 msgr2=0x7f78a81038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.211+0000 7f78acc75640 1 -- 172.21.15.129:0/1415173180 shutdown_connections 2024-05-20T17:58:34.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.211+0000 7f78acc75640 1 -- 172.21.15.129:0/1415173180 wait complete. 2024-05-20T17:58:34.221 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-20T17:58:34.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.336+0000 7fb47758f640 1 -- 172.21.15.129:0/3465512107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb470105ee0 msgr2=0x7fb470108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.336+0000 7fb47758f640 1 --2- 172.21.15.129:0/3465512107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb470105ee0 0x7fb470108300 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb460009900 tx=0x7fb46002a410 comp rx=0 tx=0).stop 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.336+0000 7fb47758f640 1 -- 172.21.15.129:0/3465512107 shutdown_connections 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.336+0000 7fb47758f640 1 --2- 172.21.15.129:0/3465512107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb470108840 0x7fb47010ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.336+0000 7fb47758f640 1 --2- 172.21.15.129:0/3465512107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb470105ee0 0x7fb470108300 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.336+0000 7fb47758f640 1 --2- 172.21.15.129:0/3465512107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb470103580 0x7fb4701059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.336+0000 7fb47758f640 1 -- 172.21.15.129:0/3465512107 >> 172.21.15.129:0/3465512107 conn(0x7fb4700fcb80 msgr2=0x7fb4700ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.336+0000 7fb47758f640 1 -- 172.21.15.129:0/3465512107 shutdown_connections 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.336+0000 7fb47758f640 1 -- 172.21.15.129:0/3465512107 wait complete. 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.336+0000 7fb47758f640 1 Processor -- start 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.336+0000 7fb47758f640 1 -- start start 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb47758f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb470103580 0x7fb470199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb47758f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb470105ee0 0x7fb470199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb47758f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb470108840 0x7fb4701a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb47758f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb47010e6f0 con 0x7fb470105ee0 2024-05-20T17:58:34.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb47758f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb47010e570 con 0x7fb470103580 2024-05-20T17:58:34.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb47758f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb47010e870 con 0x7fb470108840 2024-05-20T17:58:34.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb474b2a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb470105ee0 0x7fb470199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb474b2a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb470105ee0 0x7fb470199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34532/0 (socket says 172.21.15.129:34532) 2024-05-20T17:58:34.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb474b2a640 1 -- 172.21.15.129:0/1914022573 learned_addr learned my addr 172.21.15.129:0/1914022573 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:34.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb47532b640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb470103580 0x7fb470199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb475b2c640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb470108840 0x7fb4701a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb474b2a640 1 -- 172.21.15.129:0/1914022573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb470108840 msgr2=0x7fb4701a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb474b2a640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb470108840 0x7fb4701a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb474b2a640 1 -- 172.21.15.129:0/1914022573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb470103580 msgr2=0x7fb470199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb474b2a640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb470103580 0x7fb470199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb474b2a640 1 -- 172.21.15.129:0/1914022573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4701a1150 con 0x7fb470105ee0 2024-05-20T17:58:34.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb474b2a640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb470105ee0 0x7fb470199b50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb460006010 tx=0x7fb460004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:34.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb4667fc640 1 -- 172.21.15.129:0/1914022573 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46003f030 con 0x7fb470105ee0 2024-05-20T17:58:34.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb4667fc640 1 -- 172.21.15.129:0/1914022573 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb460003ce0 con 0x7fb470105ee0 2024-05-20T17:58:34.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.337+0000 7fb47758f640 1 -- 172.21.15.129:0/1914022573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4701a13e0 con 0x7fb470105ee0 2024-05-20T17:58:34.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.338+0000 7fb4667fc640 1 -- 172.21.15.129:0/1914022573 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb460035070 con 0x7fb470105ee0 2024-05-20T17:58:34.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.338+0000 7fb47758f640 1 -- 172.21.15.129:0/1914022573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4701a1920 con 0x7fb470105ee0 2024-05-20T17:58:34.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.338+0000 7fb47758f640 1 -- 172.21.15.129:0/1914022573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb47006b250 con 0x7fb470105ee0 2024-05-20T17:58:34.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.340+0000 7fb4667fc640 1 -- 172.21.15.129:0/1914022573 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb460042030 con 0x7fb470105ee0 2024-05-20T17:58:34.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.340+0000 7fb4667fc640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb45c0a2c10 0x7fb45c0a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.341+0000 7fb47532b640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb45c0a2c10 0x7fb45c0a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.341+0000 7fb4667fc640 1 -- 172.21.15.129:0/1914022573 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fb4600efb30 con 0x7fb470105ee0 2024-05-20T17:58:34.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.341+0000 7fb47532b640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb45c0a2c10 0x7fb45c0a5120 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb470069860 tx=0x7fb46c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:34.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.344+0000 7fb4667fc640 1 -- 172.21.15.129:0/1914022573 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb4600bc430 con 0x7fb470105ee0 2024-05-20T17:58:34.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.520+0000 7fb47758f640 1 -- 172.21.15.129:0/1914022573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb4701a1be0 con 0x7fb470105ee0 2024-05-20T17:58:34.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.520+0000 7fb4667fc640 1 -- 172.21.15.129:0/1914022573 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fb4600c0730 con 0x7fb470105ee0 2024-05-20T17:58:34.519 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-05-20T17:58:34.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 -- 172.21.15.129:0/1914022573 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb45c0a2c10 msgr2=0x7fb45c0a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb45c0a2c10 0x7fb45c0a5120 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb470069860 tx=0x7fb46c00b390 comp rx=0 tx=0).stop 2024-05-20T17:58:34.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 -- 172.21.15.129:0/1914022573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb470105ee0 msgr2=0x7fb470199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb470105ee0 0x7fb470199b50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb460006010 tx=0x7fb460004360 comp rx=0 tx=0).stop 2024-05-20T17:58:34.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 -- 172.21.15.129:0/1914022573 shutdown_connections 2024-05-20T17:58:34.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb45c0a2c10 0x7fb45c0a5120 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb470108840 0x7fb4701a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb470105ee0 0x7fb470199b50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 --2- 172.21.15.129:0/1914022573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb470103580 0x7fb470199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 -- 172.21.15.129:0/1914022573 >> 172.21.15.129:0/1914022573 conn(0x7fb4700fcb80 msgr2=0x7fb470104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:34.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 -- 172.21.15.129:0/1914022573 shutdown_connections 2024-05-20T17:58:34.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.523+0000 7fb47758f640 1 -- 172.21.15.129:0/1914022573 wait complete. 2024-05-20T17:58:34.532 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:34 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3475685473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:34.532 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:34 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1980821298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:34.532 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:34 smithi129 ceph-mon[32029]: pgmap v341: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:34.532 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:34 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2315238355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:34.532 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:34 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1415173180' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:34.533 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-20T17:58:34.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:34 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3475685473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:34.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:34 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1980821298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:34.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:34 smithi154 ceph-mon[36482]: pgmap v341: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:34.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:34 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2315238355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:34.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:34 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1415173180' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:34.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.649+0000 7f3043626640 1 -- 172.21.15.129:0/1471202519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f303c101990 msgr2=0x7f303c102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.649+0000 7f3043626640 1 --2- 172.21.15.129:0/1471202519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f303c101990 0x7f303c102690 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f302c009ca0 tx=0x7f302c02a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.649+0000 7f3043626640 1 -- 172.21.15.129:0/1471202519 shutdown_connections 2024-05-20T17:58:34.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.649+0000 7f3043626640 1 --2- 172.21.15.129:0/1471202519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f303c102bd0 0x7f303c102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.649+0000 7f3043626640 1 --2- 172.21.15.129:0/1471202519 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f303c1078d0 0x7f303c109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.649+0000 7f3043626640 1 --2- 172.21.15.129:0/1471202519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f303c101990 0x7f303c102690 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.649+0000 7f3043626640 1 -- 172.21.15.129:0/1471202519 >> 172.21.15.129:0/1471202519 conn(0x7f303c0fcb80 msgr2=0x7f303c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:34.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.649+0000 7f3043626640 1 -- 172.21.15.129:0/1471202519 shutdown_connections 2024-05-20T17:58:34.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.649+0000 7f3043626640 1 -- 172.21.15.129:0/1471202519 wait complete. 2024-05-20T17:58:34.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.649+0000 7f3043626640 1 Processor -- start 2024-05-20T17:58:34.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f3043626640 1 -- start start 2024-05-20T17:58:34.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f3043626640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f303c101990 0x7f303c1995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f3043626640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f303c102bd0 0x7f303c199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f3043626640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f303c1078d0 0x7f303c1a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f3043626640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f303c10d5e0 con 0x7f303c101990 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f3043626640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f303c10d460 con 0x7f303c102bd0 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f3043626640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f303c10d760 con 0x7f303c1078d0 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f30413c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f303c101990 0x7f303c1995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f30413c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f303c101990 0x7f303c1995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34550/0 (socket says 172.21.15.129:34550) 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f30413c2640 1 -- 172.21.15.129:0/1728004906 learned_addr learned my addr 172.21.15.129:0/1728004906 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f30413c2640 1 -- 172.21.15.129:0/1728004906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f303c1078d0 msgr2=0x7f303c1a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f30413c2640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f303c1078d0 0x7f303c1a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f30413c2640 1 -- 172.21.15.129:0/1728004906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f303c102bd0 msgr2=0x7f303c199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f3040bc1640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f303c102bd0 0x7f303c199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f30413c2640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f303c102bd0 0x7f303c199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f30413c2640 1 -- 172.21.15.129:0/1728004906 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303c1a1100 con 0x7f303c101990 2024-05-20T17:58:34.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f30413c2640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f303c101990 0x7f303c1995c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f302c000f80 tx=0x7f302c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:34.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.650+0000 7f303a7fc640 1 -- 172.21.15.129:0/1728004906 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c02faa0 con 0x7f303c101990 2024-05-20T17:58:34.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.651+0000 7f3043626640 1 -- 172.21.15.129:0/1728004906 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f303c1a1390 con 0x7f303c101990 2024-05-20T17:58:34.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.651+0000 7f303a7fc640 1 -- 172.21.15.129:0/1728004906 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f302c02fc40 con 0x7f303c101990 2024-05-20T17:58:34.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.651+0000 7f303a7fc640 1 -- 172.21.15.129:0/1728004906 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c03a7f0 con 0x7f303c101990 2024-05-20T17:58:34.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.651+0000 7f3043626640 1 -- 172.21.15.129:0/1728004906 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f303c1a18e0 con 0x7f303c101990 2024-05-20T17:58:34.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.652+0000 7f3043626640 1 -- 172.21.15.129:0/1728004906 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f303c06b250 con 0x7f303c101990 2024-05-20T17:58:34.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.653+0000 7f303a7fc640 1 -- 172.21.15.129:0/1728004906 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f302c030bc0 con 0x7f303c101990 2024-05-20T17:58:34.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.653+0000 7f303a7fc640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f30240a2b90 0x7f30240a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.653+0000 7f3040bc1640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f30240a2b90 0x7f30240a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.653+0000 7f303a7fc640 1 -- 172.21.15.129:0/1728004906 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f302c0f06b0 con 0x7f303c101990 2024-05-20T17:58:34.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.654+0000 7f3040bc1640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f30240a2b90 0x7f30240a50a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f30300058c0 tx=0x7f3030005c80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:34.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.658+0000 7f303a7fc640 1 -- 172.21.15.129:0/1728004906 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f302c0453e0 con 0x7f303c101990 2024-05-20T17:58:34.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:34 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3475685473' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:34.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:34 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1980821298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:34.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:34 smithi204 ceph-mon[37076]: pgmap v341: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:34.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:34 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2315238355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:34.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:34 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1415173180' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:34.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.834+0000 7f3043626640 1 -- 172.21.15.129:0/1728004906 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f303c1a1b40 con 0x7f303c101990 2024-05-20T17:58:34.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.834+0000 7f303a7fc640 1 -- 172.21.15.129:0/1728004906 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f302c0bd600 con 0x7f303c101990 2024-05-20T17:58:34.833 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-05-20T17:58:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 -- 172.21.15.129:0/1728004906 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f30240a2b90 msgr2=0x7f30240a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f30240a2b90 0x7f30240a50a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f30300058c0 tx=0x7f3030005c80 comp rx=0 tx=0).stop 2024-05-20T17:58:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 -- 172.21.15.129:0/1728004906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f303c101990 msgr2=0x7f303c1995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f303c101990 0x7f303c1995c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f302c000f80 tx=0x7f302c031ee0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 -- 172.21.15.129:0/1728004906 shutdown_connections 2024-05-20T17:58:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f30240a2b90 0x7f30240a50a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f303c102bd0 0x7f303c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f303c1078d0 0x7f303c1a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 --2- 172.21.15.129:0/1728004906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f303c101990 0x7f303c1995c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 -- 172.21.15.129:0/1728004906 >> 172.21.15.129:0/1728004906 conn(0x7f303c0fcb80 msgr2=0x7f303c068e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:34.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 -- 172.21.15.129:0/1728004906 shutdown_connections 2024-05-20T17:58:34.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.836+0000 7f3043626640 1 -- 172.21.15.129:0/1728004906 wait complete. 2024-05-20T17:58:34.846 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:58:34.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.959+0000 7f01cc606640 1 -- 172.21.15.129:0/3980779423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01c4103d00 msgr2=0x7f01c4104110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.959+0000 7f01cc606640 1 --2- 172.21.15.129:0/3980779423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01c4103d00 0x7f01c4104110 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009900 tx=0x7f01b402a410 comp rx=0 tx=0).stop 2024-05-20T17:58:34.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.959+0000 7f01cc606640 1 -- 172.21.15.129:0/3980779423 shutdown_connections 2024-05-20T17:58:34.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.959+0000 7f01cc606640 1 --2- 172.21.15.129:0/3980779423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4108a00 0x7f01c410ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.959+0000 7f01cc606640 1 --2- 172.21.15.129:0/3980779423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01c4103d00 0x7f01c4104110 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.959+0000 7f01cc606640 1 --2- 172.21.15.129:0/3980779423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01c4102ac0 0x7f01c41037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.959+0000 7f01cc606640 1 -- 172.21.15.129:0/3980779423 >> 172.21.15.129:0/3980779423 conn(0x7f01c40fdcf0 msgr2=0x7f01c4100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01cc606640 1 -- 172.21.15.129:0/3980779423 shutdown_connections 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01cc606640 1 -- 172.21.15.129:0/3980779423 wait complete. 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01cc606640 1 Processor -- start 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01cc606640 1 -- start start 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01cc606640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01c4102ac0 0x7f01c4199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01cc606640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01c4103d00 0x7f01c4199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01cc606640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4108a00 0x7f01c41a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01cc606640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f01c410e730 con 0x7f01c4102ac0 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01cc606640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f01c410e5b0 con 0x7f01c4108a00 2024-05-20T17:58:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01cc606640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f01c410e8b0 con 0x7f01c4103d00 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01ca3a2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01c4102ac0 0x7f01c4199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01ca3a2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01c4102ac0 0x7f01c4199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34566/0 (socket says 172.21.15.129:34566) 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.960+0000 7f01ca3a2640 1 -- 172.21.15.129:0/411489907 learned_addr learned my addr 172.21.15.129:0/411489907 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01c9ba1640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01c4103d00 0x7f01c4199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01caba3640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4108a00 0x7f01c41a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01ca3a2640 1 -- 172.21.15.129:0/411489907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01c4103d00 msgr2=0x7f01c4199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01ca3a2640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01c4103d00 0x7f01c4199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01ca3a2640 1 -- 172.21.15.129:0/411489907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4108a00 msgr2=0x7f01c41a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01ca3a2640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4108a00 0x7f01c41a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01ca3a2640 1 -- 172.21.15.129:0/411489907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c41a0e40 con 0x7f01c4102ac0 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01caba3640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4108a00 0x7f01c41a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:34.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01ca3a2640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01c4102ac0 0x7f01c4199300 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f01c000e9f0 tx=0x7f01c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:34.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01bb7fe640 1 -- 172.21.15.129:0/411489907 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c000ccc0 con 0x7f01c4102ac0 2024-05-20T17:58:34.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01cc606640 1 -- 172.21.15.129:0/411489907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c41a1130 con 0x7f01c4102ac0 2024-05-20T17:58:34.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01bb7fe640 1 -- 172.21.15.129:0/411489907 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01c00035f0 con 0x7f01c4102ac0 2024-05-20T17:58:34.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01bb7fe640 1 -- 172.21.15.129:0/411489907 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c000ce60 con 0x7f01c4102ac0 2024-05-20T17:58:34.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.961+0000 7f01cc606640 1 -- 172.21.15.129:0/411489907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c41a1630 con 0x7f01c4102ac0 2024-05-20T17:58:34.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.962+0000 7f01cc606640 1 -- 172.21.15.129:0/411489907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c4107810 con 0x7f01c4102ac0 2024-05-20T17:58:34.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.963+0000 7f01bb7fe640 1 -- 172.21.15.129:0/411489907 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f01c00037a0 con 0x7f01c4102ac0 2024-05-20T17:58:34.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.964+0000 7f01bb7fe640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f01b00a2b90 0x7f01b00a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:34.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.964+0000 7f01c9ba1640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f01b00a2b90 0x7f01b00a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:34.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.964+0000 7f01bb7fe640 1 -- 172.21.15.129:0/411489907 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f01c00cf710 con 0x7f01c4102ac0 2024-05-20T17:58:34.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.965+0000 7f01c9ba1640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f01b00a2b90 0x7f01b00a50a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009230 tx=0x7f01b4031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:34.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:34.968+0000 7f01bb7fe640 1 -- 172.21.15.129:0/411489907 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f01c009c660 con 0x7f01c4102ac0 2024-05-20T17:58:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.180+0000 7f01cc606640 1 -- 172.21.15.129:0/411489907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01c402d2a0 con 0x7f01c4102ac0 2024-05-20T17:58:35.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.181+0000 7f01bb7fe640 1 -- 172.21.15.129:0/411489907 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11829 (secure 0 0 0) 0x7f01c009bf60 con 0x7f01c4102ac0 2024-05-20T17:58:35.180 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:35.180 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:56:49.336282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T17:58:35.180 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 20 2024-05-20T17:58:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 -- 172.21.15.129:0/411489907 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f01b00a2b90 msgr2=0x7f01b00a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f01b00a2b90 0x7f01b00a50a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009230 tx=0x7f01b4031400 comp rx=0 tx=0).stop 2024-05-20T17:58:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 -- 172.21.15.129:0/411489907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01c4102ac0 msgr2=0x7f01c4199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01c4102ac0 0x7f01c4199300 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f01c000e9f0 tx=0x7f01c000eeb0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 -- 172.21.15.129:0/411489907 shutdown_connections 2024-05-20T17:58:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f01b00a2b90 0x7f01b00a50a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4108a00 0x7f01c41a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01c4103d00 0x7f01c4199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 --2- 172.21.15.129:0/411489907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01c4102ac0 0x7f01c4199300 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 -- 172.21.15.129:0/411489907 >> 172.21.15.129:0/411489907 conn(0x7f01c40fdcf0 msgr2=0x7f01c41049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:35.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 -- 172.21.15.129:0/411489907 shutdown_connections 2024-05-20T17:58:35.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.183+0000 7f01cc606640 1 -- 172.21.15.129:0/411489907 wait complete. 2024-05-20T17:58:35.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:58:35.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.306+0000 7fc3cb20d640 1 -- 172.21.15.129:0/2977219125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3c4102bd0 msgr2=0x7fc3c4102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.306+0000 7fc3cb20d640 1 --2- 172.21.15.129:0/2977219125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3c4102bd0 0x7fc3c4102fe0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac009900 tx=0x7fc3ac02a410 comp rx=0 tx=0).stop 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 -- 172.21.15.129:0/2977219125 shutdown_connections 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 --2- 172.21.15.129:0/2977219125 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c41078d0 0x7fc3c4109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 --2- 172.21.15.129:0/2977219125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3c4102bd0 0x7fc3c4102fe0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 --2- 172.21.15.129:0/2977219125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3c4101990 0x7fc3c4102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 -- 172.21.15.129:0/2977219125 >> 172.21.15.129:0/2977219125 conn(0x7fc3c40fcb80 msgr2=0x7fc3c40ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 -- 172.21.15.129:0/2977219125 shutdown_connections 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 -- 172.21.15.129:0/2977219125 wait complete. 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 Processor -- start 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 -- start start 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3c4101990 0x7fc3c4199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3c4102bd0 0x7fc3c4199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c41078d0 0x7fc3c41a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c410d660 con 0x7fc3c4101990 2024-05-20T17:58:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c410d4e0 con 0x7fc3c4102bd0 2024-05-20T17:58:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.307+0000 7fc3cb20d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c410d7e0 con 0x7fc3c41078d0 2024-05-20T17:58:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3c4102bd0 0x7fc3c4199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3c4102bd0 0x7fc3c4199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:48038/0 (socket says 172.21.15.129:48038) 2024-05-20T17:58:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c3fff640 1 -- 172.21.15.129:0/148720422 learned_addr learned my addr 172.21.15.129:0/148720422 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c97aa640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c41078d0 0x7fc3c41a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c8fa9640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3c4101990 0x7fc3c4199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c3fff640 1 -- 172.21.15.129:0/148720422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c41078d0 msgr2=0x7fc3c41a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c3fff640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c41078d0 0x7fc3c41a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c3fff640 1 -- 172.21.15.129:0/148720422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3c4101990 msgr2=0x7fc3c4199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c3fff640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3c4101990 0x7fc3c4199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c3fff640 1 -- 172.21.15.129:0/148720422 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c41a0e70 con 0x7fc3c4102bd0 2024-05-20T17:58:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c97aa640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c41078d0 0x7fc3c41a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c3fff640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3c4102bd0 0x7fc3c4199870 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac005ab0 tx=0x7fc3ac032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c1ffb640 1 -- 172.21.15.129:0/148720422 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac03f030 con 0x7fc3c4102bd0 2024-05-20T17:58:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3cb20d640 1 -- 172.21.15.129:0/148720422 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c41a1100 con 0x7fc3c4102bd0 2024-05-20T17:58:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.308+0000 7fc3c1ffb640 1 -- 172.21.15.129:0/148720422 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3ac003ce0 con 0x7fc3c4102bd0 2024-05-20T17:58:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.309+0000 7fc3cb20d640 1 -- 172.21.15.129:0/148720422 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c41a15f0 con 0x7fc3c4102bd0 2024-05-20T17:58:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.309+0000 7fc3c1ffb640 1 -- 172.21.15.129:0/148720422 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac049060 con 0x7fc3c4102bd0 2024-05-20T17:58:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.310+0000 7fc3cb20d640 1 -- 172.21.15.129:0/148720422 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3c41a18a0 con 0x7fc3c4102bd0 2024-05-20T17:58:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.311+0000 7fc3c1ffb640 1 -- 172.21.15.129:0/148720422 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fc3ac042030 con 0x7fc3c4102bd0 2024-05-20T17:58:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.312+0000 7fc3c1ffb640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3b00a2c10 0x7fc3b00a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.312+0000 7fc3c8fa9640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3b00a2c10 0x7fc3b00a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.312+0000 7fc3c1ffb640 1 -- 172.21.15.129:0/148720422 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fc3ac0ef950 con 0x7fc3c4102bd0 2024-05-20T17:58:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.312+0000 7fc3c8fa9640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3b00a2c10 0x7fc3b00a5120 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4102590 tx=0x7fc3b400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.316+0000 7fc3c1ffb640 1 -- 172.21.15.129:0/148720422 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc3ac0bc1f0 con 0x7fc3c4102bd0 2024-05-20T17:58:35.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.493+0000 7fc3cb20d640 1 -- 172.21.15.129:0/148720422 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3c41a1a70 con 0x7fc3c4102bd0 2024-05-20T17:58:35.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.495+0000 7fc3c1ffb640 1 -- 172.21.15.129:0/148720422 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22680 (secure 0 0 0) 0x7fc3ac0c04f0 con 0x7fc3c4102bd0 2024-05-20T17:58:35.494 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:35.494 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":88,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:57:49.033961+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:58:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 -- 172.21.15.129:0/148720422 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3b00a2c10 msgr2=0x7fc3b00a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3b00a2c10 0x7fc3b00a5120 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4102590 tx=0x7fc3b400b390 comp rx=0 tx=0).stop 2024-05-20T17:58:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 -- 172.21.15.129:0/148720422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3c4102bd0 msgr2=0x7fc3c4199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3c4102bd0 0x7fc3c4199870 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac005ab0 tx=0x7fc3ac032ce0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 -- 172.21.15.129:0/148720422 shutdown_connections 2024-05-20T17:58:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3b00a2c10 0x7fc3b00a5120 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3c4102bd0 0x7fc3c4199870 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc3c41078d0 0x7fc3c41a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 --2- 172.21.15.129:0/148720422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3c4101990 0x7fc3c4199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 -- 172.21.15.129:0/148720422 >> 172.21.15.129:0/148720422 conn(0x7fc3c40fcb80 msgr2=0x7fc3c4073d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 -- 172.21.15.129:0/148720422 shutdown_connections 2024-05-20T17:58:35.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.497+0000 7fc3cb20d640 1 -- 172.21.15.129:0/148720422 wait complete. 2024-05-20T17:58:35.506 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:35 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1914022573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:35.506 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:35 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1728004906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:35.506 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:35 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/411489907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:35.508 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-20T17:56:45.096192+0000', 'modified': '2024-05-20T17:56:49.336282+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24560}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24521': {'gid': 24521, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.154:6837/1478533749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 1478533749}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 1478533749}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.154:6839/793468142', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 793468142}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 793468142}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24560, 'qdb_cluster': [24560]} 2024-05-20T17:58:35.508 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-20T17:58:35.508 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-20T17:58:35.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.621+0000 7fc331f79640 1 -- 172.21.15.129:0/632858152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc32c102bd0 msgr2=0x7fc32c102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.621+0000 7fc331f79640 1 --2- 172.21.15.129:0/632858152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc32c102bd0 0x7fc32c102fe0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc320009ca0 tx=0x7fc32002a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.622+0000 7fc331f79640 1 -- 172.21.15.129:0/632858152 shutdown_connections 2024-05-20T17:58:35.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.622+0000 7fc331f79640 1 --2- 172.21.15.129:0/632858152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc32c1078d0 0x7fc32c109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.622+0000 7fc331f79640 1 --2- 172.21.15.129:0/632858152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc32c102bd0 0x7fc32c102fe0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.622+0000 7fc331f79640 1 --2- 172.21.15.129:0/632858152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc32c101990 0x7fc32c102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.622+0000 7fc331f79640 1 -- 172.21.15.129:0/632858152 >> 172.21.15.129:0/632858152 conn(0x7fc32c0fcb80 msgr2=0x7fc32c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:35.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.622+0000 7fc331f79640 1 -- 172.21.15.129:0/632858152 shutdown_connections 2024-05-20T17:58:35.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.622+0000 7fc331f79640 1 -- 172.21.15.129:0/632858152 wait complete. 2024-05-20T17:58:35.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.622+0000 7fc331f79640 1 Processor -- start 2024-05-20T17:58:35.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.622+0000 7fc331f79640 1 -- start start 2024-05-20T17:58:35.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc331f79640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc32c101990 0x7fc32c199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc331f79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc32c102bd0 0x7fc32c199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc331f79640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc32c1078d0 0x7fc32c1a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc331f79640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c10d5e0 con 0x7fc32c101990 2024-05-20T17:58:35.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc331f79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c10d460 con 0x7fc32c102bd0 2024-05-20T17:58:35.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc331f79640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c10d760 con 0x7fc32c1078d0 2024-05-20T17:58:35.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc32c101990 0x7fc32c199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc32c101990 0x7fc32c199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34608/0 (socket says 172.21.15.129:34608) 2024-05-20T17:58:35.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32b7fe640 1 -- 172.21.15.129:0/3826909145 learned_addr learned my addr 172.21.15.129:0/3826909145 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:35.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32bfff640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc32c1078d0 0x7fc32c1a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32affd640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc32c102bd0 0x7fc32c199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32b7fe640 1 -- 172.21.15.129:0/3826909145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc32c1078d0 msgr2=0x7fc32c1a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc32c1078d0 0x7fc32c1a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32b7fe640 1 -- 172.21.15.129:0/3826909145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc32c102bd0 msgr2=0x7fc32c199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc32c102bd0 0x7fc32c199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32b7fe640 1 -- 172.21.15.129:0/3826909145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c1a0d90 con 0x7fc32c101990 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32bfff640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc32c1078d0 0x7fc32c1a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32affd640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc32c102bd0 0x7fc32c199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc32c101990 0x7fc32c199300 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00c970 tx=0x7fc31c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc328ff9640 1 -- 172.21.15.129:0/3826909145 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c007c70 con 0x7fc32c101990 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc331f79640 1 -- 172.21.15.129:0/3826909145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc32c1a1080 con 0x7fc32c101990 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc328ff9640 1 -- 172.21.15.129:0/3826909145 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc31c007e10 con 0x7fc32c101990 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc328ff9640 1 -- 172.21.15.129:0/3826909145 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c0188c0 con 0x7fc32c101990 2024-05-20T17:58:35.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.623+0000 7fc331f79640 1 -- 172.21.15.129:0/3826909145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32c1a15d0 con 0x7fc32c101990 2024-05-20T17:58:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc338e19740 1 Processor -- start 2024-05-20T17:58:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc338e19740 1 -- start start 2024-05-20T17:58:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc328ff9640 1 -- 172.21.15.129:0/3826909145 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fc31c0213f0 con 0x7fc32c101990 2024-05-20T17:58:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc338e19740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b79270 0x561a12f19470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc338e19740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b503b0 0x561a12f199f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc338e19740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b58590 0x561a12f208f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc338e19740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561a12e7d4d0 con 0x561a12b58590 2024-05-20T17:58:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc338e19740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561a12e253e0 con 0x561a12b503b0 2024-05-20T17:58:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc338e19740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561a12e79490 con 0x561a12b79270 2024-05-20T17:58:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc32affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b58590 0x561a12f208f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc32affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b58590 0x561a12f208f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34624/0 (socket says 172.21.15.129:34624) 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc32affd640 1 -- 172.21.15.129:0/313935102 learned_addr learned my addr 172.21.15.129:0/313935102 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.626+0000 7fc32bfff640 1 --2- 172.21.15.129:0/313935102 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b79270 0x561a12f19470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/313935102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b503b0 0x561a12f199f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc32affd640 1 -- 172.21.15.129:0/313935102 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b79270 msgr2=0x561a12f19470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc32affd640 1 --2- 172.21.15.129:0/313935102 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b79270 0x561a12f19470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc32affd640 1 -- 172.21.15.129:0/313935102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b503b0 msgr2=0x561a12f199f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc32affd640 1 --2- 172.21.15.129:0/313935102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b503b0 0x561a12f199f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc32affd640 1 -- 172.21.15.129:0/313935102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a12b912d0 con 0x561a12b58590 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc32bfff640 1 --2- 172.21.15.129:0/313935102 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b79270 0x561a12f19470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc328ff9640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3140a2b90 0x7fc3140a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc32affd640 1 --2- 172.21.15.129:0/313935102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b58590 0x561a12f208f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc320000f80 tx=0x7fc32002f500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989114c0d7912ad4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc32bfff640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3140a2b90 0x7fc3140a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc328ff9640 1 -- 172.21.15.129:0/3826909145 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fc31c0cf1b0 con 0x7fc32c101990 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc311ffb640 1 -- 172.21.15.129:0/313935102 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320003b70 con 0x561a12b58590 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc311ffb640 1 -- 172.21.15.129:0/313935102 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc320036050 con 0x561a12b58590 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.627+0000 7fc311ffb640 1 -- 172.21.15.129:0/313935102 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320004610 con 0x561a12b58590 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.628+0000 7fc338e19740 1 -- 172.21.15.129:0/313935102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b58590 msgr2=0x561a12f208f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.628+0000 7fc32bfff640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3140a2b90 0x7fc3140a50a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc324005d00 tx=0x7fc324005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.628+0000 7fc338e19740 1 --2- 172.21.15.129:0/313935102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b58590 0x561a12f208f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc320000f80 tx=0x7fc32002f500 comp rx=0 tx=0).stop 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.628+0000 7fc338e19740 1 -- 172.21.15.129:0/313935102 shutdown_connections 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.628+0000 7fc338e19740 1 --2- 172.21.15.129:0/313935102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b503b0 0x561a12f199f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.628+0000 7fc338e19740 1 --2- 172.21.15.129:0/313935102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b58590 0x561a12f208f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.628+0000 7fc338e19740 1 --2- 172.21.15.129:0/313935102 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b79270 0x561a12f19470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.628+0000 7fc338e19740 1 -- 172.21.15.129:0/313935102 >> 172.21.15.129:0/313935102 conn(0x561a12b6d3c0 msgr2=0x561a12f14e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.628+0000 7fc338e19740 1 -- 172.21.15.129:0/313935102 shutdown_connections 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.628+0000 7fc338e19740 1 -- 172.21.15.129:0/313935102 wait complete. 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.628+0000 7fc338e19740 1 -- start start 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc338e19740 1 Processor -- start 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc338e19740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b503b0 0x561a12f230c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc338e19740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b58590 0x561a12f23600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc338e19740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f2a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc338e19740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561a12f21160 con 0x561a12b79270 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc338e19740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561a12e388e0 con 0x561a12b58590 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc338e19740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561a12f212e0 con 0x561a12b503b0 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc338e19740 10 client.0 _ll_register_callbacks cb 0x561a12e059c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f2a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f2a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34634/0 (socket says 172.21.15.129:34634) 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc32bfff640 1 -- 172.21.15.129:0/251285685 learned_addr learned my addr 172.21.15.129:0/251285685 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc32affd640 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b58590 0x561a12f23600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b503b0 0x561a12f230c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc32bfff640 1 -- 172.21.15.129:0/251285685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b503b0 msgr2=0x561a12f230c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc32bfff640 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b503b0 0x561a12f230c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc32bfff640 1 -- 172.21.15.129:0/251285685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b58590 msgr2=0x561a12f23600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc32bfff640 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b58590 0x561a12f23600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc32bfff640 1 -- 172.21.15.129:0/251285685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a12c5b6b0 con 0x561a12b79270 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc32bfff640 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f2a560 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc3240126d0 tx=0x7fc3240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a98263f0b8eaebc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.629+0000 7fc30affd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc30affd640 1 -- 172.21.15.129:0/251285685 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32401f8f0 con 0x561a12b79270 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc30affd640 1 -- 172.21.15.129:0/251285685 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fc32401fa90 con 0x561a12b79270 2024-05-20T17:58:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc338e19740 1 -- 172.21.15.129:0/251285685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc320036050 con 0x561a12b79270 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc30affd640 1 -- 172.21.15.129:0/251285685 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc32401fc90 con 0x561a12b79270 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc338e19740 10 client.15201 fetch_fsmap learned FSMap version 20 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc338e19740 1 -- 172.21.15.129:0/251285685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x561a12f2d560 con 0x561a12b79270 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc30affd640 1 -- 172.21.15.129:0/251285685 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc320036050 con 0x561a12b79270 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc30affd640 1 -- 172.21.15.129:0/251285685 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc324027540 con 0x561a12b79270 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc338e19740 10 client.15201 fetch_fsmap finished waiting for FSMap version 20 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc338e19740 10 client.15201 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc338e19740 20 client.15201 populate_metadata read hostname 'smithi129' 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc338e19740 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561a12b37e40 0x561a12f310a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc338e19740 4 client.15201 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc338e19740 1 -- 172.21.15.129:0/251285685 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a12e8ac00 con 0x561a12b37e40 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.630+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561a12b37e40 0x561a12f310a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.631+0000 7fc30affd640 10 client.15201 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.631+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561a12b37e40 0x561a12f310a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbfc7aad29c8729 server_cookie=91661172c27144ee in_seq=0 out_seq=0 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc30affd640 1 -- 172.21.15.129:0/251285685 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561a12e8ac00 con 0x561a12b37e40 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc30affd640 10 client.15201 handle_command_reply: tid=0 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 1 client.15201 shutdown 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 1 -- 172.21.15.129:0/251285685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 msgr2=0x561a12f2a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f2a560 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc3240126d0 tx=0x7fc3240046c0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 1 -- 172.21.15.129:0/251285685 shutdown_connections 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561a12b37e40 0x561a12f310a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b58590 0x561a12f23600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f2a560 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc30affd640 0 client.15201 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 1 --2- 172.21.15.129:0/251285685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b503b0 0x561a12f230c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 1 -- 172.21.15.129:0/251285685 >> 172.21.15.129:0/251285685 conn(0x561a12b6d3c0 msgr2=0x561a12f13ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:35.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 1 -- 172.21.15.129:0/251285685 shutdown_connections 2024-05-20T17:58:35.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 1 -- 172.21.15.129:0/251285685 wait complete. 2024-05-20T17:58:35.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.632+0000 7fc338e19740 20 client.15201 trim_cache size 0 max 16384 2024-05-20T17:58:35.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:35 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1914022573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:35.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:35 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1728004906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:35.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:35 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/411489907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:35.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.650+0000 7fc338e19740 1 Processor -- start 2024-05-20T17:58:35.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.650+0000 7fc338e19740 1 -- start start 2024-05-20T17:58:35.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc338e19740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b503b0 0x561a12f2ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc338e19740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b58590 0x561a12f2a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc338e19740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f1d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc338e19740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561a12f2f3e0 con 0x561a12b79270 2024-05-20T17:58:35.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc338e19740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561a12f2fab0 con 0x561a12b503b0 2024-05-20T17:58:35.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc338e19740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561a12f30180 con 0x561a12b58590 2024-05-20T17:58:35.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f1d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f1d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34650/0 (socket says 172.21.15.129:34650) 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32bfff640 1 -- 172.21.15.129:0/3004222508 learned_addr learned my addr 172.21.15.129:0/3004222508 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32affd640 1 --2- 172.21.15.129:0/3004222508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b58590 0x561a12f2a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/3004222508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b503b0 0x561a12f2ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32bfff640 1 -- 172.21.15.129:0/3004222508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b58590 msgr2=0x561a12f2a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32bfff640 1 --2- 172.21.15.129:0/3004222508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b58590 0x561a12f2a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32bfff640 1 -- 172.21.15.129:0/3004222508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b503b0 msgr2=0x561a12f2ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32bfff640 1 --2- 172.21.15.129:0/3004222508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b503b0 0x561a12f2ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32bfff640 1 -- 172.21.15.129:0/3004222508 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a12f19020 con 0x561a12b79270 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/3004222508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b503b0 0x561a12f2ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc32bfff640 1 --2- 172.21.15.129:0/3004222508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f1d670 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc3240279b0 tx=0x7fc324000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ccd5f1f8b0b4edc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc3117fa640 1 -- 172.21.15.129:0/3004222508 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324027da0 con 0x561a12b79270 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc3117fa640 1 -- 172.21.15.129:0/3004222508 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc32401f6f0 con 0x561a12b79270 2024-05-20T17:58:35.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.651+0000 7fc3117fa640 1 -- 172.21.15.129:0/3004222508 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324004730 con 0x561a12b79270 2024-05-20T17:58:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.652+0000 7fc338e19740 1 -- 172.21.15.129:0/3004222508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 msgr2=0x561a12f1d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.652+0000 7fc338e19740 1 --2- 172.21.15.129:0/3004222508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f1d670 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc3240279b0 tx=0x7fc324000bc0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.652+0000 7fc338e19740 1 -- 172.21.15.129:0/3004222508 shutdown_connections 2024-05-20T17:58:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.652+0000 7fc338e19740 1 --2- 172.21.15.129:0/3004222508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b79270 0x561a12f1d670 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.652+0000 7fc338e19740 1 --2- 172.21.15.129:0/3004222508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b58590 0x561a12f2a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.652+0000 7fc338e19740 1 --2- 172.21.15.129:0/3004222508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b503b0 0x561a12f2ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.652+0000 7fc338e19740 1 -- 172.21.15.129:0/3004222508 >> 172.21.15.129:0/3004222508 conn(0x561a12b6d3c0 msgr2=0x561a12f230c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.652+0000 7fc338e19740 1 -- 172.21.15.129:0/3004222508 shutdown_connections 2024-05-20T17:58:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.652+0000 7fc338e19740 1 -- 172.21.15.129:0/3004222508 wait complete. 2024-05-20T17:58:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.652+0000 7fc338e19740 1 -- start start 2024-05-20T17:58:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc338e19740 1 Processor -- start 2024-05-20T17:58:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc338e19740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b503b0 0x561a12f25820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc338e19740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b58590 0x561a12f25d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc338e19740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b79270 0x561a12f28ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc338e19740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561a12f30f20 con 0x561a12b503b0 2024-05-20T17:58:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc338e19740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561a12f30850 con 0x561a12b79270 2024-05-20T17:58:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc338e19740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561a12f315f0 con 0x561a12b58590 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc338e19740 10 client.0 _ll_register_callbacks cb 0x7fc32400b970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b503b0 0x561a12f25820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b503b0 0x561a12f25820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34666/0 (socket says 172.21.15.129:34666) 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32b7fe640 1 -- 172.21.15.129:0/2272184192 learned_addr learned my addr 172.21.15.129:0/2272184192 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32bfff640 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b79270 0x561a12f28ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32affd640 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b58590 0x561a12f25d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32b7fe640 1 -- 172.21.15.129:0/2272184192 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b58590 msgr2=0x561a12f25d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b58590 0x561a12f25d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32b7fe640 1 -- 172.21.15.129:0/2272184192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b79270 msgr2=0x561a12f28ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b79270 0x561a12f28ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32b7fe640 1 -- 172.21.15.129:0/2272184192 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a12f29260 con 0x561a12b503b0 2024-05-20T17:58:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32affd640 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b58590 0x561a12f25d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc32b7fe640 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b503b0 0x561a12f25820 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc31c097e60 tx=0x7fc31c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8abab6bba5a9bd6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.653+0000 7fc30affd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc30affd640 1 -- 172.21.15.129:0/2272184192 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c09c100 con 0x561a12b503b0 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc30affd640 1 -- 172.21.15.129:0/2272184192 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fc31c00fb50 con 0x561a12b503b0 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc338e19740 1 -- 172.21.15.129:0/2272184192 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc32401f6f0 con 0x561a12b503b0 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc30affd640 1 -- 172.21.15.129:0/2272184192 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc31c09ba00 con 0x561a12b503b0 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc338e19740 10 client.15207 fetch_fsmap learned FSMap version 20 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc338e19740 1 -- 172.21.15.129:0/2272184192 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x561a12f2a520 con 0x561a12b503b0 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc30affd640 1 -- 172.21.15.129:0/2272184192 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc32401f6f0 con 0x561a12b503b0 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc30affd640 1 -- 172.21.15.129:0/2272184192 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc31c020030 con 0x561a12b503b0 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc338e19740 10 client.15207 fetch_fsmap finished waiting for FSMap version 20 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc338e19740 10 client.15207 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-20T17:58:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc338e19740 20 client.15207 populate_metadata read hostname 'smithi129' 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc338e19740 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561a12b0d280 0x561a12f9d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc338e19740 4 client.15207 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.654+0000 7fc338e19740 1 -- 172.21.15.129:0/2272184192 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "status"}) -- 0x561a12e84f50 con 0x561a12b0d280 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc32affd640 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561a12b0d280 0x561a12f9d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc30affd640 10 client.15207 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc32affd640 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561a12b0d280 0x561a12f9d2e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9f6c127a62e6b92 server_cookie=6cd31ccd7cc533c6 in_seq=0 out_seq=0 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc30affd640 1 -- 172.21.15.129:0/2272184192 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561a12e84f50 con 0x561a12b0d280 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc30affd640 10 client.15207 handle_command_reply: tid=0 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc338e19740 1 client.15207 shutdown 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc338e19740 1 -- 172.21.15.129:0/2272184192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b503b0 msgr2=0x561a12f25820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc338e19740 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b503b0 0x561a12f25820 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc31c097e60 tx=0x7fc31c000bc0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc338e19740 1 -- 172.21.15.129:0/2272184192 shutdown_connections 2024-05-20T17:58:35.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc338e19740 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561a12b0d280 0x561a12f9d2e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc338e19740 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561a12b79270 0x561a12f28ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc338e19740 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561a12b58590 0x561a12f25d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc30affd640 0 client.15207 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:35.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc338e19740 1 --2- 172.21.15.129:0/2272184192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a12b503b0 0x561a12f25820 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc338e19740 1 -- 172.21.15.129:0/2272184192 >> 172.21.15.129:0/2272184192 conn(0x561a12b6d3c0 msgr2=0x561a12f2ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:35.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.655+0000 7fc338e19740 1 -- 172.21.15.129:0/2272184192 shutdown_connections 2024-05-20T17:58:35.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc338e19740 1 -- 172.21.15.129:0/2272184192 wait complete. 2024-05-20T17:58:35.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc338e19740 20 client.15207 trim_cache size 0 max 16384 2024-05-20T17:58:35.661 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T17:58:35.661 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "0f1f0dd2-16d1-11ef-bc9a-c7b262605968", 2024-05-20T17:58:35.662 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-05-20T17:58:35.662 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 24560, 2024-05-20T17:58:35.662 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-05-20T17:58:35.662 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-05-20T17:58:35.662 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-05-20T17:58:35.662 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 109.696872423, 2024-05-20T17:58:35.662 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 20, 2024-05-20T17:58:35.662 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 88, 2024-05-20T17:58:35.662 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 86, 2024-05-20T17:58:35.662 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 115.49866759299999 2024-05-20T17:58:35.662 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T17:58:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 -- 172.21.15.129:0/3826909145 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3140a2b90 msgr2=0x7fc3140a50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3140a2b90 0x7fc3140a50a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc324005d00 tx=0x7fc324005c50 comp rx=0 tx=0).stop 2024-05-20T17:58:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 -- 172.21.15.129:0/3826909145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc32c101990 msgr2=0x7fc32c199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc32c101990 0x7fc32c199300 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00c970 tx=0x7fc31c00ce30 comp rx=0 tx=0).stop 2024-05-20T17:58:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 -- 172.21.15.129:0/3826909145 shutdown_connections 2024-05-20T17:58:35.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fc3140a2b90 0x7fc3140a50a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc32c102bd0 0x7fc32c199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc32c1078d0 0x7fc32c1a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 --2- 172.21.15.129:0/3826909145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc32c101990 0x7fc32c199300 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 -- 172.21.15.129:0/3826909145 >> 172.21.15.129:0/3826909145 conn(0x7fc32c0fcb80 msgr2=0x7fc32c108d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 -- 172.21.15.129:0/3826909145 shutdown_connections 2024-05-20T17:58:35.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.656+0000 7fc331f79640 1 -- 172.21.15.129:0/3826909145 wait complete. 2024-05-20T17:58:35.666 INFO:teuthology.run_tasks:Running task sequential... 2024-05-20T17:58:35.675 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-20T17:58:35.675 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-20T17:58:35.676 INFO:teuthology.task.print:Enabling mgr modules 2024-05-20T17:58:35.676 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-20T17:58:35.677 INFO:teuthology.task.exec:Executing custom commands... 2024-05-20T17:58:35.677 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi129.front.sepia.ceph.com 2024-05-20T17:58:35.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-20T17:58:35.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:35 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1914022573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:35.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:35 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1728004906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:35.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:35 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/411489907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:35.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.789+0000 7fb5a10a8640 1 -- 172.21.15.129:0/842354898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb59c1019c0 msgr2=0x7fb59c1026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.789+0000 7fb5a10a8640 1 --2- 172.21.15.129:0/842354898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb59c1019c0 0x7fb59c1026c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb58c009900 tx=0x7fb58c02a410 comp rx=0 tx=0).stop 2024-05-20T17:58:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.789+0000 7fb5a10a8640 1 -- 172.21.15.129:0/842354898 shutdown_connections 2024-05-20T17:58:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.789+0000 7fb5a10a8640 1 --2- 172.21.15.129:0/842354898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb59c107900 0x7fb59c109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.789+0000 7fb5a10a8640 1 --2- 172.21.15.129:0/842354898 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb59c102c00 0x7fb59c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.789+0000 7fb5a10a8640 1 --2- 172.21.15.129:0/842354898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb59c1019c0 0x7fb59c1026c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.789+0000 7fb5a10a8640 1 -- 172.21.15.129:0/842354898 >> 172.21.15.129:0/842354898 conn(0x7fb59c0fcbb0 msgr2=0x7fb59c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.789+0000 7fb5a10a8640 1 -- 172.21.15.129:0/842354898 shutdown_connections 2024-05-20T17:58:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.789+0000 7fb5a10a8640 1 -- 172.21.15.129:0/842354898 wait complete. 2024-05-20T17:58:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.790+0000 7fb5a10a8640 1 Processor -- start 2024-05-20T17:58:35.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.790+0000 7fb5a10a8640 1 -- start start 2024-05-20T17:58:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb5a10a8640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb59c1019c0 0x7fb59c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb5a10a8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb59c102c00 0x7fb59c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb5a10a8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb59c107900 0x7fb59c19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb5a10a8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb59c10d5f0 con 0x7fb59c102c00 2024-05-20T17:58:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb5a10a8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb59c10d470 con 0x7fb59c107900 2024-05-20T17:58:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb5a10a8640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb59c10d770 con 0x7fb59c1019c0 2024-05-20T17:58:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb59c102c00 0x7fb59c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb59c102c00 0x7fb59c198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34680/0 (socket says 172.21.15.129:34680) 2024-05-20T17:58:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59a59c640 1 -- 172.21.15.129:0/1705522519 learned_addr learned my addr 172.21.15.129:0/1705522519 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59ad9d640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb59c1019c0 0x7fb59c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59a59c640 1 -- 172.21.15.129:0/1705522519 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb59c1019c0 msgr2=0x7fb59c198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59a59c640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb59c1019c0 0x7fb59c198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59b59e640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb59c107900 0x7fb59c19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59a59c640 1 -- 172.21.15.129:0/1705522519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb59c107900 msgr2=0x7fb59c19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59a59c640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb59c107900 0x7fb59c19f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59a59c640 1 -- 172.21.15.129:0/1705522519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb59c19fc90 con 0x7fb59c102c00 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59ad9d640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb59c1019c0 0x7fb59c198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb59a59c640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb59c102c00 0x7fb59c198740 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb58400d950 tx=0x7fb58400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb593fff640 1 -- 172.21.15.129:0/1705522519 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb584017070 con 0x7fb59c102c00 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb593fff640 1 -- 172.21.15.129:0/1705522519 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb584009c30 con 0x7fb59c102c00 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb593fff640 1 -- 172.21.15.129:0/1705522519 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58400f6b0 con 0x7fb59c102c00 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.791+0000 7fb5a10a8640 1 -- 172.21.15.129:0/1705522519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb59c19ff80 con 0x7fb59c102c00 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.792+0000 7fb5a10a8640 1 -- 172.21.15.129:0/1705522519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb59c1a03e0 con 0x7fb59c102c00 2024-05-20T17:58:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.793+0000 7fb5a10a8640 1 -- 172.21.15.129:0/1705522519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb59c106710 con 0x7fb59c102c00 2024-05-20T17:58:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.794+0000 7fb593fff640 1 -- 172.21.15.129:0/1705522519 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb5840109d0 con 0x7fb59c102c00 2024-05-20T17:58:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.795+0000 7fb593fff640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb5880a2b80 0x7fb5880a5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.795+0000 7fb59ad9d640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb5880a2b80 0x7fb5880a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:35.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.795+0000 7fb593fff640 1 -- 172.21.15.129:0/1705522519 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fb5840ced10 con 0x7fb59c102c00 2024-05-20T17:58:35.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.795+0000 7fb59ad9d640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb5880a2b80 0x7fb5880a5090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb58c009230 tx=0x7fb58c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:35.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:35.798+0000 7fb593fff640 1 -- 172.21.15.129:0/1705522519 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb584009de0 con 0x7fb59c102c00 2024-05-20T17:58:36.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.025+0000 7fb5a10a8640 1 -- 172.21.15.129:0/1705522519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fb59c02d450 con 0x7fb59c102c00 2024-05-20T17:58:36.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.236+0000 7fb593fff640 1 -- 172.21.15.129:0/1705522519 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fb58409bc60 con 0x7fb59c102c00 2024-05-20T17:58:36.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.238+0000 7fb5a10a8640 1 -- 172.21.15.129:0/1705522519 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb5880a2b80 msgr2=0x7fb5880a5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:36.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.238+0000 7fb5a10a8640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb5880a2b80 0x7fb5880a5090 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb58c009230 tx=0x7fb58c000bc0 comp rx=0 tx=0).stop 2024-05-20T17:58:36.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.238+0000 7fb5a10a8640 1 -- 172.21.15.129:0/1705522519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb59c102c00 msgr2=0x7fb59c198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:36.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.238+0000 7fb5a10a8640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb59c102c00 0x7fb59c198740 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb58400d950 tx=0x7fb58400de10 comp rx=0 tx=0).stop 2024-05-20T17:58:36.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.238+0000 7fb5a10a8640 1 -- 172.21.15.129:0/1705522519 shutdown_connections 2024-05-20T17:58:36.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.238+0000 7fb5a10a8640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fb5880a2b80 0x7fb5880a5090 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:36.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.238+0000 7fb5a10a8640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb59c107900 0x7fb59c19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:36.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.238+0000 7fb5a10a8640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb59c102c00 0x7fb59c198740 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:36.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.238+0000 7fb5a10a8640 1 --2- 172.21.15.129:0/1705522519 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb59c1019c0 0x7fb59c198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:36.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.238+0000 7fb5a10a8640 1 -- 172.21.15.129:0/1705522519 >> 172.21.15.129:0/1705522519 conn(0x7fb59c0fcbb0 msgr2=0x7fb59c1038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:36.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.239+0000 7fb5a10a8640 1 -- 172.21.15.129:0/1705522519 shutdown_connections 2024-05-20T17:58:36.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:36.239+0000 7fb5a10a8640 1 -- 172.21.15.129:0/1705522519 wait complete. 2024-05-20T17:58:36.254 INFO:teuthology.run_tasks:Running task kclient... 2024-05-20T17:58:36.281 INFO:tasks.kclient:Mounting kernel clients... 2024-05-20T17:58:36.281 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-20T17:58:36.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:58:36.282 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-05-20T17:58:36.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:58:36.299 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link delete ceph-brx 2024-05-20T17:58:36.367 INFO:teuthology.orchestra.run.smithi129.stderr:Cannot find device "ceph-brx" 2024-05-20T17:58:36.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T17:58:36.368 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-05-20T17:58:36.369 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-20T17:58:36.369 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-20T17:58:36.369 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph fs ls 2024-05-20T17:58:36.624 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:36 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/148720422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:36.624 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:36 smithi154 ceph-mon[36482]: pgmap v342: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:36.624 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:36 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1705522519' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-20T17:58:36.624 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: ignoring --setuser ceph since I am not root 2024-05-20T17:58:36.624 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: ignoring --setgroup ceph since I am not root 2024-05-20T17:58:36.624 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:36 smithi154 ceph-mgr[37858]: -- 172.21.15.154:0/579650891 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b92a9f05a0 con 0x55b92ab2e480 2024-05-20T17:58:36.624 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:36.437+0000 7fc09a40b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-20T17:58:36.625 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:36.625+0000 7fc09a40b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:58:36.636 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:36 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/148720422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:36.636 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:36 smithi204 ceph-mon[37076]: pgmap v342: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:36.636 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:36 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1705522519' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-20T17:58:36.636 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: ignoring --setuser ceph since I am not root 2024-05-20T17:58:36.636 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: ignoring --setgroup ceph since I am not root 2024-05-20T17:58:36.636 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:36 smithi204 ceph-mgr[38906]: -- 172.21.15.204:0/2502153815 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bc76dae5a0 con 0x55bc76eec000 2024-05-20T17:58:36.636 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:36.450+0000 7f44af1fc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-20T17:58:36.636 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:36.637+0000 7f44af1fc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:58:36.652 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:36 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/148720422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:36.652 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:36 smithi129 ceph-mon[32029]: pgmap v342: 97 pgs: 97 active+clean; 637 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T17:58:36.652 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:36 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1705522519' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-20T17:58:36.653 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:36 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: ignoring --setuser ceph since I am not root 2024-05-20T17:58:36.653 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:36 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: ignoring --setgroup ceph since I am not root 2024-05-20T17:58:36.653 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:36 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:36.452+0000 7fddf53ad200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-20T17:58:36.653 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:36 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:36.654+0000 7fddf53ad200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:58:36.656 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T17:58:36.893 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:36.731+0000 7fc09a40b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-20T17:58:36.912 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:36 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:36.770+0000 7fddf53ad200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-20T17:58:36.929 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:36.744+0000 7f44af1fc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-20T17:58:37.180 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:37 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:37.094+0000 7fddf53ad200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-20T17:58:37.181 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:37 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:37.182+0000 7fddf53ad200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-20T17:58:37.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:37 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1705522519' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-20T17:58:37.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:37 smithi154 ceph-mon[36482]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-05-20T17:58:37.393 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:37 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:37.044+0000 7fc09a40b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-20T17:58:37.393 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:37 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:37.131+0000 7fc09a40b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-20T17:58:37.429 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:37.059+0000 7f44af1fc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-20T17:58:37.429 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:37.145+0000 7f44af1fc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-20T17:58:37.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:37 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1705522519' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-20T17:58:37.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:37 smithi204 ceph-mon[37076]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-05-20T17:58:37.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:37 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1705522519' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-20T17:58:37.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:37 smithi129 ceph-mon[32029]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-05-20T17:58:38.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.453+0000 7f19ae897700 1 -- 172.21.15.129:0/4206224834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19a8106c70 msgr2=0x7f19a8107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:38.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.453+0000 7f19ae897700 1 --2- 172.21.15.129:0/4206224834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19a8106c70 0x7f19a8107080 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f19900099c0 tx=0x7f1990009cd0 comp rx=0 tx=0).stop 2024-05-20T17:58:38.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.453+0000 7f19ae897700 1 -- 172.21.15.129:0/4206224834 shutdown_connections 2024-05-20T17:58:38.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.453+0000 7f19ae897700 1 --2- 172.21.15.129:0/4206224834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19a81075c0 0x7f19a810ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:38.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.453+0000 7f19ae897700 1 --2- 172.21.15.129:0/4206224834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a8102340 0x7f19a8102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:38.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.453+0000 7f19ae897700 1 --2- 172.21.15.129:0/4206224834 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19a8106c70 0x7f19a8107080 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:38.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.453+0000 7f19ae897700 1 -- 172.21.15.129:0/4206224834 >> 172.21.15.129:0/4206224834 conn(0x7f19a80fdc40 msgr2=0x7f19a8100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:38.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.453+0000 7f19ae897700 1 -- 172.21.15.129:0/4206224834 shutdown_connections 2024-05-20T17:58:38.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.453+0000 7f19ae897700 1 -- 172.21.15.129:0/4206224834 wait complete. 2024-05-20T17:58:38.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.454+0000 7f19ae897700 1 Processor -- start 2024-05-20T17:58:38.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.454+0000 7f19ae897700 1 -- start start 2024-05-20T17:58:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.454+0000 7f19ae897700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a8102340 0x7f19a819e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.454+0000 7f19ae897700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19a8106c70 0x7f19a819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.454+0000 7f19ae897700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19a81075c0 0x7f19a8198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.454+0000 7f19ae897700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f19a8112320 con 0x7f19a8106c70 2024-05-20T17:58:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.454+0000 7f19ae897700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f19a81121a0 con 0x7f19a81075c0 2024-05-20T17:58:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.454+0000 7f19ae897700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f19a81124a0 con 0x7f19a8102340 2024-05-20T17:58:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.454+0000 7f19a77fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19a8106c70 0x7f19a819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a77fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19a8106c70 0x7f19a819ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34726/0 (socket says 172.21.15.129:34726) 2024-05-20T17:58:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19ace34700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19a81075c0 0x7f19a8198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a7fff700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a8102340 0x7f19a819e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a77fe700 1 -- 172.21.15.129:0/3548725729 learned_addr learned my addr 172.21.15.129:0/3548725729 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a7fff700 1 -- 172.21.15.129:0/3548725729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19a81075c0 msgr2=0x7f19a8198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a7fff700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19a81075c0 0x7f19a8198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a7fff700 1 -- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19a8106c70 msgr2=0x7f19a819ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a7fff700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19a8106c70 0x7f19a819ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a7fff700 1 -- 172.21.15.129:0/3548725729 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a81991c0 con 0x7f19a8102340 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19ace34700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19a81075c0 0x7f19a8198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a77fe700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19a8106c70 0x7f19a819ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a7fff700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a8102340 0x7f19a819e8a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f199800e9f0 tx=0x7f199800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a57fa700 1 -- 172.21.15.129:0/3548725729 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199800c9c0 con 0x7f19a8102340 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19ae897700 1 -- 172.21.15.129:0/3548725729 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19a81994b0 con 0x7f19a8102340 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a57fa700 1 -- 172.21.15.129:0/3548725729 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f199800cb60 con 0x7f19a8102340 2024-05-20T17:58:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19a57fa700 1 -- 172.21.15.129:0/3548725729 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1998018680 con 0x7f19a8102340 2024-05-20T17:58:38.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.455+0000 7f19ae897700 1 -- 172.21.15.129:0/3548725729 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19a806a490 con 0x7f19a8102340 2024-05-20T17:58:38.457 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:38 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:38.409+0000 7fddf53ad200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-20T17:58:38.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.456+0000 7f19ae897700 1 -- 172.21.15.129:0/3548725729 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19a8050170 con 0x7f19a8102340 2024-05-20T17:58:38.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.458+0000 7f19a57fa700 1 -- 172.21.15.129:0/3548725729 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f1998010430 con 0x7f19a8102340 2024-05-20T17:58:38.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.458+0000 7f19a57fa700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f19940a2c70 0x7f19940a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:38.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.458+0000 7f19a77fe700 1 -- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f19940a2c70 msgr2=0x7f19940a5180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2870089245 2024-05-20T17:58:38.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.458+0000 7f19a57fa700 1 -- 172.21.15.129:0/3548725729 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f19980cf4e0 con 0x7f19a8102340 2024-05-20T17:58:38.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.458+0000 7f19a77fe700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f19940a2c70 0x7f19940a5180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:58:38.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.462+0000 7f19a57fa700 1 -- 172.21.15.129:0/3548725729 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f199809c3d0 con 0x7f19a8102340 2024-05-20T17:58:38.643 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:38 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:38.335+0000 7fc09a40b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-20T17:58:38.643 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:38 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:38.509+0000 7fc09a40b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-20T17:58:38.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.659+0000 7f19a77fe700 1 -- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f19940a2c70 msgr2=0x7f19940a5180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2870089245 2024-05-20T17:58:38.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.659+0000 7f19a77fe700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f19940a2c70 0x7f19940a5180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-20T17:58:38.679 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:38.348+0000 7f44af1fc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-20T17:58:38.679 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:38.522+0000 7f44af1fc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-20T17:58:38.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.707+0000 7f19ae897700 1 -- 172.21.15.129:0/3548725729 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f19a8199fd0 con 0x7f19a8102340 2024-05-20T17:58:38.705 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:38 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:38.593+0000 7fddf53ad200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-20T17:58:38.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.707+0000 7f19a57fa700 1 -- 172.21.15.129:0/3548725729 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) ==== 53+0+73 (secure 0 0 0) 0x7f199809bcd0 con 0x7f19a8102340 2024-05-20T17:58:38.706 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-20T17:58:38.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.709+0000 7f19ae897700 1 -- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f19940a2c70 msgr2=0x7f19940a5180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:58:38.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.709+0000 7f19ae897700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f19940a2c70 0x7f19940a5180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:38.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.709+0000 7f19ae897700 1 -- 172.21.15.129:0/3548725729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a8102340 msgr2=0x7f19a819e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:38.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.709+0000 7f19ae897700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a8102340 0x7f19a819e8a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f199800e9f0 tx=0x7f199800ed00 comp rx=0 tx=0).stop 2024-05-20T17:58:38.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.710+0000 7f19ae897700 1 -- 172.21.15.129:0/3548725729 shutdown_connections 2024-05-20T17:58:38.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.710+0000 7f19ae897700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f19940a2c70 0x7f19940a5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.710+0000 7f19ae897700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19a81075c0 0x7f19a8198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.710+0000 7f19ae897700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19a8102340 0x7f19a819e8a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.710+0000 7f19ae897700 1 --2- 172.21.15.129:0/3548725729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19a8106c70 0x7f19a819ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.710+0000 7f19ae897700 1 -- 172.21.15.129:0/3548725729 >> 172.21.15.129:0/3548725729 conn(0x7f19a80fdc40 msgr2=0x7f19a8106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.710+0000 7f19ae897700 1 -- 172.21.15.129:0/3548725729 shutdown_connections 2024-05-20T17:58:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:38.710+0000 7f19ae897700 1 -- 172.21.15.129:0/3548725729 wait complete. 2024-05-20T17:58:38.978 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:38 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3548725729' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-20T17:58:38.979 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:38 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:38.684+0000 7fc09a40b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-20T17:58:38.993 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:38 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3548725729' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-20T17:58:38.993 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:38.697+0000 7f44af1fc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-20T17:58:39.075 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:38 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3548725729' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-20T17:58:39.075 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:38 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:38.777+0000 7fddf53ad200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-20T17:58:39.281 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:38 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:38.980+0000 7fc09a40b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-20T17:58:39.281 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:39 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:39.074+0000 7fc09a40b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-20T17:58:39.281 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:39 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:39.282+0000 7fc09a40b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-20T17:58:39.295 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:38.994+0000 7f44af1fc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-20T17:58:39.296 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:39 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:39.089+0000 7f44af1fc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-20T17:58:39.377 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:39 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:39.076+0000 7fddf53ad200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-20T17:58:39.377 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:39 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:39.172+0000 7fddf53ad200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-20T17:58:39.479 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-20T17:58:39.479 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-20T17:58:39.479 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-05-20T17:58:39.480 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-20T17:58:39.480 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-20T17:58:39.480 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-20T17:58:39.480 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-20T17:58:39.480 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-20T17:58:39.480 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-20T17:58:39.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:58:39.480 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-05-20T17:58:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-05-20T17:58:39.498 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-20T17:58:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-05-20T17:58:39.498 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-20T17:58:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-05-20T17:58:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-20T17:58:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42127sec preferred_lft 42127sec 2024-05-20T17:58:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link 2024-05-20T17:58:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-05-20T17:58:39.498 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-20T17:58:39.499 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-20T17:58:39.499 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-05-20T17:58:39.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link add name ceph-brx type bridge 2024-05-20T17:58:39.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip addr flush dev ceph-brx 2024-05-20T17:58:39.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set ceph-brx up 2024-05-20T17:58:39.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-20T17:58:39.499 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-05-20T17:58:39.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-20T17:58:39.642 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:39 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:39.479+0000 7fc09a40b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:58:39.644 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:39 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:39.379+0000 7fddf53ad200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-20T17:58:39.644 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:39 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:39.598+0000 7fddf53ad200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:58:39.679 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:39 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:39.297+0000 7f44af1fc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-20T17:58:39.679 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:39 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:39.495+0000 7f44af1fc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-20T17:58:39.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-20T17:58:39.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:58:39.688 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-20T17:58:39.712 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-05-20T17:58:39.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:58:39.714 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-05-20T17:58:39.781 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-05-20T17:58:39.781 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-20T17:58:39.781 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-20T17:58:39.781 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-20T17:58:39.781 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-20T17:58:39.781 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-20T17:58:39.782 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-05-20T17:58:39.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-20T17:58:39.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-20T17:58:39.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-20T17:58:39.782 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-05-20T17:58:39.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-20T17:58:40.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-20T17:58:40.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:58:40.118 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-05-20T17:58:40.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:58:40.134 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list-id 2024-05-20T17:58:40.190 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-20T17:58:40.190 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-05-20T17:58:40.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-20T17:58:40.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-20T17:58:40.190 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-05-20T17:58:40.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-20T17:58:40.288 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:40 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:40.243+0000 7fddf53ad200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-20T17:58:40.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-20T17:58:40.311 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-20T17:58:40.311 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-20T17:58:40.311 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-05-20T17:58:40.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-20T17:58:40.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-20T17:58:40.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-20T17:58:40.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-20T17:58:40.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-20T17:58:40.311 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-05-20T17:58:40.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-20T17:58:40.392 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:40 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:40.093+0000 7fc09a40b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-20T17:58:40.429 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:40.105+0000 7f44af1fc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-20T17:58:40.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-20T17:58:40.455 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-20T17:58:40.455 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-05-20T17:58:40.455 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-05-20T17:58:40.455 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set dev brx.0 master ceph-brx 2024-05-20T17:58:40.455 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-05-20T17:58:40.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-20T17:58:40.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-20T17:58:40.533 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-20T17:58:40.533 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-20T17:58:40.548 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-20T17:58:40.548 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-20T17:58:40.548 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-20T17:58:40.603 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-20T17:58:40.603 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-20T17:58:40.603 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/dab013c0-2723-47a5-b465-c3539a792f5b 2024-05-20T17:58:40.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:58:40.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/dab013c0-2723-47a5-b465-c3539a792f5b /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-05-20T17:58:40.705 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-05-20T17:58:40.705 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-05-20T17:58:40.705 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-05-20T17:58:40.823 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:40 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:40.521+0000 7fc09a40b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-20T17:58:40.823 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:40 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:40.619+0000 7fc09a40b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-20T17:58:40.839 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:40.535+0000 7f44af1fc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-20T17:58:40.839 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:40.635+0000 7f44af1fc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-20T17:58:40.840 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:40.841+0000 7f44af1fc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-20T17:58:40.842 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-05-20T17:58:40.842 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-05-20T17:58:40.842 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-05-20T17:58:40.843 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.154:3300,172.21.15.204:3300" 2024-05-20T17:58:40.843 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:3300,172.21.15.154:3300,172.21.15.204:3300:/volumes/qa/sv_1/dab013c0-2723-47a5-b465-c3539a792f5b 2024-05-20T17:58:40.843 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=0f1f0dd2-16d1-11ef-bc9a-c7b262605968" will pass to kernel 2024-05-20T17:58:40.843 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-20T17:58:40.843 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-05-20T17:58:40.844 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-20T17:58:40.844 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-05-20T17:58:40.844 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-20T17:58:40.845 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-20T17:58:40.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:58:40.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-20T17:58:40.871 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-20T17:58:40.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-20T17:58:40.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-20T17:58:40.969 INFO:teuthology.orchestra.run.smithi129.stdout:client15240 2024-05-20T17:58:40.969 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T17:58:40.969 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-20T17:58:40.986 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-20T17:58:41.037 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:40 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:40.712+0000 7fddf53ad200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-20T17:58:41.037 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:40 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:40.818+0000 7fddf53ad200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-20T17:58:41.037 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:41 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:41.039+0000 7fddf53ad200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-20T17:58:41.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-20T17:58:41.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-20T17:58:41.084 INFO:teuthology.orchestra.run.smithi129.stdout:client15240 2024-05-20T17:58:41.084 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-20T17:58:41.142 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:40 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:40.824+0000 7fc09a40b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-20T17:58:41.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-20T17:58:41.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-20T17:58:41.183 INFO:teuthology.orchestra.run.smithi129.stdout:client15240 2024-05-20T17:58:41.183 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968.client15240/status') 2024-05-20T17:58:41.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-20T17:58:41.245 INFO:teuthology.orchestra.run.smithi129.stdout:instance: client.15240 (3)192.168.0.1:0/2570498741 2024-05-20T17:58:41.245 INFO:teuthology.orchestra.run.smithi129.stdout:blocklisted: false 2024-05-20T17:58:41.245 INFO:teuthology.orchestra.run.smithi129.stderr:0+1 records in 2024-05-20T17:58:41.245 INFO:teuthology.orchestra.run.smithi129.stderr:0+1 records out 2024-05-20T17:58:41.245 INFO:teuthology.orchestra.run.smithi129.stderr:70 bytes copied, 2.6895e-05 s, 2.6 MB/s 2024-05-20T17:58:41.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-20T17:58:41.252 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-20T17:58:41.263 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-20T17:58:41.303 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:58:41.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.418+0000 7fd318efe640 1 -- 172.21.15.129:0/2411655950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd314104030 msgr2=0x7fd314104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:41.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.418+0000 7fd318efe640 1 --2- 172.21.15.129:0/2411655950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd314104030 0x7fd314104440 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd304009900 tx=0x7fd30402a410 comp rx=0 tx=0).stop 2024-05-20T17:58:41.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.419+0000 7fd318efe640 1 -- 172.21.15.129:0/2411655950 shutdown_connections 2024-05-20T17:58:41.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.419+0000 7fd318efe640 1 --2- 172.21.15.129:0/2411655950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3141006e0 0x7fd314100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.419+0000 7fd318efe640 1 --2- 172.21.15.129:0/2411655950 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3141011e0 0x7fd31410ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.419+0000 7fd318efe640 1 --2- 172.21.15.129:0/2411655950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd314104030 0x7fd314104440 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.419+0000 7fd318efe640 1 -- 172.21.15.129:0/2411655950 >> 172.21.15.129:0/2411655950 conn(0x7fd3140fc2e0 msgr2=0x7fd3140fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:41.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.419+0000 7fd318efe640 1 -- 172.21.15.129:0/2411655950 shutdown_connections 2024-05-20T17:58:41.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.419+0000 7fd318efe640 1 -- 172.21.15.129:0/2411655950 wait complete. 2024-05-20T17:58:41.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.419+0000 7fd318efe640 1 Processor -- start 2024-05-20T17:58:41.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.419+0000 7fd318efe640 1 -- start start 2024-05-20T17:58:41.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd318efe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3141006e0 0x7fd3141992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:41.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd318efe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3141011e0 0x7fd3141997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:41.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd318efe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd314104030 0x7fd3141a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd318efe640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd31410e6d0 con 0x7fd3141011e0 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd318efe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd31410e550 con 0x7fd314104030 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd318efe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd31410e850 con 0x7fd3141006e0 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd311d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3141011e0 0x7fd3141997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd311d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3141011e0 0x7fd3141997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42200/0 (socket says 172.21.15.129:42200) 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd311d9b640 1 -- 172.21.15.129:0/2879238370 learned_addr learned my addr 172.21.15.129:0/2879238370 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd31259c640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3141006e0 0x7fd3141992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd312d9d640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd314104030 0x7fd3141a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd311d9b640 1 -- 172.21.15.129:0/2879238370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3141006e0 msgr2=0x7fd3141992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd311d9b640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3141006e0 0x7fd3141992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd311d9b640 1 -- 172.21.15.129:0/2879238370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd314104030 msgr2=0x7fd3141a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd311d9b640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd314104030 0x7fd3141a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd311d9b640 1 -- 172.21.15.129:0/2879238370 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3141a0df0 con 0x7fd3141011e0 2024-05-20T17:58:41.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd311d9b640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3141011e0 0x7fd3141997f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd30800e9f0 tx=0x7fd30800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:41.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.420+0000 7fd3037fe640 1 -- 172.21.15.129:0/2879238370 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30800cd80 con 0x7fd3141011e0 2024-05-20T17:58:41.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.421+0000 7fd3037fe640 1 -- 172.21.15.129:0/2879238370 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd308010910 con 0x7fd3141011e0 2024-05-20T17:58:41.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.421+0000 7fd318efe640 1 -- 172.21.15.129:0/2879238370 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3141a10e0 con 0x7fd3141011e0 2024-05-20T17:58:41.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.421+0000 7fd3037fe640 1 -- 172.21.15.129:0/2879238370 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd308018990 con 0x7fd3141011e0 2024-05-20T17:58:41.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.421+0000 7fd318efe640 1 -- 172.21.15.129:0/2879238370 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3141a1630 con 0x7fd3141011e0 2024-05-20T17:58:41.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.422+0000 7fd318efe640 1 -- 172.21.15.129:0/2879238370 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3141079c0 con 0x7fd3141011e0 2024-05-20T17:58:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.424+0000 7fd3037fe640 1 -- 172.21.15.129:0/2879238370 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7fd308010ac0 con 0x7fd3141011e0 2024-05-20T17:58:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.424+0000 7fd3037fe640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2fc0a2be0 0x7fd2fc0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.425+0000 7fd31259c640 1 -- 172.21.15.129:0/2879238370 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2fc0a2be0 msgr2=0x7fd2fc0a50f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2870089245 2024-05-20T17:58:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.425+0000 7fd3037fe640 1 -- 172.21.15.129:0/2879238370 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fd3080cf860 con 0x7fd3141011e0 2024-05-20T17:58:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.425+0000 7fd31259c640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2fc0a2be0 0x7fd2fc0a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:58:41.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.427+0000 7fd3037fe640 1 -- 172.21.15.129:0/2879238370 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd30809c100 con 0x7fd3141011e0 2024-05-20T17:58:41.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.609+0000 7fd318efe640 1 -- 172.21.15.129:0/2879238370 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd314062750 con 0x7fd3141011e0 2024-05-20T17:58:41.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.610+0000 7fd3037fe640 1 -- 172.21.15.129:0/2879238370 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22680 (secure 0 0 0) 0x7fd3080a0400 con 0x7fd3141011e0 2024-05-20T17:58:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:41.609 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":88,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:57:49.033961+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:58:41.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.612+0000 7fd318efe640 1 -- 172.21.15.129:0/2879238370 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2fc0a2be0 msgr2=0x7fd2fc0a50f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:58:41.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.612+0000 7fd318efe640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2fc0a2be0 0x7fd2fc0a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.612+0000 7fd318efe640 1 -- 172.21.15.129:0/2879238370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3141011e0 msgr2=0x7fd3141997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:41.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.612+0000 7fd318efe640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3141011e0 0x7fd3141997f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd30800e9f0 tx=0x7fd30800eeb0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.613+0000 7fd318efe640 1 -- 172.21.15.129:0/2879238370 shutdown_connections 2024-05-20T17:58:41.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.613+0000 7fd318efe640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fd2fc0a2be0 0x7fd2fc0a50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.613+0000 7fd318efe640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd314104030 0x7fd3141a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.613+0000 7fd318efe640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3141011e0 0x7fd3141997f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.613+0000 7fd318efe640 1 --2- 172.21.15.129:0/2879238370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd3141006e0 0x7fd3141992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.613+0000 7fd318efe640 1 -- 172.21.15.129:0/2879238370 >> 172.21.15.129:0/2879238370 conn(0x7fd3140fc2e0 msgr2=0x7fd314104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:41.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.613+0000 7fd318efe640 1 -- 172.21.15.129:0/2879238370 shutdown_connections 2024-05-20T17:58:41.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.613+0000 7fd318efe640 1 -- 172.21.15.129:0/2879238370 wait complete. 2024-05-20T17:58:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-20T17:56:42.965073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-20T17:56:44.277499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T17:58:41.624 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-20T17:58:41.643 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:41 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:41.358+0000 7fc09a40b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-20T17:58:41.643 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:41 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:41.453+0000 7fc09a40b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-20T17:58:41.660 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:41.374+0000 7f44af1fc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-20T17:58:41.660 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:41.470+0000 7f44af1fc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-20T17:58:41.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.741+0000 7f68f2c06640 1 -- 172.21.15.129:0/2207129382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68ec1078d0 msgr2=0x7f68ec109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:41.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.741+0000 7f68f2c06640 1 --2- 172.21.15.129:0/2207129382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68ec1078d0 0x7f68ec109cf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f68e4009ca0 tx=0x7f68e402a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.742+0000 7f68f2c06640 1 -- 172.21.15.129:0/2207129382 shutdown_connections 2024-05-20T17:58:41.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.742+0000 7f68f2c06640 1 --2- 172.21.15.129:0/2207129382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68ec1078d0 0x7f68ec109cf0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.742+0000 7f68f2c06640 1 --2- 172.21.15.129:0/2207129382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ec102bd0 0x7f68ec102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.742+0000 7f68f2c06640 1 --2- 172.21.15.129:0/2207129382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ec101990 0x7f68ec102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.742+0000 7f68f2c06640 1 -- 172.21.15.129:0/2207129382 >> 172.21.15.129:0/2207129382 conn(0x7f68ec0fcb80 msgr2=0x7f68ec0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:41.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.742+0000 7f68f2c06640 1 -- 172.21.15.129:0/2207129382 shutdown_connections 2024-05-20T17:58:41.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.742+0000 7f68f2c06640 1 -- 172.21.15.129:0/2207129382 wait complete. 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.742+0000 7f68f2c06640 1 Processor -- start 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.742+0000 7f68f2c06640 1 -- start start 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.742+0000 7f68f2c06640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68ec101990 0x7f68ec0698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.742+0000 7f68f2c06640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ec102bd0 0x7f68ec19e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f2c06640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ec1078d0 0x7f68ec1a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f2c06640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f68ec10d610 con 0x7f68ec101990 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f2c06640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68ec10d490 con 0x7f68ec102bd0 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f2c06640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f68ec10d790 con 0x7f68ec1078d0 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f09a2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68ec101990 0x7f68ec0698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f09a2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68ec101990 0x7f68ec0698b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42212/0 (socket says 172.21.15.129:42212) 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f09a2640 1 -- 172.21.15.129:0/2649479562 learned_addr learned my addr 172.21.15.129:0/2649479562 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f11a3640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ec1078d0 0x7f68ec1a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68ebfff640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ec102bd0 0x7f68ec19e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f09a2640 1 -- 172.21.15.129:0/2649479562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ec1078d0 msgr2=0x7f68ec1a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f09a2640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ec1078d0 0x7f68ec1a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f09a2640 1 -- 172.21.15.129:0/2649479562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ec102bd0 msgr2=0x7f68ec19e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f09a2640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ec102bd0 0x7f68ec19e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f09a2640 1 -- 172.21.15.129:0/2649479562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68ec1a1050 con 0x7f68ec101990 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68ebfff640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ec102bd0 0x7f68ec19e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f09a2640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68ec101990 0x7f68ec0698b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f68e0005fd0 tx=0x7f68e000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68e9ffb640 1 -- 172.21.15.129:0/2649479562 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e0017070 con 0x7f68ec101990 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68e9ffb640 1 -- 172.21.15.129:0/2649479562 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68e0009c30 con 0x7f68ec101990 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68f2c06640 1 -- 172.21.15.129:0/2649479562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68ec1a1340 con 0x7f68ec101990 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.743+0000 7f68e9ffb640 1 -- 172.21.15.129:0/2649479562 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e0003e40 con 0x7f68ec101990 2024-05-20T17:58:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.744+0000 7f68f2c06640 1 -- 172.21.15.129:0/2649479562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68ec1a1890 con 0x7f68ec101990 2024-05-20T17:58:41.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.745+0000 7f68f2c06640 1 -- 172.21.15.129:0/2649479562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68ec1066e0 con 0x7f68ec101990 2024-05-20T17:58:41.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.746+0000 7f68e9ffb640 1 -- 172.21.15.129:0/2649479562 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f68e0004000 con 0x7f68ec101990 2024-05-20T17:58:41.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.746+0000 7f68e9ffb640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68dc0a2be0 0x7f68dc0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:41.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.746+0000 7f68ebfff640 1 -- 172.21.15.129:0/2649479562 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68dc0a2be0 msgr2=0x7f68dc0a50f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2870089245 2024-05-20T17:58:41.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.747+0000 7f68e9ffb640 1 -- 172.21.15.129:0/2649479562 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f68e00ced20 con 0x7f68ec101990 2024-05-20T17:58:41.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.747+0000 7f68ebfff640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68dc0a2be0 0x7f68dc0a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:58:41.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.751+0000 7f68e9ffb640 1 -- 172.21.15.129:0/2649479562 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f68e00d3070 con 0x7f68ec101990 2024-05-20T17:58:41.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:41 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2879238370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:41.912 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:41 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:41.594+0000 7fddf53ad200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-20T17:58:41.912 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:41 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:41.698+0000 7fddf53ad200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-20T17:58:41.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:41 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2879238370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:41.929 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:41.723+0000 7f44af1fc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-20T17:58:41.929 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:41.862+0000 7f44af1fc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-20T17:58:41.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.931+0000 7f68f2c06640 1 -- 172.21.15.129:0/2649479562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f68ec06b2a0 con 0x7f68ec101990 2024-05-20T17:58:41.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.931+0000 7f68e9ffb640 1 -- 172.21.15.129:0/2649479562 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f68e00bb070 con 0x7f68ec101990 2024-05-20T17:58:41.930 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T17:58:41.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:41 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2879238370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:41.931 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:41 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:41.702+0000 7fc09a40b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-20T17:58:41.931 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:41 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:41.836+0000 7fc09a40b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-20T17:58:41.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 -- 172.21.15.129:0/2649479562 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68dc0a2be0 msgr2=0x7f68dc0a50f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:58:41.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68dc0a2be0 0x7f68dc0a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 -- 172.21.15.129:0/2649479562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68ec101990 msgr2=0x7f68ec0698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:41.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68ec101990 0x7f68ec0698b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f68e0005fd0 tx=0x7f68e000dde0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 -- 172.21.15.129:0/2649479562 shutdown_connections 2024-05-20T17:58:41.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f68dc0a2be0 0x7f68dc0a50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ec102bd0 0x7f68ec19e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ec1078d0 0x7f68ec1a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 --2- 172.21.15.129:0/2649479562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68ec101990 0x7f68ec0698b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:41.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 -- 172.21.15.129:0/2649479562 >> 172.21.15.129:0/2649479562 conn(0x7f68ec0fcb80 msgr2=0x7f68ec1038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:41.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 -- 172.21.15.129:0/2649479562 shutdown_connections 2024-05-20T17:58:41.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:41.934+0000 7f68f2c06640 1 -- 172.21.15.129:0/2649479562 wait complete. 2024-05-20T17:58:41.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-20T17:58:42.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.061+0000 7fa5a6565640 1 -- 172.21.15.129:0/220157876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5a0102a90 msgr2=0x7fa5a0103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.061+0000 7fa5a6565640 1 --2- 172.21.15.129:0/220157876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5a0102a90 0x7fa5a0103790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa590009860 tx=0x7fa59002a340 comp rx=0 tx=0).stop 2024-05-20T17:58:42.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.061+0000 7fa5a6565640 1 -- 172.21.15.129:0/220157876 shutdown_connections 2024-05-20T17:58:42.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.061+0000 7fa5a6565640 1 --2- 172.21.15.129:0/220157876 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a01089d0 0x7fa5a010adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.061+0000 7fa5a6565640 1 --2- 172.21.15.129:0/220157876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0103cd0 0x7fa5a01040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.061+0000 7fa5a6565640 1 --2- 172.21.15.129:0/220157876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5a0102a90 0x7fa5a0103790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.061+0000 7fa5a6565640 1 -- 172.21.15.129:0/220157876 >> 172.21.15.129:0/220157876 conn(0x7fa5a00fdc80 msgr2=0x7fa5a0100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:42.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.061+0000 7fa5a6565640 1 -- 172.21.15.129:0/220157876 shutdown_connections 2024-05-20T17:58:42.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.061+0000 7fa5a6565640 1 -- 172.21.15.129:0/220157876 wait complete. 2024-05-20T17:58:42.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa5a6565640 1 Processor -- start 2024-05-20T17:58:42.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa5a6565640 1 -- start start 2024-05-20T17:58:42.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa5a6565640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0102a90 0x7fa5a01992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa5a6565640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a0103cd0 0x7fa5a01997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa5a6565640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5a01089d0 0x7fa5a01a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa5a6565640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a010e6e0 con 0x7fa5a0102a90 2024-05-20T17:58:42.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa5a6565640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a010e560 con 0x7fa5a01089d0 2024-05-20T17:58:42.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa5a6565640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a010e860 con 0x7fa5a0103cd0 2024-05-20T17:58:42.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa59ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0102a90 0x7fa5a01992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:42.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa59ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0102a90 0x7fa5a01992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42220/0 (socket says 172.21.15.129:42220) 2024-05-20T17:58:42.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa59ffff640 1 -- 172.21.15.129:0/2684955144 learned_addr learned my addr 172.21.15.129:0/2684955144 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:42.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa59f7fe640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a0103cd0 0x7fa5a01997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.062+0000 7fa5a4b02640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5a01089d0 0x7fa5a01a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa59f7fe640 1 -- 172.21.15.129:0/2684955144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5a01089d0 msgr2=0x7fa5a01a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa59f7fe640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5a01089d0 0x7fa5a01a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa59f7fe640 1 -- 172.21.15.129:0/2684955144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0102a90 msgr2=0x7fa5a01992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa59f7fe640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0102a90 0x7fa5a01992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa59f7fe640 1 -- 172.21.15.129:0/2684955144 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a01a0df0 con 0x7fa5a0103cd0 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa59ffff640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0102a90 0x7fa5a01992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa59f7fe640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a0103cd0 0x7fa5a01997f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa594005fd0 tx=0x7fa59400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa59d7fa640 1 -- 172.21.15.129:0/2684955144 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59400cd00 con 0x7fa5a0103cd0 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa5a6565640 1 -- 172.21.15.129:0/2684955144 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a01a10e0 con 0x7fa5a0103cd0 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa59d7fa640 1 -- 172.21.15.129:0/2684955144 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5940035f0 con 0x7fa5a0103cd0 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa59d7fa640 1 -- 172.21.15.129:0/2684955144 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594010670 con 0x7fa5a0103cd0 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.063+0000 7fa5a6565640 1 -- 172.21.15.129:0/2684955144 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a01a1680 con 0x7fa5a0103cd0 2024-05-20T17:58:42.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.064+0000 7fa5a6565640 1 -- 172.21.15.129:0/2684955144 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a0068290 con 0x7fa5a0103cd0 2024-05-20T17:58:42.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.065+0000 7fa59d7fa640 1 -- 172.21.15.129:0/2684955144 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7fa594004020 con 0x7fa5a0103cd0 2024-05-20T17:58:42.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.066+0000 7fa59d7fa640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa5880a2be0 0x7fa5880a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.066+0000 7fa59ffff640 1 -- 172.21.15.129:0/2684955144 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa5880a2be0 msgr2=0x7fa5880a50f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2870089245 2024-05-20T17:58:42.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.066+0000 7fa59d7fa640 1 -- 172.21.15.129:0/2684955144 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fa5940cf290 con 0x7fa5a0103cd0 2024-05-20T17:58:42.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.066+0000 7fa59ffff640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa5880a2be0 0x7fa5880a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:58:42.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.071+0000 7fa59d7fa640 1 -- 172.21.15.129:0/2684955144 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa594009510 con 0x7fa5a0103cd0 2024-05-20T17:58:42.213 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:41 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:41.965+0000 7fddf53ad200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-20T17:58:42.213 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:42 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:42.113+0000 7fddf53ad200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-20T17:58:42.214 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:42 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:42.215+0000 7fddf53ad200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-20T17:58:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.249+0000 7fa5a6565640 1 -- 172.21.15.129:0/2684955144 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa5a0062750 con 0x7fa5a0103cd0 2024-05-20T17:58:42.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.250+0000 7fa59d7fa640 1 -- 172.21.15.129:0/2684955144 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fa59409c1e0 con 0x7fa5a0103cd0 2024-05-20T17:58:42.249 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-05-20T17:58:42.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 -- 172.21.15.129:0/2684955144 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa5880a2be0 msgr2=0x7fa5880a50f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:58:42.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa5880a2be0 0x7fa5880a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 -- 172.21.15.129:0/2684955144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a0103cd0 msgr2=0x7fa5a01997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a0103cd0 0x7fa5a01997f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa594005fd0 tx=0x7fa59400ee50 comp rx=0 tx=0).stop 2024-05-20T17:58:42.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 -- 172.21.15.129:0/2684955144 shutdown_connections 2024-05-20T17:58:42.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa5880a2be0 0x7fa5880a50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5a01089d0 0x7fa5a01a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5a0103cd0 0x7fa5a01997f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 --2- 172.21.15.129:0/2684955144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5a0102a90 0x7fa5a01992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 -- 172.21.15.129:0/2684955144 >> 172.21.15.129:0/2684955144 conn(0x7fa5a00fdc80 msgr2=0x7fa5a01049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:42.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 -- 172.21.15.129:0/2684955144 shutdown_connections 2024-05-20T17:58:42.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.252+0000 7fa5a6565640 1 -- 172.21.15.129:0/2684955144 wait complete. 2024-05-20T17:58:42.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-20T17:58:42.354 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:41 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:41.933+0000 7fc09a40b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-20T17:58:42.355 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:42 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:42.051+0000 7fc09a40b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-20T17:58:42.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.380+0000 7f231def7640 1 -- 172.21.15.129:0/3171042890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2318101a00 msgr2=0x7f2318102700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.380+0000 7f231def7640 1 --2- 172.21.15.129:0/3171042890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2318101a00 0x7f2318102700 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2308009900 tx=0x7f230802a410 comp rx=0 tx=0).stop 2024-05-20T17:58:42.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.380+0000 7f231def7640 1 -- 172.21.15.129:0/3171042890 shutdown_connections 2024-05-20T17:58:42.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.380+0000 7f231def7640 1 --2- 172.21.15.129:0/3171042890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2318102c40 0x7f2318103050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.380+0000 7f231def7640 1 --2- 172.21.15.129:0/3171042890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2318107940 0x7f2318109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.380+0000 7f231def7640 1 --2- 172.21.15.129:0/3171042890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2318101a00 0x7f2318102700 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.380+0000 7f231def7640 1 -- 172.21.15.129:0/3171042890 >> 172.21.15.129:0/3171042890 conn(0x7f23180fcbf0 msgr2=0x7f23180ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:42.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.381+0000 7f231def7640 1 -- 172.21.15.129:0/3171042890 shutdown_connections 2024-05-20T17:58:42.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.381+0000 7f231def7640 1 -- 172.21.15.129:0/3171042890 wait complete. 2024-05-20T17:58:42.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.381+0000 7f231def7640 1 Processor -- start 2024-05-20T17:58:42.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.381+0000 7f231def7640 1 -- start start 2024-05-20T17:58:42.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.381+0000 7f231def7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2318101a00 0x7f2318074810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.381+0000 7f231def7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2318102c40 0x7f231819e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.381+0000 7f231def7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2318107940 0x7f23181a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.381+0000 7f231def7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f231810d5e0 con 0x7f2318107940 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.381+0000 7f231def7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f231810d460 con 0x7f2318101a00 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.381+0000 7f231def7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f231810d760 con 0x7f2318102c40 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23177fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2318101a00 0x7f2318074810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f230ffff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2318102c40 0x7f231819e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f2317fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2318107940 0x7f23181a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23177fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2318101a00 0x7f2318074810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:53782/0 (socket says 172.21.15.129:53782) 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f230ffff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2318102c40 0x7f231819e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:51620/0 (socket says 172.21.15.129:51620) 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23177fe640 1 -- 172.21.15.129:0/492402220 learned_addr learned my addr 172.21.15.129:0/492402220 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23177fe640 1 -- 172.21.15.129:0/492402220 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2318102c40 msgr2=0x7f231819e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23177fe640 1 --2- 172.21.15.129:0/492402220 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2318102c40 0x7f231819e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23177fe640 1 -- 172.21.15.129:0/492402220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2318107940 msgr2=0x7f23181a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23177fe640 1 --2- 172.21.15.129:0/492402220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2318107940 0x7f23181a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23177fe640 1 -- 172.21.15.129:0/492402220 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23181a0e00 con 0x7f2318101a00 2024-05-20T17:58:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23177fe640 1 --2- 172.21.15.129:0/492402220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2318101a00 0x7f2318074810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2308005ab0 tx=0x7f2308004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23157fa640 1 -- 172.21.15.129:0/492402220 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230803f030 con 0x7f2318101a00 2024-05-20T17:58:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f231def7640 1 -- 172.21.15.129:0/492402220 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23181a1090 con 0x7f2318101a00 2024-05-20T17:58:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23157fa640 1 -- 172.21.15.129:0/492402220 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2308003ad0 con 0x7f2318101a00 2024-05-20T17:58:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f23157fa640 1 -- 172.21.15.129:0/492402220 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308035070 con 0x7f2318101a00 2024-05-20T17:58:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.382+0000 7f231def7640 1 -- 172.21.15.129:0/492402220 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23181a1580 con 0x7f2318101a00 2024-05-20T17:58:42.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.383+0000 7f231def7640 1 -- 172.21.15.129:0/492402220 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2318106750 con 0x7f2318101a00 2024-05-20T17:58:42.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.385+0000 7f23157fa640 1 -- 172.21.15.129:0/492402220 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f2308042030 con 0x7f2318101a00 2024-05-20T17:58:42.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.385+0000 7f23157fa640 1 --2- 172.21.15.129:0/492402220 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f23040a2be0 0x7f23040a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.385+0000 7f230ffff640 1 -- 172.21.15.129:0/492402220 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f23040a2be0 msgr2=0x7f23040a50f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2870089245 2024-05-20T17:58:42.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.385+0000 7f23157fa640 1 -- 172.21.15.129:0/492402220 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f23080ef8f0 con 0x7f2318101a00 2024-05-20T17:58:42.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.385+0000 7f230ffff640 1 --2- 172.21.15.129:0/492402220 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f23040a2be0 0x7f23040a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:58:42.386 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:41.961+0000 7f44af1fc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-20T17:58:42.387 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:42.082+0000 7f44af1fc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-20T17:58:42.387 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:42.388+0000 7f44af1fc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-20T17:58:42.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.389+0000 7f23157fa640 1 -- 172.21.15.129:0/492402220 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f23080bc1d0 con 0x7f2318101a00 2024-05-20T17:58:42.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.571+0000 7f231def7640 1 -- 172.21.15.129:0/492402220 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f231802d430 con 0x7f2318101a00 2024-05-20T17:58:42.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.572+0000 7f23157fa640 1 -- 172.21.15.129:0/492402220 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f23080c04d0 con 0x7f2318101a00 2024-05-20T17:58:42.571 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-05-20T17:58:42.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 -- 172.21.15.129:0/492402220 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f23040a2be0 msgr2=0x7f23040a50f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:58:42.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 --2- 172.21.15.129:0/492402220 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f23040a2be0 0x7f23040a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 -- 172.21.15.129:0/492402220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2318101a00 msgr2=0x7f2318074810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 --2- 172.21.15.129:0/492402220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2318101a00 0x7f2318074810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2308005ab0 tx=0x7f2308004270 comp rx=0 tx=0).stop 2024-05-20T17:58:42.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 -- 172.21.15.129:0/492402220 shutdown_connections 2024-05-20T17:58:42.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 --2- 172.21.15.129:0/492402220 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f23040a2be0 0x7f23040a50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 --2- 172.21.15.129:0/492402220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2318107940 0x7f23181a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 --2- 172.21.15.129:0/492402220 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2318102c40 0x7f231819e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 --2- 172.21.15.129:0/492402220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2318101a00 0x7f2318074810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 -- 172.21.15.129:0/492402220 >> 172.21.15.129:0/492402220 conn(0x7f23180fcbf0 msgr2=0x7f2318103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:42.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 -- 172.21.15.129:0/492402220 shutdown_connections 2024-05-20T17:58:42.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.574+0000 7f231def7640 1 -- 172.21.15.129:0/492402220 wait complete. 2024-05-20T17:58:42.583 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:42 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:42.343+0000 7fddf53ad200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-20T17:58:42.584 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:58:42.642 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:42 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:42.356+0000 7fc09a40b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-20T17:58:42.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.700+0000 7f7085bd5640 1 -- 172.21.15.129:0/3109904082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7080107050 msgr2=0x7f7080109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.700+0000 7f7085bd5640 1 --2- 172.21.15.129:0/3109904082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7080107050 0x7f7080109470 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7078009ca0 tx=0x7f707802a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.700+0000 7f7085bd5640 1 -- 172.21.15.129:0/3109904082 shutdown_connections 2024-05-20T17:58:42.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.700+0000 7f7085bd5640 1 --2- 172.21.15.129:0/3109904082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7080107050 0x7f7080109470 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.700+0000 7f7085bd5640 1 --2- 172.21.15.129:0/3109904082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7080102350 0x7f7080102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.700+0000 7f7085bd5640 1 --2- 172.21.15.129:0/3109904082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7080101110 0x7f7080101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.700+0000 7f7085bd5640 1 -- 172.21.15.129:0/3109904082 >> 172.21.15.129:0/3109904082 conn(0x7f70800fc2e0 msgr2=0x7f70800fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:42.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.700+0000 7f7085bd5640 1 -- 172.21.15.129:0/3109904082 shutdown_connections 2024-05-20T17:58:42.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.700+0000 7f7085bd5640 1 -- 172.21.15.129:0/3109904082 wait complete. 2024-05-20T17:58:42.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f7085bd5640 1 Processor -- start 2024-05-20T17:58:42.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f7085bd5640 1 -- start start 2024-05-20T17:58:42.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f7085bd5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7080101110 0x7f70801992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f7085bd5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7080102350 0x7f70801997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f7085bd5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7080107050 0x7f70801a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f7085bd5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f708010c4f0 con 0x7f7080107050 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f7085bd5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f708010c370 con 0x7f7080102350 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f7085bd5640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f708010c670 con 0x7f7080101110 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f707ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7080107050 0x7f70801a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f707ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7080107050 0x7f70801a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42248/0 (socket says 172.21.15.129:42248) 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f707f7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7080101110 0x7f70801992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f707ffff640 1 -- 172.21.15.129:0/685489274 learned_addr learned my addr 172.21.15.129:0/685489274 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.701+0000 7f707effd640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7080102350 0x7f70801997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f707f7fe640 1 -- 172.21.15.129:0/685489274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7080102350 msgr2=0x7f70801997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f707f7fe640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7080102350 0x7f70801997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f707f7fe640 1 -- 172.21.15.129:0/685489274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7080107050 msgr2=0x7f70801a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f707f7fe640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7080107050 0x7f70801a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f707f7fe640 1 -- 172.21.15.129:0/685489274 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70801a0de0 con 0x7f7080101110 2024-05-20T17:58:42.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f707ffff640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7080107050 0x7f70801a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:42.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f707f7fe640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7080101110 0x7f70801992a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f70801009e0 tx=0x7f707400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:42.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f707cff9640 1 -- 172.21.15.129:0/685489274 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707400cd00 con 0x7f7080101110 2024-05-20T17:58:42.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f707cff9640 1 -- 172.21.15.129:0/685489274 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7074010910 con 0x7f7080101110 2024-05-20T17:58:42.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f7085bd5640 1 -- 172.21.15.129:0/685489274 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70801a10d0 con 0x7f7080101110 2024-05-20T17:58:42.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f707cff9640 1 -- 172.21.15.129:0/685489274 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70740188b0 con 0x7f7080101110 2024-05-20T17:58:42.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.702+0000 7f7085bd5640 1 -- 172.21.15.129:0/685489274 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70801a1620 con 0x7f7080101110 2024-05-20T17:58:42.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.703+0000 7f7085bd5640 1 -- 172.21.15.129:0/685489274 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70801a1880 con 0x7f7080101110 2024-05-20T17:58:42.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.704+0000 7f707cff9640 1 -- 172.21.15.129:0/685489274 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f707401f020 con 0x7f7080101110 2024-05-20T17:58:42.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.705+0000 7f707cff9640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f70680a2be0 0x7f70680a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:42.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.705+0000 7f707effd640 1 -- 172.21.15.129:0/685489274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f70680a2be0 msgr2=0x7f70680a50f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2870089245 2024-05-20T17:58:42.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.705+0000 7f707cff9640 1 -- 172.21.15.129:0/685489274 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f70740cf1e0 con 0x7f7080101110 2024-05-20T17:58:42.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.705+0000 7f707effd640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f70680a2be0 0x7f70680a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:58:42.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.709+0000 7f707cff9640 1 -- 172.21.15.129:0/685489274 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f707409c130 con 0x7f7080101110 2024-05-20T17:58:42.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.889+0000 7f7085bd5640 1 -- 172.21.15.129:0/685489274 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70801a1a50 con 0x7f7080101110 2024-05-20T17:58:42.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.891+0000 7f707cff9640 1 -- 172.21.15.129:0/685489274 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22680 (secure 0 0 0) 0x7f707409ba30 con 0x7f7080101110 2024-05-20T17:58:42.890 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":88,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:57:49.033961+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:58:42.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.893+0000 7f7085bd5640 1 -- 172.21.15.129:0/685489274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f70680a2be0 msgr2=0x7f70680a50f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:58:42.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.893+0000 7f7085bd5640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f70680a2be0 0x7f70680a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.893+0000 7f7085bd5640 1 -- 172.21.15.129:0/685489274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7080101110 msgr2=0x7f70801992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:42.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.894+0000 7f7085bd5640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7080101110 0x7f70801992a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f70801009e0 tx=0x7f707400ee50 comp rx=0 tx=0).stop 2024-05-20T17:58:42.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.894+0000 7f7085bd5640 1 -- 172.21.15.129:0/685489274 shutdown_connections 2024-05-20T17:58:42.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.894+0000 7f7085bd5640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f70680a2be0 0x7f70680a50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.894+0000 7f7085bd5640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7080102350 0x7f70801997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.894+0000 7f7085bd5640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7080107050 0x7f70801a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.894+0000 7f7085bd5640 1 --2- 172.21.15.129:0/685489274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7080101110 0x7f70801992a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:42.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.894+0000 7f7085bd5640 1 -- 172.21.15.129:0/685489274 >> 172.21.15.129:0/685489274 conn(0x7f70800fc2e0 msgr2=0x7f7080105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:42.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.894+0000 7f7085bd5640 1 -- 172.21.15.129:0/685489274 shutdown_connections 2024-05-20T17:58:42.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:42.894+0000 7f7085bd5640 1 -- 172.21.15.129:0/685489274 wait complete. 2024-05-20T17:58:42.903 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:42 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2649479562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:42.903 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:42 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2684955144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:42.903 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:42 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/492402220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:42.903 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:42 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:42.661+0000 7fddf53ad200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-20T17:58:42.904 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-20T17:56:42.965073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-20T17:56:44.277499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T17:58:42.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-20T17:58:42.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:42 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2649479562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:42.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:42 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2684955144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:42.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:42 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/492402220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:42.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:42 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2649479562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:42.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:42 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2684955144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:42.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:42 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/492402220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:43.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.022+0000 7f5bf94fe640 1 -- 172.21.15.129:0/706273542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf41019b0 msgr2=0x7f5bf41026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.022+0000 7f5bf94fe640 1 --2- 172.21.15.129:0/706273542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf41019b0 0x7f5bf41026b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5be4009900 tx=0x7f5be402a410 comp rx=0 tx=0).stop 2024-05-20T17:58:43.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.022+0000 7f5bf94fe640 1 -- 172.21.15.129:0/706273542 shutdown_connections 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.022+0000 7f5bf94fe640 1 --2- 172.21.15.129:0/706273542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bf41078f0 0x7f5bf4109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.022+0000 7f5bf94fe640 1 --2- 172.21.15.129:0/706273542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf4102bf0 0x7f5bf4103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.022+0000 7f5bf94fe640 1 --2- 172.21.15.129:0/706273542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf41019b0 0x7f5bf41026b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.022+0000 7f5bf94fe640 1 -- 172.21.15.129:0/706273542 >> 172.21.15.129:0/706273542 conn(0x7f5bf40fcba0 msgr2=0x7f5bf40ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.022+0000 7f5bf94fe640 1 -- 172.21.15.129:0/706273542 shutdown_connections 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.022+0000 7f5bf94fe640 1 -- 172.21.15.129:0/706273542 wait complete. 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf94fe640 1 Processor -- start 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf94fe640 1 -- start start 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf94fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf41019b0 0x7f5bf4198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf94fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bf4102bf0 0x7f5bf4198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf94fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf41078f0 0x7f5bf419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf94fe640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf410d5e0 con 0x7f5bf41019b0 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf94fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf410d460 con 0x7f5bf4102bf0 2024-05-20T17:58:43.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf94fe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf410d760 con 0x7f5bf41078f0 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf41019b0 0x7f5bf4198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf41019b0 0x7f5bf4198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42268/0 (socket says 172.21.15.129:42268) 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf2ffd640 1 -- 172.21.15.129:0/940314689 learned_addr learned my addr 172.21.15.129:0/940314689 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bebfff640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bf4102bf0 0x7f5bf4198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf37fe640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf41078f0 0x7f5bf419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf2ffd640 1 -- 172.21.15.129:0/940314689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf41078f0 msgr2=0x7f5bf419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf2ffd640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf41078f0 0x7f5bf419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf2ffd640 1 -- 172.21.15.129:0/940314689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bf4102bf0 msgr2=0x7f5bf4198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf2ffd640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bf4102bf0 0x7f5bf4198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.023+0000 7f5bf2ffd640 1 -- 172.21.15.129:0/940314689 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf419fd50 con 0x7f5bf41019b0 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.024+0000 7f5bf2ffd640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf41019b0 0x7f5bf4198210 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5be4006010 tx=0x7f5be40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.024+0000 7f5bf0ff9640 1 -- 172.21.15.129:0/940314689 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be403f030 con 0x7f5bf41019b0 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.024+0000 7f5bf0ff9640 1 -- 172.21.15.129:0/940314689 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5be4031a00 con 0x7f5bf41019b0 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.024+0000 7f5bf94fe640 1 -- 172.21.15.129:0/940314689 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf419ffe0 con 0x7f5bf41019b0 2024-05-20T17:58:43.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.024+0000 7f5bf0ff9640 1 -- 172.21.15.129:0/940314689 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be4035070 con 0x7f5bf41019b0 2024-05-20T17:58:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.024+0000 7f5bf94fe640 1 -- 172.21.15.129:0/940314689 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf41a03e0 con 0x7f5bf41019b0 2024-05-20T17:58:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.025+0000 7f5bf94fe640 1 -- 172.21.15.129:0/940314689 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bf4106700 con 0x7f5bf41019b0 2024-05-20T17:58:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.026+0000 7f5bf0ff9640 1 -- 172.21.15.129:0/940314689 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f5be4042030 con 0x7f5bf41019b0 2024-05-20T17:58:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.027+0000 7f5bf0ff9640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5be00a2be0 0x7f5be00a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.027+0000 7f5bebfff640 1 -- 172.21.15.129:0/940314689 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5be00a2be0 msgr2=0x7f5be00a50f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2870089245 2024-05-20T17:58:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.028+0000 7f5bf0ff9640 1 -- 172.21.15.129:0/940314689 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7f5be40ef5f0 con 0x7f5bf41019b0 2024-05-20T17:58:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.028+0000 7f5bebfff640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5be00a2be0 0x7f5be00a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:58:43.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.031+0000 7f5bf0ff9640 1 -- 172.21.15.129:0/940314689 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5be4005dc0 con 0x7f5bf41019b0 2024-05-20T17:58:43.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.213+0000 7f5bf94fe640 1 -- 172.21.15.129:0/940314689 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5bf4062780 con 0x7f5bf41019b0 2024-05-20T17:58:43.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.213+0000 7f5bf0ff9640 1 -- 172.21.15.129:0/940314689 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f5be40bbed0 con 0x7f5bf41019b0 2024-05-20T17:58:43.212 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T17:58:43.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 -- 172.21.15.129:0/940314689 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5be00a2be0 msgr2=0x7f5be00a50f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:58:43.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5be00a2be0 0x7f5be00a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 -- 172.21.15.129:0/940314689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf41019b0 msgr2=0x7f5bf4198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf41019b0 0x7f5bf4198210 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5be4006010 tx=0x7f5be40042f0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 -- 172.21.15.129:0/940314689 shutdown_connections 2024-05-20T17:58:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7f5be00a2be0 0x7f5be00a50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bf4102bf0 0x7f5bf4198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf41078f0 0x7f5bf419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 --2- 172.21.15.129:0/940314689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf41019b0 0x7f5bf4198210 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 -- 172.21.15.129:0/940314689 >> 172.21.15.129:0/940314689 conn(0x7f5bf40fcba0 msgr2=0x7f5bf41038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 -- 172.21.15.129:0/940314689 shutdown_connections 2024-05-20T17:58:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.216+0000 7f5bf94fe640 1 -- 172.21.15.129:0/940314689 wait complete. 2024-05-20T17:58:43.226 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-20T17:58:43.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.343+0000 7fa519510640 1 -- 172.21.15.129:0/1672913952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa514101990 msgr2=0x7fa514102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.343+0000 7fa519510640 1 --2- 172.21.15.129:0/1672913952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa514101990 0x7fa514102690 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa504009900 tx=0x7fa50402a410 comp rx=0 tx=0).stop 2024-05-20T17:58:43.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.344+0000 7fa519510640 1 -- 172.21.15.129:0/1672913952 shutdown_connections 2024-05-20T17:58:43.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.344+0000 7fa519510640 1 --2- 172.21.15.129:0/1672913952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5141078d0 0x7fa514109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.344+0000 7fa519510640 1 --2- 172.21.15.129:0/1672913952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa514102bd0 0x7fa514102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.344+0000 7fa519510640 1 --2- 172.21.15.129:0/1672913952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa514101990 0x7fa514102690 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.344+0000 7fa519510640 1 -- 172.21.15.129:0/1672913952 >> 172.21.15.129:0/1672913952 conn(0x7fa5140fcb80 msgr2=0x7fa5140ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:43.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.344+0000 7fa519510640 1 -- 172.21.15.129:0/1672913952 shutdown_connections 2024-05-20T17:58:43.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.344+0000 7fa519510640 1 -- 172.21.15.129:0/1672913952 wait complete. 2024-05-20T17:58:43.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.344+0000 7fa519510640 1 Processor -- start 2024-05-20T17:58:43.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.344+0000 7fa519510640 1 -- start start 2024-05-20T17:58:43.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa519510640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa514101990 0x7fa514199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:43.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa519510640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa514102bd0 0x7fa5141998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:43.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa519510640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5141078d0 0x7fa5141a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:43.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa519510640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa51410d5e0 con 0x7fa514101990 2024-05-20T17:58:43.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa519510640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa51410d460 con 0x7fa5141078d0 2024-05-20T17:58:43.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa519510640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa51410d760 con 0x7fa514102bd0 2024-05-20T17:58:43.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa512ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa514101990 0x7fa514199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:43.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa512ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa514101990 0x7fa514199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42286/0 (socket says 172.21.15.129:42286) 2024-05-20T17:58:43.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa512ffd640 1 -- 172.21.15.129:0/272841763 learned_addr learned my addr 172.21.15.129:0/272841763 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:43.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa5127fc640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa514102bd0 0x7fa5141998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa5137fe640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5141078d0 0x7fa5141a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa512ffd640 1 -- 172.21.15.129:0/272841763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa514102bd0 msgr2=0x7fa5141998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa512ffd640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa514102bd0 0x7fa5141998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa512ffd640 1 -- 172.21.15.129:0/272841763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5141078d0 msgr2=0x7fa5141a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa512ffd640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5141078d0 0x7fa5141a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa512ffd640 1 -- 172.21.15.129:0/272841763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5141a0e20 con 0x7fa514101990 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa512ffd640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa514101990 0x7fa514199390 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa504006010 tx=0x7fa5040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa4fbfff640 1 -- 172.21.15.129:0/272841763 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50403f030 con 0x7fa514101990 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.345+0000 7fa519510640 1 -- 172.21.15.129:0/272841763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5141a10b0 con 0x7fa514101990 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.346+0000 7fa519510640 1 -- 172.21.15.129:0/272841763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5141a15f0 con 0x7fa514101990 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.346+0000 7fa4fbfff640 1 -- 172.21.15.129:0/272841763 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa504031a20 con 0x7fa514101990 2024-05-20T17:58:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.346+0000 7fa4fbfff640 1 -- 172.21.15.129:0/272841763 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa504049060 con 0x7fa514101990 2024-05-20T17:58:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.347+0000 7fa519510640 1 -- 172.21.15.129:0/272841763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5141a1870 con 0x7fa514101990 2024-05-20T17:58:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.348+0000 7fa4fbfff640 1 -- 172.21.15.129:0/272841763 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7fa504042030 con 0x7fa514101990 2024-05-20T17:58:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.349+0000 7fa4fbfff640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa4fc0a2be0 0x7fa4fc0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.349+0000 7fa5127fc640 1 -- 172.21.15.129:0/272841763 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa4fc0a2be0 msgr2=0x7fa4fc0a50f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2870089245 2024-05-20T17:58:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.349+0000 7fa4fbfff640 1 -- 172.21.15.129:0/272841763 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8248+0+0 (secure 0 0 0) 0x7fa5040ef160 con 0x7fa514101990 2024-05-20T17:58:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.349+0000 7fa5127fc640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa4fc0a2be0 0x7fa4fc0a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-20T17:58:43.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.358+0000 7fa4fbfff640 1 -- 172.21.15.129:0/272841763 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa5040bba60 con 0x7fa514101990 2024-05-20T17:58:43.393 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:42 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:42.967+0000 7fc09a40b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-20T17:58:43.393 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 17:58:43 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y[37853]: 2024-05-20T17:58:43.188+0000 7fc09a40b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-20T17:58:43.429 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:43 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:43.002+0000 7f44af1fc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-20T17:58:43.429 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 17:58:43 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z[38902]: 2024-05-20T17:58:43.226+0000 7f44af1fc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-20T17:58:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.542+0000 7fa519510640 1 -- 172.21.15.129:0/272841763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa5141a1a40 con 0x7fa514101990 2024-05-20T17:58:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.542+0000 7fa4fbfff640 1 -- 172.21.15.129:0/272841763 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fa5040bfd60 con 0x7fa514101990 2024-05-20T17:58:43.540 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-05-20T17:58:43.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.544+0000 7fa519510640 1 -- 172.21.15.129:0/272841763 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa4fc0a2be0 msgr2=0x7fa4fc0a50f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T17:58:43.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.544+0000 7fa519510640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa4fc0a2be0 0x7fa4fc0a50f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.544+0000 7fa519510640 1 -- 172.21.15.129:0/272841763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa514101990 msgr2=0x7fa514199390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.544+0000 7fa519510640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa514101990 0x7fa514199390 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa504006010 tx=0x7fa5040042f0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.545+0000 7fa519510640 1 -- 172.21.15.129:0/272841763 shutdown_connections 2024-05-20T17:58:43.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.545+0000 7fa519510640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.129:6800/2870089245,v1:172.21.15.129:6801/2870089245] conn(0x7fa4fc0a2be0 0x7fa4fc0a50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.545+0000 7fa519510640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5141078d0 0x7fa5141a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.545+0000 7fa519510640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa514102bd0 0x7fa5141998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.545+0000 7fa519510640 1 --2- 172.21.15.129:0/272841763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa514101990 0x7fa514199390 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.545+0000 7fa519510640 1 -- 172.21.15.129:0/272841763 >> 172.21.15.129:0/272841763 conn(0x7fa5140fcb80 msgr2=0x7fa514108e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:43.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.545+0000 7fa519510640 1 -- 172.21.15.129:0/272841763 shutdown_connections 2024-05-20T17:58:43.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.545+0000 7fa519510640 1 -- 172.21.15.129:0/272841763 wait complete. 2024-05-20T17:58:43.552 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.317+0000 7fddf53ad200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-20T17:58:43.554 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-20T17:58:43.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.686+0000 7f96d16f0640 1 -- 172.21.15.129:0/2661352077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc101990 msgr2=0x7f96cc102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.686+0000 7f96d16f0640 1 --2- 172.21.15.129:0/2661352077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc101990 0x7f96cc102690 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009900 tx=0x7f96bc02a410 comp rx=0 tx=0).stop 2024-05-20T17:58:43.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.686+0000 7f96d16f0640 1 -- 172.21.15.129:0/2661352077 shutdown_connections 2024-05-20T17:58:43.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.686+0000 7f96d16f0640 1 --2- 172.21.15.129:0/2661352077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96cc102bd0 0x7f96cc102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.686+0000 7f96d16f0640 1 --2- 172.21.15.129:0/2661352077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96cc1078d0 0x7f96cc109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.686+0000 7f96d16f0640 1 --2- 172.21.15.129:0/2661352077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc101990 0x7f96cc102690 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.686+0000 7f96d16f0640 1 -- 172.21.15.129:0/2661352077 >> 172.21.15.129:0/2661352077 conn(0x7f96cc0fcb80 msgr2=0x7f96cc0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:43.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.687+0000 7f96d16f0640 1 -- 172.21.15.129:0/2661352077 shutdown_connections 2024-05-20T17:58:43.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.687+0000 7f96d16f0640 1 -- 172.21.15.129:0/2661352077 wait complete. 2024-05-20T17:58:43.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.687+0000 7f96d16f0640 1 Processor -- start 2024-05-20T17:58:43.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96d16f0640 1 -- start start 2024-05-20T17:58:43.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96d16f0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96cc101990 0x7f96cc0fea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:43.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96d16f0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc102bd0 0x7f96cc0fd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:43.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96d16f0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96cc1078d0 0x7f96cc0fd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:43.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96d16f0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f96cc10d5e0 con 0x7f96cc102bd0 2024-05-20T17:58:43.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96d16f0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96cc10d460 con 0x7f96cc1078d0 2024-05-20T17:58:43.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96d16f0640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f96cc10d760 con 0x7f96cc101990 2024-05-20T17:58:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96caffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96cc101990 0x7f96cc0fea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96caffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96cc101990 0x7f96cc0fea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:51740/0 (socket says 172.21.15.129:51740) 2024-05-20T17:58:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96caffd640 1 -- 172.21.15.129:0/225046053 learned_addr learned my addr 172.21.15.129:0/225046053 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96ca7fc640 1 --2- 172.21.15.129:0/225046053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc102bd0 0x7f96cc0fd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.688+0000 7f96cb7fe640 1 --2- 172.21.15.129:0/225046053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96cc1078d0 0x7f96cc0fd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96caffd640 1 -- 172.21.15.129:0/225046053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96cc1078d0 msgr2=0x7f96cc0fd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96caffd640 1 --2- 172.21.15.129:0/225046053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96cc1078d0 0x7f96cc0fd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96caffd640 1 -- 172.21.15.129:0/225046053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc102bd0 msgr2=0x7f96cc0fd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96caffd640 1 --2- 172.21.15.129:0/225046053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc102bd0 0x7f96cc0fd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96caffd640 1 -- 172.21.15.129:0/225046053 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96cc0fdd20 con 0x7f96cc101990 2024-05-20T17:58:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96cb7fe640 1 --2- 172.21.15.129:0/225046053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96cc1078d0 0x7f96cc0fd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:43.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96caffd640 1 --2- 172.21.15.129:0/225046053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96cc101990 0x7f96cc0fea10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f96bc02a930 tx=0x7f96bc0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:43.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96b3fff640 1 -- 172.21.15.129:0/225046053 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc03f030 con 0x7f96cc101990 2024-05-20T17:58:43.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96d16f0640 1 -- 172.21.15.129:0/225046053 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96cc1a34b0 con 0x7f96cc101990 2024-05-20T17:58:43.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96b3fff640 1 -- 172.21.15.129:0/225046053 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96bc003ce0 con 0x7f96cc101990 2024-05-20T17:58:43.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96d16f0640 1 -- 172.21.15.129:0/225046053 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96cc1a3910 con 0x7f96cc101990 2024-05-20T17:58:43.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.689+0000 7f96b3fff640 1 -- 172.21.15.129:0/225046053 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96bc035070 con 0x7f96cc101990 2024-05-20T17:58:43.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.690+0000 7f96d16f0640 1 -- 172.21.15.129:0/225046053 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96cc069f50 con 0x7f96cc101990 2024-05-20T17:58:43.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.692+0000 7f96b3fff640 1 -- 172.21.15.129:0/225046053 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7f96bc042030 con 0x7f96cc101990 2024-05-20T17:58:43.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.693+0000 7f96b3fff640 1 -- 172.21.15.129:0/225046053 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f96bc0ef4d0 con 0x7f96cc101990 2024-05-20T17:58:43.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.696+0000 7f96b3fff640 1 -- 172.21.15.129:0/225046053 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f96bc049030 con 0x7f96cc101990 2024-05-20T17:58:43.811 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.561+0000 7fddf53ad200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/685489274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: Standby manager daemon y restarted 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: Standby manager daemon y started 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/940314689' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: Standby manager daemon z restarted 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: Standby manager daemon z started 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:58:43.812 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/272841763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: Active manager daemon x restarted 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: Activating manager daemon x 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: osdmap e89: 12 total, 12 up, 12 in 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: mgrmap e20: x(active, starting, since 0.0155205s), standbys: z, y 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-20T17:58:43.813 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-20T17:58:43.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-20T17:58:43.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-20T17:58:43.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-20T17:58:43.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-20T17:58:43.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T17:58:43.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T17:58:43.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-20T17:58:43.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-20T17:58:43.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:58:43.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-20T17:58:43.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-20T17:58:43.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-20T17:58:43.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: Manager daemon x is now available 2024-05-20T17:58:43.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:43 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.925+0000 7f96d16f0640 1 -- 172.21.15.129:0/225046053 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f96cc1a3c50 con 0x7f96cc101990 2024-05-20T17:58:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.925+0000 7f96b3fff640 1 -- 172.21.15.129:0/225046053 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f96bc0bc720 con 0x7f96cc101990 2024-05-20T17:58:43.923 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-05-20T17:58:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.928+0000 7f96d16f0640 1 -- 172.21.15.129:0/225046053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96cc101990 msgr2=0x7f96cc0fea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.928+0000 7f96d16f0640 1 --2- 172.21.15.129:0/225046053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96cc101990 0x7f96cc0fea10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f96bc02a930 tx=0x7f96bc0042f0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.928+0000 7f96d16f0640 1 -- 172.21.15.129:0/225046053 shutdown_connections 2024-05-20T17:58:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.928+0000 7f96d16f0640 1 --2- 172.21.15.129:0/225046053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96cc1078d0 0x7f96cc0fd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.928+0000 7f96d16f0640 1 --2- 172.21.15.129:0/225046053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96cc102bd0 0x7f96cc0fd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.928+0000 7f96d16f0640 1 --2- 172.21.15.129:0/225046053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96cc101990 0x7f96cc0fea10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:43.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.928+0000 7f96d16f0640 1 -- 172.21.15.129:0/225046053 >> 172.21.15.129:0/225046053 conn(0x7f96cc0fcb80 msgr2=0x7f96cc108cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:43.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.928+0000 7f96d16f0640 1 -- 172.21.15.129:0/225046053 shutdown_connections 2024-05-20T17:58:43.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:43.928+0000 7f96d16f0640 1 -- 172.21.15.129:0/225046053 wait complete. 2024-05-20T17:58:43.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/685489274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:43.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: Standby manager daemon y restarted 2024-05-20T17:58:43.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: Standby manager daemon y started 2024-05-20T17:58:43.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/940314689' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: Standby manager daemon z restarted 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: Standby manager daemon z started 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/272841763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: Active manager daemon x restarted 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: Activating manager daemon x 2024-05-20T17:58:43.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: osdmap e89: 12 total, 12 up, 12 in 2024-05-20T17:58:43.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: mgrmap e20: x(active, starting, since 0.0155205s), standbys: z, y 2024-05-20T17:58:43.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:58:43.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:58:43.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:58:43.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-20T17:58:43.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-20T17:58:43.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-20T17:58:43.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-20T17:58:43.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-20T17:58:43.931 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:58:43.932 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: Manager daemon x is now available 2024-05-20T17:58:43.933 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:43 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:43.937 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-20T17:58:43.937 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:58:44.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.055+0000 7ff69cd8b640 1 -- 172.21.15.129:0/3703717921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff698101f60 msgr2=0x7ff698102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.055+0000 7ff69cd8b640 1 --2- 172.21.15.129:0/3703717921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff698101f60 0x7ff698102370 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff688009900 tx=0x7ff68802a410 comp rx=0 tx=0).stop 2024-05-20T17:58:44.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.055+0000 7ff69cd8b640 1 -- 172.21.15.129:0/3703717921 shutdown_connections 2024-05-20T17:58:44.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.055+0000 7ff69cd8b640 1 --2- 172.21.15.129:0/3703717921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff698102ac0 0x7ff698102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.055+0000 7ff69cd8b640 1 --2- 172.21.15.129:0/3703717921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff698103620 0x7ff698109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.055+0000 7ff69cd8b640 1 --2- 172.21.15.129:0/3703717921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff698101f60 0x7ff698102370 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.055+0000 7ff69cd8b640 1 -- 172.21.15.129:0/3703717921 >> 172.21.15.129:0/3703717921 conn(0x7ff6980fc2e0 msgr2=0x7ff6980fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:44.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.055+0000 7ff69cd8b640 1 -- 172.21.15.129:0/3703717921 shutdown_connections 2024-05-20T17:58:44.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.055+0000 7ff69cd8b640 1 -- 172.21.15.129:0/3703717921 wait complete. 2024-05-20T17:58:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.056+0000 7ff69cd8b640 1 Processor -- start 2024-05-20T17:58:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.056+0000 7ff69cd8b640 1 -- start start 2024-05-20T17:58:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.056+0000 7ff69cd8b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff698101f60 0x7ff6981990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.056+0000 7ff69cd8b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff698102ac0 0x7ff698199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.056+0000 7ff69cd8b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69819e5b0 0x7ff6981a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff69cd8b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff69810d810 con 0x7ff698102ac0 2024-05-20T17:58:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff69cd8b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff69810d690 con 0x7ff69819e5b0 2024-05-20T17:58:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff69cd8b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff69810d990 con 0x7ff698101f60 2024-05-20T17:58:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff68ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff698102ac0 0x7ff698199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff69659c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff698101f60 0x7ff6981990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff68ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff698102ac0 0x7ff698199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42406/0 (socket says 172.21.15.129:42406) 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff68ffff640 1 -- 172.21.15.129:0/1562265075 learned_addr learned my addr 172.21.15.129:0/1562265075 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff696d9d640 1 --2- 172.21.15.129:0/1562265075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69819e5b0 0x7ff6981a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff696d9d640 1 -- 172.21.15.129:0/1562265075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff698101f60 msgr2=0x7ff6981990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff696d9d640 1 --2- 172.21.15.129:0/1562265075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff698101f60 0x7ff6981990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff696d9d640 1 -- 172.21.15.129:0/1562265075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff698102ac0 msgr2=0x7ff698199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff696d9d640 1 --2- 172.21.15.129:0/1562265075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff698102ac0 0x7ff698199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff696d9d640 1 -- 172.21.15.129:0/1562265075 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6981a1040 con 0x7ff69819e5b0 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff69659c640 1 --2- 172.21.15.129:0/1562265075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff698101f60 0x7ff6981990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff696d9d640 1 --2- 172.21.15.129:0/1562265075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69819e5b0 0x7ff6981a09d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff690009fd0 tx=0x7ff69000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff68f7fe640 1 -- 172.21.15.129:0/1562265075 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69000ce20 con 0x7ff69819e5b0 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff68f7fe640 1 -- 172.21.15.129:0/1562265075 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff690005020 con 0x7ff69819e5b0 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff69cd8b640 1 -- 172.21.15.129:0/1562265075 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6981a1330 con 0x7ff69819e5b0 2024-05-20T17:58:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff68f7fe640 1 -- 172.21.15.129:0/1562265075 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff690010720 con 0x7ff69819e5b0 2024-05-20T17:58:44.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.057+0000 7ff69cd8b640 1 -- 172.21.15.129:0/1562265075 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6981a18d0 con 0x7ff69819e5b0 2024-05-20T17:58:44.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.058+0000 7ff69cd8b640 1 -- 172.21.15.129:0/1562265075 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff698068c20 con 0x7ff69819e5b0 2024-05-20T17:58:44.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.060+0000 7ff68f7fe640 1 -- 172.21.15.129:0/1562265075 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7ff690005540 con 0x7ff69819e5b0 2024-05-20T17:58:44.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.061+0000 7ff68f7fe640 1 -- 172.21.15.129:0/1562265075 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7ff6900cf150 con 0x7ff69819e5b0 2024-05-20T17:58:44.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.065+0000 7ff68f7fe640 1 -- 172.21.15.129:0/1562265075 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff69009c3a0 con 0x7ff69819e5b0 2024-05-20T17:58:44.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/685489274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:44.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: Standby manager daemon y restarted 2024-05-20T17:58:44.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: Standby manager daemon y started 2024-05-20T17:58:44.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-20T17:58:44.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:58:44.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-20T17:58:44.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.154:0/1596616396' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/940314689' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: Standby manager daemon z restarted 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: Standby manager daemon z started 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.? 172.21.15.204:0/2690685389' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/272841763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: Active manager daemon x restarted 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: Activating manager daemon x 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: osdmap e89: 12 total, 12 up, 12 in 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: mgrmap e20: x(active, starting, since 0.0155205s), standbys: z, y 2024-05-20T17:58:44.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T17:58:44.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-20T17:58:44.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-20T17:58:44.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-20T17:58:44.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-20T17:58:44.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-20T17:58:44.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-20T17:58:44.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-20T17:58:44.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: Manager daemon x is now available 2024-05-20T17:58:44.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:43 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:44.162 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.848+0000 7fdd53a66700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:58:44.162 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.848+0000 7fdd53a66700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:58:44.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.848+0000 7fdd53a66700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:58:44.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.848+0000 7fdd53a66700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:58:44.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.848+0000 7fdd53a66700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:58:44.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.849+0000 7fdd56a6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:58:44.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.849+0000 7fdd56a6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:58:44.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.849+0000 7fdd56a6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:58:44.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.849+0000 7fdd56a6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:58:44.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:58:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:58:43.849+0000 7fdd56a6c700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:58:44.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.289+0000 7ff69cd8b640 1 -- 172.21.15.129:0/1562265075 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff698107370 con 0x7ff69819e5b0 2024-05-20T17:58:44.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.290+0000 7ff68f7fe640 1 -- 172.21.15.129:0/1562265075 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11829 (secure 0 0 0) 0x7ff69009bca0 con 0x7ff69819e5b0 2024-05-20T17:58:44.288 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:44.288 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:56:49.336282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T17:58:44.288 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 20 2024-05-20T17:58:44.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.293+0000 7ff69cd8b640 1 -- 172.21.15.129:0/1562265075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69819e5b0 msgr2=0x7ff6981a09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.293+0000 7ff69cd8b640 1 --2- 172.21.15.129:0/1562265075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69819e5b0 0x7ff6981a09d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff690009fd0 tx=0x7ff69000ef10 comp rx=0 tx=0).stop 2024-05-20T17:58:44.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.293+0000 7ff69cd8b640 1 -- 172.21.15.129:0/1562265075 shutdown_connections 2024-05-20T17:58:44.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.293+0000 7ff69cd8b640 1 --2- 172.21.15.129:0/1562265075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff69819e5b0 0x7ff6981a09d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.293+0000 7ff69cd8b640 1 --2- 172.21.15.129:0/1562265075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff698102ac0 0x7ff698199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.293+0000 7ff69cd8b640 1 --2- 172.21.15.129:0/1562265075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff698101f60 0x7ff6981990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.293+0000 7ff69cd8b640 1 -- 172.21.15.129:0/1562265075 >> 172.21.15.129:0/1562265075 conn(0x7ff6980fc2e0 msgr2=0x7ff698108bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:44.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.293+0000 7ff69cd8b640 1 -- 172.21.15.129:0/1562265075 shutdown_connections 2024-05-20T17:58:44.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.293+0000 7ff69cd8b640 1 -- 172.21.15.129:0/1562265075 wait complete. 2024-05-20T17:58:44.302 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-20T17:58:44.302 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-20T17:58:44.302 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-20T17:58:44.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.420+0000 7f486ec61640 1 -- 172.21.15.129:0/2614747034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4868101970 msgr2=0x7f4868102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.420+0000 7f486ec61640 1 --2- 172.21.15.129:0/2614747034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4868101970 0x7f4868102670 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f485c009900 tx=0x7f485c02a410 comp rx=0 tx=0).stop 2024-05-20T17:58:44.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.420+0000 7f486ec61640 1 -- 172.21.15.129:0/2614747034 shutdown_connections 2024-05-20T17:58:44.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.420+0000 7f486ec61640 1 --2- 172.21.15.129:0/2614747034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4868102bb0 0x7f4868102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.420+0000 7f486ec61640 1 --2- 172.21.15.129:0/2614747034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f48681078b0 0x7f4868109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.420+0000 7f486ec61640 1 --2- 172.21.15.129:0/2614747034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4868101970 0x7f4868102670 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.420+0000 7f486ec61640 1 -- 172.21.15.129:0/2614747034 >> 172.21.15.129:0/2614747034 conn(0x7f48680fcb80 msgr2=0x7f48680ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:44.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.420+0000 7f486ec61640 1 -- 172.21.15.129:0/2614747034 shutdown_connections 2024-05-20T17:58:44.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.420+0000 7f486ec61640 1 -- 172.21.15.129:0/2614747034 wait complete. 2024-05-20T17:58:44.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486ec61640 1 Processor -- start 2024-05-20T17:58:44.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486ec61640 1 -- start start 2024-05-20T17:58:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486ec61640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4868101970 0x7f48680698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486ec61640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4868102bb0 0x7f486819e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486ec61640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48681078b0 0x7f48681a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486ec61640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f486810d5c0 con 0x7f4868101970 2024-05-20T17:58:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486ec61640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f486810d440 con 0x7f48681078b0 2024-05-20T17:58:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486ec61640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f486810d740 con 0x7f4868102bb0 2024-05-20T17:58:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486c9fd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4868101970 0x7f48680698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486c9fd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4868101970 0x7f48680698b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42432/0 (socket says 172.21.15.129:42432) 2024-05-20T17:58:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486c9fd640 1 -- 172.21.15.129:0/3233257805 learned_addr learned my addr 172.21.15.129:0/3233257805 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:44.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486d1fe640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48681078b0 0x7f48681a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:44.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f4867fff640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4868102bb0 0x7f486819e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:44.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486c9fd640 1 -- 172.21.15.129:0/3233257805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4868102bb0 msgr2=0x7f486819e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486c9fd640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4868102bb0 0x7f486819e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486c9fd640 1 -- 172.21.15.129:0/3233257805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48681078b0 msgr2=0x7f48681a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486c9fd640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48681078b0 0x7f48681a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.421+0000 7f486c9fd640 1 -- 172.21.15.129:0/3233257805 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48681a0dc0 con 0x7f4868101970 2024-05-20T17:58:44.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.422+0000 7f486c9fd640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4868101970 0x7f48680698b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f485c005ab0 tx=0x7f485c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:44.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.422+0000 7f4865ffb640 1 -- 172.21.15.129:0/3233257805 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485c03f030 con 0x7f4868101970 2024-05-20T17:58:44.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.422+0000 7f4865ffb640 1 -- 172.21.15.129:0/3233257805 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f485c003ad0 con 0x7f4868101970 2024-05-20T17:58:44.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.422+0000 7f486ec61640 1 -- 172.21.15.129:0/3233257805 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48681a1050 con 0x7f4868101970 2024-05-20T17:58:44.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.422+0000 7f4865ffb640 1 -- 172.21.15.129:0/3233257805 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485c035070 con 0x7f4868101970 2024-05-20T17:58:44.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.422+0000 7f486ec61640 1 -- 172.21.15.129:0/3233257805 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48681a1590 con 0x7f4868101970 2024-05-20T17:58:44.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.423+0000 7f486ec61640 1 -- 172.21.15.129:0/3233257805 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48681066c0 con 0x7f4868101970 2024-05-20T17:58:44.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.424+0000 7f4865ffb640 1 -- 172.21.15.129:0/3233257805 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7f485c042030 con 0x7f4868101970 2024-05-20T17:58:44.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.425+0000 7f4865ffb640 1 -- 172.21.15.129:0/3233257805 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f485c0ef340 con 0x7f4868101970 2024-05-20T17:58:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.433+0000 7f4865ffb640 1 -- 172.21.15.129:0/3233257805 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f485c0bbf40 con 0x7f4868101970 2024-05-20T17:58:44.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.579+0000 7f4865ffb640 1 -- 172.21.15.129:0/3233257805 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f485c0b7c90 con 0x7f4868101970 2024-05-20T17:58:44.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.580+0000 7f4865ffb640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f48580a9980 0x7f48580abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:44.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.582+0000 7f4867fff640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f48580a9980 0x7f48580abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:44.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.584+0000 7f4867fff640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f48580a9980 0x7f48580abda0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4868069250 tx=0x7f4854005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:44.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.637+0000 7f486ec61640 1 -- 172.21.15.129:0/3233257805 --> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4868061f90 con 0x7f48580a9980 2024-05-20T17:58:44.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.644+0000 7f4865ffb640 1 -- 172.21.15.129:0/3233257805 <== mgr.15228 v2:172.21.15.129:6800/3515662734 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257946 (secure 0 0 0) 0x7f4868061f90 con 0x7f48580a9980 2024-05-20T17:58:44.644 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:44.646 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-05-20T17:58:44.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 -- 172.21.15.129:0/3233257805 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f48580a9980 msgr2=0x7f48580abda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f48580a9980 0x7f48580abda0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4868069250 tx=0x7f4854005f90 comp rx=0 tx=0).stop 2024-05-20T17:58:44.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 -- 172.21.15.129:0/3233257805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4868101970 msgr2=0x7f48680698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4868101970 0x7f48680698b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f485c005ab0 tx=0x7f485c004270 comp rx=0 tx=0).stop 2024-05-20T17:58:44.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 -- 172.21.15.129:0/3233257805 shutdown_connections 2024-05-20T17:58:44.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f48580a9980 0x7f48580abda0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48681078b0 0x7f48681a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4868102bb0 0x7f486819e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 --2- 172.21.15.129:0/3233257805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4868101970 0x7f48680698b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 -- 172.21.15.129:0/3233257805 >> 172.21.15.129:0/3233257805 conn(0x7f48680fcb80 msgr2=0x7f4868103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:44.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 -- 172.21.15.129:0/3233257805 shutdown_connections 2024-05-20T17:58:44.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.648+0000 7f486ec61640 1 -- 172.21.15.129:0/3233257805 wait complete. 2024-05-20T17:58:44.658 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":3,"stamp":"2024-05-20T17:58:44.594311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1183974,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":476,"num_read_kb":391,"num_write":379,"num_write_kb":4343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365084,"kb_used_data":11744,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576740,"statfs":{"total":1151940427776,"available":1151566581760,"internally_reserved":0,"allocated":12025856,"data_stored":8515694,"data_compressed":250131,"data_compressed_allocated":335872,"data_compressed_original":2965504,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859196+0000","last_change":"2024-05-20T17:56:45.746793+0000","last_active":"2024-05-20T17:58:43.859196+0000","last_peered":"2024-05-20T17:58:43.859196+0000","last_clean":"2024-05-20T17:58:43.859196+0000","last_became_active":"2024-05-20T17:56:45.746244+0000","last_became_peered":"2024-05-20T17:56:45.746244+0000","last_unstale":"2024-05-20T17:58:43.859196+0000","last_undegraded":"2024-05-20T17:58:43.859196+0000","last_fullsized":"2024-05-20T17:58:43.859196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:37:03.703599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113926+0000","last_change":"2024-05-20T17:56:45.744538+0000","last_active":"2024-05-20T17:58:44.113926+0000","last_peered":"2024-05-20T17:58:44.113926+0000","last_clean":"2024-05-20T17:58:44.113926+0000","last_became_active":"2024-05-20T17:56:45.744213+0000","last_became_peered":"2024-05-20T17:56:45.744213+0000","last_unstale":"2024-05-20T17:58:44.113926+0000","last_undegraded":"2024-05-20T17:58:44.113926+0000","last_fullsized":"2024-05-20T17:58:44.113926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:19:26.163003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113332+0000","last_change":"2024-05-20T17:56:45.749725+0000","last_active":"2024-05-20T17:58:44.113332+0000","last_peered":"2024-05-20T17:58:44.113332+0000","last_clean":"2024-05-20T17:58:44.113332+0000","last_became_active":"2024-05-20T17:56:45.749392+0000","last_became_peered":"2024-05-20T17:56:45.749392+0000","last_unstale":"2024-05-20T17:58:44.113332+0000","last_undegraded":"2024-05-20T17:58:44.113332+0000","last_fullsized":"2024-05-20T17:58:44.113332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:43:33.184192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113454+0000","last_change":"2024-05-20T17:56:45.755191+0000","last_active":"2024-05-20T17:58:44.113454+0000","last_peered":"2024-05-20T17:58:44.113454+0000","last_clean":"2024-05-20T17:58:44.113454+0000","last_became_active":"2024-05-20T17:56:45.753524+0000","last_became_peered":"2024-05-20T17:56:45.753524+0000","last_unstale":"2024-05-20T17:58:44.113454+0000","last_undegraded":"2024-05-20T17:58:44.113454+0000","last_fullsized":"2024-05-20T17:58:44.113454+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:11:23.099919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681633+0000","last_change":"2024-05-20T17:56:45.756291+0000","last_active":"2024-05-20T17:58:43.681633+0000","last_peered":"2024-05-20T17:58:43.681633+0000","last_clean":"2024-05-20T17:58:43.681633+0000","last_became_active":"2024-05-20T17:56:45.755910+0000","last_became_peered":"2024-05-20T17:56:45.755910+0000","last_unstale":"2024-05-20T17:58:43.681633+0000","last_undegraded":"2024-05-20T17:58:43.681633+0000","last_fullsized":"2024-05-20T17:58:43.681633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:37:36.941012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694392+0000","last_change":"2024-05-20T17:56:45.748756+0000","last_active":"2024-05-20T17:58:43.694392+0000","last_peered":"2024-05-20T17:58:43.694392+0000","last_clean":"2024-05-20T17:58:43.694392+0000","last_became_active":"2024-05-20T17:56:45.748406+0000","last_became_peered":"2024-05-20T17:56:45.748406+0000","last_unstale":"2024-05-20T17:58:43.694392+0000","last_undegraded":"2024-05-20T17:58:43.694392+0000","last_fullsized":"2024-05-20T17:58:43.694392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:16:36.527832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571098+0000","last_change":"2024-05-20T17:57:49.051459+0000","last_active":"2024-05-20T17:58:43.571098+0000","last_peered":"2024-05-20T17:58:43.571098+0000","last_clean":"2024-05-20T17:58:43.571098+0000","last_became_active":"2024-05-20T17:57:49.050933+0000","last_became_peered":"2024-05-20T17:57:49.050933+0000","last_unstale":"2024-05-20T17:58:43.571098+0000","last_undegraded":"2024-05-20T17:58:43.571098+0000","last_fullsized":"2024-05-20T17:58:43.571098+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:18.214102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681494+0000","last_change":"2024-05-20T17:56:45.757387+0000","last_active":"2024-05-20T17:58:43.681494+0000","last_peered":"2024-05-20T17:58:43.681494+0000","last_clean":"2024-05-20T17:58:43.681494+0000","last_became_active":"2024-05-20T17:56:45.756374+0000","last_became_peered":"2024-05-20T17:56:45.756374+0000","last_unstale":"2024-05-20T17:58:43.681494+0000","last_undegraded":"2024-05-20T17:58:43.681494+0000","last_fullsized":"2024-05-20T17:58:43.681494+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:53:52.751019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.857216+0000","last_change":"2024-05-20T17:56:45.751933+0000","last_active":"2024-05-20T17:58:43.857216+0000","last_peered":"2024-05-20T17:58:43.857216+0000","last_clean":"2024-05-20T17:58:43.857216+0000","last_became_active":"2024-05-20T17:56:45.751414+0000","last_became_peered":"2024-05-20T17:56:45.751414+0000","last_unstale":"2024-05-20T17:58:43.857216+0000","last_undegraded":"2024-05-20T17:58:43.857216+0000","last_fullsized":"2024-05-20T17:58:43.857216+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:56:52.974830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694627+0000","last_change":"2024-05-20T17:56:45.749511+0000","last_active":"2024-05-20T17:58:43.694627+0000","last_peered":"2024-05-20T17:58:43.694627+0000","last_clean":"2024-05-20T17:58:43.694627+0000","last_became_active":"2024-05-20T17:56:45.749211+0000","last_became_peered":"2024-05-20T17:56:45.749211+0000","last_unstale":"2024-05-20T17:58:43.694627+0000","last_undegraded":"2024-05-20T17:58:43.694627+0000","last_fullsized":"2024-05-20T17:58:43.694627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:15.633296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571341+0000","last_change":"2024-05-20T17:56:45.753743+0000","last_active":"2024-05-20T17:58:43.571341+0000","last_peered":"2024-05-20T17:58:43.571341+0000","last_clean":"2024-05-20T17:58:43.571341+0000","last_became_active":"2024-05-20T17:56:45.753378+0000","last_became_peered":"2024-05-20T17:56:45.753378+0000","last_unstale":"2024-05-20T17:58:43.571341+0000","last_undegraded":"2024-05-20T17:58:43.571341+0000","last_fullsized":"2024-05-20T17:58:43.571341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:54.563895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113327+0000","last_change":"2024-05-20T17:56:45.754890+0000","last_active":"2024-05-20T17:58:44.113327+0000","last_peered":"2024-05-20T17:58:44.113327+0000","last_clean":"2024-05-20T17:58:44.113327+0000","last_became_active":"2024-05-20T17:56:45.754333+0000","last_became_peered":"2024-05-20T17:56:45.754333+0000","last_unstale":"2024-05-20T17:58:44.113327+0000","last_undegraded":"2024-05-20T17:58:44.113327+0000","last_fullsized":"2024-05-20T17:58:44.113327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:51.025516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113630+0000","last_change":"2024-05-20T17:56:45.745662+0000","last_active":"2024-05-20T17:58:44.113630+0000","last_peered":"2024-05-20T17:58:44.113630+0000","last_clean":"2024-05-20T17:58:44.113630+0000","last_became_active":"2024-05-20T17:56:45.745281+0000","last_became_peered":"2024-05-20T17:56:45.745281+0000","last_unstale":"2024-05-20T17:58:44.113630+0000","last_undegraded":"2024-05-20T17:58:44.113630+0000","last_fullsized":"2024-05-20T17:58:44.113630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:38:57.057301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.112954+0000","last_change":"2024-05-20T17:56:45.749164+0000","last_active":"2024-05-20T17:58:44.112954+0000","last_peered":"2024-05-20T17:58:44.112954+0000","last_clean":"2024-05-20T17:58:44.112954+0000","last_became_active":"2024-05-20T17:56:45.748822+0000","last_became_peered":"2024-05-20T17:56:45.748822+0000","last_unstale":"2024-05-20T17:58:44.112954+0000","last_undegraded":"2024-05-20T17:58:44.112954+0000","last_fullsized":"2024-05-20T17:58:44.112954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:45:47.487116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.112953+0000","last_change":"2024-05-20T17:56:45.751185+0000","last_active":"2024-05-20T17:58:44.112953+0000","last_peered":"2024-05-20T17:58:44.112953+0000","last_clean":"2024-05-20T17:58:44.112953+0000","last_became_active":"2024-05-20T17:56:45.750851+0000","last_became_peered":"2024-05-20T17:56:45.750851+0000","last_unstale":"2024-05-20T17:58:44.112953+0000","last_undegraded":"2024-05-20T17:58:44.112953+0000","last_fullsized":"2024-05-20T17:58:44.112953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:31:07.792717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856607+0000","last_change":"2024-05-20T17:56:45.748101+0000","last_active":"2024-05-20T17:58:43.856607+0000","last_peered":"2024-05-20T17:58:43.856607+0000","last_clean":"2024-05-20T17:58:43.856607+0000","last_became_active":"2024-05-20T17:56:45.747754+0000","last_became_peered":"2024-05-20T17:56:45.747754+0000","last_unstale":"2024-05-20T17:58:43.856607+0000","last_undegraded":"2024-05-20T17:58:43.856607+0000","last_fullsized":"2024-05-20T17:58:43.856607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:24:54.702601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859350+0000","last_change":"2024-05-20T17:56:45.747497+0000","last_active":"2024-05-20T17:58:43.859350+0000","last_peered":"2024-05-20T17:58:43.859350+0000","last_clean":"2024-05-20T17:58:43.859350+0000","last_became_active":"2024-05-20T17:56:45.747139+0000","last_became_peered":"2024-05-20T17:56:45.747139+0000","last_unstale":"2024-05-20T17:58:43.859350+0000","last_undegraded":"2024-05-20T17:58:43.859350+0000","last_fullsized":"2024-05-20T17:58:43.859350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:39:13.102413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570446+0000","last_change":"2024-05-20T17:56:45.746194+0000","last_active":"2024-05-20T17:58:43.570446+0000","last_peered":"2024-05-20T17:58:43.570446+0000","last_clean":"2024-05-20T17:58:43.570446+0000","last_became_active":"2024-05-20T17:56:45.745809+0000","last_became_peered":"2024-05-20T17:56:45.745809+0000","last_unstale":"2024-05-20T17:58:43.570446+0000","last_undegraded":"2024-05-20T17:58:43.570446+0000","last_fullsized":"2024-05-20T17:58:43.570446+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:26:29.133790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569306+0000","last_change":"2024-05-20T17:56:45.749918+0000","last_active":"2024-05-20T17:58:43.569306+0000","last_peered":"2024-05-20T17:58:43.569306+0000","last_clean":"2024-05-20T17:58:43.569306+0000","last_became_active":"2024-05-20T17:56:45.749560+0000","last_became_peered":"2024-05-20T17:56:45.749560+0000","last_unstale":"2024-05-20T17:58:43.569306+0000","last_undegraded":"2024-05-20T17:58:43.569306+0000","last_fullsized":"2024-05-20T17:58:43.569306+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:33:12.737206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570253+0000","last_change":"2024-05-20T17:57:49.082268+0000","last_active":"2024-05-20T17:58:43.570253+0000","last_peered":"2024-05-20T17:58:43.570253+0000","last_clean":"2024-05-20T17:58:43.570253+0000","last_became_active":"2024-05-20T17:57:49.078471+0000","last_became_peered":"2024-05-20T17:57:49.078471+0000","last_unstale":"2024-05-20T17:58:43.570253+0000","last_undegraded":"2024-05-20T17:58:43.570253+0000","last_fullsized":"2024-05-20T17:58:43.570253+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:50:14.617477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682939+0000","last_change":"2024-05-20T17:56:45.758330+0000","last_active":"2024-05-20T17:58:43.682939+0000","last_peered":"2024-05-20T17:58:43.682939+0000","last_clean":"2024-05-20T17:58:43.682939+0000","last_became_active":"2024-05-20T17:56:45.755873+0000","last_became_peered":"2024-05-20T17:56:45.755873+0000","last_unstale":"2024-05-20T17:58:43.682939+0000","last_undegraded":"2024-05-20T17:58:43.682939+0000","last_fullsized":"2024-05-20T17:58:43.682939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:53:31.653497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.683149+0000","last_change":"2024-05-20T17:56:44.728508+0000","last_active":"2024-05-20T17:58:43.683149+0000","last_peered":"2024-05-20T17:58:43.683149+0000","last_clean":"2024-05-20T17:58:43.683149+0000","last_became_active":"2024-05-20T17:56:44.727875+0000","last_became_peered":"2024-05-20T17:56:44.727875+0000","last_unstale":"2024-05-20T17:58:43.683149+0000","last_undegraded":"2024-05-20T17:58:43.683149+0000","last_fullsized":"2024-05-20T17:58:43.683149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:12:14.349039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.114265+0000","last_change":"2024-05-20T17:56:45.750000+0000","last_active":"2024-05-20T17:58:44.114265+0000","last_peered":"2024-05-20T17:58:44.114265+0000","last_clean":"2024-05-20T17:58:44.114265+0000","last_became_active":"2024-05-20T17:56:45.749644+0000","last_became_peered":"2024-05-20T17:56:45.749644+0000","last_unstale":"2024-05-20T17:58:44.114265+0000","last_undegraded":"2024-05-20T17:58:44.114265+0000","last_fullsized":"2024-05-20T17:58:44.114265+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:32:46.085081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682012+0000","last_change":"2024-05-20T17:56:44.722692+0000","last_active":"2024-05-20T17:58:43.682012+0000","last_peered":"2024-05-20T17:58:43.682012+0000","last_clean":"2024-05-20T17:58:43.682012+0000","last_became_active":"2024-05-20T17:56:44.722170+0000","last_became_peered":"2024-05-20T17:56:44.722170+0000","last_unstale":"2024-05-20T17:58:43.682012+0000","last_undegraded":"2024-05-20T17:58:43.682012+0000","last_fullsized":"2024-05-20T17:58:43.682012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:38:03.679562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856783+0000","last_change":"2024-05-20T17:56:45.751666+0000","last_active":"2024-05-20T17:58:43.856783+0000","last_peered":"2024-05-20T17:58:43.856783+0000","last_clean":"2024-05-20T17:58:43.856783+0000","last_became_active":"2024-05-20T17:56:45.751291+0000","last_became_peered":"2024-05-20T17:56:45.751291+0000","last_unstale":"2024-05-20T17:58:43.856783+0000","last_undegraded":"2024-05-20T17:58:43.856783+0000","last_fullsized":"2024-05-20T17:58:43.856783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:29:51.517363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681871+0000","last_change":"2024-05-20T17:56:44.721121+0000","last_active":"2024-05-20T17:58:43.681871+0000","last_peered":"2024-05-20T17:58:43.681871+0000","last_clean":"2024-05-20T17:58:43.681871+0000","last_became_active":"2024-05-20T17:56:44.720693+0000","last_became_peered":"2024-05-20T17:56:44.720693+0000","last_unstale":"2024-05-20T17:58:43.681871+0000","last_undegraded":"2024-05-20T17:58:43.681871+0000","last_fullsized":"2024-05-20T17:58:43.681871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:44.223828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691476+0000","last_change":"2024-05-20T17:56:45.745925+0000","last_active":"2024-05-20T17:58:43.691476+0000","last_peered":"2024-05-20T17:58:43.691476+0000","last_clean":"2024-05-20T17:58:43.691476+0000","last_became_active":"2024-05-20T17:56:45.745303+0000","last_became_peered":"2024-05-20T17:56:45.745303+0000","last_unstale":"2024-05-20T17:58:43.691476+0000","last_undegraded":"2024-05-20T17:58:43.691476+0000","last_fullsized":"2024-05-20T17:58:43.691476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:41:00.230681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691275+0000","last_change":"2024-05-20T17:56:44.719048+0000","last_active":"2024-05-20T17:58:43.691275+0000","last_peered":"2024-05-20T17:58:43.691275+0000","last_clean":"2024-05-20T17:58:43.691275+0000","last_became_active":"2024-05-20T17:56:44.718397+0000","last_became_peered":"2024-05-20T17:56:44.718397+0000","last_unstale":"2024-05-20T17:58:43.691275+0000","last_undegraded":"2024-05-20T17:58:43.691275+0000","last_fullsized":"2024-05-20T17:58:43.691275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:11:53.898226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681642+0000","last_change":"2024-05-20T17:56:45.752395+0000","last_active":"2024-05-20T17:58:43.681642+0000","last_peered":"2024-05-20T17:58:43.681642+0000","last_clean":"2024-05-20T17:58:43.681642+0000","last_became_active":"2024-05-20T17:56:45.751940+0000","last_became_peered":"2024-05-20T17:56:45.751940+0000","last_unstale":"2024-05-20T17:58:43.681642+0000","last_undegraded":"2024-05-20T17:58:43.681642+0000","last_fullsized":"2024-05-20T17:58:43.681642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:33:57.780042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694331+0000","last_change":"2024-05-20T17:56:44.714712+0000","last_active":"2024-05-20T17:58:43.694331+0000","last_peered":"2024-05-20T17:58:43.694331+0000","last_clean":"2024-05-20T17:58:43.694331+0000","last_became_active":"2024-05-20T17:56:44.714396+0000","last_became_peered":"2024-05-20T17:56:44.714396+0000","last_unstale":"2024-05-20T17:58:43.694331+0000","last_undegraded":"2024-05-20T17:58:43.694331+0000","last_fullsized":"2024-05-20T17:58:43.694331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:21:04.831289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691360+0000","last_change":"2024-05-20T17:56:45.758160+0000","last_active":"2024-05-20T17:58:43.691360+0000","last_peered":"2024-05-20T17:58:43.691360+0000","last_clean":"2024-05-20T17:58:43.691360+0000","last_became_active":"2024-05-20T17:56:45.757405+0000","last_became_peered":"2024-05-20T17:56:45.757405+0000","last_unstale":"2024-05-20T17:58:43.691360+0000","last_undegraded":"2024-05-20T17:58:43.691360+0000","last_fullsized":"2024-05-20T17:58:43.691360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:17.625246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113310+0000","last_change":"2024-05-20T17:56:44.716342+0000","last_active":"2024-05-20T17:58:44.113310+0000","last_peered":"2024-05-20T17:58:44.113310+0000","last_clean":"2024-05-20T17:58:44.113310+0000","last_became_active":"2024-05-20T17:56:44.716010+0000","last_became_peered":"2024-05-20T17:56:44.716010+0000","last_unstale":"2024-05-20T17:58:44.113310+0000","last_undegraded":"2024-05-20T17:58:44.113310+0000","last_fullsized":"2024-05-20T17:58:44.113310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:00:25.073312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681615+0000","last_change":"2024-05-20T17:56:45.755288+0000","last_active":"2024-05-20T17:58:43.681615+0000","last_peered":"2024-05-20T17:58:43.681615+0000","last_clean":"2024-05-20T17:58:43.681615+0000","last_became_active":"2024-05-20T17:56:45.754952+0000","last_became_peered":"2024-05-20T17:56:45.754952+0000","last_unstale":"2024-05-20T17:58:43.681615+0000","last_undegraded":"2024-05-20T17:58:43.681615+0000","last_fullsized":"2024-05-20T17:58:43.681615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:44:27.631722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691102+0000","last_change":"2024-05-20T17:56:44.717887+0000","last_active":"2024-05-20T17:58:43.691102+0000","last_peered":"2024-05-20T17:58:43.691102+0000","last_clean":"2024-05-20T17:58:43.691102+0000","last_became_active":"2024-05-20T17:56:44.717581+0000","last_became_peered":"2024-05-20T17:56:44.717581+0000","last_unstale":"2024-05-20T17:58:43.691102+0000","last_undegraded":"2024-05-20T17:58:43.691102+0000","last_fullsized":"2024-05-20T17:58:43.691102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:42:54.703112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570033+0000","last_change":"2024-05-20T17:56:45.748448+0000","last_active":"2024-05-20T17:58:43.570033+0000","last_peered":"2024-05-20T17:58:43.570033+0000","last_clean":"2024-05-20T17:58:43.570033+0000","last_became_active":"2024-05-20T17:56:45.748016+0000","last_became_peered":"2024-05-20T17:56:45.748016+0000","last_unstale":"2024-05-20T17:58:43.570033+0000","last_undegraded":"2024-05-20T17:58:43.570033+0000","last_fullsized":"2024-05-20T17:58:43.570033+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:31:39.350477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694679+0000","last_change":"2024-05-20T17:56:44.709483+0000","last_active":"2024-05-20T17:58:43.694679+0000","last_peered":"2024-05-20T17:58:43.694679+0000","last_clean":"2024-05-20T17:58:43.694679+0000","last_became_active":"2024-05-20T17:56:44.708649+0000","last_became_peered":"2024-05-20T17:56:44.708649+0000","last_unstale":"2024-05-20T17:58:43.694679+0000","last_undegraded":"2024-05-20T17:58:43.694679+0000","last_fullsized":"2024-05-20T17:58:43.694679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:31:46.580256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.857082+0000","last_change":"2024-05-20T17:56:45.752279+0000","last_active":"2024-05-20T17:58:43.857082+0000","last_peered":"2024-05-20T17:58:43.857082+0000","last_clean":"2024-05-20T17:58:43.857082+0000","last_became_active":"2024-05-20T17:56:45.751790+0000","last_became_peered":"2024-05-20T17:56:45.751790+0000","last_unstale":"2024-05-20T17:58:43.857082+0000","last_undegraded":"2024-05-20T17:58:43.857082+0000","last_fullsized":"2024-05-20T17:58:43.857082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:06:38.113559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571437+0000","last_change":"2024-05-20T17:57:49.077752+0000","last_active":"2024-05-20T17:58:43.571437+0000","last_peered":"2024-05-20T17:58:43.571437+0000","last_clean":"2024-05-20T17:58:43.571437+0000","last_became_active":"2024-05-20T17:57:49.077277+0000","last_became_peered":"2024-05-20T17:57:49.077277+0000","last_unstale":"2024-05-20T17:58:43.571437+0000","last_undegraded":"2024-05-20T17:58:43.571437+0000","last_fullsized":"2024-05-20T17:58:43.571437+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:24.954772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.114092+0000","last_change":"2024-05-20T17:57:49.051693+0000","last_active":"2024-05-20T17:58:44.114092+0000","last_peered":"2024-05-20T17:58:44.114092+0000","last_clean":"2024-05-20T17:58:44.114092+0000","last_became_active":"2024-05-20T17:57:49.051215+0000","last_became_peered":"2024-05-20T17:57:49.051215+0000","last_unstale":"2024-05-20T17:58:44.114092+0000","last_undegraded":"2024-05-20T17:58:44.114092+0000","last_fullsized":"2024-05-20T17:58:44.114092+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:52:35.406908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691588+0000","last_change":"2024-05-20T17:56:44.709548+0000","last_active":"2024-05-20T17:58:43.691588+0000","last_peered":"2024-05-20T17:58:43.691588+0000","last_clean":"2024-05-20T17:58:43.691588+0000","last_became_active":"2024-05-20T17:56:44.709209+0000","last_became_peered":"2024-05-20T17:56:44.709209+0000","last_unstale":"2024-05-20T17:58:43.691588+0000","last_undegraded":"2024-05-20T17:58:43.691588+0000","last_fullsized":"2024-05-20T17:58:43.691588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:28:13.118336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856926+0000","last_change":"2024-05-20T17:56:45.755141+0000","last_active":"2024-05-20T17:58:43.856926+0000","last_peered":"2024-05-20T17:58:43.856926+0000","last_clean":"2024-05-20T17:58:43.856926+0000","last_became_active":"2024-05-20T17:56:45.754786+0000","last_became_peered":"2024-05-20T17:56:45.754786+0000","last_unstale":"2024-05-20T17:58:43.856926+0000","last_undegraded":"2024-05-20T17:58:43.856926+0000","last_fullsized":"2024-05-20T17:58:43.856926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:45:21.044602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":296,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.171070+0000","last_change":"2024-05-20T17:56:44.713578+0000","last_active":"2024-05-20T17:58:44.171070+0000","last_peered":"2024-05-20T17:58:44.171070+0000","last_clean":"2024-05-20T17:58:44.171070+0000","last_became_active":"2024-05-20T17:56:44.713203+0000","last_became_peered":"2024-05-20T17:56:44.713203+0000","last_unstale":"2024-05-20T17:58:44.171070+0000","last_undegraded":"2024-05-20T17:58:44.171070+0000","last_fullsized":"2024-05-20T17:58:44.171070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:36:54.386036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":242,"num_read_kb":187,"num_write":33,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681854+0000","last_change":"2024-05-20T17:56:45.752023+0000","last_active":"2024-05-20T17:58:43.681854+0000","last_peered":"2024-05-20T17:58:43.681854+0000","last_clean":"2024-05-20T17:58:43.681854+0000","last_became_active":"2024-05-20T17:56:45.751672+0000","last_became_peered":"2024-05-20T17:56:45.751672+0000","last_unstale":"2024-05-20T17:58:43.681854+0000","last_undegraded":"2024-05-20T17:58:43.681854+0000","last_fullsized":"2024-05-20T17:58:43.681854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:23:49.127016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569077+0000","last_change":"2024-05-20T17:56:44.717336+0000","last_active":"2024-05-20T17:58:43.569077+0000","last_peered":"2024-05-20T17:58:43.569077+0000","last_clean":"2024-05-20T17:58:43.569077+0000","last_became_active":"2024-05-20T17:56:44.717016+0000","last_became_peered":"2024-05-20T17:56:44.717016+0000","last_unstale":"2024-05-20T17:58:43.569077+0000","last_undegraded":"2024-05-20T17:58:43.569077+0000","last_fullsized":"2024-05-20T17:58:43.569077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:50:46.433461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113461+0000","last_change":"2024-05-20T17:56:45.752411+0000","last_active":"2024-05-20T17:58:44.113461+0000","last_peered":"2024-05-20T17:58:44.113461+0000","last_clean":"2024-05-20T17:58:44.113461+0000","last_became_active":"2024-05-20T17:56:45.752078+0000","last_became_peered":"2024-05-20T17:56:45.752078+0000","last_unstale":"2024-05-20T17:58:44.113461+0000","last_undegraded":"2024-05-20T17:58:44.113461+0000","last_fullsized":"2024-05-20T17:58:44.113461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:22:15.560666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691295+0000","last_change":"2024-05-20T17:56:44.723262+0000","last_active":"2024-05-20T17:58:43.691295+0000","last_peered":"2024-05-20T17:58:43.691295+0000","last_clean":"2024-05-20T17:58:43.691295+0000","last_became_active":"2024-05-20T17:56:44.722865+0000","last_became_peered":"2024-05-20T17:56:44.722865+0000","last_unstale":"2024-05-20T17:58:43.691295+0000","last_undegraded":"2024-05-20T17:58:43.691295+0000","last_fullsized":"2024-05-20T17:58:43.691295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:15:02.820718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571429+0000","last_change":"2024-05-20T17:57:49.052110+0000","last_active":"2024-05-20T17:58:43.571429+0000","last_peered":"2024-05-20T17:58:43.571429+0000","last_clean":"2024-05-20T17:58:43.571429+0000","last_became_active":"2024-05-20T17:57:49.051356+0000","last_became_peered":"2024-05-20T17:57:49.051356+0000","last_unstale":"2024-05-20T17:58:43.571429+0000","last_undegraded":"2024-05-20T17:58:43.571429+0000","last_fullsized":"2024-05-20T17:58:43.571429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:32.899615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113252+0000","last_change":"2024-05-20T17:56:44.718396+0000","last_active":"2024-05-20T17:58:44.113252+0000","last_peered":"2024-05-20T17:58:44.113252+0000","last_clean":"2024-05-20T17:58:44.113252+0000","last_became_active":"2024-05-20T17:56:44.718031+0000","last_became_peered":"2024-05-20T17:56:44.718031+0000","last_unstale":"2024-05-20T17:58:44.113252+0000","last_undegraded":"2024-05-20T17:58:44.113252+0000","last_fullsized":"2024-05-20T17:58:44.113252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:42:40.955520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571035+0000","last_change":"2024-05-20T17:56:45.759302+0000","last_active":"2024-05-20T17:58:43.571035+0000","last_peered":"2024-05-20T17:58:43.571035+0000","last_clean":"2024-05-20T17:58:43.571035+0000","last_became_active":"2024-05-20T17:56:45.758889+0000","last_became_peered":"2024-05-20T17:56:45.758889+0000","last_unstale":"2024-05-20T17:58:43.571035+0000","last_undegraded":"2024-05-20T17:58:43.571035+0000","last_fullsized":"2024-05-20T17:58:43.571035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:40.685471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681563+0000","last_change":"2024-05-20T17:56:44.726502+0000","last_active":"2024-05-20T17:58:43.681563+0000","last_peered":"2024-05-20T17:58:43.681563+0000","last_clean":"2024-05-20T17:58:43.681563+0000","last_became_active":"2024-05-20T17:56:44.726168+0000","last_became_peered":"2024-05-20T17:56:44.726168+0000","last_unstale":"2024-05-20T17:58:43.681563+0000","last_undegraded":"2024-05-20T17:58:43.681563+0000","last_fullsized":"2024-05-20T17:58:43.681563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:45:14.523968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691604+0000","last_change":"2024-05-20T17:56:45.759355+0000","last_active":"2024-05-20T17:58:43.691604+0000","last_peered":"2024-05-20T17:58:43.691604+0000","last_clean":"2024-05-20T17:58:43.691604+0000","last_became_active":"2024-05-20T17:56:45.758920+0000","last_became_peered":"2024-05-20T17:56:45.758920+0000","last_unstale":"2024-05-20T17:58:43.691604+0000","last_undegraded":"2024-05-20T17:58:43.691604+0000","last_fullsized":"2024-05-20T17:58:43.691604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:46:02.372250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682113+0000","last_change":"2024-05-20T17:56:44.718507+0000","last_active":"2024-05-20T17:58:43.682113+0000","last_peered":"2024-05-20T17:58:43.682113+0000","last_clean":"2024-05-20T17:58:43.682113+0000","last_became_active":"2024-05-20T17:56:44.718188+0000","last_became_peered":"2024-05-20T17:56:44.718188+0000","last_unstale":"2024-05-20T17:58:43.682113+0000","last_undegraded":"2024-05-20T17:58:43.682113+0000","last_fullsized":"2024-05-20T17:58:43.682113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:43:59.792942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859084+0000","last_change":"2024-05-20T17:56:45.751361+0000","last_active":"2024-05-20T17:58:43.859084+0000","last_peered":"2024-05-20T17:58:43.859084+0000","last_clean":"2024-05-20T17:58:43.859084+0000","last_became_active":"2024-05-20T17:56:45.750032+0000","last_became_peered":"2024-05-20T17:56:45.750032+0000","last_unstale":"2024-05-20T17:58:43.859084+0000","last_undegraded":"2024-05-20T17:58:43.859084+0000","last_fullsized":"2024-05-20T17:58:43.859084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:01:42.729156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570328+0000","last_change":"2024-05-20T17:56:44.714404+0000","last_active":"2024-05-20T17:58:43.570328+0000","last_peered":"2024-05-20T17:58:43.570328+0000","last_clean":"2024-05-20T17:58:43.570328+0000","last_became_active":"2024-05-20T17:56:44.714072+0000","last_became_peered":"2024-05-20T17:56:44.714072+0000","last_unstale":"2024-05-20T17:58:43.570328+0000","last_undegraded":"2024-05-20T17:58:43.570328+0000","last_fullsized":"2024-05-20T17:58:43.570328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:39:53.983513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694551+0000","last_change":"2024-05-20T17:56:45.749785+0000","last_active":"2024-05-20T17:58:43.694551+0000","last_peered":"2024-05-20T17:58:43.694551+0000","last_clean":"2024-05-20T17:58:43.694551+0000","last_became_active":"2024-05-20T17:56:45.749457+0000","last_became_peered":"2024-05-20T17:56:45.749457+0000","last_unstale":"2024-05-20T17:58:43.694551+0000","last_undegraded":"2024-05-20T17:58:43.694551+0000","last_fullsized":"2024-05-20T17:58:43.694551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:53:05.136882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571253+0000","last_change":"2024-05-20T17:56:44.722787+0000","last_active":"2024-05-20T17:58:43.571253+0000","last_peered":"2024-05-20T17:58:43.571253+0000","last_clean":"2024-05-20T17:58:43.571253+0000","last_became_active":"2024-05-20T17:56:44.722214+0000","last_became_peered":"2024-05-20T17:56:44.722214+0000","last_unstale":"2024-05-20T17:58:43.571253+0000","last_undegraded":"2024-05-20T17:58:43.571253+0000","last_fullsized":"2024-05-20T17:58:43.571253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:40:16.705718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694565+0000","last_change":"2024-05-20T17:56:45.757515+0000","last_active":"2024-05-20T17:58:43.694565+0000","last_peered":"2024-05-20T17:58:43.694565+0000","last_clean":"2024-05-20T17:58:43.694565+0000","last_became_active":"2024-05-20T17:56:45.757174+0000","last_became_peered":"2024-05-20T17:56:45.757174+0000","last_unstale":"2024-05-20T17:58:43.694565+0000","last_undegraded":"2024-05-20T17:58:43.694565+0000","last_fullsized":"2024-05-20T17:58:43.694565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:18:14.154827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569893+0000","last_change":"2024-05-20T17:57:49.051489+0000","last_active":"2024-05-20T17:58:43.569893+0000","last_peered":"2024-05-20T17:58:43.569893+0000","last_clean":"2024-05-20T17:58:43.569893+0000","last_became_active":"2024-05-20T17:57:49.051070+0000","last_became_peered":"2024-05-20T17:57:49.051070+0000","last_unstale":"2024-05-20T17:58:43.569893+0000","last_undegraded":"2024-05-20T17:58:43.569893+0000","last_fullsized":"2024-05-20T17:58:43.569893+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:18.745185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113217+0000","last_change":"2024-05-20T17:56:44.724642+0000","last_active":"2024-05-20T17:58:44.113217+0000","last_peered":"2024-05-20T17:58:44.113217+0000","last_clean":"2024-05-20T17:58:44.113217+0000","last_became_active":"2024-05-20T17:56:44.724311+0000","last_became_peered":"2024-05-20T17:56:44.724311+0000","last_unstale":"2024-05-20T17:58:44.113217+0000","last_undegraded":"2024-05-20T17:58:44.113217+0000","last_fullsized":"2024-05-20T17:58:44.113217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:47:35.336236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681864+0000","last_change":"2024-05-20T17:56:45.758466+0000","last_active":"2024-05-20T17:58:43.681864+0000","last_peered":"2024-05-20T17:58:43.681864+0000","last_clean":"2024-05-20T17:58:43.681864+0000","last_became_active":"2024-05-20T17:56:45.758126+0000","last_became_peered":"2024-05-20T17:56:45.758126+0000","last_unstale":"2024-05-20T17:58:43.681864+0000","last_undegraded":"2024-05-20T17:58:43.681864+0000","last_fullsized":"2024-05-20T17:58:43.681864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:36.830828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113214+0000","last_change":"2024-05-20T17:56:44.711173+0000","last_active":"2024-05-20T17:58:44.113214+0000","last_peered":"2024-05-20T17:58:44.113214+0000","last_clean":"2024-05-20T17:58:44.113214+0000","last_became_active":"2024-05-20T17:56:44.710778+0000","last_became_peered":"2024-05-20T17:56:44.710778+0000","last_unstale":"2024-05-20T17:58:44.113214+0000","last_undegraded":"2024-05-20T17:58:44.113214+0000","last_fullsized":"2024-05-20T17:58:44.113214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:20:14.150346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571144+0000","last_change":"2024-05-20T17:55:34.871483+0000","last_active":"2024-05-20T17:58:43.571144+0000","last_peered":"2024-05-20T17:58:43.571144+0000","last_clean":"2024-05-20T17:58:43.571144+0000","last_became_active":"2024-05-20T17:55:34.871028+0000","last_became_peered":"2024-05-20T17:55:34.871028+0000","last_unstale":"2024-05-20T17:58:43.571144+0000","last_undegraded":"2024-05-20T17:58:43.571144+0000","last_fullsized":"2024-05-20T17:58:43.571144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_clean_scrub_stamp":"2024-05-20T17:52:08.599715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:29:40.159577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113167+0000","last_change":"2024-05-20T17:56:45.747725+0000","last_active":"2024-05-20T17:58:44.113167+0000","last_peered":"2024-05-20T17:58:44.113167+0000","last_clean":"2024-05-20T17:58:44.113167+0000","last_became_active":"2024-05-20T17:56:45.747232+0000","last_became_peered":"2024-05-20T17:56:45.747232+0000","last_unstale":"2024-05-20T17:58:44.113167+0000","last_undegraded":"2024-05-20T17:58:44.113167+0000","last_fullsized":"2024-05-20T17:58:44.113167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:09:26.589905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571349+0000","last_change":"2024-05-20T17:56:44.721004+0000","last_active":"2024-05-20T17:58:43.571349+0000","last_peered":"2024-05-20T17:58:43.571349+0000","last_clean":"2024-05-20T17:58:43.571349+0000","last_became_active":"2024-05-20T17:56:44.720620+0000","last_became_peered":"2024-05-20T17:56:44.720620+0000","last_unstale":"2024-05-20T17:58:43.571349+0000","last_undegraded":"2024-05-20T17:58:43.571349+0000","last_fullsized":"2024-05-20T17:58:43.571349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:42:14.710663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681867+0000","last_change":"2024-05-20T17:56:44.714754+0000","last_active":"2024-05-20T17:58:43.681867+0000","last_peered":"2024-05-20T17:58:43.681867+0000","last_clean":"2024-05-20T17:58:43.681867+0000","last_became_active":"2024-05-20T17:56:44.714378+0000","last_became_peered":"2024-05-20T17:56:44.714378+0000","last_unstale":"2024-05-20T17:58:43.681867+0000","last_undegraded":"2024-05-20T17:58:43.681867+0000","last_fullsized":"2024-05-20T17:58:43.681867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:46:53.969960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113922+0000","last_change":"2024-05-20T17:56:45.742415+0000","last_active":"2024-05-20T17:58:44.113922+0000","last_peered":"2024-05-20T17:58:44.113922+0000","last_clean":"2024-05-20T17:58:44.113922+0000","last_became_active":"2024-05-20T17:56:45.741941+0000","last_became_peered":"2024-05-20T17:56:45.741941+0000","last_unstale":"2024-05-20T17:58:44.113922+0000","last_undegraded":"2024-05-20T17:58:44.113922+0000","last_fullsized":"2024-05-20T17:58:44.113922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:20:03.191021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'43","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.813704+0000","last_change":"2024-05-20T17:56:44.724361+0000","last_active":"2024-05-20T17:58:43.813704+0000","last_peered":"2024-05-20T17:58:43.813704+0000","last_clean":"2024-05-20T17:58:43.813704+0000","last_became_active":"2024-05-20T17:56:44.724033+0000","last_became_peered":"2024-05-20T17:56:44.724033+0000","last_unstale":"2024-05-20T17:58:43.813704+0000","last_undegraded":"2024-05-20T17:58:43.813704+0000","last_fullsized":"2024-05-20T17:58:43.813704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:10:13.756606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570095+0000","last_change":"2024-05-20T17:57:49.051338+0000","last_active":"2024-05-20T17:58:43.570095+0000","last_peered":"2024-05-20T17:58:43.570095+0000","last_clean":"2024-05-20T17:58:43.570095+0000","last_became_active":"2024-05-20T17:57:49.050618+0000","last_became_peered":"2024-05-20T17:57:49.050618+0000","last_unstale":"2024-05-20T17:58:43.570095+0000","last_undegraded":"2024-05-20T17:58:43.570095+0000","last_fullsized":"2024-05-20T17:58:43.570095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:02:09.846175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113481+0000","last_change":"2024-05-20T17:56:44.716296+0000","last_active":"2024-05-20T17:58:44.113481+0000","last_peered":"2024-05-20T17:58:44.113481+0000","last_clean":"2024-05-20T17:58:44.113481+0000","last_became_active":"2024-05-20T17:56:44.715952+0000","last_became_peered":"2024-05-20T17:56:44.715952+0000","last_unstale":"2024-05-20T17:58:44.113481+0000","last_undegraded":"2024-05-20T17:58:44.113481+0000","last_fullsized":"2024-05-20T17:58:44.113481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:52:37.986399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113744+0000","last_change":"2024-05-20T17:56:45.751874+0000","last_active":"2024-05-20T17:58:44.113744+0000","last_peered":"2024-05-20T17:58:44.113744+0000","last_clean":"2024-05-20T17:58:44.113744+0000","last_became_active":"2024-05-20T17:56:45.751534+0000","last_became_peered":"2024-05-20T17:56:45.751534+0000","last_unstale":"2024-05-20T17:58:44.113744+0000","last_undegraded":"2024-05-20T17:58:44.113744+0000","last_fullsized":"2024-05-20T17:58:44.113744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:05:27.248115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856682+0000","last_change":"2024-05-20T17:56:44.724271+0000","last_active":"2024-05-20T17:58:43.856682+0000","last_peered":"2024-05-20T17:58:43.856682+0000","last_clean":"2024-05-20T17:58:43.856682+0000","last_became_active":"2024-05-20T17:56:44.723947+0000","last_became_peered":"2024-05-20T17:56:44.723947+0000","last_unstale":"2024-05-20T17:58:43.856682+0000","last_undegraded":"2024-05-20T17:58:43.856682+0000","last_fullsized":"2024-05-20T17:58:43.856682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:32:13.482059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681563+0000","last_change":"2024-05-20T17:56:45.751296+0000","last_active":"2024-05-20T17:58:43.681563+0000","last_peered":"2024-05-20T17:58:43.681563+0000","last_clean":"2024-05-20T17:58:43.681563+0000","last_became_active":"2024-05-20T17:56:45.748035+0000","last_became_peered":"2024-05-20T17:56:45.748035+0000","last_unstale":"2024-05-20T17:58:43.681563+0000","last_undegraded":"2024-05-20T17:58:43.681563+0000","last_fullsized":"2024-05-20T17:58:43.681563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:37:43.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681705+0000","last_change":"2024-05-20T17:56:44.720461+0000","last_active":"2024-05-20T17:58:43.681705+0000","last_peered":"2024-05-20T17:58:43.681705+0000","last_clean":"2024-05-20T17:58:43.681705+0000","last_became_active":"2024-05-20T17:56:44.720115+0000","last_became_peered":"2024-05-20T17:56:44.720115+0000","last_unstale":"2024-05-20T17:58:43.681705+0000","last_undegraded":"2024-05-20T17:58:43.681705+0000","last_fullsized":"2024-05-20T17:58:43.681705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:14:47.330193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.858870+0000","last_change":"2024-05-20T17:56:45.745025+0000","last_active":"2024-05-20T17:58:43.858870+0000","last_peered":"2024-05-20T17:58:43.858870+0000","last_clean":"2024-05-20T17:58:43.858870+0000","last_became_active":"2024-05-20T17:56:45.744626+0000","last_became_peered":"2024-05-20T17:56:45.744626+0000","last_unstale":"2024-05-20T17:58:43.858870+0000","last_undegraded":"2024-05-20T17:58:43.858870+0000","last_fullsized":"2024-05-20T17:58:43.858870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:08:51.052423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569045+0000","last_change":"2024-05-20T17:56:44.726099+0000","last_active":"2024-05-20T17:58:43.569045+0000","last_peered":"2024-05-20T17:58:43.569045+0000","last_clean":"2024-05-20T17:58:43.569045+0000","last_became_active":"2024-05-20T17:56:44.725714+0000","last_became_peered":"2024-05-20T17:56:44.725714+0000","last_unstale":"2024-05-20T17:58:43.569045+0000","last_undegraded":"2024-05-20T17:58:43.569045+0000","last_fullsized":"2024-05-20T17:58:43.569045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:51.326907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682248+0000","last_change":"2024-05-20T17:56:45.757376+0000","last_active":"2024-05-20T17:58:43.682248+0000","last_peered":"2024-05-20T17:58:43.682248+0000","last_clean":"2024-05-20T17:58:43.682248+0000","last_became_active":"2024-05-20T17:56:45.757053+0000","last_became_peered":"2024-05-20T17:56:45.757053+0000","last_unstale":"2024-05-20T17:58:43.682248+0000","last_undegraded":"2024-05-20T17:58:43.682248+0000","last_fullsized":"2024-05-20T17:58:43.682248+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:50:27.727055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681658+0000","last_change":"2024-05-20T17:56:44.726341+0000","last_active":"2024-05-20T17:58:43.681658+0000","last_peered":"2024-05-20T17:58:43.681658+0000","last_clean":"2024-05-20T17:58:43.681658+0000","last_became_active":"2024-05-20T17:56:44.726008+0000","last_became_peered":"2024-05-20T17:56:44.726008+0000","last_unstale":"2024-05-20T17:58:43.681658+0000","last_undegraded":"2024-05-20T17:58:43.681658+0000","last_fullsized":"2024-05-20T17:58:43.681658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:25:07.520179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571377+0000","last_change":"2024-05-20T17:56:45.760469+0000","last_active":"2024-05-20T17:58:43.571377+0000","last_peered":"2024-05-20T17:58:43.571377+0000","last_clean":"2024-05-20T17:58:43.571377+0000","last_became_active":"2024-05-20T17:56:45.760100+0000","last_became_peered":"2024-05-20T17:56:45.760100+0000","last_unstale":"2024-05-20T17:58:43.571377+0000","last_undegraded":"2024-05-20T17:58:43.571377+0000","last_fullsized":"2024-05-20T17:58:43.571377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:54:06.309355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113346+0000","last_change":"2024-05-20T17:57:49.051674+0000","last_active":"2024-05-20T17:58:44.113346+0000","last_peered":"2024-05-20T17:58:44.113346+0000","last_clean":"2024-05-20T17:58:44.113346+0000","last_became_active":"2024-05-20T17:57:49.051182+0000","last_became_peered":"2024-05-20T17:57:49.051182+0000","last_unstale":"2024-05-20T17:58:44.113346+0000","last_undegraded":"2024-05-20T17:58:44.113346+0000","last_fullsized":"2024-05-20T17:58:44.113346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:29:35.458339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571593+0000","last_change":"2024-05-20T17:56:45.757302+0000","last_active":"2024-05-20T17:58:43.571593+0000","last_peered":"2024-05-20T17:58:43.571593+0000","last_clean":"2024-05-20T17:58:43.571593+0000","last_became_active":"2024-05-20T17:56:45.754169+0000","last_became_peered":"2024-05-20T17:56:45.754169+0000","last_unstale":"2024-05-20T17:58:43.571593+0000","last_undegraded":"2024-05-20T17:58:43.571593+0000","last_fullsized":"2024-05-20T17:58:43.571593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:03:10.794342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.807590+0000","last_change":"2024-05-20T17:56:44.724478+0000","last_active":"2024-05-20T17:58:43.807590+0000","last_peered":"2024-05-20T17:58:43.807590+0000","last_clean":"2024-05-20T17:58:43.807590+0000","last_became_active":"2024-05-20T17:56:44.723927+0000","last_became_peered":"2024-05-20T17:56:44.723927+0000","last_unstale":"2024-05-20T17:58:43.807590+0000","last_undegraded":"2024-05-20T17:58:43.807590+0000","last_fullsized":"2024-05-20T17:58:43.807590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:04:12.003957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571188+0000","last_change":"2024-05-20T17:56:45.758135+0000","last_active":"2024-05-20T17:58:43.571188+0000","last_peered":"2024-05-20T17:58:43.571188+0000","last_clean":"2024-05-20T17:58:43.571188+0000","last_became_active":"2024-05-20T17:56:45.757358+0000","last_became_peered":"2024-05-20T17:56:45.757358+0000","last_unstale":"2024-05-20T17:58:43.571188+0000","last_undegraded":"2024-05-20T17:58:43.571188+0000","last_fullsized":"2024-05-20T17:58:43.571188+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:13:06.435055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.114107+0000","last_change":"2024-05-20T17:56:44.719822+0000","last_active":"2024-05-20T17:58:44.114107+0000","last_peered":"2024-05-20T17:58:44.114107+0000","last_clean":"2024-05-20T17:58:44.114107+0000","last_became_active":"2024-05-20T17:56:44.719454+0000","last_became_peered":"2024-05-20T17:56:44.719454+0000","last_unstale":"2024-05-20T17:58:44.114107+0000","last_undegraded":"2024-05-20T17:58:44.114107+0000","last_fullsized":"2024-05-20T17:58:44.114107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:03:21.048330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571553+0000","last_change":"2024-05-20T17:56:45.753510+0000","last_active":"2024-05-20T17:58:43.571553+0000","last_peered":"2024-05-20T17:58:43.571553+0000","last_clean":"2024-05-20T17:58:43.571553+0000","last_became_active":"2024-05-20T17:56:45.752718+0000","last_became_peered":"2024-05-20T17:56:45.752718+0000","last_unstale":"2024-05-20T17:58:43.571553+0000","last_undegraded":"2024-05-20T17:58:43.571553+0000","last_fullsized":"2024-05-20T17:58:43.571553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:05:44.677138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.858961+0000","last_change":"2024-05-20T17:56:45.752528+0000","last_active":"2024-05-20T17:58:43.858961+0000","last_peered":"2024-05-20T17:58:43.858961+0000","last_clean":"2024-05-20T17:58:43.858961+0000","last_became_active":"2024-05-20T17:56:45.752171+0000","last_became_peered":"2024-05-20T17:56:45.752171+0000","last_unstale":"2024-05-20T17:58:43.858961+0000","last_undegraded":"2024-05-20T17:58:43.858961+0000","last_fullsized":"2024-05-20T17:58:43.858961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:12:55.826122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569068+0000","last_change":"2024-05-20T17:56:45.749228+0000","last_active":"2024-05-20T17:58:43.569068+0000","last_peered":"2024-05-20T17:58:43.569068+0000","last_clean":"2024-05-20T17:58:43.569068+0000","last_became_active":"2024-05-20T17:56:45.748886+0000","last_became_peered":"2024-05-20T17:56:45.748886+0000","last_unstale":"2024-05-20T17:58:43.569068+0000","last_undegraded":"2024-05-20T17:58:43.569068+0000","last_fullsized":"2024-05-20T17:58:43.569068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:23:10.569720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569035+0000","last_change":"2024-05-20T17:56:45.751105+0000","last_active":"2024-05-20T17:58:43.569035+0000","last_peered":"2024-05-20T17:58:43.569035+0000","last_clean":"2024-05-20T17:58:43.569035+0000","last_became_active":"2024-05-20T17:56:45.750754+0000","last_became_peered":"2024-05-20T17:56:45.750754+0000","last_unstale":"2024-05-20T17:58:43.569035+0000","last_undegraded":"2024-05-20T17:58:43.569035+0000","last_fullsized":"2024-05-20T17:58:43.569035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:57:48.327428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113562+0000","last_change":"2024-05-20T17:57:49.050574+0000","last_active":"2024-05-20T17:58:44.113562+0000","last_peered":"2024-05-20T17:58:44.113562+0000","last_clean":"2024-05-20T17:58:44.113562+0000","last_became_active":"2024-05-20T17:57:49.050037+0000","last_became_peered":"2024-05-20T17:57:49.050037+0000","last_unstale":"2024-05-20T17:58:44.113562+0000","last_undegraded":"2024-05-20T17:58:44.113562+0000","last_fullsized":"2024-05-20T17:58:44.113562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:01:56.453842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113800+0000","last_change":"2024-05-20T17:56:45.746122+0000","last_active":"2024-05-20T17:58:44.113800+0000","last_peered":"2024-05-20T17:58:44.113800+0000","last_clean":"2024-05-20T17:58:44.113800+0000","last_became_active":"2024-05-20T17:56:45.745773+0000","last_became_peered":"2024-05-20T17:56:45.745773+0000","last_unstale":"2024-05-20T17:58:44.113800+0000","last_undegraded":"2024-05-20T17:58:44.113800+0000","last_fullsized":"2024-05-20T17:58:44.113800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:21:27.909974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859042+0000","last_change":"2024-05-20T17:56:45.755018+0000","last_active":"2024-05-20T17:58:43.859042+0000","last_peered":"2024-05-20T17:58:43.859042+0000","last_clean":"2024-05-20T17:58:43.859042+0000","last_became_active":"2024-05-20T17:56:45.754615+0000","last_became_peered":"2024-05-20T17:56:45.754615+0000","last_unstale":"2024-05-20T17:58:43.859042+0000","last_undegraded":"2024-05-20T17:58:43.859042+0000","last_fullsized":"2024-05-20T17:58:43.859042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:36:39.100566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569612+0000","last_change":"2024-05-20T17:56:45.751091+0000","last_active":"2024-05-20T17:58:43.569612+0000","last_peered":"2024-05-20T17:58:43.569612+0000","last_clean":"2024-05-20T17:58:43.569612+0000","last_became_active":"2024-05-20T17:56:45.748573+0000","last_became_peered":"2024-05-20T17:56:45.748573+0000","last_unstale":"2024-05-20T17:58:43.569612+0000","last_undegraded":"2024-05-20T17:58:43.569612+0000","last_fullsized":"2024-05-20T17:58:43.569612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:11:16.140967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682866+0000","last_change":"2024-05-20T17:56:45.756373+0000","last_active":"2024-05-20T17:58:43.682866+0000","last_peered":"2024-05-20T17:58:43.682866+0000","last_clean":"2024-05-20T17:58:43.682866+0000","last_became_active":"2024-05-20T17:56:45.755993+0000","last_became_peered":"2024-05-20T17:56:45.755993+0000","last_unstale":"2024-05-20T17:58:43.682866+0000","last_undegraded":"2024-05-20T17:58:43.682866+0000","last_fullsized":"2024-05-20T17:58:43.682866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:34:44.491341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691184+0000","last_change":"2024-05-20T17:56:45.751866+0000","last_active":"2024-05-20T17:58:43.691184+0000","last_peered":"2024-05-20T17:58:43.691184+0000","last_clean":"2024-05-20T17:58:43.691184+0000","last_became_active":"2024-05-20T17:56:45.751518+0000","last_became_peered":"2024-05-20T17:56:45.751518+0000","last_unstale":"2024-05-20T17:58:43.691184+0000","last_undegraded":"2024-05-20T17:58:43.691184+0000","last_fullsized":"2024-05-20T17:58:43.691184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:38:18.736177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681682+0000","last_change":"2024-05-20T17:56:45.749903+0000","last_active":"2024-05-20T17:58:43.681682+0000","last_peered":"2024-05-20T17:58:43.681682+0000","last_clean":"2024-05-20T17:58:43.681682+0000","last_became_active":"2024-05-20T17:56:45.749500+0000","last_became_peered":"2024-05-20T17:56:45.749500+0000","last_unstale":"2024-05-20T17:58:43.681682+0000","last_undegraded":"2024-05-20T17:58:43.681682+0000","last_fullsized":"2024-05-20T17:58:43.681682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:39:34.875111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856827+0000","last_change":"2024-05-20T17:56:45.747370+0000","last_active":"2024-05-20T17:58:43.856827+0000","last_peered":"2024-05-20T17:58:43.856827+0000","last_clean":"2024-05-20T17:58:43.856827+0000","last_became_active":"2024-05-20T17:56:45.747036+0000","last_became_peered":"2024-05-20T17:56:45.747036+0000","last_unstale":"2024-05-20T17:58:43.856827+0000","last_undegraded":"2024-05-20T17:58:43.856827+0000","last_fullsized":"2024-05-20T17:58:43.856827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:02:02.220522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113692+0000","last_change":"2024-05-20T17:56:45.752762+0000","last_active":"2024-05-20T17:58:44.113692+0000","last_peered":"2024-05-20T17:58:44.113692+0000","last_clean":"2024-05-20T17:58:44.113692+0000","last_became_active":"2024-05-20T17:56:45.752345+0000","last_became_peered":"2024-05-20T17:56:45.752345+0000","last_unstale":"2024-05-20T17:58:44.113692+0000","last_undegraded":"2024-05-20T17:58:44.113692+0000","last_fullsized":"2024-05-20T17:58:44.113692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:14:49.861456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.858880+0000","last_change":"2024-05-20T17:56:45.750229+0000","last_active":"2024-05-20T17:58:43.858880+0000","last_peered":"2024-05-20T17:58:43.858880+0000","last_clean":"2024-05-20T17:58:43.858880+0000","last_became_active":"2024-05-20T17:56:45.749872+0000","last_became_peered":"2024-05-20T17:56:45.749872+0000","last_unstale":"2024-05-20T17:58:43.858880+0000","last_undegraded":"2024-05-20T17:58:43.858880+0000","last_fullsized":"2024-05-20T17:58:43.858880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:56:58.015339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106947,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":593372,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":340,"num_read_kb":272,"num_write":196,"num_write_kb":1403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":1780116,"data_compressed":81486,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":462373,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612649,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1122304,"data_stored":1261374,"data_compressed":52401,"data_compressed_allocated":69632,"data_compressed_original":745472,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808879,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":865135,"data_compressed":27882,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070516,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":864575,"data_compressed":27882,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":1062404,"data_compressed":44385,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":462490,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":539677,"data_compressed":10596,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":471427,"data_compressed":8736,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":539129,"data_compressed":10596,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":462711,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":462395,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":1062004,"data_compressed":44385,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68340,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67792,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393238,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":199552,"data_compressed":8016,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-20T17:58:44.662 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-20T17:58:44.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.815+0000 7f09a4e0d640 1 -- 172.21.15.129:0/570704777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09a0106050 msgr2=0x7f09a0108470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.815+0000 7f09a4e0d640 1 --2- 172.21.15.129:0/570704777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09a0106050 0x7f09a0108470 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0988009900 tx=0x7f098802a410 comp rx=0 tx=0).stop 2024-05-20T17:58:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.815+0000 7f09a4e0d640 1 -- 172.21.15.129:0/570704777 shutdown_connections 2024-05-20T17:58:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.815+0000 7f09a4e0d640 1 --2- 172.21.15.129:0/570704777 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09a01089b0 0x7f09a010add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.815+0000 7f09a4e0d640 1 --2- 172.21.15.129:0/570704777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09a0106050 0x7f09a0108470 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.815+0000 7f09a4e0d640 1 --2- 172.21.15.129:0/570704777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a0100ea0 0x7f09a0105b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.815+0000 7f09a4e0d640 1 -- 172.21.15.129:0/570704777 >> 172.21.15.129:0/570704777 conn(0x7f09a00fc2e0 msgr2=0x7f09a00fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.815+0000 7f09a4e0d640 1 -- 172.21.15.129:0/570704777 shutdown_connections 2024-05-20T17:58:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.815+0000 7f09a4e0d640 1 -- 172.21.15.129:0/570704777 wait complete. 2024-05-20T17:58:44.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f09a4e0d640 1 Processor -- start 2024-05-20T17:58:44.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f09a4e0d640 1 -- start start 2024-05-20T17:58:44.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f09a4e0d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09a0100ea0 0x7f09a0199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:44.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f09a4e0d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a0106050 0x7f09a0199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:44.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f09a4e0d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09a01089b0 0x7f09a01a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:44.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f09a4e0d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f09a010e7a0 con 0x7f09a0100ea0 2024-05-20T17:58:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f09a4e0d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09a010e620 con 0x7f09a0106050 2024-05-20T17:58:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f09a4e0d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f09a010e920 con 0x7f09a01089b0 2024-05-20T17:58:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f099e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09a0100ea0 0x7f09a0199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f099e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09a0100ea0 0x7f09a0199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42462/0 (socket says 172.21.15.129:42462) 2024-05-20T17:58:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f099e59c640 1 -- 172.21.15.129:0/2464719099 learned_addr learned my addr 172.21.15.129:0/2464719099 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f099ed9d640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09a01089b0 0x7f09a01a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f099dd9b640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a0106050 0x7f09a0199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f099e59c640 1 -- 172.21.15.129:0/2464719099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09a01089b0 msgr2=0x7f09a01a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f099e59c640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09a01089b0 0x7f09a01a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f099e59c640 1 -- 172.21.15.129:0/2464719099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a0106050 msgr2=0x7f09a0199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f099e59c640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a0106050 0x7f09a0199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:44.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f099e59c640 1 -- 172.21.15.129:0/2464719099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a01a0e50 con 0x7f09a0100ea0 2024-05-20T17:58:44.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.816+0000 7f099e59c640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09a0100ea0 0x7f09a0199310 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f099000eb90 tx=0x7f099000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:44.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.817+0000 7f09977fe640 1 -- 172.21.15.129:0/2464719099 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990010630 con 0x7f09a0100ea0 2024-05-20T17:58:44.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.817+0000 7f09977fe640 1 -- 172.21.15.129:0/2464719099 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099000ee90 con 0x7f09a0100ea0 2024-05-20T17:58:44.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.817+0000 7f09a4e0d640 1 -- 172.21.15.129:0/2464719099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a01a1140 con 0x7f09a0100ea0 2024-05-20T17:58:44.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.817+0000 7f09977fe640 1 -- 172.21.15.129:0/2464719099 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990018b90 con 0x7f09a0100ea0 2024-05-20T17:58:44.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.817+0000 7f09a4e0d640 1 -- 172.21.15.129:0/2464719099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a01a1690 con 0x7f09a0100ea0 2024-05-20T17:58:44.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.818+0000 7f09a4e0d640 1 -- 172.21.15.129:0/2464719099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a00fde00 con 0x7f09a0100ea0 2024-05-20T17:58:44.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.820+0000 7f09977fe640 1 -- 172.21.15.129:0/2464719099 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f09900107d0 con 0x7f09a0100ea0 2024-05-20T17:58:44.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.820+0000 7f09977fe640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f098c0a2990 0x7f098c0a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:44.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.820+0000 7f09977fe640 1 -- 172.21.15.129:0/2464719099 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f09900cf700 con 0x7f09a0100ea0 2024-05-20T17:58:44.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.820+0000 7f099dd9b640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f098c0a2990 0x7f098c0a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:44.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.821+0000 7f099dd9b640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f098c0a2990 0x7f098c0a4ea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0988009230 tx=0x7f0988031400 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:44.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:44.824+0000 7f09977fe640 1 -- 172.21.15.129:0/2464719099 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f099009c1f0 con 0x7f09a0100ea0 2024-05-20T17:58:44.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:44 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-20T17:58:44.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:44 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-20T17:58:44.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:44 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/225046053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:44.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:44 smithi129 ceph-mon[32029]: [20/May/2024:17:58:44] ENGINE Bus STARTING 2024-05-20T17:58:44.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:44 smithi129 ceph-mon[32029]: [20/May/2024:17:58:44] ENGINE Serving on https://172.21.15.129:7150 2024-05-20T17:58:44.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:44 smithi129 ceph-mon[32029]: [20/May/2024:17:58:44] ENGINE Serving on http://172.21.15.129:8765 2024-05-20T17:58:44.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:44 smithi129 ceph-mon[32029]: [20/May/2024:17:58:44] ENGINE Bus STARTED 2024-05-20T17:58:44.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:44 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1562265075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:44.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:44 smithi129 ceph-mon[32029]: mgrmap e21: x(active, since 1.02215s), standbys: z, y 2024-05-20T17:58:44.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:44 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-20T17:58:44.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:44 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-20T17:58:44.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:44 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/225046053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:44.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:44 smithi204 ceph-mon[37076]: [20/May/2024:17:58:44] ENGINE Bus STARTING 2024-05-20T17:58:44.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:44 smithi204 ceph-mon[37076]: [20/May/2024:17:58:44] ENGINE Serving on https://172.21.15.129:7150 2024-05-20T17:58:44.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:44 smithi204 ceph-mon[37076]: [20/May/2024:17:58:44] ENGINE Serving on http://172.21.15.129:8765 2024-05-20T17:58:44.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:44 smithi204 ceph-mon[37076]: [20/May/2024:17:58:44] ENGINE Bus STARTED 2024-05-20T17:58:44.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:44 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1562265075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:44.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:44 smithi204 ceph-mon[37076]: mgrmap e21: x(active, since 1.02215s), standbys: z, y 2024-05-20T17:58:44.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:44 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-20T17:58:44.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:44 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-20T17:58:44.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:44 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/225046053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:44.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:44 smithi154 ceph-mon[36482]: [20/May/2024:17:58:44] ENGINE Bus STARTING 2024-05-20T17:58:44.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:44 smithi154 ceph-mon[36482]: [20/May/2024:17:58:44] ENGINE Serving on https://172.21.15.129:7150 2024-05-20T17:58:44.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:44 smithi154 ceph-mon[36482]: [20/May/2024:17:58:44] ENGINE Serving on http://172.21.15.129:8765 2024-05-20T17:58:44.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:44 smithi154 ceph-mon[36482]: [20/May/2024:17:58:44] ENGINE Bus STARTED 2024-05-20T17:58:44.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:44 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1562265075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:44.998 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:44 smithi154 ceph-mon[36482]: mgrmap e21: x(active, since 1.02215s), standbys: z, y 2024-05-20T17:58:45.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.008+0000 7f09a4e0d640 1 -- 172.21.15.129:0/2464719099 --> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f09a0061f90 con 0x7f098c0a2990 2024-05-20T17:58:45.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.015+0000 7f09977fe640 1 -- 172.21.15.129:0/2464719099 <== mgr.15228 v2:172.21.15.129:6800/3515662734 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257946 (secure 0 0 0) 0x7f09a0061f90 con 0x7f098c0a2990 2024-05-20T17:58:45.015 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:45.016 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-05-20T17:58:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.018+0000 7f09a4e0d640 1 -- 172.21.15.129:0/2464719099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f098c0a2990 msgr2=0x7f098c0a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.018+0000 7f09a4e0d640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f098c0a2990 0x7f098c0a4ea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0988009230 tx=0x7f0988031400 comp rx=0 tx=0).stop 2024-05-20T17:58:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.018+0000 7f09a4e0d640 1 -- 172.21.15.129:0/2464719099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09a0100ea0 msgr2=0x7f09a0199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.018+0000 7f09a4e0d640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09a0100ea0 0x7f09a0199310 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f099000eb90 tx=0x7f099000c6a0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.018+0000 7f09a4e0d640 1 -- 172.21.15.129:0/2464719099 shutdown_connections 2024-05-20T17:58:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.018+0000 7f09a4e0d640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f098c0a2990 0x7f098c0a4ea0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.018+0000 7f09a4e0d640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a0106050 0x7f09a0199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.018+0000 7f09a4e0d640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09a01089b0 0x7f09a01a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.018+0000 7f09a4e0d640 1 --2- 172.21.15.129:0/2464719099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09a0100ea0 0x7f09a0199310 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.019+0000 7f09a4e0d640 1 -- 172.21.15.129:0/2464719099 >> 172.21.15.129:0/2464719099 conn(0x7f09a00fc2e0 msgr2=0x7f09a01045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.019+0000 7f09a4e0d640 1 -- 172.21.15.129:0/2464719099 shutdown_connections 2024-05-20T17:58:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.019+0000 7f09a4e0d640 1 -- 172.21.15.129:0/2464719099 wait complete. 2024-05-20T17:58:45.028 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":3,"stamp":"2024-05-20T17:58:44.594311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1183974,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":476,"num_read_kb":391,"num_write":379,"num_write_kb":4343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365084,"kb_used_data":11744,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576740,"statfs":{"total":1151940427776,"available":1151566581760,"internally_reserved":0,"allocated":12025856,"data_stored":8515694,"data_compressed":250131,"data_compressed_allocated":335872,"data_compressed_original":2965504,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859196+0000","last_change":"2024-05-20T17:56:45.746793+0000","last_active":"2024-05-20T17:58:43.859196+0000","last_peered":"2024-05-20T17:58:43.859196+0000","last_clean":"2024-05-20T17:58:43.859196+0000","last_became_active":"2024-05-20T17:56:45.746244+0000","last_became_peered":"2024-05-20T17:56:45.746244+0000","last_unstale":"2024-05-20T17:58:43.859196+0000","last_undegraded":"2024-05-20T17:58:43.859196+0000","last_fullsized":"2024-05-20T17:58:43.859196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:37:03.703599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113926+0000","last_change":"2024-05-20T17:56:45.744538+0000","last_active":"2024-05-20T17:58:44.113926+0000","last_peered":"2024-05-20T17:58:44.113926+0000","last_clean":"2024-05-20T17:58:44.113926+0000","last_became_active":"2024-05-20T17:56:45.744213+0000","last_became_peered":"2024-05-20T17:56:45.744213+0000","last_unstale":"2024-05-20T17:58:44.113926+0000","last_undegraded":"2024-05-20T17:58:44.113926+0000","last_fullsized":"2024-05-20T17:58:44.113926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:19:26.163003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113332+0000","last_change":"2024-05-20T17:56:45.749725+0000","last_active":"2024-05-20T17:58:44.113332+0000","last_peered":"2024-05-20T17:58:44.113332+0000","last_clean":"2024-05-20T17:58:44.113332+0000","last_became_active":"2024-05-20T17:56:45.749392+0000","last_became_peered":"2024-05-20T17:56:45.749392+0000","last_unstale":"2024-05-20T17:58:44.113332+0000","last_undegraded":"2024-05-20T17:58:44.113332+0000","last_fullsized":"2024-05-20T17:58:44.113332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:43:33.184192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113454+0000","last_change":"2024-05-20T17:56:45.755191+0000","last_active":"2024-05-20T17:58:44.113454+0000","last_peered":"2024-05-20T17:58:44.113454+0000","last_clean":"2024-05-20T17:58:44.113454+0000","last_became_active":"2024-05-20T17:56:45.753524+0000","last_became_peered":"2024-05-20T17:56:45.753524+0000","last_unstale":"2024-05-20T17:58:44.113454+0000","last_undegraded":"2024-05-20T17:58:44.113454+0000","last_fullsized":"2024-05-20T17:58:44.113454+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:11:23.099919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681633+0000","last_change":"2024-05-20T17:56:45.756291+0000","last_active":"2024-05-20T17:58:43.681633+0000","last_peered":"2024-05-20T17:58:43.681633+0000","last_clean":"2024-05-20T17:58:43.681633+0000","last_became_active":"2024-05-20T17:56:45.755910+0000","last_became_peered":"2024-05-20T17:56:45.755910+0000","last_unstale":"2024-05-20T17:58:43.681633+0000","last_undegraded":"2024-05-20T17:58:43.681633+0000","last_fullsized":"2024-05-20T17:58:43.681633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:37:36.941012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694392+0000","last_change":"2024-05-20T17:56:45.748756+0000","last_active":"2024-05-20T17:58:43.694392+0000","last_peered":"2024-05-20T17:58:43.694392+0000","last_clean":"2024-05-20T17:58:43.694392+0000","last_became_active":"2024-05-20T17:56:45.748406+0000","last_became_peered":"2024-05-20T17:56:45.748406+0000","last_unstale":"2024-05-20T17:58:43.694392+0000","last_undegraded":"2024-05-20T17:58:43.694392+0000","last_fullsized":"2024-05-20T17:58:43.694392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:16:36.527832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571098+0000","last_change":"2024-05-20T17:57:49.051459+0000","last_active":"2024-05-20T17:58:43.571098+0000","last_peered":"2024-05-20T17:58:43.571098+0000","last_clean":"2024-05-20T17:58:43.571098+0000","last_became_active":"2024-05-20T17:57:49.050933+0000","last_became_peered":"2024-05-20T17:57:49.050933+0000","last_unstale":"2024-05-20T17:58:43.571098+0000","last_undegraded":"2024-05-20T17:58:43.571098+0000","last_fullsized":"2024-05-20T17:58:43.571098+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:18.214102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681494+0000","last_change":"2024-05-20T17:56:45.757387+0000","last_active":"2024-05-20T17:58:43.681494+0000","last_peered":"2024-05-20T17:58:43.681494+0000","last_clean":"2024-05-20T17:58:43.681494+0000","last_became_active":"2024-05-20T17:56:45.756374+0000","last_became_peered":"2024-05-20T17:56:45.756374+0000","last_unstale":"2024-05-20T17:58:43.681494+0000","last_undegraded":"2024-05-20T17:58:43.681494+0000","last_fullsized":"2024-05-20T17:58:43.681494+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:53:52.751019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.857216+0000","last_change":"2024-05-20T17:56:45.751933+0000","last_active":"2024-05-20T17:58:43.857216+0000","last_peered":"2024-05-20T17:58:43.857216+0000","last_clean":"2024-05-20T17:58:43.857216+0000","last_became_active":"2024-05-20T17:56:45.751414+0000","last_became_peered":"2024-05-20T17:56:45.751414+0000","last_unstale":"2024-05-20T17:58:43.857216+0000","last_undegraded":"2024-05-20T17:58:43.857216+0000","last_fullsized":"2024-05-20T17:58:43.857216+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:56:52.974830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694627+0000","last_change":"2024-05-20T17:56:45.749511+0000","last_active":"2024-05-20T17:58:43.694627+0000","last_peered":"2024-05-20T17:58:43.694627+0000","last_clean":"2024-05-20T17:58:43.694627+0000","last_became_active":"2024-05-20T17:56:45.749211+0000","last_became_peered":"2024-05-20T17:56:45.749211+0000","last_unstale":"2024-05-20T17:58:43.694627+0000","last_undegraded":"2024-05-20T17:58:43.694627+0000","last_fullsized":"2024-05-20T17:58:43.694627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:15.633296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571341+0000","last_change":"2024-05-20T17:56:45.753743+0000","last_active":"2024-05-20T17:58:43.571341+0000","last_peered":"2024-05-20T17:58:43.571341+0000","last_clean":"2024-05-20T17:58:43.571341+0000","last_became_active":"2024-05-20T17:56:45.753378+0000","last_became_peered":"2024-05-20T17:56:45.753378+0000","last_unstale":"2024-05-20T17:58:43.571341+0000","last_undegraded":"2024-05-20T17:58:43.571341+0000","last_fullsized":"2024-05-20T17:58:43.571341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:54.563895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113327+0000","last_change":"2024-05-20T17:56:45.754890+0000","last_active":"2024-05-20T17:58:44.113327+0000","last_peered":"2024-05-20T17:58:44.113327+0000","last_clean":"2024-05-20T17:58:44.113327+0000","last_became_active":"2024-05-20T17:56:45.754333+0000","last_became_peered":"2024-05-20T17:56:45.754333+0000","last_unstale":"2024-05-20T17:58:44.113327+0000","last_undegraded":"2024-05-20T17:58:44.113327+0000","last_fullsized":"2024-05-20T17:58:44.113327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:51.025516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113630+0000","last_change":"2024-05-20T17:56:45.745662+0000","last_active":"2024-05-20T17:58:44.113630+0000","last_peered":"2024-05-20T17:58:44.113630+0000","last_clean":"2024-05-20T17:58:44.113630+0000","last_became_active":"2024-05-20T17:56:45.745281+0000","last_became_peered":"2024-05-20T17:56:45.745281+0000","last_unstale":"2024-05-20T17:58:44.113630+0000","last_undegraded":"2024-05-20T17:58:44.113630+0000","last_fullsized":"2024-05-20T17:58:44.113630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:38:57.057301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.112954+0000","last_change":"2024-05-20T17:56:45.749164+0000","last_active":"2024-05-20T17:58:44.112954+0000","last_peered":"2024-05-20T17:58:44.112954+0000","last_clean":"2024-05-20T17:58:44.112954+0000","last_became_active":"2024-05-20T17:56:45.748822+0000","last_became_peered":"2024-05-20T17:56:45.748822+0000","last_unstale":"2024-05-20T17:58:44.112954+0000","last_undegraded":"2024-05-20T17:58:44.112954+0000","last_fullsized":"2024-05-20T17:58:44.112954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:45:47.487116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.112953+0000","last_change":"2024-05-20T17:56:45.751185+0000","last_active":"2024-05-20T17:58:44.112953+0000","last_peered":"2024-05-20T17:58:44.112953+0000","last_clean":"2024-05-20T17:58:44.112953+0000","last_became_active":"2024-05-20T17:56:45.750851+0000","last_became_peered":"2024-05-20T17:56:45.750851+0000","last_unstale":"2024-05-20T17:58:44.112953+0000","last_undegraded":"2024-05-20T17:58:44.112953+0000","last_fullsized":"2024-05-20T17:58:44.112953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:31:07.792717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856607+0000","last_change":"2024-05-20T17:56:45.748101+0000","last_active":"2024-05-20T17:58:43.856607+0000","last_peered":"2024-05-20T17:58:43.856607+0000","last_clean":"2024-05-20T17:58:43.856607+0000","last_became_active":"2024-05-20T17:56:45.747754+0000","last_became_peered":"2024-05-20T17:56:45.747754+0000","last_unstale":"2024-05-20T17:58:43.856607+0000","last_undegraded":"2024-05-20T17:58:43.856607+0000","last_fullsized":"2024-05-20T17:58:43.856607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:24:54.702601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859350+0000","last_change":"2024-05-20T17:56:45.747497+0000","last_active":"2024-05-20T17:58:43.859350+0000","last_peered":"2024-05-20T17:58:43.859350+0000","last_clean":"2024-05-20T17:58:43.859350+0000","last_became_active":"2024-05-20T17:56:45.747139+0000","last_became_peered":"2024-05-20T17:56:45.747139+0000","last_unstale":"2024-05-20T17:58:43.859350+0000","last_undegraded":"2024-05-20T17:58:43.859350+0000","last_fullsized":"2024-05-20T17:58:43.859350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:39:13.102413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570446+0000","last_change":"2024-05-20T17:56:45.746194+0000","last_active":"2024-05-20T17:58:43.570446+0000","last_peered":"2024-05-20T17:58:43.570446+0000","last_clean":"2024-05-20T17:58:43.570446+0000","last_became_active":"2024-05-20T17:56:45.745809+0000","last_became_peered":"2024-05-20T17:56:45.745809+0000","last_unstale":"2024-05-20T17:58:43.570446+0000","last_undegraded":"2024-05-20T17:58:43.570446+0000","last_fullsized":"2024-05-20T17:58:43.570446+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:26:29.133790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569306+0000","last_change":"2024-05-20T17:56:45.749918+0000","last_active":"2024-05-20T17:58:43.569306+0000","last_peered":"2024-05-20T17:58:43.569306+0000","last_clean":"2024-05-20T17:58:43.569306+0000","last_became_active":"2024-05-20T17:56:45.749560+0000","last_became_peered":"2024-05-20T17:56:45.749560+0000","last_unstale":"2024-05-20T17:58:43.569306+0000","last_undegraded":"2024-05-20T17:58:43.569306+0000","last_fullsized":"2024-05-20T17:58:43.569306+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:33:12.737206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570253+0000","last_change":"2024-05-20T17:57:49.082268+0000","last_active":"2024-05-20T17:58:43.570253+0000","last_peered":"2024-05-20T17:58:43.570253+0000","last_clean":"2024-05-20T17:58:43.570253+0000","last_became_active":"2024-05-20T17:57:49.078471+0000","last_became_peered":"2024-05-20T17:57:49.078471+0000","last_unstale":"2024-05-20T17:58:43.570253+0000","last_undegraded":"2024-05-20T17:58:43.570253+0000","last_fullsized":"2024-05-20T17:58:43.570253+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:50:14.617477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682939+0000","last_change":"2024-05-20T17:56:45.758330+0000","last_active":"2024-05-20T17:58:43.682939+0000","last_peered":"2024-05-20T17:58:43.682939+0000","last_clean":"2024-05-20T17:58:43.682939+0000","last_became_active":"2024-05-20T17:56:45.755873+0000","last_became_peered":"2024-05-20T17:56:45.755873+0000","last_unstale":"2024-05-20T17:58:43.682939+0000","last_undegraded":"2024-05-20T17:58:43.682939+0000","last_fullsized":"2024-05-20T17:58:43.682939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:53:31.653497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.683149+0000","last_change":"2024-05-20T17:56:44.728508+0000","last_active":"2024-05-20T17:58:43.683149+0000","last_peered":"2024-05-20T17:58:43.683149+0000","last_clean":"2024-05-20T17:58:43.683149+0000","last_became_active":"2024-05-20T17:56:44.727875+0000","last_became_peered":"2024-05-20T17:56:44.727875+0000","last_unstale":"2024-05-20T17:58:43.683149+0000","last_undegraded":"2024-05-20T17:58:43.683149+0000","last_fullsized":"2024-05-20T17:58:43.683149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:12:14.349039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.114265+0000","last_change":"2024-05-20T17:56:45.750000+0000","last_active":"2024-05-20T17:58:44.114265+0000","last_peered":"2024-05-20T17:58:44.114265+0000","last_clean":"2024-05-20T17:58:44.114265+0000","last_became_active":"2024-05-20T17:56:45.749644+0000","last_became_peered":"2024-05-20T17:56:45.749644+0000","last_unstale":"2024-05-20T17:58:44.114265+0000","last_undegraded":"2024-05-20T17:58:44.114265+0000","last_fullsized":"2024-05-20T17:58:44.114265+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:32:46.085081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682012+0000","last_change":"2024-05-20T17:56:44.722692+0000","last_active":"2024-05-20T17:58:43.682012+0000","last_peered":"2024-05-20T17:58:43.682012+0000","last_clean":"2024-05-20T17:58:43.682012+0000","last_became_active":"2024-05-20T17:56:44.722170+0000","last_became_peered":"2024-05-20T17:56:44.722170+0000","last_unstale":"2024-05-20T17:58:43.682012+0000","last_undegraded":"2024-05-20T17:58:43.682012+0000","last_fullsized":"2024-05-20T17:58:43.682012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:38:03.679562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856783+0000","last_change":"2024-05-20T17:56:45.751666+0000","last_active":"2024-05-20T17:58:43.856783+0000","last_peered":"2024-05-20T17:58:43.856783+0000","last_clean":"2024-05-20T17:58:43.856783+0000","last_became_active":"2024-05-20T17:56:45.751291+0000","last_became_peered":"2024-05-20T17:56:45.751291+0000","last_unstale":"2024-05-20T17:58:43.856783+0000","last_undegraded":"2024-05-20T17:58:43.856783+0000","last_fullsized":"2024-05-20T17:58:43.856783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:29:51.517363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681871+0000","last_change":"2024-05-20T17:56:44.721121+0000","last_active":"2024-05-20T17:58:43.681871+0000","last_peered":"2024-05-20T17:58:43.681871+0000","last_clean":"2024-05-20T17:58:43.681871+0000","last_became_active":"2024-05-20T17:56:44.720693+0000","last_became_peered":"2024-05-20T17:56:44.720693+0000","last_unstale":"2024-05-20T17:58:43.681871+0000","last_undegraded":"2024-05-20T17:58:43.681871+0000","last_fullsized":"2024-05-20T17:58:43.681871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:44.223828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691476+0000","last_change":"2024-05-20T17:56:45.745925+0000","last_active":"2024-05-20T17:58:43.691476+0000","last_peered":"2024-05-20T17:58:43.691476+0000","last_clean":"2024-05-20T17:58:43.691476+0000","last_became_active":"2024-05-20T17:56:45.745303+0000","last_became_peered":"2024-05-20T17:56:45.745303+0000","last_unstale":"2024-05-20T17:58:43.691476+0000","last_undegraded":"2024-05-20T17:58:43.691476+0000","last_fullsized":"2024-05-20T17:58:43.691476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:41:00.230681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691275+0000","last_change":"2024-05-20T17:56:44.719048+0000","last_active":"2024-05-20T17:58:43.691275+0000","last_peered":"2024-05-20T17:58:43.691275+0000","last_clean":"2024-05-20T17:58:43.691275+0000","last_became_active":"2024-05-20T17:56:44.718397+0000","last_became_peered":"2024-05-20T17:56:44.718397+0000","last_unstale":"2024-05-20T17:58:43.691275+0000","last_undegraded":"2024-05-20T17:58:43.691275+0000","last_fullsized":"2024-05-20T17:58:43.691275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:11:53.898226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681642+0000","last_change":"2024-05-20T17:56:45.752395+0000","last_active":"2024-05-20T17:58:43.681642+0000","last_peered":"2024-05-20T17:58:43.681642+0000","last_clean":"2024-05-20T17:58:43.681642+0000","last_became_active":"2024-05-20T17:56:45.751940+0000","last_became_peered":"2024-05-20T17:56:45.751940+0000","last_unstale":"2024-05-20T17:58:43.681642+0000","last_undegraded":"2024-05-20T17:58:43.681642+0000","last_fullsized":"2024-05-20T17:58:43.681642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:33:57.780042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694331+0000","last_change":"2024-05-20T17:56:44.714712+0000","last_active":"2024-05-20T17:58:43.694331+0000","last_peered":"2024-05-20T17:58:43.694331+0000","last_clean":"2024-05-20T17:58:43.694331+0000","last_became_active":"2024-05-20T17:56:44.714396+0000","last_became_peered":"2024-05-20T17:56:44.714396+0000","last_unstale":"2024-05-20T17:58:43.694331+0000","last_undegraded":"2024-05-20T17:58:43.694331+0000","last_fullsized":"2024-05-20T17:58:43.694331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:21:04.831289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691360+0000","last_change":"2024-05-20T17:56:45.758160+0000","last_active":"2024-05-20T17:58:43.691360+0000","last_peered":"2024-05-20T17:58:43.691360+0000","last_clean":"2024-05-20T17:58:43.691360+0000","last_became_active":"2024-05-20T17:56:45.757405+0000","last_became_peered":"2024-05-20T17:56:45.757405+0000","last_unstale":"2024-05-20T17:58:43.691360+0000","last_undegraded":"2024-05-20T17:58:43.691360+0000","last_fullsized":"2024-05-20T17:58:43.691360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:17.625246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113310+0000","last_change":"2024-05-20T17:56:44.716342+0000","last_active":"2024-05-20T17:58:44.113310+0000","last_peered":"2024-05-20T17:58:44.113310+0000","last_clean":"2024-05-20T17:58:44.113310+0000","last_became_active":"2024-05-20T17:56:44.716010+0000","last_became_peered":"2024-05-20T17:56:44.716010+0000","last_unstale":"2024-05-20T17:58:44.113310+0000","last_undegraded":"2024-05-20T17:58:44.113310+0000","last_fullsized":"2024-05-20T17:58:44.113310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:00:25.073312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681615+0000","last_change":"2024-05-20T17:56:45.755288+0000","last_active":"2024-05-20T17:58:43.681615+0000","last_peered":"2024-05-20T17:58:43.681615+0000","last_clean":"2024-05-20T17:58:43.681615+0000","last_became_active":"2024-05-20T17:56:45.754952+0000","last_became_peered":"2024-05-20T17:56:45.754952+0000","last_unstale":"2024-05-20T17:58:43.681615+0000","last_undegraded":"2024-05-20T17:58:43.681615+0000","last_fullsized":"2024-05-20T17:58:43.681615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:44:27.631722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691102+0000","last_change":"2024-05-20T17:56:44.717887+0000","last_active":"2024-05-20T17:58:43.691102+0000","last_peered":"2024-05-20T17:58:43.691102+0000","last_clean":"2024-05-20T17:58:43.691102+0000","last_became_active":"2024-05-20T17:56:44.717581+0000","last_became_peered":"2024-05-20T17:56:44.717581+0000","last_unstale":"2024-05-20T17:58:43.691102+0000","last_undegraded":"2024-05-20T17:58:43.691102+0000","last_fullsized":"2024-05-20T17:58:43.691102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:42:54.703112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570033+0000","last_change":"2024-05-20T17:56:45.748448+0000","last_active":"2024-05-20T17:58:43.570033+0000","last_peered":"2024-05-20T17:58:43.570033+0000","last_clean":"2024-05-20T17:58:43.570033+0000","last_became_active":"2024-05-20T17:56:45.748016+0000","last_became_peered":"2024-05-20T17:56:45.748016+0000","last_unstale":"2024-05-20T17:58:43.570033+0000","last_undegraded":"2024-05-20T17:58:43.570033+0000","last_fullsized":"2024-05-20T17:58:43.570033+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:31:39.350477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694679+0000","last_change":"2024-05-20T17:56:44.709483+0000","last_active":"2024-05-20T17:58:43.694679+0000","last_peered":"2024-05-20T17:58:43.694679+0000","last_clean":"2024-05-20T17:58:43.694679+0000","last_became_active":"2024-05-20T17:56:44.708649+0000","last_became_peered":"2024-05-20T17:56:44.708649+0000","last_unstale":"2024-05-20T17:58:43.694679+0000","last_undegraded":"2024-05-20T17:58:43.694679+0000","last_fullsized":"2024-05-20T17:58:43.694679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:31:46.580256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.857082+0000","last_change":"2024-05-20T17:56:45.752279+0000","last_active":"2024-05-20T17:58:43.857082+0000","last_peered":"2024-05-20T17:58:43.857082+0000","last_clean":"2024-05-20T17:58:43.857082+0000","last_became_active":"2024-05-20T17:56:45.751790+0000","last_became_peered":"2024-05-20T17:56:45.751790+0000","last_unstale":"2024-05-20T17:58:43.857082+0000","last_undegraded":"2024-05-20T17:58:43.857082+0000","last_fullsized":"2024-05-20T17:58:43.857082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:06:38.113559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571437+0000","last_change":"2024-05-20T17:57:49.077752+0000","last_active":"2024-05-20T17:58:43.571437+0000","last_peered":"2024-05-20T17:58:43.571437+0000","last_clean":"2024-05-20T17:58:43.571437+0000","last_became_active":"2024-05-20T17:57:49.077277+0000","last_became_peered":"2024-05-20T17:57:49.077277+0000","last_unstale":"2024-05-20T17:58:43.571437+0000","last_undegraded":"2024-05-20T17:58:43.571437+0000","last_fullsized":"2024-05-20T17:58:43.571437+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:24.954772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.114092+0000","last_change":"2024-05-20T17:57:49.051693+0000","last_active":"2024-05-20T17:58:44.114092+0000","last_peered":"2024-05-20T17:58:44.114092+0000","last_clean":"2024-05-20T17:58:44.114092+0000","last_became_active":"2024-05-20T17:57:49.051215+0000","last_became_peered":"2024-05-20T17:57:49.051215+0000","last_unstale":"2024-05-20T17:58:44.114092+0000","last_undegraded":"2024-05-20T17:58:44.114092+0000","last_fullsized":"2024-05-20T17:58:44.114092+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:52:35.406908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691588+0000","last_change":"2024-05-20T17:56:44.709548+0000","last_active":"2024-05-20T17:58:43.691588+0000","last_peered":"2024-05-20T17:58:43.691588+0000","last_clean":"2024-05-20T17:58:43.691588+0000","last_became_active":"2024-05-20T17:56:44.709209+0000","last_became_peered":"2024-05-20T17:56:44.709209+0000","last_unstale":"2024-05-20T17:58:43.691588+0000","last_undegraded":"2024-05-20T17:58:43.691588+0000","last_fullsized":"2024-05-20T17:58:43.691588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:28:13.118336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856926+0000","last_change":"2024-05-20T17:56:45.755141+0000","last_active":"2024-05-20T17:58:43.856926+0000","last_peered":"2024-05-20T17:58:43.856926+0000","last_clean":"2024-05-20T17:58:43.856926+0000","last_became_active":"2024-05-20T17:56:45.754786+0000","last_became_peered":"2024-05-20T17:56:45.754786+0000","last_unstale":"2024-05-20T17:58:43.856926+0000","last_undegraded":"2024-05-20T17:58:43.856926+0000","last_fullsized":"2024-05-20T17:58:43.856926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:45:21.044602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":296,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.171070+0000","last_change":"2024-05-20T17:56:44.713578+0000","last_active":"2024-05-20T17:58:44.171070+0000","last_peered":"2024-05-20T17:58:44.171070+0000","last_clean":"2024-05-20T17:58:44.171070+0000","last_became_active":"2024-05-20T17:56:44.713203+0000","last_became_peered":"2024-05-20T17:56:44.713203+0000","last_unstale":"2024-05-20T17:58:44.171070+0000","last_undegraded":"2024-05-20T17:58:44.171070+0000","last_fullsized":"2024-05-20T17:58:44.171070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:36:54.386036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":242,"num_read_kb":187,"num_write":33,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681854+0000","last_change":"2024-05-20T17:56:45.752023+0000","last_active":"2024-05-20T17:58:43.681854+0000","last_peered":"2024-05-20T17:58:43.681854+0000","last_clean":"2024-05-20T17:58:43.681854+0000","last_became_active":"2024-05-20T17:56:45.751672+0000","last_became_peered":"2024-05-20T17:56:45.751672+0000","last_unstale":"2024-05-20T17:58:43.681854+0000","last_undegraded":"2024-05-20T17:58:43.681854+0000","last_fullsized":"2024-05-20T17:58:43.681854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:23:49.127016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569077+0000","last_change":"2024-05-20T17:56:44.717336+0000","last_active":"2024-05-20T17:58:43.569077+0000","last_peered":"2024-05-20T17:58:43.569077+0000","last_clean":"2024-05-20T17:58:43.569077+0000","last_became_active":"2024-05-20T17:56:44.717016+0000","last_became_peered":"2024-05-20T17:56:44.717016+0000","last_unstale":"2024-05-20T17:58:43.569077+0000","last_undegraded":"2024-05-20T17:58:43.569077+0000","last_fullsized":"2024-05-20T17:58:43.569077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:50:46.433461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113461+0000","last_change":"2024-05-20T17:56:45.752411+0000","last_active":"2024-05-20T17:58:44.113461+0000","last_peered":"2024-05-20T17:58:44.113461+0000","last_clean":"2024-05-20T17:58:44.113461+0000","last_became_active":"2024-05-20T17:56:45.752078+0000","last_became_peered":"2024-05-20T17:56:45.752078+0000","last_unstale":"2024-05-20T17:58:44.113461+0000","last_undegraded":"2024-05-20T17:58:44.113461+0000","last_fullsized":"2024-05-20T17:58:44.113461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:22:15.560666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691295+0000","last_change":"2024-05-20T17:56:44.723262+0000","last_active":"2024-05-20T17:58:43.691295+0000","last_peered":"2024-05-20T17:58:43.691295+0000","last_clean":"2024-05-20T17:58:43.691295+0000","last_became_active":"2024-05-20T17:56:44.722865+0000","last_became_peered":"2024-05-20T17:56:44.722865+0000","last_unstale":"2024-05-20T17:58:43.691295+0000","last_undegraded":"2024-05-20T17:58:43.691295+0000","last_fullsized":"2024-05-20T17:58:43.691295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:15:02.820718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571429+0000","last_change":"2024-05-20T17:57:49.052110+0000","last_active":"2024-05-20T17:58:43.571429+0000","last_peered":"2024-05-20T17:58:43.571429+0000","last_clean":"2024-05-20T17:58:43.571429+0000","last_became_active":"2024-05-20T17:57:49.051356+0000","last_became_peered":"2024-05-20T17:57:49.051356+0000","last_unstale":"2024-05-20T17:58:43.571429+0000","last_undegraded":"2024-05-20T17:58:43.571429+0000","last_fullsized":"2024-05-20T17:58:43.571429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:32.899615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113252+0000","last_change":"2024-05-20T17:56:44.718396+0000","last_active":"2024-05-20T17:58:44.113252+0000","last_peered":"2024-05-20T17:58:44.113252+0000","last_clean":"2024-05-20T17:58:44.113252+0000","last_became_active":"2024-05-20T17:56:44.718031+0000","last_became_peered":"2024-05-20T17:56:44.718031+0000","last_unstale":"2024-05-20T17:58:44.113252+0000","last_undegraded":"2024-05-20T17:58:44.113252+0000","last_fullsized":"2024-05-20T17:58:44.113252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:42:40.955520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571035+0000","last_change":"2024-05-20T17:56:45.759302+0000","last_active":"2024-05-20T17:58:43.571035+0000","last_peered":"2024-05-20T17:58:43.571035+0000","last_clean":"2024-05-20T17:58:43.571035+0000","last_became_active":"2024-05-20T17:56:45.758889+0000","last_became_peered":"2024-05-20T17:56:45.758889+0000","last_unstale":"2024-05-20T17:58:43.571035+0000","last_undegraded":"2024-05-20T17:58:43.571035+0000","last_fullsized":"2024-05-20T17:58:43.571035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:40.685471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681563+0000","last_change":"2024-05-20T17:56:44.726502+0000","last_active":"2024-05-20T17:58:43.681563+0000","last_peered":"2024-05-20T17:58:43.681563+0000","last_clean":"2024-05-20T17:58:43.681563+0000","last_became_active":"2024-05-20T17:56:44.726168+0000","last_became_peered":"2024-05-20T17:56:44.726168+0000","last_unstale":"2024-05-20T17:58:43.681563+0000","last_undegraded":"2024-05-20T17:58:43.681563+0000","last_fullsized":"2024-05-20T17:58:43.681563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:45:14.523968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691604+0000","last_change":"2024-05-20T17:56:45.759355+0000","last_active":"2024-05-20T17:58:43.691604+0000","last_peered":"2024-05-20T17:58:43.691604+0000","last_clean":"2024-05-20T17:58:43.691604+0000","last_became_active":"2024-05-20T17:56:45.758920+0000","last_became_peered":"2024-05-20T17:56:45.758920+0000","last_unstale":"2024-05-20T17:58:43.691604+0000","last_undegraded":"2024-05-20T17:58:43.691604+0000","last_fullsized":"2024-05-20T17:58:43.691604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:46:02.372250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682113+0000","last_change":"2024-05-20T17:56:44.718507+0000","last_active":"2024-05-20T17:58:43.682113+0000","last_peered":"2024-05-20T17:58:43.682113+0000","last_clean":"2024-05-20T17:58:43.682113+0000","last_became_active":"2024-05-20T17:56:44.718188+0000","last_became_peered":"2024-05-20T17:56:44.718188+0000","last_unstale":"2024-05-20T17:58:43.682113+0000","last_undegraded":"2024-05-20T17:58:43.682113+0000","last_fullsized":"2024-05-20T17:58:43.682113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:43:59.792942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859084+0000","last_change":"2024-05-20T17:56:45.751361+0000","last_active":"2024-05-20T17:58:43.859084+0000","last_peered":"2024-05-20T17:58:43.859084+0000","last_clean":"2024-05-20T17:58:43.859084+0000","last_became_active":"2024-05-20T17:56:45.750032+0000","last_became_peered":"2024-05-20T17:56:45.750032+0000","last_unstale":"2024-05-20T17:58:43.859084+0000","last_undegraded":"2024-05-20T17:58:43.859084+0000","last_fullsized":"2024-05-20T17:58:43.859084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:01:42.729156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570328+0000","last_change":"2024-05-20T17:56:44.714404+0000","last_active":"2024-05-20T17:58:43.570328+0000","last_peered":"2024-05-20T17:58:43.570328+0000","last_clean":"2024-05-20T17:58:43.570328+0000","last_became_active":"2024-05-20T17:56:44.714072+0000","last_became_peered":"2024-05-20T17:56:44.714072+0000","last_unstale":"2024-05-20T17:58:43.570328+0000","last_undegraded":"2024-05-20T17:58:43.570328+0000","last_fullsized":"2024-05-20T17:58:43.570328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:39:53.983513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694551+0000","last_change":"2024-05-20T17:56:45.749785+0000","last_active":"2024-05-20T17:58:43.694551+0000","last_peered":"2024-05-20T17:58:43.694551+0000","last_clean":"2024-05-20T17:58:43.694551+0000","last_became_active":"2024-05-20T17:56:45.749457+0000","last_became_peered":"2024-05-20T17:56:45.749457+0000","last_unstale":"2024-05-20T17:58:43.694551+0000","last_undegraded":"2024-05-20T17:58:43.694551+0000","last_fullsized":"2024-05-20T17:58:43.694551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:53:05.136882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571253+0000","last_change":"2024-05-20T17:56:44.722787+0000","last_active":"2024-05-20T17:58:43.571253+0000","last_peered":"2024-05-20T17:58:43.571253+0000","last_clean":"2024-05-20T17:58:43.571253+0000","last_became_active":"2024-05-20T17:56:44.722214+0000","last_became_peered":"2024-05-20T17:56:44.722214+0000","last_unstale":"2024-05-20T17:58:43.571253+0000","last_undegraded":"2024-05-20T17:58:43.571253+0000","last_fullsized":"2024-05-20T17:58:43.571253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:40:16.705718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694565+0000","last_change":"2024-05-20T17:56:45.757515+0000","last_active":"2024-05-20T17:58:43.694565+0000","last_peered":"2024-05-20T17:58:43.694565+0000","last_clean":"2024-05-20T17:58:43.694565+0000","last_became_active":"2024-05-20T17:56:45.757174+0000","last_became_peered":"2024-05-20T17:56:45.757174+0000","last_unstale":"2024-05-20T17:58:43.694565+0000","last_undegraded":"2024-05-20T17:58:43.694565+0000","last_fullsized":"2024-05-20T17:58:43.694565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:18:14.154827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569893+0000","last_change":"2024-05-20T17:57:49.051489+0000","last_active":"2024-05-20T17:58:43.569893+0000","last_peered":"2024-05-20T17:58:43.569893+0000","last_clean":"2024-05-20T17:58:43.569893+0000","last_became_active":"2024-05-20T17:57:49.051070+0000","last_became_peered":"2024-05-20T17:57:49.051070+0000","last_unstale":"2024-05-20T17:58:43.569893+0000","last_undegraded":"2024-05-20T17:58:43.569893+0000","last_fullsized":"2024-05-20T17:58:43.569893+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:18.745185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113217+0000","last_change":"2024-05-20T17:56:44.724642+0000","last_active":"2024-05-20T17:58:44.113217+0000","last_peered":"2024-05-20T17:58:44.113217+0000","last_clean":"2024-05-20T17:58:44.113217+0000","last_became_active":"2024-05-20T17:56:44.724311+0000","last_became_peered":"2024-05-20T17:56:44.724311+0000","last_unstale":"2024-05-20T17:58:44.113217+0000","last_undegraded":"2024-05-20T17:58:44.113217+0000","last_fullsized":"2024-05-20T17:58:44.113217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:47:35.336236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681864+0000","last_change":"2024-05-20T17:56:45.758466+0000","last_active":"2024-05-20T17:58:43.681864+0000","last_peered":"2024-05-20T17:58:43.681864+0000","last_clean":"2024-05-20T17:58:43.681864+0000","last_became_active":"2024-05-20T17:56:45.758126+0000","last_became_peered":"2024-05-20T17:56:45.758126+0000","last_unstale":"2024-05-20T17:58:43.681864+0000","last_undegraded":"2024-05-20T17:58:43.681864+0000","last_fullsized":"2024-05-20T17:58:43.681864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:36.830828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113214+0000","last_change":"2024-05-20T17:56:44.711173+0000","last_active":"2024-05-20T17:58:44.113214+0000","last_peered":"2024-05-20T17:58:44.113214+0000","last_clean":"2024-05-20T17:58:44.113214+0000","last_became_active":"2024-05-20T17:56:44.710778+0000","last_became_peered":"2024-05-20T17:56:44.710778+0000","last_unstale":"2024-05-20T17:58:44.113214+0000","last_undegraded":"2024-05-20T17:58:44.113214+0000","last_fullsized":"2024-05-20T17:58:44.113214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:20:14.150346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571144+0000","last_change":"2024-05-20T17:55:34.871483+0000","last_active":"2024-05-20T17:58:43.571144+0000","last_peered":"2024-05-20T17:58:43.571144+0000","last_clean":"2024-05-20T17:58:43.571144+0000","last_became_active":"2024-05-20T17:55:34.871028+0000","last_became_peered":"2024-05-20T17:55:34.871028+0000","last_unstale":"2024-05-20T17:58:43.571144+0000","last_undegraded":"2024-05-20T17:58:43.571144+0000","last_fullsized":"2024-05-20T17:58:43.571144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_clean_scrub_stamp":"2024-05-20T17:52:08.599715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:29:40.159577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113167+0000","last_change":"2024-05-20T17:56:45.747725+0000","last_active":"2024-05-20T17:58:44.113167+0000","last_peered":"2024-05-20T17:58:44.113167+0000","last_clean":"2024-05-20T17:58:44.113167+0000","last_became_active":"2024-05-20T17:56:45.747232+0000","last_became_peered":"2024-05-20T17:56:45.747232+0000","last_unstale":"2024-05-20T17:58:44.113167+0000","last_undegraded":"2024-05-20T17:58:44.113167+0000","last_fullsized":"2024-05-20T17:58:44.113167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:09:26.589905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571349+0000","last_change":"2024-05-20T17:56:44.721004+0000","last_active":"2024-05-20T17:58:43.571349+0000","last_peered":"2024-05-20T17:58:43.571349+0000","last_clean":"2024-05-20T17:58:43.571349+0000","last_became_active":"2024-05-20T17:56:44.720620+0000","last_became_peered":"2024-05-20T17:56:44.720620+0000","last_unstale":"2024-05-20T17:58:43.571349+0000","last_undegraded":"2024-05-20T17:58:43.571349+0000","last_fullsized":"2024-05-20T17:58:43.571349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:42:14.710663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681867+0000","last_change":"2024-05-20T17:56:44.714754+0000","last_active":"2024-05-20T17:58:43.681867+0000","last_peered":"2024-05-20T17:58:43.681867+0000","last_clean":"2024-05-20T17:58:43.681867+0000","last_became_active":"2024-05-20T17:56:44.714378+0000","last_became_peered":"2024-05-20T17:56:44.714378+0000","last_unstale":"2024-05-20T17:58:43.681867+0000","last_undegraded":"2024-05-20T17:58:43.681867+0000","last_fullsized":"2024-05-20T17:58:43.681867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:46:53.969960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113922+0000","last_change":"2024-05-20T17:56:45.742415+0000","last_active":"2024-05-20T17:58:44.113922+0000","last_peered":"2024-05-20T17:58:44.113922+0000","last_clean":"2024-05-20T17:58:44.113922+0000","last_became_active":"2024-05-20T17:56:45.741941+0000","last_became_peered":"2024-05-20T17:56:45.741941+0000","last_unstale":"2024-05-20T17:58:44.113922+0000","last_undegraded":"2024-05-20T17:58:44.113922+0000","last_fullsized":"2024-05-20T17:58:44.113922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:20:03.191021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'43","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.813704+0000","last_change":"2024-05-20T17:56:44.724361+0000","last_active":"2024-05-20T17:58:43.813704+0000","last_peered":"2024-05-20T17:58:43.813704+0000","last_clean":"2024-05-20T17:58:43.813704+0000","last_became_active":"2024-05-20T17:56:44.724033+0000","last_became_peered":"2024-05-20T17:56:44.724033+0000","last_unstale":"2024-05-20T17:58:43.813704+0000","last_undegraded":"2024-05-20T17:58:43.813704+0000","last_fullsized":"2024-05-20T17:58:43.813704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:10:13.756606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570095+0000","last_change":"2024-05-20T17:57:49.051338+0000","last_active":"2024-05-20T17:58:43.570095+0000","last_peered":"2024-05-20T17:58:43.570095+0000","last_clean":"2024-05-20T17:58:43.570095+0000","last_became_active":"2024-05-20T17:57:49.050618+0000","last_became_peered":"2024-05-20T17:57:49.050618+0000","last_unstale":"2024-05-20T17:58:43.570095+0000","last_undegraded":"2024-05-20T17:58:43.570095+0000","last_fullsized":"2024-05-20T17:58:43.570095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:02:09.846175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113481+0000","last_change":"2024-05-20T17:56:44.716296+0000","last_active":"2024-05-20T17:58:44.113481+0000","last_peered":"2024-05-20T17:58:44.113481+0000","last_clean":"2024-05-20T17:58:44.113481+0000","last_became_active":"2024-05-20T17:56:44.715952+0000","last_became_peered":"2024-05-20T17:56:44.715952+0000","last_unstale":"2024-05-20T17:58:44.113481+0000","last_undegraded":"2024-05-20T17:58:44.113481+0000","last_fullsized":"2024-05-20T17:58:44.113481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:52:37.986399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113744+0000","last_change":"2024-05-20T17:56:45.751874+0000","last_active":"2024-05-20T17:58:44.113744+0000","last_peered":"2024-05-20T17:58:44.113744+0000","last_clean":"2024-05-20T17:58:44.113744+0000","last_became_active":"2024-05-20T17:56:45.751534+0000","last_became_peered":"2024-05-20T17:56:45.751534+0000","last_unstale":"2024-05-20T17:58:44.113744+0000","last_undegraded":"2024-05-20T17:58:44.113744+0000","last_fullsized":"2024-05-20T17:58:44.113744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:05:27.248115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856682+0000","last_change":"2024-05-20T17:56:44.724271+0000","last_active":"2024-05-20T17:58:43.856682+0000","last_peered":"2024-05-20T17:58:43.856682+0000","last_clean":"2024-05-20T17:58:43.856682+0000","last_became_active":"2024-05-20T17:56:44.723947+0000","last_became_peered":"2024-05-20T17:56:44.723947+0000","last_unstale":"2024-05-20T17:58:43.856682+0000","last_undegraded":"2024-05-20T17:58:43.856682+0000","last_fullsized":"2024-05-20T17:58:43.856682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:32:13.482059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681563+0000","last_change":"2024-05-20T17:56:45.751296+0000","last_active":"2024-05-20T17:58:43.681563+0000","last_peered":"2024-05-20T17:58:43.681563+0000","last_clean":"2024-05-20T17:58:43.681563+0000","last_became_active":"2024-05-20T17:56:45.748035+0000","last_became_peered":"2024-05-20T17:56:45.748035+0000","last_unstale":"2024-05-20T17:58:43.681563+0000","last_undegraded":"2024-05-20T17:58:43.681563+0000","last_fullsized":"2024-05-20T17:58:43.681563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:37:43.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681705+0000","last_change":"2024-05-20T17:56:44.720461+0000","last_active":"2024-05-20T17:58:43.681705+0000","last_peered":"2024-05-20T17:58:43.681705+0000","last_clean":"2024-05-20T17:58:43.681705+0000","last_became_active":"2024-05-20T17:56:44.720115+0000","last_became_peered":"2024-05-20T17:56:44.720115+0000","last_unstale":"2024-05-20T17:58:43.681705+0000","last_undegraded":"2024-05-20T17:58:43.681705+0000","last_fullsized":"2024-05-20T17:58:43.681705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:14:47.330193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.858870+0000","last_change":"2024-05-20T17:56:45.745025+0000","last_active":"2024-05-20T17:58:43.858870+0000","last_peered":"2024-05-20T17:58:43.858870+0000","last_clean":"2024-05-20T17:58:43.858870+0000","last_became_active":"2024-05-20T17:56:45.744626+0000","last_became_peered":"2024-05-20T17:56:45.744626+0000","last_unstale":"2024-05-20T17:58:43.858870+0000","last_undegraded":"2024-05-20T17:58:43.858870+0000","last_fullsized":"2024-05-20T17:58:43.858870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:08:51.052423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569045+0000","last_change":"2024-05-20T17:56:44.726099+0000","last_active":"2024-05-20T17:58:43.569045+0000","last_peered":"2024-05-20T17:58:43.569045+0000","last_clean":"2024-05-20T17:58:43.569045+0000","last_became_active":"2024-05-20T17:56:44.725714+0000","last_became_peered":"2024-05-20T17:56:44.725714+0000","last_unstale":"2024-05-20T17:58:43.569045+0000","last_undegraded":"2024-05-20T17:58:43.569045+0000","last_fullsized":"2024-05-20T17:58:43.569045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:51.326907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682248+0000","last_change":"2024-05-20T17:56:45.757376+0000","last_active":"2024-05-20T17:58:43.682248+0000","last_peered":"2024-05-20T17:58:43.682248+0000","last_clean":"2024-05-20T17:58:43.682248+0000","last_became_active":"2024-05-20T17:56:45.757053+0000","last_became_peered":"2024-05-20T17:56:45.757053+0000","last_unstale":"2024-05-20T17:58:43.682248+0000","last_undegraded":"2024-05-20T17:58:43.682248+0000","last_fullsized":"2024-05-20T17:58:43.682248+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:50:27.727055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681658+0000","last_change":"2024-05-20T17:56:44.726341+0000","last_active":"2024-05-20T17:58:43.681658+0000","last_peered":"2024-05-20T17:58:43.681658+0000","last_clean":"2024-05-20T17:58:43.681658+0000","last_became_active":"2024-05-20T17:56:44.726008+0000","last_became_peered":"2024-05-20T17:56:44.726008+0000","last_unstale":"2024-05-20T17:58:43.681658+0000","last_undegraded":"2024-05-20T17:58:43.681658+0000","last_fullsized":"2024-05-20T17:58:43.681658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:25:07.520179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571377+0000","last_change":"2024-05-20T17:56:45.760469+0000","last_active":"2024-05-20T17:58:43.571377+0000","last_peered":"2024-05-20T17:58:43.571377+0000","last_clean":"2024-05-20T17:58:43.571377+0000","last_became_active":"2024-05-20T17:56:45.760100+0000","last_became_peered":"2024-05-20T17:56:45.760100+0000","last_unstale":"2024-05-20T17:58:43.571377+0000","last_undegraded":"2024-05-20T17:58:43.571377+0000","last_fullsized":"2024-05-20T17:58:43.571377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:54:06.309355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113346+0000","last_change":"2024-05-20T17:57:49.051674+0000","last_active":"2024-05-20T17:58:44.113346+0000","last_peered":"2024-05-20T17:58:44.113346+0000","last_clean":"2024-05-20T17:58:44.113346+0000","last_became_active":"2024-05-20T17:57:49.051182+0000","last_became_peered":"2024-05-20T17:57:49.051182+0000","last_unstale":"2024-05-20T17:58:44.113346+0000","last_undegraded":"2024-05-20T17:58:44.113346+0000","last_fullsized":"2024-05-20T17:58:44.113346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:29:35.458339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571593+0000","last_change":"2024-05-20T17:56:45.757302+0000","last_active":"2024-05-20T17:58:43.571593+0000","last_peered":"2024-05-20T17:58:43.571593+0000","last_clean":"2024-05-20T17:58:43.571593+0000","last_became_active":"2024-05-20T17:56:45.754169+0000","last_became_peered":"2024-05-20T17:56:45.754169+0000","last_unstale":"2024-05-20T17:58:43.571593+0000","last_undegraded":"2024-05-20T17:58:43.571593+0000","last_fullsized":"2024-05-20T17:58:43.571593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:03:10.794342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.807590+0000","last_change":"2024-05-20T17:56:44.724478+0000","last_active":"2024-05-20T17:58:43.807590+0000","last_peered":"2024-05-20T17:58:43.807590+0000","last_clean":"2024-05-20T17:58:43.807590+0000","last_became_active":"2024-05-20T17:56:44.723927+0000","last_became_peered":"2024-05-20T17:56:44.723927+0000","last_unstale":"2024-05-20T17:58:43.807590+0000","last_undegraded":"2024-05-20T17:58:43.807590+0000","last_fullsized":"2024-05-20T17:58:43.807590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:04:12.003957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571188+0000","last_change":"2024-05-20T17:56:45.758135+0000","last_active":"2024-05-20T17:58:43.571188+0000","last_peered":"2024-05-20T17:58:43.571188+0000","last_clean":"2024-05-20T17:58:43.571188+0000","last_became_active":"2024-05-20T17:56:45.757358+0000","last_became_peered":"2024-05-20T17:56:45.757358+0000","last_unstale":"2024-05-20T17:58:43.571188+0000","last_undegraded":"2024-05-20T17:58:43.571188+0000","last_fullsized":"2024-05-20T17:58:43.571188+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:13:06.435055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.114107+0000","last_change":"2024-05-20T17:56:44.719822+0000","last_active":"2024-05-20T17:58:44.114107+0000","last_peered":"2024-05-20T17:58:44.114107+0000","last_clean":"2024-05-20T17:58:44.114107+0000","last_became_active":"2024-05-20T17:56:44.719454+0000","last_became_peered":"2024-05-20T17:56:44.719454+0000","last_unstale":"2024-05-20T17:58:44.114107+0000","last_undegraded":"2024-05-20T17:58:44.114107+0000","last_fullsized":"2024-05-20T17:58:44.114107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:03:21.048330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571553+0000","last_change":"2024-05-20T17:56:45.753510+0000","last_active":"2024-05-20T17:58:43.571553+0000","last_peered":"2024-05-20T17:58:43.571553+0000","last_clean":"2024-05-20T17:58:43.571553+0000","last_became_active":"2024-05-20T17:56:45.752718+0000","last_became_peered":"2024-05-20T17:56:45.752718+0000","last_unstale":"2024-05-20T17:58:43.571553+0000","last_undegraded":"2024-05-20T17:58:43.571553+0000","last_fullsized":"2024-05-20T17:58:43.571553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:05:44.677138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.858961+0000","last_change":"2024-05-20T17:56:45.752528+0000","last_active":"2024-05-20T17:58:43.858961+0000","last_peered":"2024-05-20T17:58:43.858961+0000","last_clean":"2024-05-20T17:58:43.858961+0000","last_became_active":"2024-05-20T17:56:45.752171+0000","last_became_peered":"2024-05-20T17:56:45.752171+0000","last_unstale":"2024-05-20T17:58:43.858961+0000","last_undegraded":"2024-05-20T17:58:43.858961+0000","last_fullsized":"2024-05-20T17:58:43.858961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:12:55.826122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569068+0000","last_change":"2024-05-20T17:56:45.749228+0000","last_active":"2024-05-20T17:58:43.569068+0000","last_peered":"2024-05-20T17:58:43.569068+0000","last_clean":"2024-05-20T17:58:43.569068+0000","last_became_active":"2024-05-20T17:56:45.748886+0000","last_became_peered":"2024-05-20T17:56:45.748886+0000","last_unstale":"2024-05-20T17:58:43.569068+0000","last_undegraded":"2024-05-20T17:58:43.569068+0000","last_fullsized":"2024-05-20T17:58:43.569068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:23:10.569720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569035+0000","last_change":"2024-05-20T17:56:45.751105+0000","last_active":"2024-05-20T17:58:43.569035+0000","last_peered":"2024-05-20T17:58:43.569035+0000","last_clean":"2024-05-20T17:58:43.569035+0000","last_became_active":"2024-05-20T17:56:45.750754+0000","last_became_peered":"2024-05-20T17:56:45.750754+0000","last_unstale":"2024-05-20T17:58:43.569035+0000","last_undegraded":"2024-05-20T17:58:43.569035+0000","last_fullsized":"2024-05-20T17:58:43.569035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:57:48.327428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113562+0000","last_change":"2024-05-20T17:57:49.050574+0000","last_active":"2024-05-20T17:58:44.113562+0000","last_peered":"2024-05-20T17:58:44.113562+0000","last_clean":"2024-05-20T17:58:44.113562+0000","last_became_active":"2024-05-20T17:57:49.050037+0000","last_became_peered":"2024-05-20T17:57:49.050037+0000","last_unstale":"2024-05-20T17:58:44.113562+0000","last_undegraded":"2024-05-20T17:58:44.113562+0000","last_fullsized":"2024-05-20T17:58:44.113562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:01:56.453842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113800+0000","last_change":"2024-05-20T17:56:45.746122+0000","last_active":"2024-05-20T17:58:44.113800+0000","last_peered":"2024-05-20T17:58:44.113800+0000","last_clean":"2024-05-20T17:58:44.113800+0000","last_became_active":"2024-05-20T17:56:45.745773+0000","last_became_peered":"2024-05-20T17:56:45.745773+0000","last_unstale":"2024-05-20T17:58:44.113800+0000","last_undegraded":"2024-05-20T17:58:44.113800+0000","last_fullsized":"2024-05-20T17:58:44.113800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:21:27.909974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859042+0000","last_change":"2024-05-20T17:56:45.755018+0000","last_active":"2024-05-20T17:58:43.859042+0000","last_peered":"2024-05-20T17:58:43.859042+0000","last_clean":"2024-05-20T17:58:43.859042+0000","last_became_active":"2024-05-20T17:56:45.754615+0000","last_became_peered":"2024-05-20T17:56:45.754615+0000","last_unstale":"2024-05-20T17:58:43.859042+0000","last_undegraded":"2024-05-20T17:58:43.859042+0000","last_fullsized":"2024-05-20T17:58:43.859042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:36:39.100566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569612+0000","last_change":"2024-05-20T17:56:45.751091+0000","last_active":"2024-05-20T17:58:43.569612+0000","last_peered":"2024-05-20T17:58:43.569612+0000","last_clean":"2024-05-20T17:58:43.569612+0000","last_became_active":"2024-05-20T17:56:45.748573+0000","last_became_peered":"2024-05-20T17:56:45.748573+0000","last_unstale":"2024-05-20T17:58:43.569612+0000","last_undegraded":"2024-05-20T17:58:43.569612+0000","last_fullsized":"2024-05-20T17:58:43.569612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:11:16.140967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682866+0000","last_change":"2024-05-20T17:56:45.756373+0000","last_active":"2024-05-20T17:58:43.682866+0000","last_peered":"2024-05-20T17:58:43.682866+0000","last_clean":"2024-05-20T17:58:43.682866+0000","last_became_active":"2024-05-20T17:56:45.755993+0000","last_became_peered":"2024-05-20T17:56:45.755993+0000","last_unstale":"2024-05-20T17:58:43.682866+0000","last_undegraded":"2024-05-20T17:58:43.682866+0000","last_fullsized":"2024-05-20T17:58:43.682866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:34:44.491341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691184+0000","last_change":"2024-05-20T17:56:45.751866+0000","last_active":"2024-05-20T17:58:43.691184+0000","last_peered":"2024-05-20T17:58:43.691184+0000","last_clean":"2024-05-20T17:58:43.691184+0000","last_became_active":"2024-05-20T17:56:45.751518+0000","last_became_peered":"2024-05-20T17:56:45.751518+0000","last_unstale":"2024-05-20T17:58:43.691184+0000","last_undegraded":"2024-05-20T17:58:43.691184+0000","last_fullsized":"2024-05-20T17:58:43.691184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:38:18.736177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681682+0000","last_change":"2024-05-20T17:56:45.749903+0000","last_active":"2024-05-20T17:58:43.681682+0000","last_peered":"2024-05-20T17:58:43.681682+0000","last_clean":"2024-05-20T17:58:43.681682+0000","last_became_active":"2024-05-20T17:56:45.749500+0000","last_became_peered":"2024-05-20T17:56:45.749500+0000","last_unstale":"2024-05-20T17:58:43.681682+0000","last_undegraded":"2024-05-20T17:58:43.681682+0000","last_fullsized":"2024-05-20T17:58:43.681682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:39:34.875111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856827+0000","last_change":"2024-05-20T17:56:45.747370+0000","last_active":"2024-05-20T17:58:43.856827+0000","last_peered":"2024-05-20T17:58:43.856827+0000","last_clean":"2024-05-20T17:58:43.856827+0000","last_became_active":"2024-05-20T17:56:45.747036+0000","last_became_peered":"2024-05-20T17:56:45.747036+0000","last_unstale":"2024-05-20T17:58:43.856827+0000","last_undegraded":"2024-05-20T17:58:43.856827+0000","last_fullsized":"2024-05-20T17:58:43.856827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:02:02.220522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113692+0000","last_change":"2024-05-20T17:56:45.752762+0000","last_active":"2024-05-20T17:58:44.113692+0000","last_peered":"2024-05-20T17:58:44.113692+0000","last_clean":"2024-05-20T17:58:44.113692+0000","last_became_active":"2024-05-20T17:56:45.752345+0000","last_became_peered":"2024-05-20T17:56:45.752345+0000","last_unstale":"2024-05-20T17:58:44.113692+0000","last_undegraded":"2024-05-20T17:58:44.113692+0000","last_fullsized":"2024-05-20T17:58:44.113692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:14:49.861456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.858880+0000","last_change":"2024-05-20T17:56:45.750229+0000","last_active":"2024-05-20T17:58:43.858880+0000","last_peered":"2024-05-20T17:58:43.858880+0000","last_clean":"2024-05-20T17:58:43.858880+0000","last_became_active":"2024-05-20T17:56:45.749872+0000","last_became_peered":"2024-05-20T17:56:45.749872+0000","last_unstale":"2024-05-20T17:58:43.858880+0000","last_undegraded":"2024-05-20T17:58:43.858880+0000","last_fullsized":"2024-05-20T17:58:43.858880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:56:58.015339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106947,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":593372,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":340,"num_read_kb":272,"num_write":196,"num_write_kb":1403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":1780116,"data_compressed":81486,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":462373,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612649,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1122304,"data_stored":1261374,"data_compressed":52401,"data_compressed_allocated":69632,"data_compressed_original":745472,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808879,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":865135,"data_compressed":27882,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070516,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":864575,"data_compressed":27882,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":1062404,"data_compressed":44385,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":462490,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":539677,"data_compressed":10596,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":471427,"data_compressed":8736,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":539129,"data_compressed":10596,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":462711,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":462395,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":1062004,"data_compressed":44385,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68340,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67792,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393238,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":199552,"data_compressed":8016,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-20T17:58:45.032 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-20T17:58:45.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-20T17:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.184+0000 7f638a814640 1 -- 172.21.15.129:0/1252480084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63841078b0 msgr2=0x7f6384109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.184+0000 7f638a814640 1 --2- 172.21.15.129:0/1252480084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63841078b0 0x7f6384109cd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f637c009ca0 tx=0x7f637c02a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.184+0000 7f638a814640 1 -- 172.21.15.129:0/1252480084 shutdown_connections 2024-05-20T17:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.184+0000 7f638a814640 1 --2- 172.21.15.129:0/1252480084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6384102bb0 0x7f6384102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.184+0000 7f638a814640 1 --2- 172.21.15.129:0/1252480084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63841078b0 0x7f6384109cd0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.184+0000 7f638a814640 1 --2- 172.21.15.129:0/1252480084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6384101970 0x7f6384102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.184+0000 7f638a814640 1 -- 172.21.15.129:0/1252480084 >> 172.21.15.129:0/1252480084 conn(0x7f63840fcb80 msgr2=0x7f63840ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.184+0000 7f638a814640 1 -- 172.21.15.129:0/1252480084 shutdown_connections 2024-05-20T17:58:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.184+0000 7f638a814640 1 -- 172.21.15.129:0/1252480084 wait complete. 2024-05-20T17:58:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f638a814640 1 Processor -- start 2024-05-20T17:58:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f638a814640 1 -- start start 2024-05-20T17:58:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f638a814640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6384101970 0x7f6384198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f638a814640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6384102bb0 0x7f6384198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f638a814640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63841078b0 0x7f638419f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f638a814640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f638410d5f0 con 0x7f6384102bb0 2024-05-20T17:58:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f638a814640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f638410d470 con 0x7f6384101970 2024-05-20T17:58:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f638a814640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f638410d770 con 0x7f63841078b0 2024-05-20T17:58:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f63837fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6384102bb0 0x7f6384198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f63837fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6384102bb0 0x7f6384198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42476/0 (socket says 172.21.15.129:42476) 2024-05-20T17:58:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f63837fe640 1 -- 172.21.15.129:0/3416245117 learned_addr learned my addr 172.21.15.129:0/3416245117 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f6388db1640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63841078b0 0x7f638419f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f6383fff640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6384101970 0x7f6384198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f63837fe640 1 -- 172.21.15.129:0/3416245117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63841078b0 msgr2=0x7f638419f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f63837fe640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63841078b0 0x7f638419f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.185+0000 7f63837fe640 1 -- 172.21.15.129:0/3416245117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6384101970 msgr2=0x7f6384198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.186+0000 7f63837fe640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6384101970 0x7f6384198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.186+0000 7f63837fe640 1 -- 172.21.15.129:0/3416245117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638419fc80 con 0x7f6384102bb0 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.186+0000 7f6388db1640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63841078b0 0x7f638419f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.186+0000 7f6383fff640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6384101970 0x7f6384198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.186+0000 7f63837fe640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6384102bb0 0x7f6384198680 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f636c00e9f0 tx=0x7f636c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.186+0000 7f63817fa640 1 -- 172.21.15.129:0/3416245117 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c00cd80 con 0x7f6384102bb0 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.186+0000 7f63817fa640 1 -- 172.21.15.129:0/3416245117 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f636c010910 con 0x7f6384102bb0 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.186+0000 7f638a814640 1 -- 172.21.15.129:0/3416245117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638419ff70 con 0x7f6384102bb0 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.186+0000 7f63817fa640 1 -- 172.21.15.129:0/3416245117 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c0188f0 con 0x7f6384102bb0 2024-05-20T17:58:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.186+0000 7f638a814640 1 -- 172.21.15.129:0/3416245117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63841a03e0 con 0x7f6384102bb0 2024-05-20T17:58:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.187+0000 7f638a814640 1 -- 172.21.15.129:0/3416245117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63841066c0 con 0x7f6384102bb0 2024-05-20T17:58:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.188+0000 7f63817fa640 1 -- 172.21.15.129:0/3416245117 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f636c010ac0 con 0x7f6384102bb0 2024-05-20T17:58:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.189+0000 7f63817fa640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f63700a2990 0x7f63700a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.189+0000 7f6383fff640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f63700a2990 0x7f63700a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.189+0000 7f63817fa640 1 -- 172.21.15.129:0/3416245117 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f636c0cf670 con 0x7f6384102bb0 2024-05-20T17:58:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.190+0000 7f6383fff640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f63700a2990 0x7f63700a4ea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6384102570 tx=0x7f6374005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.193+0000 7f63817fa640 1 -- 172.21.15.129:0/3416245117 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f636c09c7b0 con 0x7f6384102bb0 2024-05-20T17:58:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.377+0000 7f638a814640 1 -- 172.21.15.129:0/3416245117 --> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6384061f90 con 0x7f63700a2990 2024-05-20T17:58:45.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.386+0000 7f63817fa640 1 -- 172.21.15.129:0/3416245117 <== mgr.15228 v2:172.21.15.129:6800/3515662734 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257946 (secure 0 0 0) 0x7f6384061f90 con 0x7f63700a2990 2024-05-20T17:58:45.386 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:45.387 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-05-20T17:58:45.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 -- 172.21.15.129:0/3416245117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f63700a2990 msgr2=0x7f63700a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f63700a2990 0x7f63700a4ea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6384102570 tx=0x7f6374005f90 comp rx=0 tx=0).stop 2024-05-20T17:58:45.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 -- 172.21.15.129:0/3416245117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6384102bb0 msgr2=0x7f6384198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6384102bb0 0x7f6384198680 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f636c00e9f0 tx=0x7f636c00eeb0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 -- 172.21.15.129:0/3416245117 shutdown_connections 2024-05-20T17:58:45.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f63700a2990 0x7f63700a4ea0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f63841078b0 0x7f638419f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6384102bb0 0x7f6384198680 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 --2- 172.21.15.129:0/3416245117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6384101970 0x7f6384198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 -- 172.21.15.129:0/3416245117 >> 172.21.15.129:0/3416245117 conn(0x7f63840fcb80 msgr2=0x7f6384103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:45.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 -- 172.21.15.129:0/3416245117 shutdown_connections 2024-05-20T17:58:45.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.390+0000 7f638a814640 1 -- 172.21.15.129:0/3416245117 wait complete. 2024-05-20T17:58:45.400 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":3,"stamp":"2024-05-20T17:58:44.594311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1183974,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":476,"num_read_kb":391,"num_write":379,"num_write_kb":4343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365084,"kb_used_data":11744,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576740,"statfs":{"total":1151940427776,"available":1151566581760,"internally_reserved":0,"allocated":12025856,"data_stored":8515694,"data_compressed":250131,"data_compressed_allocated":335872,"data_compressed_original":2965504,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859196+0000","last_change":"2024-05-20T17:56:45.746793+0000","last_active":"2024-05-20T17:58:43.859196+0000","last_peered":"2024-05-20T17:58:43.859196+0000","last_clean":"2024-05-20T17:58:43.859196+0000","last_became_active":"2024-05-20T17:56:45.746244+0000","last_became_peered":"2024-05-20T17:56:45.746244+0000","last_unstale":"2024-05-20T17:58:43.859196+0000","last_undegraded":"2024-05-20T17:58:43.859196+0000","last_fullsized":"2024-05-20T17:58:43.859196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:37:03.703599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113926+0000","last_change":"2024-05-20T17:56:45.744538+0000","last_active":"2024-05-20T17:58:44.113926+0000","last_peered":"2024-05-20T17:58:44.113926+0000","last_clean":"2024-05-20T17:58:44.113926+0000","last_became_active":"2024-05-20T17:56:45.744213+0000","last_became_peered":"2024-05-20T17:56:45.744213+0000","last_unstale":"2024-05-20T17:58:44.113926+0000","last_undegraded":"2024-05-20T17:58:44.113926+0000","last_fullsized":"2024-05-20T17:58:44.113926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:19:26.163003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113332+0000","last_change":"2024-05-20T17:56:45.749725+0000","last_active":"2024-05-20T17:58:44.113332+0000","last_peered":"2024-05-20T17:58:44.113332+0000","last_clean":"2024-05-20T17:58:44.113332+0000","last_became_active":"2024-05-20T17:56:45.749392+0000","last_became_peered":"2024-05-20T17:56:45.749392+0000","last_unstale":"2024-05-20T17:58:44.113332+0000","last_undegraded":"2024-05-20T17:58:44.113332+0000","last_fullsized":"2024-05-20T17:58:44.113332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:43:33.184192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113454+0000","last_change":"2024-05-20T17:56:45.755191+0000","last_active":"2024-05-20T17:58:44.113454+0000","last_peered":"2024-05-20T17:58:44.113454+0000","last_clean":"2024-05-20T17:58:44.113454+0000","last_became_active":"2024-05-20T17:56:45.753524+0000","last_became_peered":"2024-05-20T17:56:45.753524+0000","last_unstale":"2024-05-20T17:58:44.113454+0000","last_undegraded":"2024-05-20T17:58:44.113454+0000","last_fullsized":"2024-05-20T17:58:44.113454+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:11:23.099919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681633+0000","last_change":"2024-05-20T17:56:45.756291+0000","last_active":"2024-05-20T17:58:43.681633+0000","last_peered":"2024-05-20T17:58:43.681633+0000","last_clean":"2024-05-20T17:58:43.681633+0000","last_became_active":"2024-05-20T17:56:45.755910+0000","last_became_peered":"2024-05-20T17:56:45.755910+0000","last_unstale":"2024-05-20T17:58:43.681633+0000","last_undegraded":"2024-05-20T17:58:43.681633+0000","last_fullsized":"2024-05-20T17:58:43.681633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:37:36.941012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694392+0000","last_change":"2024-05-20T17:56:45.748756+0000","last_active":"2024-05-20T17:58:43.694392+0000","last_peered":"2024-05-20T17:58:43.694392+0000","last_clean":"2024-05-20T17:58:43.694392+0000","last_became_active":"2024-05-20T17:56:45.748406+0000","last_became_peered":"2024-05-20T17:56:45.748406+0000","last_unstale":"2024-05-20T17:58:43.694392+0000","last_undegraded":"2024-05-20T17:58:43.694392+0000","last_fullsized":"2024-05-20T17:58:43.694392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:16:36.527832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571098+0000","last_change":"2024-05-20T17:57:49.051459+0000","last_active":"2024-05-20T17:58:43.571098+0000","last_peered":"2024-05-20T17:58:43.571098+0000","last_clean":"2024-05-20T17:58:43.571098+0000","last_became_active":"2024-05-20T17:57:49.050933+0000","last_became_peered":"2024-05-20T17:57:49.050933+0000","last_unstale":"2024-05-20T17:58:43.571098+0000","last_undegraded":"2024-05-20T17:58:43.571098+0000","last_fullsized":"2024-05-20T17:58:43.571098+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:18.214102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681494+0000","last_change":"2024-05-20T17:56:45.757387+0000","last_active":"2024-05-20T17:58:43.681494+0000","last_peered":"2024-05-20T17:58:43.681494+0000","last_clean":"2024-05-20T17:58:43.681494+0000","last_became_active":"2024-05-20T17:56:45.756374+0000","last_became_peered":"2024-05-20T17:56:45.756374+0000","last_unstale":"2024-05-20T17:58:43.681494+0000","last_undegraded":"2024-05-20T17:58:43.681494+0000","last_fullsized":"2024-05-20T17:58:43.681494+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:53:52.751019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.857216+0000","last_change":"2024-05-20T17:56:45.751933+0000","last_active":"2024-05-20T17:58:43.857216+0000","last_peered":"2024-05-20T17:58:43.857216+0000","last_clean":"2024-05-20T17:58:43.857216+0000","last_became_active":"2024-05-20T17:56:45.751414+0000","last_became_peered":"2024-05-20T17:56:45.751414+0000","last_unstale":"2024-05-20T17:58:43.857216+0000","last_undegraded":"2024-05-20T17:58:43.857216+0000","last_fullsized":"2024-05-20T17:58:43.857216+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:56:52.974830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694627+0000","last_change":"2024-05-20T17:56:45.749511+0000","last_active":"2024-05-20T17:58:43.694627+0000","last_peered":"2024-05-20T17:58:43.694627+0000","last_clean":"2024-05-20T17:58:43.694627+0000","last_became_active":"2024-05-20T17:56:45.749211+0000","last_became_peered":"2024-05-20T17:56:45.749211+0000","last_unstale":"2024-05-20T17:58:43.694627+0000","last_undegraded":"2024-05-20T17:58:43.694627+0000","last_fullsized":"2024-05-20T17:58:43.694627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:15.633296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571341+0000","last_change":"2024-05-20T17:56:45.753743+0000","last_active":"2024-05-20T17:58:43.571341+0000","last_peered":"2024-05-20T17:58:43.571341+0000","last_clean":"2024-05-20T17:58:43.571341+0000","last_became_active":"2024-05-20T17:56:45.753378+0000","last_became_peered":"2024-05-20T17:56:45.753378+0000","last_unstale":"2024-05-20T17:58:43.571341+0000","last_undegraded":"2024-05-20T17:58:43.571341+0000","last_fullsized":"2024-05-20T17:58:43.571341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:44:54.563895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113327+0000","last_change":"2024-05-20T17:56:45.754890+0000","last_active":"2024-05-20T17:58:44.113327+0000","last_peered":"2024-05-20T17:58:44.113327+0000","last_clean":"2024-05-20T17:58:44.113327+0000","last_became_active":"2024-05-20T17:56:45.754333+0000","last_became_peered":"2024-05-20T17:56:45.754333+0000","last_unstale":"2024-05-20T17:58:44.113327+0000","last_undegraded":"2024-05-20T17:58:44.113327+0000","last_fullsized":"2024-05-20T17:58:44.113327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:51.025516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113630+0000","last_change":"2024-05-20T17:56:45.745662+0000","last_active":"2024-05-20T17:58:44.113630+0000","last_peered":"2024-05-20T17:58:44.113630+0000","last_clean":"2024-05-20T17:58:44.113630+0000","last_became_active":"2024-05-20T17:56:45.745281+0000","last_became_peered":"2024-05-20T17:56:45.745281+0000","last_unstale":"2024-05-20T17:58:44.113630+0000","last_undegraded":"2024-05-20T17:58:44.113630+0000","last_fullsized":"2024-05-20T17:58:44.113630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:38:57.057301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.112954+0000","last_change":"2024-05-20T17:56:45.749164+0000","last_active":"2024-05-20T17:58:44.112954+0000","last_peered":"2024-05-20T17:58:44.112954+0000","last_clean":"2024-05-20T17:58:44.112954+0000","last_became_active":"2024-05-20T17:56:45.748822+0000","last_became_peered":"2024-05-20T17:56:45.748822+0000","last_unstale":"2024-05-20T17:58:44.112954+0000","last_undegraded":"2024-05-20T17:58:44.112954+0000","last_fullsized":"2024-05-20T17:58:44.112954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:45:47.487116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.112953+0000","last_change":"2024-05-20T17:56:45.751185+0000","last_active":"2024-05-20T17:58:44.112953+0000","last_peered":"2024-05-20T17:58:44.112953+0000","last_clean":"2024-05-20T17:58:44.112953+0000","last_became_active":"2024-05-20T17:56:45.750851+0000","last_became_peered":"2024-05-20T17:56:45.750851+0000","last_unstale":"2024-05-20T17:58:44.112953+0000","last_undegraded":"2024-05-20T17:58:44.112953+0000","last_fullsized":"2024-05-20T17:58:44.112953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:31:07.792717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856607+0000","last_change":"2024-05-20T17:56:45.748101+0000","last_active":"2024-05-20T17:58:43.856607+0000","last_peered":"2024-05-20T17:58:43.856607+0000","last_clean":"2024-05-20T17:58:43.856607+0000","last_became_active":"2024-05-20T17:56:45.747754+0000","last_became_peered":"2024-05-20T17:56:45.747754+0000","last_unstale":"2024-05-20T17:58:43.856607+0000","last_undegraded":"2024-05-20T17:58:43.856607+0000","last_fullsized":"2024-05-20T17:58:43.856607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:24:54.702601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859350+0000","last_change":"2024-05-20T17:56:45.747497+0000","last_active":"2024-05-20T17:58:43.859350+0000","last_peered":"2024-05-20T17:58:43.859350+0000","last_clean":"2024-05-20T17:58:43.859350+0000","last_became_active":"2024-05-20T17:56:45.747139+0000","last_became_peered":"2024-05-20T17:56:45.747139+0000","last_unstale":"2024-05-20T17:58:43.859350+0000","last_undegraded":"2024-05-20T17:58:43.859350+0000","last_fullsized":"2024-05-20T17:58:43.859350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:39:13.102413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570446+0000","last_change":"2024-05-20T17:56:45.746194+0000","last_active":"2024-05-20T17:58:43.570446+0000","last_peered":"2024-05-20T17:58:43.570446+0000","last_clean":"2024-05-20T17:58:43.570446+0000","last_became_active":"2024-05-20T17:56:45.745809+0000","last_became_peered":"2024-05-20T17:56:45.745809+0000","last_unstale":"2024-05-20T17:58:43.570446+0000","last_undegraded":"2024-05-20T17:58:43.570446+0000","last_fullsized":"2024-05-20T17:58:43.570446+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:26:29.133790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569306+0000","last_change":"2024-05-20T17:56:45.749918+0000","last_active":"2024-05-20T17:58:43.569306+0000","last_peered":"2024-05-20T17:58:43.569306+0000","last_clean":"2024-05-20T17:58:43.569306+0000","last_became_active":"2024-05-20T17:56:45.749560+0000","last_became_peered":"2024-05-20T17:56:45.749560+0000","last_unstale":"2024-05-20T17:58:43.569306+0000","last_undegraded":"2024-05-20T17:58:43.569306+0000","last_fullsized":"2024-05-20T17:58:43.569306+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:33:12.737206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570253+0000","last_change":"2024-05-20T17:57:49.082268+0000","last_active":"2024-05-20T17:58:43.570253+0000","last_peered":"2024-05-20T17:58:43.570253+0000","last_clean":"2024-05-20T17:58:43.570253+0000","last_became_active":"2024-05-20T17:57:49.078471+0000","last_became_peered":"2024-05-20T17:57:49.078471+0000","last_unstale":"2024-05-20T17:58:43.570253+0000","last_undegraded":"2024-05-20T17:58:43.570253+0000","last_fullsized":"2024-05-20T17:58:43.570253+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:50:14.617477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682939+0000","last_change":"2024-05-20T17:56:45.758330+0000","last_active":"2024-05-20T17:58:43.682939+0000","last_peered":"2024-05-20T17:58:43.682939+0000","last_clean":"2024-05-20T17:58:43.682939+0000","last_became_active":"2024-05-20T17:56:45.755873+0000","last_became_peered":"2024-05-20T17:56:45.755873+0000","last_unstale":"2024-05-20T17:58:43.682939+0000","last_undegraded":"2024-05-20T17:58:43.682939+0000","last_fullsized":"2024-05-20T17:58:43.682939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:53:31.653497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.683149+0000","last_change":"2024-05-20T17:56:44.728508+0000","last_active":"2024-05-20T17:58:43.683149+0000","last_peered":"2024-05-20T17:58:43.683149+0000","last_clean":"2024-05-20T17:58:43.683149+0000","last_became_active":"2024-05-20T17:56:44.727875+0000","last_became_peered":"2024-05-20T17:56:44.727875+0000","last_unstale":"2024-05-20T17:58:43.683149+0000","last_undegraded":"2024-05-20T17:58:43.683149+0000","last_fullsized":"2024-05-20T17:58:43.683149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:12:14.349039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.114265+0000","last_change":"2024-05-20T17:56:45.750000+0000","last_active":"2024-05-20T17:58:44.114265+0000","last_peered":"2024-05-20T17:58:44.114265+0000","last_clean":"2024-05-20T17:58:44.114265+0000","last_became_active":"2024-05-20T17:56:45.749644+0000","last_became_peered":"2024-05-20T17:56:45.749644+0000","last_unstale":"2024-05-20T17:58:44.114265+0000","last_undegraded":"2024-05-20T17:58:44.114265+0000","last_fullsized":"2024-05-20T17:58:44.114265+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:32:46.085081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682012+0000","last_change":"2024-05-20T17:56:44.722692+0000","last_active":"2024-05-20T17:58:43.682012+0000","last_peered":"2024-05-20T17:58:43.682012+0000","last_clean":"2024-05-20T17:58:43.682012+0000","last_became_active":"2024-05-20T17:56:44.722170+0000","last_became_peered":"2024-05-20T17:56:44.722170+0000","last_unstale":"2024-05-20T17:58:43.682012+0000","last_undegraded":"2024-05-20T17:58:43.682012+0000","last_fullsized":"2024-05-20T17:58:43.682012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:38:03.679562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856783+0000","last_change":"2024-05-20T17:56:45.751666+0000","last_active":"2024-05-20T17:58:43.856783+0000","last_peered":"2024-05-20T17:58:43.856783+0000","last_clean":"2024-05-20T17:58:43.856783+0000","last_became_active":"2024-05-20T17:56:45.751291+0000","last_became_peered":"2024-05-20T17:56:45.751291+0000","last_unstale":"2024-05-20T17:58:43.856783+0000","last_undegraded":"2024-05-20T17:58:43.856783+0000","last_fullsized":"2024-05-20T17:58:43.856783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:29:51.517363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681871+0000","last_change":"2024-05-20T17:56:44.721121+0000","last_active":"2024-05-20T17:58:43.681871+0000","last_peered":"2024-05-20T17:58:43.681871+0000","last_clean":"2024-05-20T17:58:43.681871+0000","last_became_active":"2024-05-20T17:56:44.720693+0000","last_became_peered":"2024-05-20T17:56:44.720693+0000","last_unstale":"2024-05-20T17:58:43.681871+0000","last_undegraded":"2024-05-20T17:58:43.681871+0000","last_fullsized":"2024-05-20T17:58:43.681871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:44.223828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691476+0000","last_change":"2024-05-20T17:56:45.745925+0000","last_active":"2024-05-20T17:58:43.691476+0000","last_peered":"2024-05-20T17:58:43.691476+0000","last_clean":"2024-05-20T17:58:43.691476+0000","last_became_active":"2024-05-20T17:56:45.745303+0000","last_became_peered":"2024-05-20T17:56:45.745303+0000","last_unstale":"2024-05-20T17:58:43.691476+0000","last_undegraded":"2024-05-20T17:58:43.691476+0000","last_fullsized":"2024-05-20T17:58:43.691476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:41:00.230681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691275+0000","last_change":"2024-05-20T17:56:44.719048+0000","last_active":"2024-05-20T17:58:43.691275+0000","last_peered":"2024-05-20T17:58:43.691275+0000","last_clean":"2024-05-20T17:58:43.691275+0000","last_became_active":"2024-05-20T17:56:44.718397+0000","last_became_peered":"2024-05-20T17:56:44.718397+0000","last_unstale":"2024-05-20T17:58:43.691275+0000","last_undegraded":"2024-05-20T17:58:43.691275+0000","last_fullsized":"2024-05-20T17:58:43.691275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:11:53.898226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681642+0000","last_change":"2024-05-20T17:56:45.752395+0000","last_active":"2024-05-20T17:58:43.681642+0000","last_peered":"2024-05-20T17:58:43.681642+0000","last_clean":"2024-05-20T17:58:43.681642+0000","last_became_active":"2024-05-20T17:56:45.751940+0000","last_became_peered":"2024-05-20T17:56:45.751940+0000","last_unstale":"2024-05-20T17:58:43.681642+0000","last_undegraded":"2024-05-20T17:58:43.681642+0000","last_fullsized":"2024-05-20T17:58:43.681642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:33:57.780042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694331+0000","last_change":"2024-05-20T17:56:44.714712+0000","last_active":"2024-05-20T17:58:43.694331+0000","last_peered":"2024-05-20T17:58:43.694331+0000","last_clean":"2024-05-20T17:58:43.694331+0000","last_became_active":"2024-05-20T17:56:44.714396+0000","last_became_peered":"2024-05-20T17:56:44.714396+0000","last_unstale":"2024-05-20T17:58:43.694331+0000","last_undegraded":"2024-05-20T17:58:43.694331+0000","last_fullsized":"2024-05-20T17:58:43.694331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:21:04.831289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691360+0000","last_change":"2024-05-20T17:56:45.758160+0000","last_active":"2024-05-20T17:58:43.691360+0000","last_peered":"2024-05-20T17:58:43.691360+0000","last_clean":"2024-05-20T17:58:43.691360+0000","last_became_active":"2024-05-20T17:56:45.757405+0000","last_became_peered":"2024-05-20T17:56:45.757405+0000","last_unstale":"2024-05-20T17:58:43.691360+0000","last_undegraded":"2024-05-20T17:58:43.691360+0000","last_fullsized":"2024-05-20T17:58:43.691360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:17.625246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113310+0000","last_change":"2024-05-20T17:56:44.716342+0000","last_active":"2024-05-20T17:58:44.113310+0000","last_peered":"2024-05-20T17:58:44.113310+0000","last_clean":"2024-05-20T17:58:44.113310+0000","last_became_active":"2024-05-20T17:56:44.716010+0000","last_became_peered":"2024-05-20T17:56:44.716010+0000","last_unstale":"2024-05-20T17:58:44.113310+0000","last_undegraded":"2024-05-20T17:58:44.113310+0000","last_fullsized":"2024-05-20T17:58:44.113310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:00:25.073312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681615+0000","last_change":"2024-05-20T17:56:45.755288+0000","last_active":"2024-05-20T17:58:43.681615+0000","last_peered":"2024-05-20T17:58:43.681615+0000","last_clean":"2024-05-20T17:58:43.681615+0000","last_became_active":"2024-05-20T17:56:45.754952+0000","last_became_peered":"2024-05-20T17:56:45.754952+0000","last_unstale":"2024-05-20T17:58:43.681615+0000","last_undegraded":"2024-05-20T17:58:43.681615+0000","last_fullsized":"2024-05-20T17:58:43.681615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:44:27.631722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691102+0000","last_change":"2024-05-20T17:56:44.717887+0000","last_active":"2024-05-20T17:58:43.691102+0000","last_peered":"2024-05-20T17:58:43.691102+0000","last_clean":"2024-05-20T17:58:43.691102+0000","last_became_active":"2024-05-20T17:56:44.717581+0000","last_became_peered":"2024-05-20T17:56:44.717581+0000","last_unstale":"2024-05-20T17:58:43.691102+0000","last_undegraded":"2024-05-20T17:58:43.691102+0000","last_fullsized":"2024-05-20T17:58:43.691102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:42:54.703112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570033+0000","last_change":"2024-05-20T17:56:45.748448+0000","last_active":"2024-05-20T17:58:43.570033+0000","last_peered":"2024-05-20T17:58:43.570033+0000","last_clean":"2024-05-20T17:58:43.570033+0000","last_became_active":"2024-05-20T17:56:45.748016+0000","last_became_peered":"2024-05-20T17:56:45.748016+0000","last_unstale":"2024-05-20T17:58:43.570033+0000","last_undegraded":"2024-05-20T17:58:43.570033+0000","last_fullsized":"2024-05-20T17:58:43.570033+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:31:39.350477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694679+0000","last_change":"2024-05-20T17:56:44.709483+0000","last_active":"2024-05-20T17:58:43.694679+0000","last_peered":"2024-05-20T17:58:43.694679+0000","last_clean":"2024-05-20T17:58:43.694679+0000","last_became_active":"2024-05-20T17:56:44.708649+0000","last_became_peered":"2024-05-20T17:56:44.708649+0000","last_unstale":"2024-05-20T17:58:43.694679+0000","last_undegraded":"2024-05-20T17:58:43.694679+0000","last_fullsized":"2024-05-20T17:58:43.694679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:31:46.580256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.857082+0000","last_change":"2024-05-20T17:56:45.752279+0000","last_active":"2024-05-20T17:58:43.857082+0000","last_peered":"2024-05-20T17:58:43.857082+0000","last_clean":"2024-05-20T17:58:43.857082+0000","last_became_active":"2024-05-20T17:56:45.751790+0000","last_became_peered":"2024-05-20T17:56:45.751790+0000","last_unstale":"2024-05-20T17:58:43.857082+0000","last_undegraded":"2024-05-20T17:58:43.857082+0000","last_fullsized":"2024-05-20T17:58:43.857082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:06:38.113559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571437+0000","last_change":"2024-05-20T17:57:49.077752+0000","last_active":"2024-05-20T17:58:43.571437+0000","last_peered":"2024-05-20T17:58:43.571437+0000","last_clean":"2024-05-20T17:58:43.571437+0000","last_became_active":"2024-05-20T17:57:49.077277+0000","last_became_peered":"2024-05-20T17:57:49.077277+0000","last_unstale":"2024-05-20T17:58:43.571437+0000","last_undegraded":"2024-05-20T17:58:43.571437+0000","last_fullsized":"2024-05-20T17:58:43.571437+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:46:24.954772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.114092+0000","last_change":"2024-05-20T17:57:49.051693+0000","last_active":"2024-05-20T17:58:44.114092+0000","last_peered":"2024-05-20T17:58:44.114092+0000","last_clean":"2024-05-20T17:58:44.114092+0000","last_became_active":"2024-05-20T17:57:49.051215+0000","last_became_peered":"2024-05-20T17:57:49.051215+0000","last_unstale":"2024-05-20T17:58:44.114092+0000","last_undegraded":"2024-05-20T17:58:44.114092+0000","last_fullsized":"2024-05-20T17:58:44.114092+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:52:35.406908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691588+0000","last_change":"2024-05-20T17:56:44.709548+0000","last_active":"2024-05-20T17:58:43.691588+0000","last_peered":"2024-05-20T17:58:43.691588+0000","last_clean":"2024-05-20T17:58:43.691588+0000","last_became_active":"2024-05-20T17:56:44.709209+0000","last_became_peered":"2024-05-20T17:56:44.709209+0000","last_unstale":"2024-05-20T17:58:43.691588+0000","last_undegraded":"2024-05-20T17:58:43.691588+0000","last_fullsized":"2024-05-20T17:58:43.691588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:28:13.118336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856926+0000","last_change":"2024-05-20T17:56:45.755141+0000","last_active":"2024-05-20T17:58:43.856926+0000","last_peered":"2024-05-20T17:58:43.856926+0000","last_clean":"2024-05-20T17:58:43.856926+0000","last_became_active":"2024-05-20T17:56:45.754786+0000","last_became_peered":"2024-05-20T17:56:45.754786+0000","last_unstale":"2024-05-20T17:58:43.856926+0000","last_undegraded":"2024-05-20T17:58:43.856926+0000","last_fullsized":"2024-05-20T17:58:43.856926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:45:21.044602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":296,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.171070+0000","last_change":"2024-05-20T17:56:44.713578+0000","last_active":"2024-05-20T17:58:44.171070+0000","last_peered":"2024-05-20T17:58:44.171070+0000","last_clean":"2024-05-20T17:58:44.171070+0000","last_became_active":"2024-05-20T17:56:44.713203+0000","last_became_peered":"2024-05-20T17:56:44.713203+0000","last_unstale":"2024-05-20T17:58:44.171070+0000","last_undegraded":"2024-05-20T17:58:44.171070+0000","last_fullsized":"2024-05-20T17:58:44.171070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:36:54.386036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":242,"num_read_kb":187,"num_write":33,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681854+0000","last_change":"2024-05-20T17:56:45.752023+0000","last_active":"2024-05-20T17:58:43.681854+0000","last_peered":"2024-05-20T17:58:43.681854+0000","last_clean":"2024-05-20T17:58:43.681854+0000","last_became_active":"2024-05-20T17:56:45.751672+0000","last_became_peered":"2024-05-20T17:56:45.751672+0000","last_unstale":"2024-05-20T17:58:43.681854+0000","last_undegraded":"2024-05-20T17:58:43.681854+0000","last_fullsized":"2024-05-20T17:58:43.681854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:23:49.127016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569077+0000","last_change":"2024-05-20T17:56:44.717336+0000","last_active":"2024-05-20T17:58:43.569077+0000","last_peered":"2024-05-20T17:58:43.569077+0000","last_clean":"2024-05-20T17:58:43.569077+0000","last_became_active":"2024-05-20T17:56:44.717016+0000","last_became_peered":"2024-05-20T17:56:44.717016+0000","last_unstale":"2024-05-20T17:58:43.569077+0000","last_undegraded":"2024-05-20T17:58:43.569077+0000","last_fullsized":"2024-05-20T17:58:43.569077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:50:46.433461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113461+0000","last_change":"2024-05-20T17:56:45.752411+0000","last_active":"2024-05-20T17:58:44.113461+0000","last_peered":"2024-05-20T17:58:44.113461+0000","last_clean":"2024-05-20T17:58:44.113461+0000","last_became_active":"2024-05-20T17:56:45.752078+0000","last_became_peered":"2024-05-20T17:56:45.752078+0000","last_unstale":"2024-05-20T17:58:44.113461+0000","last_undegraded":"2024-05-20T17:58:44.113461+0000","last_fullsized":"2024-05-20T17:58:44.113461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:22:15.560666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691295+0000","last_change":"2024-05-20T17:56:44.723262+0000","last_active":"2024-05-20T17:58:43.691295+0000","last_peered":"2024-05-20T17:58:43.691295+0000","last_clean":"2024-05-20T17:58:43.691295+0000","last_became_active":"2024-05-20T17:56:44.722865+0000","last_became_peered":"2024-05-20T17:56:44.722865+0000","last_unstale":"2024-05-20T17:58:43.691295+0000","last_undegraded":"2024-05-20T17:58:43.691295+0000","last_fullsized":"2024-05-20T17:58:43.691295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:15:02.820718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571429+0000","last_change":"2024-05-20T17:57:49.052110+0000","last_active":"2024-05-20T17:58:43.571429+0000","last_peered":"2024-05-20T17:58:43.571429+0000","last_clean":"2024-05-20T17:58:43.571429+0000","last_became_active":"2024-05-20T17:57:49.051356+0000","last_became_peered":"2024-05-20T17:57:49.051356+0000","last_unstale":"2024-05-20T17:58:43.571429+0000","last_undegraded":"2024-05-20T17:58:43.571429+0000","last_fullsized":"2024-05-20T17:58:43.571429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:32.899615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113252+0000","last_change":"2024-05-20T17:56:44.718396+0000","last_active":"2024-05-20T17:58:44.113252+0000","last_peered":"2024-05-20T17:58:44.113252+0000","last_clean":"2024-05-20T17:58:44.113252+0000","last_became_active":"2024-05-20T17:56:44.718031+0000","last_became_peered":"2024-05-20T17:56:44.718031+0000","last_unstale":"2024-05-20T17:58:44.113252+0000","last_undegraded":"2024-05-20T17:58:44.113252+0000","last_fullsized":"2024-05-20T17:58:44.113252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:42:40.955520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571035+0000","last_change":"2024-05-20T17:56:45.759302+0000","last_active":"2024-05-20T17:58:43.571035+0000","last_peered":"2024-05-20T17:58:43.571035+0000","last_clean":"2024-05-20T17:58:43.571035+0000","last_became_active":"2024-05-20T17:56:45.758889+0000","last_became_peered":"2024-05-20T17:56:45.758889+0000","last_unstale":"2024-05-20T17:58:43.571035+0000","last_undegraded":"2024-05-20T17:58:43.571035+0000","last_fullsized":"2024-05-20T17:58:43.571035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:40.685471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681563+0000","last_change":"2024-05-20T17:56:44.726502+0000","last_active":"2024-05-20T17:58:43.681563+0000","last_peered":"2024-05-20T17:58:43.681563+0000","last_clean":"2024-05-20T17:58:43.681563+0000","last_became_active":"2024-05-20T17:56:44.726168+0000","last_became_peered":"2024-05-20T17:56:44.726168+0000","last_unstale":"2024-05-20T17:58:43.681563+0000","last_undegraded":"2024-05-20T17:58:43.681563+0000","last_fullsized":"2024-05-20T17:58:43.681563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:45:14.523968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691604+0000","last_change":"2024-05-20T17:56:45.759355+0000","last_active":"2024-05-20T17:58:43.691604+0000","last_peered":"2024-05-20T17:58:43.691604+0000","last_clean":"2024-05-20T17:58:43.691604+0000","last_became_active":"2024-05-20T17:56:45.758920+0000","last_became_peered":"2024-05-20T17:56:45.758920+0000","last_unstale":"2024-05-20T17:58:43.691604+0000","last_undegraded":"2024-05-20T17:58:43.691604+0000","last_fullsized":"2024-05-20T17:58:43.691604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:46:02.372250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682113+0000","last_change":"2024-05-20T17:56:44.718507+0000","last_active":"2024-05-20T17:58:43.682113+0000","last_peered":"2024-05-20T17:58:43.682113+0000","last_clean":"2024-05-20T17:58:43.682113+0000","last_became_active":"2024-05-20T17:56:44.718188+0000","last_became_peered":"2024-05-20T17:56:44.718188+0000","last_unstale":"2024-05-20T17:58:43.682113+0000","last_undegraded":"2024-05-20T17:58:43.682113+0000","last_fullsized":"2024-05-20T17:58:43.682113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:43:59.792942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859084+0000","last_change":"2024-05-20T17:56:45.751361+0000","last_active":"2024-05-20T17:58:43.859084+0000","last_peered":"2024-05-20T17:58:43.859084+0000","last_clean":"2024-05-20T17:58:43.859084+0000","last_became_active":"2024-05-20T17:56:45.750032+0000","last_became_peered":"2024-05-20T17:56:45.750032+0000","last_unstale":"2024-05-20T17:58:43.859084+0000","last_undegraded":"2024-05-20T17:58:43.859084+0000","last_fullsized":"2024-05-20T17:58:43.859084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:01:42.729156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570328+0000","last_change":"2024-05-20T17:56:44.714404+0000","last_active":"2024-05-20T17:58:43.570328+0000","last_peered":"2024-05-20T17:58:43.570328+0000","last_clean":"2024-05-20T17:58:43.570328+0000","last_became_active":"2024-05-20T17:56:44.714072+0000","last_became_peered":"2024-05-20T17:56:44.714072+0000","last_unstale":"2024-05-20T17:58:43.570328+0000","last_undegraded":"2024-05-20T17:58:43.570328+0000","last_fullsized":"2024-05-20T17:58:43.570328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:39:53.983513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694551+0000","last_change":"2024-05-20T17:56:45.749785+0000","last_active":"2024-05-20T17:58:43.694551+0000","last_peered":"2024-05-20T17:58:43.694551+0000","last_clean":"2024-05-20T17:58:43.694551+0000","last_became_active":"2024-05-20T17:56:45.749457+0000","last_became_peered":"2024-05-20T17:56:45.749457+0000","last_unstale":"2024-05-20T17:58:43.694551+0000","last_undegraded":"2024-05-20T17:58:43.694551+0000","last_fullsized":"2024-05-20T17:58:43.694551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:53:05.136882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571253+0000","last_change":"2024-05-20T17:56:44.722787+0000","last_active":"2024-05-20T17:58:43.571253+0000","last_peered":"2024-05-20T17:58:43.571253+0000","last_clean":"2024-05-20T17:58:43.571253+0000","last_became_active":"2024-05-20T17:56:44.722214+0000","last_became_peered":"2024-05-20T17:56:44.722214+0000","last_unstale":"2024-05-20T17:58:43.571253+0000","last_undegraded":"2024-05-20T17:58:43.571253+0000","last_fullsized":"2024-05-20T17:58:43.571253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:40:16.705718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.694565+0000","last_change":"2024-05-20T17:56:45.757515+0000","last_active":"2024-05-20T17:58:43.694565+0000","last_peered":"2024-05-20T17:58:43.694565+0000","last_clean":"2024-05-20T17:58:43.694565+0000","last_became_active":"2024-05-20T17:56:45.757174+0000","last_became_peered":"2024-05-20T17:56:45.757174+0000","last_unstale":"2024-05-20T17:58:43.694565+0000","last_undegraded":"2024-05-20T17:58:43.694565+0000","last_fullsized":"2024-05-20T17:58:43.694565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:18:14.154827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569893+0000","last_change":"2024-05-20T17:57:49.051489+0000","last_active":"2024-05-20T17:58:43.569893+0000","last_peered":"2024-05-20T17:58:43.569893+0000","last_clean":"2024-05-20T17:58:43.569893+0000","last_became_active":"2024-05-20T17:57:49.051070+0000","last_became_peered":"2024-05-20T17:57:49.051070+0000","last_unstale":"2024-05-20T17:58:43.569893+0000","last_undegraded":"2024-05-20T17:58:43.569893+0000","last_fullsized":"2024-05-20T17:58:43.569893+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:54:18.745185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113217+0000","last_change":"2024-05-20T17:56:44.724642+0000","last_active":"2024-05-20T17:58:44.113217+0000","last_peered":"2024-05-20T17:58:44.113217+0000","last_clean":"2024-05-20T17:58:44.113217+0000","last_became_active":"2024-05-20T17:56:44.724311+0000","last_became_peered":"2024-05-20T17:56:44.724311+0000","last_unstale":"2024-05-20T17:58:44.113217+0000","last_undegraded":"2024-05-20T17:58:44.113217+0000","last_fullsized":"2024-05-20T17:58:44.113217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:47:35.336236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681864+0000","last_change":"2024-05-20T17:56:45.758466+0000","last_active":"2024-05-20T17:58:43.681864+0000","last_peered":"2024-05-20T17:58:43.681864+0000","last_clean":"2024-05-20T17:58:43.681864+0000","last_became_active":"2024-05-20T17:56:45.758126+0000","last_became_peered":"2024-05-20T17:56:45.758126+0000","last_unstale":"2024-05-20T17:58:43.681864+0000","last_undegraded":"2024-05-20T17:58:43.681864+0000","last_fullsized":"2024-05-20T17:58:43.681864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:25:36.830828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113214+0000","last_change":"2024-05-20T17:56:44.711173+0000","last_active":"2024-05-20T17:58:44.113214+0000","last_peered":"2024-05-20T17:58:44.113214+0000","last_clean":"2024-05-20T17:58:44.113214+0000","last_became_active":"2024-05-20T17:56:44.710778+0000","last_became_peered":"2024-05-20T17:56:44.710778+0000","last_unstale":"2024-05-20T17:58:44.113214+0000","last_undegraded":"2024-05-20T17:58:44.113214+0000","last_fullsized":"2024-05-20T17:58:44.113214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:20:14.150346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571144+0000","last_change":"2024-05-20T17:55:34.871483+0000","last_active":"2024-05-20T17:58:43.571144+0000","last_peered":"2024-05-20T17:58:43.571144+0000","last_clean":"2024-05-20T17:58:43.571144+0000","last_became_active":"2024-05-20T17:55:34.871028+0000","last_became_peered":"2024-05-20T17:55:34.871028+0000","last_unstale":"2024-05-20T17:58:43.571144+0000","last_undegraded":"2024-05-20T17:58:43.571144+0000","last_fullsized":"2024-05-20T17:58:43.571144+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:52:08.599715+0000","last_clean_scrub_stamp":"2024-05-20T17:52:08.599715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:29:40.159577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113167+0000","last_change":"2024-05-20T17:56:45.747725+0000","last_active":"2024-05-20T17:58:44.113167+0000","last_peered":"2024-05-20T17:58:44.113167+0000","last_clean":"2024-05-20T17:58:44.113167+0000","last_became_active":"2024-05-20T17:56:45.747232+0000","last_became_peered":"2024-05-20T17:56:45.747232+0000","last_unstale":"2024-05-20T17:58:44.113167+0000","last_undegraded":"2024-05-20T17:58:44.113167+0000","last_fullsized":"2024-05-20T17:58:44.113167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:09:26.589905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571349+0000","last_change":"2024-05-20T17:56:44.721004+0000","last_active":"2024-05-20T17:58:43.571349+0000","last_peered":"2024-05-20T17:58:43.571349+0000","last_clean":"2024-05-20T17:58:43.571349+0000","last_became_active":"2024-05-20T17:56:44.720620+0000","last_became_peered":"2024-05-20T17:56:44.720620+0000","last_unstale":"2024-05-20T17:58:43.571349+0000","last_undegraded":"2024-05-20T17:58:43.571349+0000","last_fullsized":"2024-05-20T17:58:43.571349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:42:14.710663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681867+0000","last_change":"2024-05-20T17:56:44.714754+0000","last_active":"2024-05-20T17:58:43.681867+0000","last_peered":"2024-05-20T17:58:43.681867+0000","last_clean":"2024-05-20T17:58:43.681867+0000","last_became_active":"2024-05-20T17:56:44.714378+0000","last_became_peered":"2024-05-20T17:56:44.714378+0000","last_unstale":"2024-05-20T17:58:43.681867+0000","last_undegraded":"2024-05-20T17:58:43.681867+0000","last_fullsized":"2024-05-20T17:58:43.681867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:46:53.969960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113922+0000","last_change":"2024-05-20T17:56:45.742415+0000","last_active":"2024-05-20T17:58:44.113922+0000","last_peered":"2024-05-20T17:58:44.113922+0000","last_clean":"2024-05-20T17:58:44.113922+0000","last_became_active":"2024-05-20T17:56:45.741941+0000","last_became_peered":"2024-05-20T17:56:45.741941+0000","last_unstale":"2024-05-20T17:58:44.113922+0000","last_undegraded":"2024-05-20T17:58:44.113922+0000","last_fullsized":"2024-05-20T17:58:44.113922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:20:03.191021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'43","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.813704+0000","last_change":"2024-05-20T17:56:44.724361+0000","last_active":"2024-05-20T17:58:43.813704+0000","last_peered":"2024-05-20T17:58:43.813704+0000","last_clean":"2024-05-20T17:58:43.813704+0000","last_became_active":"2024-05-20T17:56:44.724033+0000","last_became_peered":"2024-05-20T17:56:44.724033+0000","last_unstale":"2024-05-20T17:58:43.813704+0000","last_undegraded":"2024-05-20T17:58:43.813704+0000","last_fullsized":"2024-05-20T17:58:43.813704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:10:13.756606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.570095+0000","last_change":"2024-05-20T17:57:49.051338+0000","last_active":"2024-05-20T17:58:43.570095+0000","last_peered":"2024-05-20T17:58:43.570095+0000","last_clean":"2024-05-20T17:58:43.570095+0000","last_became_active":"2024-05-20T17:57:49.050618+0000","last_became_peered":"2024-05-20T17:57:49.050618+0000","last_unstale":"2024-05-20T17:58:43.570095+0000","last_undegraded":"2024-05-20T17:58:43.570095+0000","last_fullsized":"2024-05-20T17:58:43.570095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T22:02:09.846175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113481+0000","last_change":"2024-05-20T17:56:44.716296+0000","last_active":"2024-05-20T17:58:44.113481+0000","last_peered":"2024-05-20T17:58:44.113481+0000","last_clean":"2024-05-20T17:58:44.113481+0000","last_became_active":"2024-05-20T17:56:44.715952+0000","last_became_peered":"2024-05-20T17:56:44.715952+0000","last_unstale":"2024-05-20T17:58:44.113481+0000","last_undegraded":"2024-05-20T17:58:44.113481+0000","last_fullsized":"2024-05-20T17:58:44.113481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:52:37.986399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113744+0000","last_change":"2024-05-20T17:56:45.751874+0000","last_active":"2024-05-20T17:58:44.113744+0000","last_peered":"2024-05-20T17:58:44.113744+0000","last_clean":"2024-05-20T17:58:44.113744+0000","last_became_active":"2024-05-20T17:56:45.751534+0000","last_became_peered":"2024-05-20T17:56:45.751534+0000","last_unstale":"2024-05-20T17:58:44.113744+0000","last_undegraded":"2024-05-20T17:58:44.113744+0000","last_fullsized":"2024-05-20T17:58:44.113744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:05:27.248115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856682+0000","last_change":"2024-05-20T17:56:44.724271+0000","last_active":"2024-05-20T17:58:43.856682+0000","last_peered":"2024-05-20T17:58:43.856682+0000","last_clean":"2024-05-20T17:58:43.856682+0000","last_became_active":"2024-05-20T17:56:44.723947+0000","last_became_peered":"2024-05-20T17:56:44.723947+0000","last_unstale":"2024-05-20T17:58:43.856682+0000","last_undegraded":"2024-05-20T17:58:43.856682+0000","last_fullsized":"2024-05-20T17:58:43.856682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:32:13.482059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681563+0000","last_change":"2024-05-20T17:56:45.751296+0000","last_active":"2024-05-20T17:58:43.681563+0000","last_peered":"2024-05-20T17:58:43.681563+0000","last_clean":"2024-05-20T17:58:43.681563+0000","last_became_active":"2024-05-20T17:56:45.748035+0000","last_became_peered":"2024-05-20T17:56:45.748035+0000","last_unstale":"2024-05-20T17:58:43.681563+0000","last_undegraded":"2024-05-20T17:58:43.681563+0000","last_fullsized":"2024-05-20T17:58:43.681563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:37:43.007041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681705+0000","last_change":"2024-05-20T17:56:44.720461+0000","last_active":"2024-05-20T17:58:43.681705+0000","last_peered":"2024-05-20T17:58:43.681705+0000","last_clean":"2024-05-20T17:58:43.681705+0000","last_became_active":"2024-05-20T17:56:44.720115+0000","last_became_peered":"2024-05-20T17:56:44.720115+0000","last_unstale":"2024-05-20T17:58:43.681705+0000","last_undegraded":"2024-05-20T17:58:43.681705+0000","last_fullsized":"2024-05-20T17:58:43.681705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:14:47.330193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.858870+0000","last_change":"2024-05-20T17:56:45.745025+0000","last_active":"2024-05-20T17:58:43.858870+0000","last_peered":"2024-05-20T17:58:43.858870+0000","last_clean":"2024-05-20T17:58:43.858870+0000","last_became_active":"2024-05-20T17:56:45.744626+0000","last_became_peered":"2024-05-20T17:56:45.744626+0000","last_unstale":"2024-05-20T17:58:43.858870+0000","last_undegraded":"2024-05-20T17:58:43.858870+0000","last_fullsized":"2024-05-20T17:58:43.858870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:08:51.052423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569045+0000","last_change":"2024-05-20T17:56:44.726099+0000","last_active":"2024-05-20T17:58:43.569045+0000","last_peered":"2024-05-20T17:58:43.569045+0000","last_clean":"2024-05-20T17:58:43.569045+0000","last_became_active":"2024-05-20T17:56:44.725714+0000","last_became_peered":"2024-05-20T17:56:44.725714+0000","last_unstale":"2024-05-20T17:58:43.569045+0000","last_undegraded":"2024-05-20T17:58:43.569045+0000","last_fullsized":"2024-05-20T17:58:43.569045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:40:51.326907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682248+0000","last_change":"2024-05-20T17:56:45.757376+0000","last_active":"2024-05-20T17:58:43.682248+0000","last_peered":"2024-05-20T17:58:43.682248+0000","last_clean":"2024-05-20T17:58:43.682248+0000","last_became_active":"2024-05-20T17:56:45.757053+0000","last_became_peered":"2024-05-20T17:56:45.757053+0000","last_unstale":"2024-05-20T17:58:43.682248+0000","last_undegraded":"2024-05-20T17:58:43.682248+0000","last_fullsized":"2024-05-20T17:58:43.682248+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:50:27.727055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681658+0000","last_change":"2024-05-20T17:56:44.726341+0000","last_active":"2024-05-20T17:58:43.681658+0000","last_peered":"2024-05-20T17:58:43.681658+0000","last_clean":"2024-05-20T17:58:43.681658+0000","last_became_active":"2024-05-20T17:56:44.726008+0000","last_became_peered":"2024-05-20T17:56:44.726008+0000","last_unstale":"2024-05-20T17:58:43.681658+0000","last_undegraded":"2024-05-20T17:58:43.681658+0000","last_fullsized":"2024-05-20T17:58:43.681658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:25:07.520179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571377+0000","last_change":"2024-05-20T17:56:45.760469+0000","last_active":"2024-05-20T17:58:43.571377+0000","last_peered":"2024-05-20T17:58:43.571377+0000","last_clean":"2024-05-20T17:58:43.571377+0000","last_became_active":"2024-05-20T17:56:45.760100+0000","last_became_peered":"2024-05-20T17:56:45.760100+0000","last_unstale":"2024-05-20T17:58:43.571377+0000","last_undegraded":"2024-05-20T17:58:43.571377+0000","last_fullsized":"2024-05-20T17:58:43.571377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T05:54:06.309355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113346+0000","last_change":"2024-05-20T17:57:49.051674+0000","last_active":"2024-05-20T17:58:44.113346+0000","last_peered":"2024-05-20T17:58:44.113346+0000","last_clean":"2024-05-20T17:58:44.113346+0000","last_became_active":"2024-05-20T17:57:49.051182+0000","last_became_peered":"2024-05-20T17:57:49.051182+0000","last_unstale":"2024-05-20T17:58:44.113346+0000","last_undegraded":"2024-05-20T17:58:44.113346+0000","last_fullsized":"2024-05-20T17:58:44.113346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:29:35.458339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571593+0000","last_change":"2024-05-20T17:56:45.757302+0000","last_active":"2024-05-20T17:58:43.571593+0000","last_peered":"2024-05-20T17:58:43.571593+0000","last_clean":"2024-05-20T17:58:43.571593+0000","last_became_active":"2024-05-20T17:56:45.754169+0000","last_became_peered":"2024-05-20T17:56:45.754169+0000","last_unstale":"2024-05-20T17:58:43.571593+0000","last_undegraded":"2024-05-20T17:58:43.571593+0000","last_fullsized":"2024-05-20T17:58:43.571593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T00:03:10.794342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.807590+0000","last_change":"2024-05-20T17:56:44.724478+0000","last_active":"2024-05-20T17:58:43.807590+0000","last_peered":"2024-05-20T17:58:43.807590+0000","last_clean":"2024-05-20T17:58:43.807590+0000","last_became_active":"2024-05-20T17:56:44.723927+0000","last_became_peered":"2024-05-20T17:56:44.723927+0000","last_unstale":"2024-05-20T17:58:43.807590+0000","last_undegraded":"2024-05-20T17:58:43.807590+0000","last_fullsized":"2024-05-20T17:58:43.807590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T21:04:12.003957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571188+0000","last_change":"2024-05-20T17:56:45.758135+0000","last_active":"2024-05-20T17:58:43.571188+0000","last_peered":"2024-05-20T17:58:43.571188+0000","last_clean":"2024-05-20T17:58:43.571188+0000","last_became_active":"2024-05-20T17:56:45.757358+0000","last_became_peered":"2024-05-20T17:56:45.757358+0000","last_unstale":"2024-05-20T17:58:43.571188+0000","last_undegraded":"2024-05-20T17:58:43.571188+0000","last_fullsized":"2024-05-20T17:58:43.571188+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T01:13:06.435055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.114107+0000","last_change":"2024-05-20T17:56:44.719822+0000","last_active":"2024-05-20T17:58:44.114107+0000","last_peered":"2024-05-20T17:58:44.114107+0000","last_clean":"2024-05-20T17:58:44.114107+0000","last_became_active":"2024-05-20T17:56:44.719454+0000","last_became_peered":"2024-05-20T17:56:44.719454+0000","last_unstale":"2024-05-20T17:58:44.114107+0000","last_undegraded":"2024-05-20T17:58:44.114107+0000","last_fullsized":"2024-05-20T17:58:44.114107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:43.691517+0000","last_clean_scrub_stamp":"2024-05-20T17:56:43.691517+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:03:21.048330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.571553+0000","last_change":"2024-05-20T17:56:45.753510+0000","last_active":"2024-05-20T17:58:43.571553+0000","last_peered":"2024-05-20T17:58:43.571553+0000","last_clean":"2024-05-20T17:58:43.571553+0000","last_became_active":"2024-05-20T17:56:45.752718+0000","last_became_peered":"2024-05-20T17:56:45.752718+0000","last_unstale":"2024-05-20T17:58:43.571553+0000","last_undegraded":"2024-05-20T17:58:43.571553+0000","last_fullsized":"2024-05-20T17:58:43.571553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T03:05:44.677138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.858961+0000","last_change":"2024-05-20T17:56:45.752528+0000","last_active":"2024-05-20T17:58:43.858961+0000","last_peered":"2024-05-20T17:58:43.858961+0000","last_clean":"2024-05-20T17:58:43.858961+0000","last_became_active":"2024-05-20T17:56:45.752171+0000","last_became_peered":"2024-05-20T17:56:45.752171+0000","last_unstale":"2024-05-20T17:58:43.858961+0000","last_undegraded":"2024-05-20T17:58:43.858961+0000","last_fullsized":"2024-05-20T17:58:43.858961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:12:55.826122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569068+0000","last_change":"2024-05-20T17:56:45.749228+0000","last_active":"2024-05-20T17:58:43.569068+0000","last_peered":"2024-05-20T17:58:43.569068+0000","last_clean":"2024-05-20T17:58:43.569068+0000","last_became_active":"2024-05-20T17:56:45.748886+0000","last_became_peered":"2024-05-20T17:56:45.748886+0000","last_unstale":"2024-05-20T17:58:43.569068+0000","last_undegraded":"2024-05-20T17:58:43.569068+0000","last_fullsized":"2024-05-20T17:58:43.569068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T18:23:10.569720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569035+0000","last_change":"2024-05-20T17:56:45.751105+0000","last_active":"2024-05-20T17:58:43.569035+0000","last_peered":"2024-05-20T17:58:43.569035+0000","last_clean":"2024-05-20T17:58:43.569035+0000","last_became_active":"2024-05-20T17:56:45.750754+0000","last_became_peered":"2024-05-20T17:56:45.750754+0000","last_unstale":"2024-05-20T17:58:43.569035+0000","last_undegraded":"2024-05-20T17:58:43.569035+0000","last_fullsized":"2024-05-20T17:58:43.569035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:57:48.327428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113562+0000","last_change":"2024-05-20T17:57:49.050574+0000","last_active":"2024-05-20T17:58:44.113562+0000","last_peered":"2024-05-20T17:58:44.113562+0000","last_clean":"2024-05-20T17:58:44.113562+0000","last_became_active":"2024-05-20T17:57:49.050037+0000","last_became_peered":"2024-05-20T17:57:49.050037+0000","last_unstale":"2024-05-20T17:58:44.113562+0000","last_undegraded":"2024-05-20T17:58:44.113562+0000","last_fullsized":"2024-05-20T17:58:44.113562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:01:56.453842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113800+0000","last_change":"2024-05-20T17:56:45.746122+0000","last_active":"2024-05-20T17:58:44.113800+0000","last_peered":"2024-05-20T17:58:44.113800+0000","last_clean":"2024-05-20T17:58:44.113800+0000","last_became_active":"2024-05-20T17:56:45.745773+0000","last_became_peered":"2024-05-20T17:56:45.745773+0000","last_unstale":"2024-05-20T17:58:44.113800+0000","last_undegraded":"2024-05-20T17:58:44.113800+0000","last_fullsized":"2024-05-20T17:58:44.113800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:21:27.909974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.859042+0000","last_change":"2024-05-20T17:56:45.755018+0000","last_active":"2024-05-20T17:58:43.859042+0000","last_peered":"2024-05-20T17:58:43.859042+0000","last_clean":"2024-05-20T17:58:43.859042+0000","last_became_active":"2024-05-20T17:56:45.754615+0000","last_became_peered":"2024-05-20T17:56:45.754615+0000","last_unstale":"2024-05-20T17:58:43.859042+0000","last_undegraded":"2024-05-20T17:58:43.859042+0000","last_fullsized":"2024-05-20T17:58:43.859042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:36:39.100566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.569612+0000","last_change":"2024-05-20T17:56:45.751091+0000","last_active":"2024-05-20T17:58:43.569612+0000","last_peered":"2024-05-20T17:58:43.569612+0000","last_clean":"2024-05-20T17:58:43.569612+0000","last_became_active":"2024-05-20T17:56:45.748573+0000","last_became_peered":"2024-05-20T17:56:45.748573+0000","last_unstale":"2024-05-20T17:58:43.569612+0000","last_undegraded":"2024-05-20T17:58:43.569612+0000","last_fullsized":"2024-05-20T17:58:43.569612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T23:11:16.140967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.682866+0000","last_change":"2024-05-20T17:56:45.756373+0000","last_active":"2024-05-20T17:58:43.682866+0000","last_peered":"2024-05-20T17:58:43.682866+0000","last_clean":"2024-05-20T17:58:43.682866+0000","last_became_active":"2024-05-20T17:56:45.755993+0000","last_became_peered":"2024-05-20T17:56:45.755993+0000","last_unstale":"2024-05-20T17:58:43.682866+0000","last_undegraded":"2024-05-20T17:58:43.682866+0000","last_fullsized":"2024-05-20T17:58:43.682866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T20:34:44.491341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.691184+0000","last_change":"2024-05-20T17:56:45.751866+0000","last_active":"2024-05-20T17:58:43.691184+0000","last_peered":"2024-05-20T17:58:43.691184+0000","last_clean":"2024-05-20T17:58:43.691184+0000","last_became_active":"2024-05-20T17:56:45.751518+0000","last_became_peered":"2024-05-20T17:56:45.751518+0000","last_unstale":"2024-05-20T17:58:43.691184+0000","last_undegraded":"2024-05-20T17:58:43.691184+0000","last_fullsized":"2024-05-20T17:58:43.691184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:38:18.736177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.681682+0000","last_change":"2024-05-20T17:56:45.749903+0000","last_active":"2024-05-20T17:58:43.681682+0000","last_peered":"2024-05-20T17:58:43.681682+0000","last_clean":"2024-05-20T17:58:43.681682+0000","last_became_active":"2024-05-20T17:56:45.749500+0000","last_became_peered":"2024-05-20T17:56:45.749500+0000","last_unstale":"2024-05-20T17:58:43.681682+0000","last_undegraded":"2024-05-20T17:58:43.681682+0000","last_fullsized":"2024-05-20T17:58:43.681682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:39:34.875111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.856827+0000","last_change":"2024-05-20T17:56:45.747370+0000","last_active":"2024-05-20T17:58:43.856827+0000","last_peered":"2024-05-20T17:58:43.856827+0000","last_clean":"2024-05-20T17:58:43.856827+0000","last_became_active":"2024-05-20T17:56:45.747036+0000","last_became_peered":"2024-05-20T17:56:45.747036+0000","last_unstale":"2024-05-20T17:58:43.856827+0000","last_undegraded":"2024-05-20T17:58:43.856827+0000","last_fullsized":"2024-05-20T17:58:43.856827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-21T19:02:02.220522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:44.113692+0000","last_change":"2024-05-20T17:56:45.752762+0000","last_active":"2024-05-20T17:58:44.113692+0000","last_peered":"2024-05-20T17:58:44.113692+0000","last_clean":"2024-05-20T17:58:44.113692+0000","last_became_active":"2024-05-20T17:56:45.752345+0000","last_became_peered":"2024-05-20T17:56:45.752345+0000","last_unstale":"2024-05-20T17:58:44.113692+0000","last_undegraded":"2024-05-20T17:58:44.113692+0000","last_fullsized":"2024-05-20T17:58:44.113692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T04:14:49.861456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-20T17:58:43.858880+0000","last_change":"2024-05-20T17:56:45.750229+0000","last_active":"2024-05-20T17:58:43.858880+0000","last_peered":"2024-05-20T17:58:43.858880+0000","last_clean":"2024-05-20T17:58:43.858880+0000","last_became_active":"2024-05-20T17:56:45.749872+0000","last_became_peered":"2024-05-20T17:56:45.749872+0000","last_unstale":"2024-05-20T17:58:43.858880+0000","last_undegraded":"2024-05-20T17:58:43.858880+0000","last_fullsized":"2024-05-20T17:58:43.858880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-20T17:56:44.696889+0000","last_clean_scrub_stamp":"2024-05-20T17:56:44.696889+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-22T02:56:58.015339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106947,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":593372,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":340,"num_read_kb":272,"num_write":196,"num_write_kb":1403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":1780116,"data_compressed":81486,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351012,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":462373,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612649,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1122304,"data_stored":1261374,"data_compressed":52401,"data_compressed_allocated":69632,"data_compressed_original":745472,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808879,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":865135,"data_compressed":27882,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070516,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":864575,"data_compressed":27882,"data_compressed_allocated":36864,"data_compressed_original":417792,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":1062404,"data_compressed":44385,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":462490,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":539677,"data_compressed":10596,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":966656,"data_stored":471427,"data_compressed":8736,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":539129,"data_compressed":10596,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":462711,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":462395,"data_compressed":5817,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":1062004,"data_compressed":44385,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35649,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68340,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67792,"data_compressed":1860,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393238,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":199552,"data_compressed":8016,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-20T17:58:45.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:58:45.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.564+0000 7f34863b7640 1 -- 172.21.15.129:0/2940971105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34801035b0 msgr2=0x7f34801059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.564+0000 7f34863b7640 1 --2- 172.21.15.129:0/2940971105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34801035b0 0x7f34801059d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3470009900 tx=0x7f347002a410 comp rx=0 tx=0).stop 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.564+0000 7f34863b7640 1 -- 172.21.15.129:0/2940971105 shutdown_connections 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.564+0000 7f34863b7640 1 --2- 172.21.15.129:0/2940971105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3480105f10 0x7f3480108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.564+0000 7f34863b7640 1 --2- 172.21.15.129:0/2940971105 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3480108870 0x7f348010ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.564+0000 7f34863b7640 1 --2- 172.21.15.129:0/2940971105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34801035b0 0x7f34801059d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.564+0000 7f34863b7640 1 -- 172.21.15.129:0/2940971105 >> 172.21.15.129:0/2940971105 conn(0x7f34800fcbb0 msgr2=0x7f34800ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.564+0000 7f34863b7640 1 -- 172.21.15.129:0/2940971105 shutdown_connections 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.564+0000 7f34863b7640 1 -- 172.21.15.129:0/2940971105 wait complete. 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f34863b7640 1 Processor -- start 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f34863b7640 1 -- start start 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f34863b7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34801035b0 0x7f3480199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f34863b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3480105f10 0x7f3480199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f34863b7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3480108870 0x7f34801a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f34863b7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f348010e710 con 0x7f3480108870 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f34863b7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f348010e590 con 0x7f3480105f10 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f34863b7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f348010e890 con 0x7f34801035b0 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f3484954640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3480108870 0x7f34801a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f347ffff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34801035b0 0x7f3480199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f3484954640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3480108870 0x7f34801a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42498/0 (socket says 172.21.15.129:42498) 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f3484954640 1 -- 172.21.15.129:0/690551163 learned_addr learned my addr 172.21.15.129:0/690551163 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f3477fff640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3480105f10 0x7f3480199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f347ffff640 1 -- 172.21.15.129:0/690551163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3480105f10 msgr2=0x7f3480199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f347ffff640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3480105f10 0x7f3480199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f347ffff640 1 -- 172.21.15.129:0/690551163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3480108870 msgr2=0x7f34801a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f347ffff640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3480108870 0x7f34801a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.565+0000 7f347ffff640 1 -- 172.21.15.129:0/690551163 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34801a0e50 con 0x7f34801035b0 2024-05-20T17:58:45.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.566+0000 7f347ffff640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34801035b0 0x7f3480199310 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3470006010 tx=0x7f34700042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:45.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.566+0000 7f347dffb640 1 -- 172.21.15.129:0/690551163 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347003f030 con 0x7f34801035b0 2024-05-20T17:58:45.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.566+0000 7f347dffb640 1 -- 172.21.15.129:0/690551163 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3470031a00 con 0x7f34801035b0 2024-05-20T17:58:45.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.566+0000 7f347dffb640 1 -- 172.21.15.129:0/690551163 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470035070 con 0x7f34801035b0 2024-05-20T17:58:45.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.566+0000 7f34863b7640 1 -- 172.21.15.129:0/690551163 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34801a10e0 con 0x7f34801035b0 2024-05-20T17:58:45.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.566+0000 7f34863b7640 1 -- 172.21.15.129:0/690551163 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34801a15d0 con 0x7f34801035b0 2024-05-20T17:58:45.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.567+0000 7f34863b7640 1 -- 172.21.15.129:0/690551163 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34800fe6d0 con 0x7f34801035b0 2024-05-20T17:58:45.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.568+0000 7f347dffb640 1 -- 172.21.15.129:0/690551163 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f3470042030 con 0x7f34801035b0 2024-05-20T17:58:45.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.569+0000 7f347dffb640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f346c0a2990 0x7f346c0a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.569+0000 7f3477fff640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f346c0a2990 0x7f346c0a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.569+0000 7f347dffb640 1 -- 172.21.15.129:0/690551163 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f34700eefe0 con 0x7f34801035b0 2024-05-20T17:58:45.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.569+0000 7f3477fff640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f346c0a2990 0x7f346c0a4ea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f348019a3b0 tx=0x7f3468005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:45.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.574+0000 7f347dffb640 1 -- 172.21.15.129:0/690551163 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f34700bbad0 con 0x7f34801035b0 2024-05-20T17:58:45.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.683+0000 7f347dffb640 1 -- 172.21.15.129:0/690551163 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mgrmap(e 22) ==== 137954+0+0 (secure 0 0 0) 0x7f34700b8140 con 0x7f34801035b0 2024-05-20T17:58:45.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.797+0000 7f34863b7640 1 -- 172.21.15.129:0/690551163 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3480062750 con 0x7f34801035b0 2024-05-20T17:58:45.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.799+0000 7f347dffb640 1 -- 172.21.15.129:0/690551163 <== mon.1 v2:172.21.15.204:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+23063 (secure 0 0 0) 0x7f3470071790 con 0x7f34801035b0 2024-05-20T17:58:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:45.797 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":89,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:58:43.562537+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2249940021":"2024-05-21T17:58:43.562501+0000","172.21.15.129:6800/2870089245":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/1943655536":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/2870089245":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1369259428":"2024-05-21T17:58:43.562501+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/549964701":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:58:45.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 -- 172.21.15.129:0/690551163 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f346c0a2990 msgr2=0x7f346c0a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f346c0a2990 0x7f346c0a4ea0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f348019a3b0 tx=0x7f3468005f90 comp rx=0 tx=0).stop 2024-05-20T17:58:45.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 -- 172.21.15.129:0/690551163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34801035b0 msgr2=0x7f3480199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34801035b0 0x7f3480199310 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3470006010 tx=0x7f34700042f0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 -- 172.21.15.129:0/690551163 shutdown_connections 2024-05-20T17:58:45.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f346c0a2990 0x7f346c0a4ea0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3480105f10 0x7f3480199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3480108870 0x7f34801a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 --2- 172.21.15.129:0/690551163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f34801035b0 0x7f3480199310 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 -- 172.21.15.129:0/690551163 >> 172.21.15.129:0/690551163 conn(0x7f34800fcbb0 msgr2=0x7f3480104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:45.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 -- 172.21.15.129:0/690551163 shutdown_connections 2024-05-20T17:58:45.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.801+0000 7f34863b7640 1 -- 172.21.15.129:0/690551163 wait complete. 2024-05-20T17:58:45.811 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-20T17:56:42.965073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-20T17:56:44.277499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T17:58:45.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-20T17:58:45.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.932+0000 7f31476f0640 1 -- 172.21.15.129:0/2583679285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31401028e0 msgr2=0x7f3140102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.932+0000 7f31476f0640 1 --2- 172.21.15.129:0/2583679285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31401028e0 0x7f3140102d90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3134009900 tx=0x7f313402a410 comp rx=0 tx=0).stop 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.932+0000 7f31476f0640 1 -- 172.21.15.129:0/2583679285 shutdown_connections 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.932+0000 7f31476f0640 1 --2- 172.21.15.129:0/2583679285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3140103420 0x7f3140109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.932+0000 7f31476f0640 1 --2- 172.21.15.129:0/2583679285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31401028e0 0x7f3140102d90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.932+0000 7f31476f0640 1 --2- 172.21.15.129:0/2583679285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3140101f00 0x7f3140102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.932+0000 7f31476f0640 1 -- 172.21.15.129:0/2583679285 >> 172.21.15.129:0/2583679285 conn(0x7f31400fc300 msgr2=0x7f31400fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.932+0000 7f31476f0640 1 -- 172.21.15.129:0/2583679285 shutdown_connections 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.932+0000 7f31476f0640 1 -- 172.21.15.129:0/2583679285 wait complete. 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f31476f0640 1 Processor -- start 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f31476f0640 1 -- start start 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f31476f0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3140101f00 0x7f31401011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f31476f0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31401028e0 0x7f314019e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f31476f0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3140103420 0x7f31401a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f31476f0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f314010d690 con 0x7f3140103420 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f31476f0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f314010d510 con 0x7f31401028e0 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f31476f0640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f314010d810 con 0x7f3140101f00 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f3145c8d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3140103420 0x7f31401a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f3145c8d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3140103420 0x7f31401a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42536/0 (socket says 172.21.15.129:42536) 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f3144c8b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31401028e0 0x7f314019e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f3145c8d640 1 -- 172.21.15.129:0/2318954710 learned_addr learned my addr 172.21.15.129:0/2318954710 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f314548c640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3140101f00 0x7f31401011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f3145c8d640 1 -- 172.21.15.129:0/2318954710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3140101f00 msgr2=0x7f31401011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f3145c8d640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3140101f00 0x7f31401011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f3145c8d640 1 -- 172.21.15.129:0/2318954710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31401028e0 msgr2=0x7f314019e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:45.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f3145c8d640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31401028e0 0x7f314019e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:45.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.933+0000 7f3145c8d640 1 -- 172.21.15.129:0/2318954710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31401a0e80 con 0x7f3140103420 2024-05-20T17:58:45.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.934+0000 7f3144c8b640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31401028e0 0x7f314019e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:45.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.934+0000 7f3145c8d640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3140103420 0x7f31401a0660 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f313800e970 tx=0x7f313800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:45.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.934+0000 7f313e7fc640 1 -- 172.21.15.129:0/2318954710 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313800ccc0 con 0x7f3140103420 2024-05-20T17:58:45.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.934+0000 7f313e7fc640 1 -- 172.21.15.129:0/2318954710 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f313800ce60 con 0x7f3140103420 2024-05-20T17:58:45.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.934+0000 7f31476f0640 1 -- 172.21.15.129:0/2318954710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31401a1170 con 0x7f3140103420 2024-05-20T17:58:45.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.934+0000 7f313e7fc640 1 -- 172.21.15.129:0/2318954710 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3138010490 con 0x7f3140103420 2024-05-20T17:58:45.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.934+0000 7f31476f0640 1 -- 172.21.15.129:0/2318954710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31401a1690 con 0x7f3140103420 2024-05-20T17:58:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.935+0000 7f31476f0640 1 -- 172.21.15.129:0/2318954710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3140069110 con 0x7f3140103420 2024-05-20T17:58:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.936+0000 7f313e7fc640 1 -- 172.21.15.129:0/2318954710 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) ==== 137954+0+0 (secure 0 0 0) 0x7f3138005020 con 0x7f3140103420 2024-05-20T17:58:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.937+0000 7f313e7fc640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31280a2d00 0x7f31280a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.937+0000 7f314548c640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31280a2d00 0x7f31280a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.937+0000 7f313e7fc640 1 -- 172.21.15.129:0/2318954710 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f31380cf7e0 con 0x7f3140103420 2024-05-20T17:58:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.938+0000 7f314548c640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31280a2d00 0x7f31280a5210 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f314006a4b0 tx=0x7f3130009500 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:45.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:45.942+0000 7f313e7fc640 1 -- 172.21.15.129:0/2318954710 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f313809c410 con 0x7f3140103420 2024-05-20T17:58:46.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.136+0000 7f31476f0640 1 -- 172.21.15.129:0/2318954710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f314002d2a0 con 0x7f3140103420 2024-05-20T17:58:46.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.136+0000 7f313e7fc640 1 -- 172.21.15.129:0/2318954710 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f313809bd10 con 0x7f3140103420 2024-05-20T17:58:46.134 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T17:58:46.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 -- 172.21.15.129:0/2318954710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31280a2d00 msgr2=0x7f31280a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31280a2d00 0x7f31280a5210 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f314006a4b0 tx=0x7f3130009500 comp rx=0 tx=0).stop 2024-05-20T17:58:46.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 -- 172.21.15.129:0/2318954710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3140103420 msgr2=0x7f31401a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3140103420 0x7f31401a0660 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f313800e970 tx=0x7f313800ed70 comp rx=0 tx=0).stop 2024-05-20T17:58:46.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 -- 172.21.15.129:0/2318954710 shutdown_connections 2024-05-20T17:58:46.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31280a2d00 0x7f31280a5210 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31401028e0 0x7f314019e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3140103420 0x7f31401a0660 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 --2- 172.21.15.129:0/2318954710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3140101f00 0x7f31401011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 -- 172.21.15.129:0/2318954710 >> 172.21.15.129:0/2318954710 conn(0x7f31400fc300 msgr2=0x7f3140103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:46.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 -- 172.21.15.129:0/2318954710 shutdown_connections 2024-05-20T17:58:46.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.139+0000 7f31476f0640 1 -- 172.21.15.129:0/2318954710 wait complete. 2024-05-20T17:58:46.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:45 smithi154 ceph-mon[36482]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:58:46.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:45 smithi154 ceph-mon[36482]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:46.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:45 smithi154 ceph-mon[36482]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:46.147 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:45 smithi129 ceph-mon[32029]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:58:46.147 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:45 smithi129 ceph-mon[32029]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:46.147 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:45 smithi129 ceph-mon[32029]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:46.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-20T17:58:46.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:45 smithi204 ceph-mon[37076]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:58:46.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:45 smithi204 ceph-mon[37076]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:46.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:45 smithi204 ceph-mon[37076]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:46.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.266+0000 7fa51532f640 1 -- 172.21.15.129:0/3132113069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5101031e0 msgr2=0x7fa510105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.266+0000 7fa51532f640 1 --2- 172.21.15.129:0/3132113069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5101031e0 0x7fa510105600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa500009860 tx=0x7fa50002a340 comp rx=0 tx=0).stop 2024-05-20T17:58:46.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.266+0000 7fa51532f640 1 -- 172.21.15.129:0/3132113069 shutdown_connections 2024-05-20T17:58:46.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.266+0000 7fa51532f640 1 --2- 172.21.15.129:0/3132113069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa510108740 0x7fa51010ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.266+0000 7fa51532f640 1 --2- 172.21.15.129:0/3132113069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa510105bd0 0x7fa5101080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.266+0000 7fa51532f640 1 --2- 172.21.15.129:0/3132113069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5101031e0 0x7fa510105600 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.266+0000 7fa51532f640 1 -- 172.21.15.129:0/3132113069 >> 172.21.15.129:0/3132113069 conn(0x7fa5100fcb80 msgr2=0x7fa5100ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:46.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.266+0000 7fa51532f640 1 -- 172.21.15.129:0/3132113069 shutdown_connections 2024-05-20T17:58:46.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.266+0000 7fa51532f640 1 -- 172.21.15.129:0/3132113069 wait complete. 2024-05-20T17:58:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa51532f640 1 Processor -- start 2024-05-20T17:58:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa51532f640 1 -- start start 2024-05-20T17:58:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa51532f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5101031e0 0x7fa5101995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa51532f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa510105bd0 0x7fa510199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa51532f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa510108740 0x7fa5101a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa51532f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa51010e6c0 con 0x7fa510108740 2024-05-20T17:58:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa51532f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa51010e540 con 0x7fa510105bd0 2024-05-20T17:58:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa51532f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa51010e840 con 0x7fa5101031e0 2024-05-20T17:58:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa50f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa510108740 0x7fa5101a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa50e7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa510105bd0 0x7fa510199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa50f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa510108740 0x7fa5101a0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42558/0 (socket says 172.21.15.129:42558) 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa50f7fe640 1 -- 172.21.15.129:0/3362555803 learned_addr learned my addr 172.21.15.129:0/3362555803 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.267+0000 7fa50effd640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5101031e0 0x7fa5101995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa50e7fc640 1 -- 172.21.15.129:0/3362555803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5101031e0 msgr2=0x7fa5101995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa50e7fc640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5101031e0 0x7fa5101995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa50e7fc640 1 -- 172.21.15.129:0/3362555803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa510108740 msgr2=0x7fa5101a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa50e7fc640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa510108740 0x7fa5101a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa50e7fc640 1 -- 172.21.15.129:0/3362555803 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5101a10f0 con 0x7fa510105bd0 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa50f7fe640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa510108740 0x7fa5101a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa50e7fc640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa510105bd0 0x7fa510199af0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa50400e9f0 tx=0x7fa50400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa4f7fff640 1 -- 172.21.15.129:0/3362555803 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50400ccc0 con 0x7fa510105bd0 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa4f7fff640 1 -- 172.21.15.129:0/3362555803 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa504005020 con 0x7fa510105bd0 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa51532f640 1 -- 172.21.15.129:0/3362555803 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5101a13e0 con 0x7fa510105bd0 2024-05-20T17:58:46.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa4f7fff640 1 -- 172.21.15.129:0/3362555803 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5040049e0 con 0x7fa510105bd0 2024-05-20T17:58:46.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.268+0000 7fa51532f640 1 -- 172.21.15.129:0/3362555803 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5101a1930 con 0x7fa510105bd0 2024-05-20T17:58:46.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.269+0000 7fa51532f640 1 -- 172.21.15.129:0/3362555803 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5100fe6a0 con 0x7fa510105bd0 2024-05-20T17:58:46.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.270+0000 7fa4f7fff640 1 -- 172.21.15.129:0/3362555803 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 22) ==== 137954+0+0 (secure 0 0 0) 0x7fa504005560 con 0x7fa510105bd0 2024-05-20T17:58:46.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.271+0000 7fa4f7fff640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa4f80a2d80 0x7fa4f80a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.271+0000 7fa4f7fff640 1 -- 172.21.15.129:0/3362555803 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7fa5040cf320 con 0x7fa510105bd0 2024-05-20T17:58:46.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.271+0000 7fa50effd640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa4f80a2d80 0x7fa4f80a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.272+0000 7fa50effd640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa4f80a2d80 0x7fa4f80a5290 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa5000305b0 tx=0x7fa500000bc0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:46.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.279+0000 7fa4f7fff640 1 -- 172.21.15.129:0/3362555803 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa5040051d0 con 0x7fa510105bd0 2024-05-20T17:58:46.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.465+0000 7fa51532f640 1 -- 172.21.15.129:0/3362555803 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa51002d460 con 0x7fa510105bd0 2024-05-20T17:58:46.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.466+0000 7fa4f7fff640 1 -- 172.21.15.129:0/3362555803 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fa50409bf50 con 0x7fa510105bd0 2024-05-20T17:58:46.464 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-05-20T17:58:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 -- 172.21.15.129:0/3362555803 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa4f80a2d80 msgr2=0x7fa4f80a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa4f80a2d80 0x7fa4f80a5290 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa5000305b0 tx=0x7fa500000bc0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 -- 172.21.15.129:0/3362555803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa510105bd0 msgr2=0x7fa510199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa510105bd0 0x7fa510199af0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa50400e9f0 tx=0x7fa50400eeb0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 -- 172.21.15.129:0/3362555803 shutdown_connections 2024-05-20T17:58:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa4f80a2d80 0x7fa4f80a5290 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa510105bd0 0x7fa510199af0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa510108740 0x7fa5101a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 --2- 172.21.15.129:0/3362555803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5101031e0 0x7fa5101995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 -- 172.21.15.129:0/3362555803 >> 172.21.15.129:0/3362555803 conn(0x7fa5100fcb80 msgr2=0x7fa510104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 -- 172.21.15.129:0/3362555803 shutdown_connections 2024-05-20T17:58:46.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.468+0000 7fa51532f640 1 -- 172.21.15.129:0/3362555803 wait complete. 2024-05-20T17:58:46.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-20T17:58:46.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.596+0000 7fed310b1640 1 -- 172.21.15.129:0/3061997346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed2c1011f0 msgr2=0x7fed2c10ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.596+0000 7fed310b1640 1 --2- 172.21.15.129:0/3061997346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed2c1011f0 0x7fed2c10ac50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fed24009ca0 tx=0x7fed2402a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.596+0000 7fed310b1640 1 -- 172.21.15.129:0/3061997346 shutdown_connections 2024-05-20T17:58:46.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.596+0000 7fed310b1640 1 --2- 172.21.15.129:0/3061997346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2c1006f0 0x7fed2c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.596+0000 7fed310b1640 1 --2- 172.21.15.129:0/3061997346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed2c1011f0 0x7fed2c10ac50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.596+0000 7fed310b1640 1 --2- 172.21.15.129:0/3061997346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed2c104040 0x7fed2c104450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.596+0000 7fed310b1640 1 -- 172.21.15.129:0/3061997346 >> 172.21.15.129:0/3061997346 conn(0x7fed2c0fc2f0 msgr2=0x7fed2c0fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:46.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.597+0000 7fed310b1640 1 -- 172.21.15.129:0/3061997346 shutdown_connections 2024-05-20T17:58:46.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.597+0000 7fed310b1640 1 -- 172.21.15.129:0/3061997346 wait complete. 2024-05-20T17:58:46.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.597+0000 7fed310b1640 1 Processor -- start 2024-05-20T17:58:46.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed310b1640 1 -- start start 2024-05-20T17:58:46.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed310b1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed2c1006f0 0x7fed2c199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed310b1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2c1011f0 0x7fed2c199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed310b1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed2c104040 0x7fed2c1a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed310b1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c10e700 con 0x7fed2c104040 2024-05-20T17:58:46.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed310b1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c10e580 con 0x7fed2c1011f0 2024-05-20T17:58:46.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed310b1640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c10e880 con 0x7fed2c1006f0 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2b59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed2c104040 0x7fed2c1a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2ad9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed2c1006f0 0x7fed2c199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2ad9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed2c1006f0 0x7fed2c199210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:51926/0 (socket says 172.21.15.129:51926) 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2ad9d640 1 -- 172.21.15.129:0/4022254263 learned_addr learned my addr 172.21.15.129:0/4022254263 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2c1011f0 0x7fed2c199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2b59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed2c104040 0x7fed2c1a06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42580/0 (socket says 172.21.15.129:42580) 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2ad9d640 1 -- 172.21.15.129:0/4022254263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2c1011f0 msgr2=0x7fed2c199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2ad9d640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2c1011f0 0x7fed2c199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2ad9d640 1 -- 172.21.15.129:0/4022254263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed2c104040 msgr2=0x7fed2c1a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2ad9d640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed2c104040 0x7fed2c1a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2ad9d640 1 -- 172.21.15.129:0/4022254263 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2c1a0d50 con 0x7fed2c1006f0 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.598+0000 7fed2ad9d640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed2c1006f0 0x7fed2c199210 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00e9f0 tx=0x7fed1c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.599+0000 7fed13fff640 1 -- 172.21.15.129:0/4022254263 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c00ccc0 con 0x7fed2c1006f0 2024-05-20T17:58:46.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.599+0000 7fed13fff640 1 -- 172.21.15.129:0/4022254263 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed1c0035f0 con 0x7fed2c1006f0 2024-05-20T17:58:46.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.599+0000 7fed310b1640 1 -- 172.21.15.129:0/4022254263 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed2c1a0fe0 con 0x7fed2c1006f0 2024-05-20T17:58:46.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.599+0000 7fed13fff640 1 -- 172.21.15.129:0/4022254263 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c00ce60 con 0x7fed2c1006f0 2024-05-20T17:58:46.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.599+0000 7fed310b1640 1 -- 172.21.15.129:0/4022254263 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed2c1a14a0 con 0x7fed2c1006f0 2024-05-20T17:58:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.600+0000 7fed310b1640 1 -- 172.21.15.129:0/4022254263 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed2c1099e0 con 0x7fed2c1006f0 2024-05-20T17:58:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.601+0000 7fed13fff640 1 -- 172.21.15.129:0/4022254263 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 22) ==== 137954+0+0 (secure 0 0 0) 0x7fed1c0037a0 con 0x7fed2c1006f0 2024-05-20T17:58:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.601+0000 7fed13fff640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed140a2d00 0x7fed140a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.602+0000 7fed2a59c640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed140a2d00 0x7fed140a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.602+0000 7fed13fff640 1 -- 172.21.15.129:0/4022254263 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7fed1c0cf740 con 0x7fed2c1006f0 2024-05-20T17:58:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.602+0000 7fed2a59c640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed140a2d00 0x7fed140a5210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed2c19a2b0 tx=0x7fed20005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:46.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.606+0000 7fed13fff640 1 -- 172.21.15.129:0/4022254263 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fed1c09c370 con 0x7fed2c1006f0 2024-05-20T17:58:46.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.801+0000 7fed310b1640 1 -- 172.21.15.129:0/4022254263 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fed2c062750 con 0x7fed2c1006f0 2024-05-20T17:58:46.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.802+0000 7fed13fff640 1 -- 172.21.15.129:0/4022254263 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fed1c0860a0 con 0x7fed2c1006f0 2024-05-20T17:58:46.801 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-05-20T17:58:46.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 -- 172.21.15.129:0/4022254263 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed140a2d00 msgr2=0x7fed140a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed140a2d00 0x7fed140a5210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed2c19a2b0 tx=0x7fed20005f90 comp rx=0 tx=0).stop 2024-05-20T17:58:46.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 -- 172.21.15.129:0/4022254263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed2c1006f0 msgr2=0x7fed2c199210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed2c1006f0 0x7fed2c199210 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00e9f0 tx=0x7fed1c00eeb0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 -- 172.21.15.129:0/4022254263 shutdown_connections 2024-05-20T17:58:46.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed140a2d00 0x7fed140a5210 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2c1011f0 0x7fed2c199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed2c104040 0x7fed2c1a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 --2- 172.21.15.129:0/4022254263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed2c1006f0 0x7fed2c199210 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 -- 172.21.15.129:0/4022254263 >> 172.21.15.129:0/4022254263 conn(0x7fed2c0fc2f0 msgr2=0x7fed2c106bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:46.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 -- 172.21.15.129:0/4022254263 shutdown_connections 2024-05-20T17:58:46.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.805+0000 7fed310b1640 1 -- 172.21.15.129:0/4022254263 wait complete. 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/690551163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2318954710' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:46.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:46 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3362555803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:46.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:58:46.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.938+0000 7f1fcf765640 1 -- 172.21.15.129:0/2316399849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fc81006e0 msgr2=0x7f1fc8100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.938+0000 7f1fcf765640 1 --2- 172.21.15.129:0/2316399849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fc81006e0 0x7f1fc8100b50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009900 tx=0x7f1fc402a410 comp rx=0 tx=0).stop 2024-05-20T17:58:46.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.938+0000 7f1fcf765640 1 -- 172.21.15.129:0/2316399849 shutdown_connections 2024-05-20T17:58:46.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.938+0000 7f1fcf765640 1 --2- 172.21.15.129:0/2316399849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc81011e0 0x7f1fc810ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.938+0000 7f1fcf765640 1 --2- 172.21.15.129:0/2316399849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fc81006e0 0x7f1fc8100b50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.938+0000 7f1fcf765640 1 --2- 172.21.15.129:0/2316399849 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc8104030 0x7f1fc8104440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.938+0000 7f1fcf765640 1 -- 172.21.15.129:0/2316399849 >> 172.21.15.129:0/2316399849 conn(0x7f1fc80fc2e0 msgr2=0x7f1fc80fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:46.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.938+0000 7f1fcf765640 1 -- 172.21.15.129:0/2316399849 shutdown_connections 2024-05-20T17:58:46.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.938+0000 7f1fcf765640 1 -- 172.21.15.129:0/2316399849 wait complete. 2024-05-20T17:58:46.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcf765640 1 Processor -- start 2024-05-20T17:58:46.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcf765640 1 -- start start 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcf765640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fc81006e0 0x7f1fc81992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcf765640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc81011e0 0x7f1fc8199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcf765640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc8104030 0x7f1fc81a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcf765640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc810e710 con 0x7f1fc81006e0 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcf765640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc810e590 con 0x7f1fc8104030 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcf765640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc810e890 con 0x7f1fc81011e0 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcd501640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fc81006e0 0x7f1fc81992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcd501640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fc81006e0 0x7f1fc81992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42594/0 (socket says 172.21.15.129:42594) 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcd501640 1 -- 172.21.15.129:0/4084710359 learned_addr learned my addr 172.21.15.129:0/4084710359 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fccd00640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc81011e0 0x7f1fc8199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcdd02640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc8104030 0x7f1fc81a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcd501640 1 -- 172.21.15.129:0/4084710359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc81011e0 msgr2=0x7f1fc8199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcd501640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc81011e0 0x7f1fc8199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcd501640 1 -- 172.21.15.129:0/4084710359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc8104030 msgr2=0x7f1fc81a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:46.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcd501640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc8104030 0x7f1fc81a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:46.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.939+0000 7f1fcd501640 1 -- 172.21.15.129:0/4084710359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc81a0e00 con 0x7f1fc81006e0 2024-05-20T17:58:46.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.940+0000 7f1fcd501640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fc81006e0 0x7f1fc81992c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800e9f0 tx=0x7f1fb800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:46.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.940+0000 7f1fbe7fc640 1 -- 172.21.15.129:0/4084710359 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb800ccc0 con 0x7f1fc81006e0 2024-05-20T17:58:46.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.940+0000 7f1fcf765640 1 -- 172.21.15.129:0/4084710359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc81a10f0 con 0x7f1fc81006e0 2024-05-20T17:58:46.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.940+0000 7f1fbe7fc640 1 -- 172.21.15.129:0/4084710359 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fb80035f0 con 0x7f1fc81006e0 2024-05-20T17:58:46.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.940+0000 7f1fbe7fc640 1 -- 172.21.15.129:0/4084710359 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb800ce60 con 0x7f1fc81006e0 2024-05-20T17:58:46.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.941+0000 7f1fcf765640 1 -- 172.21.15.129:0/4084710359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc81a15f0 con 0x7f1fc81006e0 2024-05-20T17:58:46.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.941+0000 7f1fcf765640 1 -- 172.21.15.129:0/4084710359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc81079c0 con 0x7f1fc81006e0 2024-05-20T17:58:46.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.943+0000 7f1fbe7fc640 1 -- 172.21.15.129:0/4084710359 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) ==== 137954+0+0 (secure 0 0 0) 0x7f1fb80037a0 con 0x7f1fc81006e0 2024-05-20T17:58:46.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.943+0000 7f1fbe7fc640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fb40a2d00 0x7f1fb40a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:46.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.943+0000 7f1fccd00640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fb40a2d00 0x7f1fb40a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:46.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.943+0000 7f1fbe7fc640 1 -- 172.21.15.129:0/4084710359 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f1fb80cf9f0 con 0x7f1fc81006e0 2024-05-20T17:58:46.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.944+0000 7f1fccd00640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fb40a2d00 0x7f1fb40a5210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009230 tx=0x7f1fc4031400 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:46.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:46.948+0000 7f1fbe7fc640 1 -- 172.21.15.129:0/4084710359 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1fb809c620 con 0x7f1fc81006e0 2024-05-20T17:58:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:58:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-05-20T17:58:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/690551163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:47.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2318954710' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:47.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:46 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3362555803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:47.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-20T17:58:47.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:58:47.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-05-20T17:58:47.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/690551163' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2318954710' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T17:58:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:46 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3362555803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T17:58:47.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.225+0000 7f1fcf765640 1 -- 172.21.15.129:0/4084710359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1fc8062750 con 0x7f1fc81006e0 2024-05-20T17:58:47.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.226+0000 7f1fbe7fc640 1 -- 172.21.15.129:0/4084710359 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11829 (secure 0 0 0) 0x7f1fb809bf20 con 0x7f1fc81006e0 2024-05-20T17:58:47.224 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:47.224 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:56:49.336282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24521":{"gid":24521,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1478533749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1478533749},{"type":"v1","addr":"172.21.15.154:6837","nonce":1478533749}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T17:58:47.224 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 20 2024-05-20T17:58:47.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.228+0000 7f1fcf765640 1 -- 172.21.15.129:0/4084710359 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fb40a2d00 msgr2=0x7f1fb40a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.228+0000 7f1fcf765640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fb40a2d00 0x7f1fb40a5210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009230 tx=0x7f1fc4031400 comp rx=0 tx=0).stop 2024-05-20T17:58:47.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.228+0000 7f1fcf765640 1 -- 172.21.15.129:0/4084710359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fc81006e0 msgr2=0x7f1fc81992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.228+0000 7f1fcf765640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fc81006e0 0x7f1fc81992c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800e9f0 tx=0x7f1fb800eeb0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.228+0000 7f1fcf765640 1 -- 172.21.15.129:0/4084710359 shutdown_connections 2024-05-20T17:58:47.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.228+0000 7f1fcf765640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fb40a2d00 0x7f1fb40a5210 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.228+0000 7f1fcf765640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fc8104030 0x7f1fc81a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.228+0000 7f1fcf765640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc81011e0 0x7f1fc8199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.228+0000 7f1fcf765640 1 --2- 172.21.15.129:0/4084710359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fc81006e0 0x7f1fc81992c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.228+0000 7f1fcf765640 1 -- 172.21.15.129:0/4084710359 >> 172.21.15.129:0/4084710359 conn(0x7f1fc80fc2e0 msgr2=0x7f1fc8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.229+0000 7f1fcf765640 1 -- 172.21.15.129:0/4084710359 shutdown_connections 2024-05-20T17:58:47.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.229+0000 7f1fcf765640 1 -- 172.21.15.129:0/4084710359 wait complete. 2024-05-20T17:58:47.245 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T17:58:47.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.366+0000 7ffaaee08640 1 -- 172.21.15.129:0/3131710368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffaa8102350 msgr2=0x7ffaa8102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.366+0000 7ffaaee08640 1 --2- 172.21.15.129:0/3131710368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffaa8102350 0x7ffaa8102760 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ffa9400aa10 tx=0x7ffa9402b720 comp rx=0 tx=0).stop 2024-05-20T17:58:47.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.367+0000 7ffaaee08640 1 -- 172.21.15.129:0/3131710368 shutdown_connections 2024-05-20T17:58:47.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.367+0000 7ffaaee08640 1 --2- 172.21.15.129:0/3131710368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa8107050 0x7ffaa8109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.367+0000 7ffaaee08640 1 --2- 172.21.15.129:0/3131710368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffaa8102350 0x7ffaa8102760 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.367+0000 7ffaaee08640 1 --2- 172.21.15.129:0/3131710368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffaa8101110 0x7ffaa8101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.367+0000 7ffaaee08640 1 -- 172.21.15.129:0/3131710368 >> 172.21.15.129:0/3131710368 conn(0x7ffaa80fc2e0 msgr2=0x7ffaa80fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.367+0000 7ffaaee08640 1 -- 172.21.15.129:0/3131710368 shutdown_connections 2024-05-20T17:58:47.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.367+0000 7ffaaee08640 1 -- 172.21.15.129:0/3131710368 wait complete. 2024-05-20T17:58:47.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.367+0000 7ffaaee08640 1 Processor -- start 2024-05-20T17:58:47.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaaee08640 1 -- start start 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaaee08640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffaa8101110 0x7ffaa81992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaaee08640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffaa8102350 0x7ffaa8199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaaee08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa8107050 0x7ffaa81a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaaee08640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa810c4f0 con 0x7ffaa8102350 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaaee08640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa810c370 con 0x7ffaa8107050 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaaee08640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa810c670 con 0x7ffaa8101110 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaa7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffaa8102350 0x7ffaa8199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaa7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffaa8102350 0x7ffaa8199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42612/0 (socket says 172.21.15.129:42612) 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaa7fff640 1 -- 172.21.15.129:0/147904422 learned_addr learned my addr 172.21.15.129:0/147904422 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaacba4640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffaa8101110 0x7ffaa81992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaad3a5640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa8107050 0x7ffaa81a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaa7fff640 1 -- 172.21.15.129:0/147904422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffaa8101110 msgr2=0x7ffaa81992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaa7fff640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffaa8101110 0x7ffaa81992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaa7fff640 1 -- 172.21.15.129:0/147904422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa8107050 msgr2=0x7ffaa81a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaa7fff640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa8107050 0x7ffaa81a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaa7fff640 1 -- 172.21.15.129:0/147904422 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa81a0dc0 con 0x7ffaa8102350 2024-05-20T17:58:47.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaacba4640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffaa8101110 0x7ffaa81992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:47.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.368+0000 7ffaa7fff640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffaa8102350 0x7ffaa8199830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ffa94005ab0 tx=0x7ffa940042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.369+0000 7ffaa5ffb640 1 -- 172.21.15.129:0/147904422 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9403f070 con 0x7ffaa8102350 2024-05-20T17:58:47.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.369+0000 7ffaa5ffb640 1 -- 172.21.15.129:0/147904422 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa940044c0 con 0x7ffaa8102350 2024-05-20T17:58:47.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.369+0000 7ffaa5ffb640 1 -- 172.21.15.129:0/147904422 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa94036040 con 0x7ffaa8102350 2024-05-20T17:58:47.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.369+0000 7ffaaee08640 1 -- 172.21.15.129:0/147904422 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa81a1050 con 0x7ffaa8102350 2024-05-20T17:58:47.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.369+0000 7ffaaee08640 1 -- 172.21.15.129:0/147904422 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa81a14f0 con 0x7ffaa8102350 2024-05-20T17:58:47.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.370+0000 7ffaaee08640 1 -- 172.21.15.129:0/147904422 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa8105e60 con 0x7ffaa8102350 2024-05-20T17:58:47.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.372+0000 7ffaa5ffb640 1 -- 172.21.15.129:0/147904422 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) ==== 137954+0+0 (secure 0 0 0) 0x7ffa94042070 con 0x7ffaa8102350 2024-05-20T17:58:47.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.373+0000 7ffaa5ffb640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa980a2d80 0x7ffa980a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.373+0000 7ffaa5ffb640 1 -- 172.21.15.129:0/147904422 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7ffa940efd70 con 0x7ffaa8102350 2024-05-20T17:58:47.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.373+0000 7ffaacba4640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa980a2d80 0x7ffa980a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.374+0000 7ffaacba4640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa980a2d80 0x7ffa980a5290 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8101d10 tx=0x7ffa9c005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.377+0000 7ffaa5ffb640 1 -- 172.21.15.129:0/147904422 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffa940bc350 con 0x7ffaa8102350 2024-05-20T17:58:47.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.618+0000 7ffaaee08640 1 -- 172.21.15.129:0/147904422 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffaa8062750 con 0x7ffaa8102350 2024-05-20T17:58:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.620+0000 7ffaa5ffb640 1 -- 172.21.15.129:0/147904422 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+23063 (secure 0 0 0) 0x7ffa940c0650 con 0x7ffaa8102350 2024-05-20T17:58:47.619 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:58:47.619 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":89,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T17:58:43.562537+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2249940021":"2024-05-21T17:58:43.562501+0000","172.21.15.129:6800/2870089245":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/1943655536":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/2870089245":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1369259428":"2024-05-21T17:58:43.562501+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/549964701":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T17:58:47.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.622+0000 7ffaaee08640 1 -- 172.21.15.129:0/147904422 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa980a2d80 msgr2=0x7ffa980a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.622+0000 7ffaaee08640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa980a2d80 0x7ffa980a5290 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8101d10 tx=0x7ffa9c005f90 comp rx=0 tx=0).stop 2024-05-20T17:58:47.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.622+0000 7ffaaee08640 1 -- 172.21.15.129:0/147904422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffaa8102350 msgr2=0x7ffaa8199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.622+0000 7ffaaee08640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffaa8102350 0x7ffaa8199830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ffa94005ab0 tx=0x7ffa940042d0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.623+0000 7ffaaee08640 1 -- 172.21.15.129:0/147904422 shutdown_connections 2024-05-20T17:58:47.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.623+0000 7ffaaee08640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa980a2d80 0x7ffa980a5290 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.623+0000 7ffaaee08640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffaa8107050 0x7ffaa81a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.623+0000 7ffaaee08640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffaa8102350 0x7ffaa8199830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.623+0000 7ffaaee08640 1 --2- 172.21.15.129:0/147904422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffaa8101110 0x7ffaa81992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.623+0000 7ffaaee08640 1 -- 172.21.15.129:0/147904422 >> 172.21.15.129:0/147904422 conn(0x7ffaa80fc2e0 msgr2=0x7ffaa8103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.623+0000 7ffaaee08640 1 -- 172.21.15.129:0/147904422 shutdown_connections 2024-05-20T17:58:47.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.623+0000 7ffaaee08640 1 -- 172.21.15.129:0/147904422 wait complete. 2024-05-20T17:58:47.635 DEBUG:tasks.fwd_scrub:Yielding 2024-05-20T17:58:47.635 INFO:teuthology.run_tasks:Running task exec... 2024-05-20T17:58:47.647 INFO:teuthology.task.exec:Executing custom commands... 2024-05-20T17:58:47.647 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi129.front.sepia.ceph.com 2024-05-20T17:58:47.647 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-05-20T17:58:47.648 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-20T17:58:47.648 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T17:58:47.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:58:47.648 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T17:58:47.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.773+0000 7f59662e2640 1 -- 172.21.15.129:0/2108261857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5960100e10 msgr2=0x7f5960101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.773+0000 7f59662e2640 1 --2- 172.21.15.129:0/2108261857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5960100e10 0x7f5960101220 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5950009900 tx=0x7f595002a410 comp rx=0 tx=0).stop 2024-05-20T17:58:47.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.773+0000 7f59662e2640 1 -- 172.21.15.129:0/2108261857 shutdown_connections 2024-05-20T17:58:47.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.773+0000 7f59662e2640 1 --2- 172.21.15.129:0/2108261857 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5960108780 0x7f596010ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.773+0000 7f59662e2640 1 --2- 172.21.15.129:0/2108261857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5960105c10 0x7f59601080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.773+0000 7f59662e2640 1 --2- 172.21.15.129:0/2108261857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5960100e10 0x7f5960101220 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.773+0000 7f59662e2640 1 -- 172.21.15.129:0/2108261857 >> 172.21.15.129:0/2108261857 conn(0x7f59600fc340 msgr2=0x7f59600fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.773+0000 7f59662e2640 1 -- 172.21.15.129:0/2108261857 shutdown_connections 2024-05-20T17:58:47.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.773+0000 7f59662e2640 1 -- 172.21.15.129:0/2108261857 wait complete. 2024-05-20T17:58:47.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.774+0000 7f59662e2640 1 Processor -- start 2024-05-20T17:58:47.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.774+0000 7f59662e2640 1 -- start start 2024-05-20T17:58:47.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.774+0000 7f59662e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5960100e10 0x7f59601992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.774+0000 7f59662e2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5960105c10 0x7f59601997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.774+0000 7f59662e2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5960108780 0x7f59601a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.774+0000 7f59662e2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f596010e730 con 0x7f5960100e10 2024-05-20T17:58:47.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.774+0000 7f59662e2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f596010e5b0 con 0x7f5960105c10 2024-05-20T17:58:47.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.774+0000 7f59662e2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f596010e8b0 con 0x7f5960108780 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5960105c10 0x7f59601997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5960100e10 0x7f59601992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5960100e10 0x7f59601992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42630/0 (socket says 172.21.15.129:42630) 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595ffff640 1 -- 172.21.15.129:0/212034662 learned_addr learned my addr 172.21.15.129:0/212034662 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5960105c10 0x7f59601997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:54208/0 (socket says 172.21.15.129:54208) 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f596487f640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5960108780 0x7f59601a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595f7fe640 1 -- 172.21.15.129:0/212034662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5960108780 msgr2=0x7f59601a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595f7fe640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5960108780 0x7f59601a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595f7fe640 1 -- 172.21.15.129:0/212034662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5960100e10 msgr2=0x7f59601992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595f7fe640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5960100e10 0x7f59601992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595f7fe640 1 -- 172.21.15.129:0/212034662 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59601a0de0 con 0x7f5960105c10 2024-05-20T17:58:47.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595ffff640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5960100e10 0x7f59601992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:47.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f596487f640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5960108780 0x7f59601a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:47.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.775+0000 7f595f7fe640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5960105c10 0x7f59601997e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f595400e9f0 tx=0x7f595400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.776+0000 7f595d7fa640 1 -- 172.21.15.129:0/212034662 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595400cd80 con 0x7f5960105c10 2024-05-20T17:58:47.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.776+0000 7f595d7fa640 1 -- 172.21.15.129:0/212034662 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59540035f0 con 0x7f5960105c10 2024-05-20T17:58:47.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.776+0000 7f59662e2640 1 -- 172.21.15.129:0/212034662 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59601a10d0 con 0x7f5960105c10 2024-05-20T17:58:47.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.776+0000 7f595d7fa640 1 -- 172.21.15.129:0/212034662 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954010690 con 0x7f5960105c10 2024-05-20T17:58:47.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.776+0000 7f59662e2640 1 -- 172.21.15.129:0/212034662 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59601a1670 con 0x7f5960105c10 2024-05-20T17:58:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.777+0000 7f59662e2640 1 -- 172.21.15.129:0/212034662 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59600fde60 con 0x7f5960105c10 2024-05-20T17:58:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.778+0000 7f595d7fa640 1 -- 172.21.15.129:0/212034662 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 22) ==== 137954+0+0 (secure 0 0 0) 0x7f5954010830 con 0x7f5960105c10 2024-05-20T17:58:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.779+0000 7f595d7fa640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f59480a2df0 0x7f59480a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.779+0000 7f595ffff640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f59480a2df0 0x7f59480a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.779+0000 7f595d7fa640 1 -- 172.21.15.129:0/212034662 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f59540cf6b0 con 0x7f5960105c10 2024-05-20T17:58:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.780+0000 7f595ffff640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f59480a2df0 0x7f59480a5300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5950004730 tx=0x7f59500046c0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.784+0000 7f595d7fa640 1 -- 172.21.15.129:0/212034662 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f595409c280 con 0x7f5960105c10 2024-05-20T17:58:47.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.785+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1842316212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d28101990 msgr2=0x7f9d28102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.785+0000 7f9d2d3c2640 1 --2- 172.21.15.129:0/1842316212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d28101990 0x7f9d28102690 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009900 tx=0x7f9d1802a410 comp rx=0 tx=0).stop 2024-05-20T17:58:47.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.785+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1842316212 shutdown_connections 2024-05-20T17:58:47.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.785+0000 7f9d2d3c2640 1 --2- 172.21.15.129:0/1842316212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d281078d0 0x7f9d28109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.785+0000 7f9d2d3c2640 1 --2- 172.21.15.129:0/1842316212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d28102bd0 0x7f9d28102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.785+0000 7f9d2d3c2640 1 --2- 172.21.15.129:0/1842316212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d28101990 0x7f9d28102690 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.785+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1842316212 >> 172.21.15.129:0/1842316212 conn(0x7f9d280fcb80 msgr2=0x7f9d280ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.785+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1842316212 shutdown_connections 2024-05-20T17:58:47.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.785+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1842316212 wait complete. 2024-05-20T17:58:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.786+0000 7f9d2d3c2640 1 Processor -- start 2024-05-20T17:58:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.786+0000 7f9d2d3c2640 1 -- start start 2024-05-20T17:58:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d2d3c2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d28101990 0x7f9d28197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d2d3c2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d28102bd0 0x7f9d281978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d2d3c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d281078d0 0x7f9d2819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d2d3c2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2810c540 con 0x7f9d281078d0 2024-05-20T17:58:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d2d3c2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2810c3c0 con 0x7f9d28102bd0 2024-05-20T17:58:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d2d3c2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2810c6c0 con 0x7f9d28101990 2024-05-20T17:58:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d26ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d28101990 0x7f9d28197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d277fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d281078d0 0x7f9d2819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d267fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d28102bd0 0x7f9d281978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d26ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d28101990 0x7f9d28197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:52002/0 (socket says 172.21.15.129:52002) 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d267fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d28102bd0 0x7f9d281978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:54222/0 (socket says 172.21.15.129:54222) 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d26ffd640 1 -- 172.21.15.129:0/1474413255 learned_addr learned my addr 172.21.15.129:0/1474413255 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d26ffd640 1 -- 172.21.15.129:0/1474413255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d28102bd0 msgr2=0x7f9d281978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d28102bd0 0x7f9d281978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d26ffd640 1 -- 172.21.15.129:0/1474413255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d281078d0 msgr2=0x7f9d2819e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d281078d0 0x7f9d2819e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.787+0000 7f9d26ffd640 1 -- 172.21.15.129:0/1474413255 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2819eed0 con 0x7f9d28101990 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.788+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d28101990 0x7f9d28197390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9d18031440 tx=0x7f9d18032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.788+0000 7f9d0ffff640 1 -- 172.21.15.129:0/1474413255 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1803f030 con 0x7f9d28101990 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.788+0000 7f9d0ffff640 1 -- 172.21.15.129:0/1474413255 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d18003ce0 con 0x7f9d28101990 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.788+0000 7f9d0ffff640 1 -- 172.21.15.129:0/1474413255 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18035070 con 0x7f9d28101990 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.788+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1474413255 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d2819f160 con 0x7f9d28101990 2024-05-20T17:58:47.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.788+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1474413255 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d2819f6a0 con 0x7f9d28101990 2024-05-20T17:58:47.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.791+0000 7f9d0ffff640 1 -- 172.21.15.129:0/1474413255 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 22) ==== 137954+0+0 (secure 0 0 0) 0x7f9d18042030 con 0x7f9d28101990 2024-05-20T17:58:47.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d3425f740 1 Processor -- start 2024-05-20T17:58:47.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d3425f740 1 -- start start 2024-05-20T17:58:47.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 0x560dc98e84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc98e8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 0x560dc98ef920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d3425f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560dc984c990 con 0x560dc9527730 2024-05-20T17:58:47.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d3425f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560dc97f45b0 con 0x560dc9548410 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d3425f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560dc9848950 con 0x560dc951f550 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d0ffff640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d100a2d80 0x7f9d100a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d277fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 0x560dc98e84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d277fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 0x560dc98e84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:54234/0 (socket says 172.21.15.129:54234) 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d277fe640 1 -- 172.21.15.129:0/3970635998 learned_addr learned my addr 172.21.15.129:0/3970635998 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d277fe640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d100a2d80 0x7f9d100a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/3970635998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc98e8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.792+0000 7f9d267fc640 1 --2- 172.21.15.129:0/3970635998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 0x560dc98ef920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.793+0000 7f9d277fe640 1 -- 172.21.15.129:0/3970635998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 msgr2=0x560dc98e8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.793+0000 7f9d277fe640 1 --2- 172.21.15.129:0/3970635998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc98e8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.793+0000 7f9d277fe640 1 -- 172.21.15.129:0/3970635998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 msgr2=0x560dc98ef920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.793+0000 7f9d277fe640 1 --2- 172.21.15.129:0/3970635998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 0x560dc98ef920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.793+0000 7f9d277fe640 1 -- 172.21.15.129:0/3970635998 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dc96287e0 con 0x560dc9548410 2024-05-20T17:58:47.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.793+0000 7f9d0ffff640 1 -- 172.21.15.129:0/1474413255 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f9d180ef7c0 con 0x7f9d28101990 2024-05-20T17:58:47.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.793+0000 7f9d277fe640 1 --2- 172.21.15.129:0/3970635998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 0x560dc98e84a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9d20012d80 tx=0x7f9d20010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d1645fa6e41220d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.793+0000 7f9d277fe640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d100a2d80 0x7f9d100a5290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9d20010c40 tx=0x7f9d2000c000 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.793+0000 7f9d0d7fa640 1 -- 172.21.15.129:0/3970635998 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d20024070 con 0x560dc9548410 2024-05-20T17:58:47.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.793+0000 7f9d0d7fa640 1 -- 172.21.15.129:0/3970635998 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d200125a0 con 0x560dc9548410 2024-05-20T17:58:47.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.794+0000 7f9d3425f740 1 -- 172.21.15.129:0/3970635998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 msgr2=0x560dc98e84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.794+0000 7f9d3425f740 1 --2- 172.21.15.129:0/3970635998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 0x560dc98e84a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9d20012d80 tx=0x7f9d20010740 comp rx=0 tx=0).stop 2024-05-20T17:58:47.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.794+0000 7f9d3425f740 1 -- 172.21.15.129:0/3970635998 shutdown_connections 2024-05-20T17:58:47.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.794+0000 7f9d3425f740 1 --2- 172.21.15.129:0/3970635998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 0x560dc98ef920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.794+0000 7f9d3425f740 1 --2- 172.21.15.129:0/3970635998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc98e8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.794+0000 7f9d3425f740 1 --2- 172.21.15.129:0/3970635998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 0x560dc98e84a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.794+0000 7f9d3425f740 1 -- 172.21.15.129:0/3970635998 >> 172.21.15.129:0/3970635998 conn(0x560dc953c560 msgr2=0x560dc98e3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.794+0000 7f9d3425f740 1 -- 172.21.15.129:0/3970635998 shutdown_connections 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.794+0000 7f9d3425f740 1 -- 172.21.15.129:0/3970635998 wait complete. 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.795+0000 7f9d3425f740 1 -- start start 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.795+0000 7f9d3425f740 1 Processor -- start 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.795+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc9903380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.795+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9527730 0x560dc99038c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.795+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9548410 0x560dc990a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.795+0000 7f9d26ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc9903380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.795+0000 7f9d26ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc9903380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:52020/0 (socket says 172.21.15.129:52020) 2024-05-20T17:58:47.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.795+0000 7f9d267fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9527730 0x560dc99038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d267fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9527730 0x560dc99038c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:54236/0 (socket says 172.21.15.129:54236) 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d3425f740 1 -- 172.21.15.129:0/1166394438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560dc9901370 con 0x560dc9548410 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d3425f740 1 -- 172.21.15.129:0/1166394438 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560dc9807b40 con 0x560dc9527730 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d3425f740 1 -- 172.21.15.129:0/1166394438 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560dc99014f0 con 0x560dc951f550 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d26ffd640 1 -- 172.21.15.129:0/1166394438 learned_addr learned my addr 172.21.15.129:0/1166394438 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d277fe640 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9548410 0x560dc990a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d3425f740 10 client.0 _ll_register_callbacks cb 0x560dc97fc0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d26ffd640 1 -- 172.21.15.129:0/1166394438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9527730 msgr2=0x560dc99038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9527730 0x560dc99038c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d26ffd640 1 -- 172.21.15.129:0/1166394438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9548410 msgr2=0x560dc990a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9548410 0x560dc990a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d26ffd640 1 -- 172.21.15.129:0/1166394438 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dc9688f70 con 0x560dc951f550 2024-05-20T17:58:47.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d3425f740 1 -- 172.21.15.129:0/1166394438 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d200125a0 con 0x560dc951f550 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc9903380 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9d18000c00 tx=0x7f9d18003f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1db61ecac86727cf server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d0aff5640 10 client.25132 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d0aff5640 1 -- 172.21.15.129:0/1166394438 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18049040 con 0x560dc951f550 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.796+0000 7f9d0aff5640 1 -- 172.21.15.129:0/1166394438 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f9d180ed1a0 con 0x560dc951f550 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.797+0000 7f9d0aff5640 1 -- 172.21.15.129:0/1166394438 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9d180bb700 con 0x560dc951f550 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.797+0000 7f9d3425f740 10 client.25132 fetch_fsmap learned FSMap version 20 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.797+0000 7f9d3425f740 1 -- 172.21.15.129:0/1166394438 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({fsmap=20}) -- 0x560dc9520170 con 0x560dc951f550 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.797+0000 7f9d0aff5640 1 -- 172.21.15.129:0/1166394438 <== mon.1 v2:172.21.15.204:3300/0 4 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9d18096030 con 0x560dc951f550 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.797+0000 7f9d3425f740 10 client.25132 fetch_fsmap finished waiting for FSMap version 20 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.797+0000 7f9d3425f740 10 client.25132 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.797+0000 7f9d3425f740 20 client.25132 populate_metadata read hostname 'smithi129' 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.797+0000 7f9d3425f740 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560dc95071f0 0x560dc9910b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.797+0000 7f9d3425f740 4 client.25132 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:58:47.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.797+0000 7f9d3425f740 1 -- 172.21.15.129:0/1166394438 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560dc985a0c0 con 0x560dc95071f0 2024-05-20T17:58:47.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.797+0000 7f9d267fc640 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560dc95071f0 0x560dc9910b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.798+0000 7f9d267fc640 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560dc95071f0 0x560dc9910b70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d679ccb352f872e0 server_cookie=60edf973cb71e18e in_seq=0 out_seq=0 2024-05-20T17:58:47.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.798+0000 7f9d0aff5640 10 client.25132 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:47.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.798+0000 7f9d0aff5640 1 -- 172.21.15.129:0/1166394438 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560dc985a0c0 con 0x560dc95071f0 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.798+0000 7f9d0aff5640 10 client.25132 handle_command_reply: tid=0 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.798+0000 7f9d3425f740 1 client.25132 shutdown 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d3425f740 1 -- 172.21.15.129:0/1166394438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 msgr2=0x560dc9903380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d3425f740 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc9903380 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9d18000c00 tx=0x7f9d18003f50 comp rx=0 tx=0).stop 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d3425f740 1 -- 172.21.15.129:0/1166394438 shutdown_connections 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d3425f740 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560dc95071f0 0x560dc9910b70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d3425f740 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9527730 0x560dc99038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d3425f740 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9548410 0x560dc990a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d3425f740 1 --2- 172.21.15.129:0/1166394438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc9903380 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d3425f740 1 -- 172.21.15.129:0/1166394438 >> 172.21.15.129:0/1166394438 conn(0x560dc953c560 msgr2=0x560dc98e2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d0aff5640 0 client.25132 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d3425f740 1 -- 172.21.15.129:0/1166394438 shutdown_connections 2024-05-20T17:58:47.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d3425f740 1 -- 172.21.15.129:0/1166394438 wait complete. 2024-05-20T17:58:47.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.799+0000 7f9d3425f740 20 client.25132 trim_cache size 0 max 16384 2024-05-20T17:58:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d3425f740 1 Processor -- start 2024-05-20T17:58:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d3425f740 1 -- start start 2024-05-20T17:58:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc98e3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 0x560dc9902f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 0x560dc9903490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d3425f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d28071fb0 con 0x560dc9527730 2024-05-20T17:58:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d3425f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d281a03b0 con 0x560dc9548410 2024-05-20T17:58:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d26ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc98e3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d267fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 0x560dc9902f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d3425f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d28072620 con 0x560dc951f550 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d26ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc98e3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:52028/0 (socket says 172.21.15.129:52028) 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d26ffd640 1 -- 172.21.15.129:0/297442203 learned_addr learned my addr 172.21.15.129:0/297442203 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.837+0000 7f9d267fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 0x560dc9902f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42678/0 (socket says 172.21.15.129:42678) 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.838+0000 7f9d277fe640 1 --2- 172.21.15.129:0/297442203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 0x560dc9903490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.838+0000 7f9d26ffd640 1 -- 172.21.15.129:0/297442203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 msgr2=0x560dc9903490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.838+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/297442203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 0x560dc9903490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.838+0000 7f9d26ffd640 1 -- 172.21.15.129:0/297442203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 msgr2=0x560dc9902f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.838+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/297442203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 0x560dc9902f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.838+0000 7f9d26ffd640 1 -- 172.21.15.129:0/297442203 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560dc9902080 con 0x560dc951f550 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.838+0000 7f9d267fc640 1 --2- 172.21.15.129:0/297442203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 0x560dc9902f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.838+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/297442203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc98e3e60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9d1802a930 tx=0x7f9d180be000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6e5d4287a02d601 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.838+0000 7f9d0cff9640 1 -- 172.21.15.129:0/297442203 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1809e020 con 0x560dc951f550 2024-05-20T17:58:47.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.838+0000 7f9d0cff9640 1 -- 172.21.15.129:0/297442203 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d200125a0 con 0x560dc951f550 2024-05-20T17:58:47.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.838+0000 7f9d0cff9640 1 -- 172.21.15.129:0/297442203 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18049040 con 0x560dc951f550 2024-05-20T17:58:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.839+0000 7f9d3425f740 1 -- 172.21.15.129:0/297442203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 msgr2=0x560dc98e3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.839+0000 7f9d3425f740 1 --2- 172.21.15.129:0/297442203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc98e3e60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9d1802a930 tx=0x7f9d180be000 comp rx=0 tx=0).stop 2024-05-20T17:58:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.839+0000 7f9d3425f740 1 -- 172.21.15.129:0/297442203 shutdown_connections 2024-05-20T17:58:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.839+0000 7f9d3425f740 1 --2- 172.21.15.129:0/297442203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9548410 0x560dc9903490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.839+0000 7f9d3425f740 1 --2- 172.21.15.129:0/297442203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9527730 0x560dc9902f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.839+0000 7f9d3425f740 1 --2- 172.21.15.129:0/297442203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc98e3e60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.839+0000 7f9d3425f740 1 -- 172.21.15.129:0/297442203 >> 172.21.15.129:0/297442203 conn(0x560dc953c560 msgr2=0x560dc98e3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.839+0000 7f9d3425f740 1 -- 172.21.15.129:0/297442203 shutdown_connections 2024-05-20T17:58:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.839+0000 7f9d3425f740 1 -- 172.21.15.129:0/297442203 wait complete. 2024-05-20T17:58:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d3425f740 1 -- start start 2024-05-20T17:58:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d3425f740 1 Processor -- start 2024-05-20T17:58:47.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc99076f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9527730 0x560dc9907c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d3425f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9548410 0x560dc9908b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d3425f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d28072cf0 con 0x560dc9548410 2024-05-20T17:58:47.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d3425f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2819fce0 con 0x560dc9527730 2024-05-20T17:58:47.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d26ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc99076f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d26ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc99076f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:52042/0 (socket says 172.21.15.129:52042) 2024-05-20T17:58:47.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d26ffd640 1 -- 172.21.15.129:0/259855905 learned_addr learned my addr 172.21.15.129:0/259855905 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:47.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d3425f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d280733c0 con 0x560dc951f550 2024-05-20T17:58:47.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d277fe640 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9548410 0x560dc9908b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d3425f740 10 client.0 _ll_register_callbacks cb 0x7f9d18005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:58:47.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.840+0000 7f9d267fc640 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9527730 0x560dc9907c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d26ffd640 1 -- 172.21.15.129:0/259855905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9527730 msgr2=0x560dc9907c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9527730 0x560dc9907c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d26ffd640 1 -- 172.21.15.129:0/259855905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9548410 msgr2=0x560dc9908b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9548410 0x560dc9908b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d26ffd640 1 -- 172.21.15.129:0/259855905 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560dc9909120 con 0x560dc951f550 2024-05-20T17:58:47.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d26ffd640 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc99076f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9d18006010 tx=0x7f9d180efb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1678b3d24312962 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d0aff5640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T17:58:47.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d0aff5640 1 -- 172.21.15.129:0/259855905 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1809e070 con 0x560dc951f550 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d0aff5640 1 -- 172.21.15.129:0/259855905 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f9d1803bdf0 con 0x560dc951f550 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d3425f740 1 -- 172.21.15.129:0/259855905 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d200125a0 con 0x560dc951f550 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d0aff5640 1 -- 172.21.15.129:0/259855905 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9d18049030 con 0x560dc951f550 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d3425f740 10 client.25138 fetch_fsmap learned FSMap version 20 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.841+0000 7f9d3425f740 1 -- 172.21.15.129:0/259855905 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x560dc9909120 con 0x560dc951f550 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.842+0000 7f9d0aff5640 1 -- 172.21.15.129:0/259855905 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d200125a0 con 0x560dc951f550 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.842+0000 7f9d0aff5640 1 -- 172.21.15.129:0/259855905 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f9d180bc000 con 0x560dc951f550 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.842+0000 7f9d3425f740 10 client.25138 fetch_fsmap finished waiting for FSMap version 20 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.842+0000 7f9d3425f740 10 client.25138 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.842+0000 7f9d3425f740 20 client.25138 populate_metadata read hostname 'smithi129' 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.842+0000 7f9d3425f740 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560dc94dc630 0x560dc9983c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.842+0000 7f9d3425f740 4 client.25138 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T17:58:47.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.842+0000 7f9d3425f740 1 -- 172.21.15.129:0/259855905 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560dc9854410 con 0x560dc94dc630 2024-05-20T17:58:47.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.842+0000 7f9d267fc640 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560dc94dc630 0x560dc9983c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.842+0000 7f9d0aff5640 10 client.25138 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:47.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.843+0000 7f9d267fc640 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560dc94dc630 0x560dc9983c50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e4943c13ed9fb27 server_cookie=a9eb4f3887918b51 in_seq=0 out_seq=0 2024-05-20T17:58:47.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d0aff5640 1 -- 172.21.15.129:0/259855905 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560dc9854410 con 0x560dc94dc630 2024-05-20T17:58:47.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d0aff5640 10 client.25138 handle_command_reply: tid=0 2024-05-20T17:58:47.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 1 client.25138 shutdown 2024-05-20T17:58:47.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 1 -- 172.21.15.129:0/259855905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 msgr2=0x560dc99076f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc99076f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9d18006010 tx=0x7f9d180efb50 comp rx=0 tx=0).stop 2024-05-20T17:58:47.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 1 -- 172.21.15.129:0/259855905 shutdown_connections 2024-05-20T17:58:47.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560dc94dc630 0x560dc9983c50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560dc9527730 0x560dc9907c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560dc9548410 0x560dc9908b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 1 --2- 172.21.15.129:0/259855905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560dc951f550 0x560dc99076f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 1 -- 172.21.15.129:0/259855905 >> 172.21.15.129:0/259855905 conn(0x560dc953c560 msgr2=0x560dc9902ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d0aff5640 0 client.25138 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:47.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 1 -- 172.21.15.129:0/259855905 shutdown_connections 2024-05-20T17:58:47.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 1 -- 172.21.15.129:0/259855905 wait complete. 2024-05-20T17:58:47.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.844+0000 7f9d3425f740 20 client.25138 trim_cache size 0 max 16384 2024-05-20T17:58:47.848 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T17:58:47.848 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T17:58:47.848 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "6f9d389c-bfb1-4100-8078-5efc77296dbe", 2024-05-20T17:58:47.848 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T17:58:47.849 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T17:58:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1474413255 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d100a2d80 msgr2=0x7f9d100a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d100a2d80 0x7f9d100a5290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9d20010c40 tx=0x7f9d2000c000 comp rx=0 tx=0).stop 2024-05-20T17:58:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1474413255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d28101990 msgr2=0x7f9d28197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d28101990 0x7f9d28197390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9d18031440 tx=0x7f9d18032ce0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1474413255 shutdown_connections 2024-05-20T17:58:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d100a2d80 0x7f9d100a5290 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d28102bd0 0x7f9d281978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d281078d0 0x7f9d2819e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 --2- 172.21.15.129:0/1474413255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d28101990 0x7f9d28197390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1474413255 >> 172.21.15.129:0/1474413255 conn(0x7f9d280fcb80 msgr2=0x7f9d281038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1474413255 shutdown_connections 2024-05-20T17:58:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.845+0000 7f9d2d3c2640 1 -- 172.21.15.129:0/1474413255 wait complete. 2024-05-20T17:58:47.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T17:58:47.857 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T17:58:47.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.981+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2035644102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc954101ac0 msgr2=0x7fc954101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.981+0000 7fc95c5ea640 1 --2- 172.21.15.129:0/2035644102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc954101ac0 0x7fc954101ed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc944009900 tx=0x7fc94402a410 comp rx=0 tx=0).stop 2024-05-20T17:58:47.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.981+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2035644102 shutdown_connections 2024-05-20T17:58:47.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.981+0000 7fc95c5ea640 1 --2- 172.21.15.129:0/2035644102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc954101ac0 0x7fc954101ed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.981+0000 7fc95c5ea640 1 --2- 172.21.15.129:0/2035644102 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc95410c350 0x7fc95410e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.981+0000 7fc95c5ea640 1 --2- 172.21.15.129:0/2035644102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9541079f0 0x7fc954101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.981+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2035644102 >> 172.21.15.129:0/2035644102 conn(0x7fc954077130 msgr2=0x7fc954077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.981+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2035644102 shutdown_connections 2024-05-20T17:58:47.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.981+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2035644102 wait complete. 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.982+0000 7fc95c5ea640 1 Processor -- start 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.982+0000 7fc95c5ea640 1 -- start start 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.982+0000 7fc95c5ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc954101ac0 0x7fc954199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.982+0000 7fc95c5ea640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9541079f0 0x7fc954199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95c5ea640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc95410c350 0x7fc95419de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95c5ea640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc954112e70 con 0x7fc95410c350 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95c5ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc954112cf0 con 0x7fc954101ac0 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95c5ea640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc954112ff0 con 0x7fc9541079f0 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95a386640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc954101ac0 0x7fc954199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95a386640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc954101ac0 0x7fc954199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:54282/0 (socket says 172.21.15.129:54282) 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95a386640 1 -- 172.21.15.129:0/2265420106 learned_addr learned my addr 172.21.15.129:0/2265420106 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc959b85640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9541079f0 0x7fc954199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95ab87640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc95410c350 0x7fc95419de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95a386640 1 -- 172.21.15.129:0/2265420106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9541079f0 msgr2=0x7fc954199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95a386640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9541079f0 0x7fc954199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95a386640 1 -- 172.21.15.129:0/2265420106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc95410c350 msgr2=0x7fc95419de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95a386640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc95410c350 0x7fc95419de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95a386640 1 -- 172.21.15.129:0/2265420106 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc95419e490 con 0x7fc954101ac0 2024-05-20T17:58:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc95a386640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc954101ac0 0x7fc954199630 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc95000e9c0 tx=0x7fc95000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.983+0000 7fc94b7fe640 1 -- 172.21.15.129:0/2265420106 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95000cd80 con 0x7fc954101ac0 2024-05-20T17:58:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.984+0000 7fc94b7fe640 1 -- 172.21.15.129:0/2265420106 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc950005500 con 0x7fc954101ac0 2024-05-20T17:58:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.984+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2265420106 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc95419e780 con 0x7fc954101ac0 2024-05-20T17:58:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.984+0000 7fc94b7fe640 1 -- 172.21.15.129:0/2265420106 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc950010690 con 0x7fc954101ac0 2024-05-20T17:58:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.984+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2265420106 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9541a6100 con 0x7fc954101ac0 2024-05-20T17:58:47.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.986+0000 7fc94b7fe640 1 -- 172.21.15.129:0/2265420106 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 22) ==== 137954+0+0 (secure 0 0 0) 0x7fc9500046c0 con 0x7fc954101ac0 2024-05-20T17:58:47.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.987+0000 7fc94b7fe640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc9400a2d80 0x7fc9400a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.987+0000 7fc94b7fe640 1 -- 172.21.15.129:0/2265420106 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7fc9500cf760 con 0x7fc954101ac0 2024-05-20T17:58:47.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.987+0000 7fc959b85640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc9400a2d80 0x7fc9400a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.988+0000 7fc96356a740 1 Processor -- start 2024-05-20T17:58:47.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.988+0000 7fc96356a740 1 -- start start 2024-05-20T17:58:47.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.988+0000 7fc959b85640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc9400a2d80 0x7fc9400a5290 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc94402a930 tx=0x7fc944000bc0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.988+0000 7fc96356a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5e4390 0x55bf4c992200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.988+0000 7fc96356a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c992740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.988+0000 7fc96356a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c98ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.988+0000 7fc96356a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bf4c8e8d00 con 0x55bf4c5bb4d0 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.988+0000 7fc96356a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bf4c8904c0 con 0x55bf4c5e4390 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.988+0000 7fc96356a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55bf4c8e4cc0 con 0x55bf4c5c36b0 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc959b85640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c992740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc95a386640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5e4390 0x55bf4c992200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc95a386640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5e4390 0x55bf4c992200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:54296/0 (socket says 172.21.15.129:54296) 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc959b85640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c992740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42722/0 (socket says 172.21.15.129:42722) 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc95a386640 1 -- 172.21.15.129:0/1674161615 learned_addr learned my addr 172.21.15.129:0/1674161615 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc95ab87640 1 --2- 172.21.15.129:0/1674161615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c98ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc959b85640 1 -- 172.21.15.129:0/1674161615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 msgr2=0x55bf4c98ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc959b85640 1 --2- 172.21.15.129:0/1674161615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c98ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc959b85640 1 -- 172.21.15.129:0/1674161615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5e4390 msgr2=0x55bf4c992200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc959b85640 1 --2- 172.21.15.129:0/1674161615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5e4390 0x55bf4c992200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc959b85640 1 -- 172.21.15.129:0/1674161615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf4c6c67d0 con 0x55bf4c5bb4d0 2024-05-20T17:58:47.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc959b85640 1 --2- 172.21.15.129:0/1674161615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c992740 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc94402aa00 tx=0x7fc94403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab89c8ab17863f93 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc948bfe640 1 -- 172.21.15.129:0/1674161615 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944035040 con 0x55bf4c5bb4d0 2024-05-20T17:58:47.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc948bfe640 1 -- 172.21.15.129:0/1674161615 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc94404f070 con 0x55bf4c5bb4d0 2024-05-20T17:58:47.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.989+0000 7fc948bfe640 1 -- 172.21.15.129:0/1674161615 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9440038e0 con 0x55bf4c5bb4d0 2024-05-20T17:58:47.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.991+0000 7fc96356a740 1 -- 172.21.15.129:0/1674161615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 msgr2=0x55bf4c992740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:47.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.991+0000 7fc96356a740 1 --2- 172.21.15.129:0/1674161615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c992740 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc94402aa00 tx=0x7fc94403d000 comp rx=0 tx=0).stop 2024-05-20T17:58:47.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.991+0000 7fc96356a740 1 -- 172.21.15.129:0/1674161615 shutdown_connections 2024-05-20T17:58:47.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.991+0000 7fc96356a740 1 --2- 172.21.15.129:0/1674161615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c98ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.991+0000 7fc96356a740 1 --2- 172.21.15.129:0/1674161615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c992740 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.991+0000 7fc96356a740 1 --2- 172.21.15.129:0/1674161615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5e4390 0x55bf4c992200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.991+0000 7fc96356a740 1 -- 172.21.15.129:0/1674161615 >> 172.21.15.129:0/1674161615 conn(0x55bf4c5d84e0 msgr2=0x55bf4c97ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:47.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.991+0000 7fc96356a740 1 -- 172.21.15.129:0/1674161615 shutdown_connections 2024-05-20T17:58:47.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.991+0000 7fc96356a740 1 -- 172.21.15.129:0/1674161615 wait complete. 2024-05-20T17:58:47.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.992+0000 7fc96356a740 1 -- start start 2024-05-20T17:58:47.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.992+0000 7fc96356a740 1 Processor -- start 2024-05-20T17:58:47.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.992+0000 7fc96356a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.992+0000 7fc95a386640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.992+0000 7fc95a386640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42738/0 (socket says 172.21.15.129:42738) 2024-05-20T17:58:47.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.992+0000 7fc96356a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c99c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.992+0000 7fc95a386640 1 -- 172.21.15.129:0/4193131378 learned_addr learned my addr 172.21.15.129:0/4193131378 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:47.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.992+0000 7fc96356a740 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5e4390 0x55bf4c996320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:47.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.992+0000 7fc96356a740 1 -- 172.21.15.129:0/4193131378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bf4c98d410 con 0x55bf4c5bb4d0 2024-05-20T17:58:47.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.992+0000 7fc96356a740 1 -- 172.21.15.129:0/4193131378 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bf4c8a3b90 con 0x55bf4c5e4390 2024-05-20T17:58:47.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.992+0000 7fc96356a740 1 -- 172.21.15.129:0/4193131378 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55bf4c98d590 con 0x55bf4c5c36b0 2024-05-20T17:58:47.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.993+0000 7fc959b85640 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c99c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.993+0000 7fc95ab87640 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5e4390 0x55bf4c996320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:47.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.993+0000 7fc96356a740 10 client.0 _ll_register_callbacks cb 0x55bf4c8dfda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:58:47.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.993+0000 7fc95a386640 1 -- 172.21.15.129:0/4193131378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 msgr2=0x55bf4c99c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.993+0000 7fc95a386640 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c99c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.993+0000 7fc95a386640 1 -- 172.21.15.129:0/4193131378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5e4390 msgr2=0x55bf4c996320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:47.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.993+0000 7fc95a386640 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5e4390 0x55bf4c996320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:47.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.993+0000 7fc95a386640 1 -- 172.21.15.129:0/4193131378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf4c5fc3f0 con 0x55bf4c5bb4d0 2024-05-20T17:58:47.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.993+0000 7fc95a386640 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99bf30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc95000cfd0 tx=0x7fc95009b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be46c127357204c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:47.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.993+0000 7fc93dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:58:47.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.995+0000 7fc93dffb640 1 -- 172.21.15.129:0/4193131378 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9500a0040 con 0x55bf4c5bb4d0 2024-05-20T17:58:47.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.995+0000 7fc93dffb640 1 -- 172.21.15.129:0/4193131378 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7fc9500a4470 con 0x55bf4c5bb4d0 2024-05-20T17:58:47.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:47.995+0000 7fc96356a740 1 -- 172.21.15.129:0/4193131378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc94404f070 con 0x55bf4c5bb4d0 2024-05-20T17:58:47.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.000+0000 7fc93dffb640 1 -- 172.21.15.129:0/4193131378 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc9500a4900 con 0x55bf4c5bb4d0 2024-05-20T17:58:47.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.001+0000 7fc96356a740 10 client.15426 fetch_fsmap learned FSMap version 20 2024-05-20T17:58:47.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.001+0000 7fc96356a740 1 -- 172.21.15.129:0/4193131378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55bf4c5bc0f0 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.001+0000 7fc93dffb640 1 -- 172.21.15.129:0/4193131378 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc94404f070 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.001+0000 7fc93dffb640 1 -- 172.21.15.129:0/4193131378 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc9500b3070 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.001+0000 7fc96356a740 10 client.15426 fetch_fsmap finished waiting for FSMap version 20 2024-05-20T17:58:48.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.001+0000 7fc96356a740 10 client.15426 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:58:48.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.002+0000 7fc96356a740 20 client.15426 populate_metadata read hostname 'smithi129' 2024-05-20T17:58:48.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.002+0000 7fc96356a740 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bf4c5a2f60 0x55bf4c9a8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.002+0000 7fc96356a740 4 client.15426 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:58:48.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.002+0000 7fc96356a740 1 -- 172.21.15.129:0/4193131378 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf4c8f6430 con 0x55bf4c5a2f60 2024-05-20T17:58:48.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.002+0000 7fc959b85640 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bf4c5a2f60 0x55bf4c9a8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.002+0000 7fc959b85640 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bf4c5a2f60 0x55bf4c9a8390 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c905ff7d2d7c2fd2 server_cookie=b6569f9108b41b23 in_seq=0 out_seq=0 2024-05-20T17:58:48.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.002+0000 7fc93dffb640 10 client.15426 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:48.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.003+0000 7fc93dffb640 1 -- 172.21.15.129:0/4193131378 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bf4c8f6430 con 0x55bf4c5a2f60 2024-05-20T17:58:48.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.003+0000 7fc93dffb640 10 client.15426 handle_command_reply: tid=0 2024-05-20T17:58:48.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.003+0000 7fc96356a740 1 client.15426 shutdown 2024-05-20T17:58:48.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc96356a740 1 -- 172.21.15.129:0/4193131378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 msgr2=0x55bf4c99bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc96356a740 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99bf30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc95000cfd0 tx=0x7fc95009b6b0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc96356a740 1 -- 172.21.15.129:0/4193131378 shutdown_connections 2024-05-20T17:58:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc96356a740 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bf4c5a2f60 0x55bf4c9a8390 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc93dffb640 0 client.15426 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc96356a740 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5e4390 0x55bf4c996320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc96356a740 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c99c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc96356a740 1 --2- 172.21.15.129:0/4193131378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99bf30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc96356a740 1 -- 172.21.15.129:0/4193131378 >> 172.21.15.129:0/4193131378 conn(0x55bf4c5d84e0 msgr2=0x55bf4c97ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc96356a740 1 -- 172.21.15.129:0/4193131378 shutdown_connections 2024-05-20T17:58:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc96356a740 1 -- 172.21.15.129:0/4193131378 wait complete. 2024-05-20T17:58:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.004+0000 7fc96356a740 20 client.15426 trim_cache size 0 max 16384 2024-05-20T17:58:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.015+0000 7fc94b7fe640 1 -- 172.21.15.129:0/2265420106 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mgrmap(e 23) ==== 138010+0+0 (secure 0 0 0) 0x7fc95009acb0 con 0x7fc954101ac0 2024-05-20T17:58:48.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.017+0000 7f595d7fa640 1 -- 172.21.15.129:0/212034662 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 23) ==== 138010+0+0 (secure 0 0 0) 0x7f59540981f0 con 0x7f5960105c10 2024-05-20T17:58:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.034+0000 7fc96356a740 1 Processor -- start 2024-05-20T17:58:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.034+0000 7fc96356a740 1 -- start start 2024-05-20T17:58:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc96356a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5a2f60 0x55bf4c99c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc96356a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c97e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc96356a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c97e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc96356a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541aa4c0 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc96356a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541acd40 con 0x55bf4c5a2f60 2024-05-20T17:58:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc96356a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541aab30 con 0x55bf4c5c36b0 2024-05-20T17:58:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc95ab87640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c97e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc959b85640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c97e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc95ab87640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c97e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:52072/0 (socket says 172.21.15.129:52072) 2024-05-20T17:58:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc95ab87640 1 -- 172.21.15.129:0/1744894453 learned_addr learned my addr 172.21.15.129:0/1744894453 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc95a386640 1 --2- 172.21.15.129:0/1744894453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5a2f60 0x55bf4c99c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc95ab87640 1 -- 172.21.15.129:0/1744894453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5a2f60 msgr2=0x55bf4c99c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc95ab87640 1 --2- 172.21.15.129:0/1744894453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5a2f60 0x55bf4c99c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc95ab87640 1 -- 172.21.15.129:0/1744894453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 msgr2=0x55bf4c97e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc95ab87640 1 --2- 172.21.15.129:0/1744894453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c97e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc95ab87640 1 -- 172.21.15.129:0/1744894453 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf4c994890 con 0x55bf4c5c36b0 2024-05-20T17:58:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc959b85640 1 --2- 172.21.15.129:0/1744894453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c97e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.035+0000 7fc95ab87640 1 --2- 172.21.15.129:0/1744894453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c97e7e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc94c003cc0 tx=0x7fc94c004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1836a465f5ec11f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.036+0000 7fc93ffff640 1 -- 172.21.15.129:0/1744894453 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c004b20 con 0x55bf4c5c36b0 2024-05-20T17:58:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.036+0000 7fc93ffff640 1 -- 172.21.15.129:0/1744894453 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc94c010910 con 0x55bf4c5c36b0 2024-05-20T17:58:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.036+0000 7fc93ffff640 1 -- 172.21.15.129:0/1744894453 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c01b990 con 0x55bf4c5c36b0 2024-05-20T17:58:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.037+0000 7f59662e2640 1 -- 172.21.15.129:0/212034662 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7f5960062750 con 0x7f5960105c10 2024-05-20T17:58:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.037+0000 7fc96356a740 1 -- 172.21.15.129:0/1744894453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 msgr2=0x55bf4c97e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.037+0000 7fc96356a740 1 --2- 172.21.15.129:0/1744894453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c97e7e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc94c003cc0 tx=0x7fc94c004410 comp rx=0 tx=0).stop 2024-05-20T17:58:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.037+0000 7fc96356a740 1 -- 172.21.15.129:0/1744894453 shutdown_connections 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.037+0000 7fc96356a740 1 --2- 172.21.15.129:0/1744894453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5c36b0 0x55bf4c97e7e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.037+0000 7fc96356a740 1 --2- 172.21.15.129:0/1744894453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c97e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.037+0000 7fc96356a740 1 --2- 172.21.15.129:0/1744894453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5a2f60 0x55bf4c99c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.037+0000 7fc96356a740 1 -- 172.21.15.129:0/1744894453 >> 172.21.15.129:0/1744894453 conn(0x55bf4c5d84e0 msgr2=0x55bf4c97db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.037+0000 7fc96356a740 1 -- 172.21.15.129:0/1744894453 shutdown_connections 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.038+0000 7fc96356a740 1 -- 172.21.15.129:0/1744894453 wait complete. 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.038+0000 7fc96356a740 1 -- start start 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.038+0000 7fc96356a740 1 Processor -- start 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc96356a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5a2f60 0x55bf4c9a26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc96356a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc96356a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5c36b0 0x55bf4c9a0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc96356a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541ab200 con 0x55bf4c5a2f60 2024-05-20T17:58:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc96356a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541ac670 con 0x55bf4c5c36b0 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc96356a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541ab8d0 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc96356a740 10 client.0 _ll_register_callbacks cb 0x7fc9500cce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc959b85640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc95ab87640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5c36b0 0x55bf4c9a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc959b85640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:52078/0 (socket says 172.21.15.129:52078) 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc959b85640 1 -- 172.21.15.129:0/142064657 learned_addr learned my addr 172.21.15.129:0/142064657 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc95a386640 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5a2f60 0x55bf4c9a26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc959b85640 1 -- 172.21.15.129:0/142064657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5c36b0 msgr2=0x55bf4c9a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc959b85640 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5c36b0 0x55bf4c9a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc959b85640 1 -- 172.21.15.129:0/142064657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5a2f60 msgr2=0x55bf4c9a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc959b85640 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5a2f60 0x55bf4c9a26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc959b85640 1 -- 172.21.15.129:0/142064657 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf4c9a0850 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc959b85640 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99fac0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55bf4c97fde0 tx=0x7fc94402a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf88ada76e624e49 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.039+0000 7fc93dffb640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T17:58:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.043+0000 7fc93dffb640 1 -- 172.21.15.129:0/142064657 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944035040 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.043+0000 7fc93dffb640 1 -- 172.21.15.129:0/142064657 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7fc944003800 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.043+0000 7fc96356a740 1 -- 172.21.15.129:0/142064657 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc94c010910 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.044+0000 7f595d7fa640 1 -- 172.21.15.129:0/212034662 <== mon.2 v2:172.21.15.154:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v30)=0 v30) ==== 129+0+0 (secure 0 0 0) 0x7f5954046ac0 con 0x7f5960105c10 2024-05-20T17:58:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.046+0000 7fc93dffb640 1 -- 172.21.15.129:0/142064657 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc944071050 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.046+0000 7fc96356a740 10 client.25156 fetch_fsmap learned FSMap version 20 2024-05-20T17:58:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.046+0000 7fc96356a740 1 -- 172.21.15.129:0/142064657 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55bf4ca10740 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 -- 172.21.15.129:0/212034662 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f59480a2df0 msgr2=0x7f59480a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f59480a2df0 0x7f59480a5300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5950004730 tx=0x7f59500046c0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 -- 172.21.15.129:0/212034662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5960105c10 msgr2=0x7f59601997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5960105c10 0x7f59601997e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f595400e9f0 tx=0x7f595400eeb0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7fc93dffb640 1 -- 172.21.15.129:0/142064657 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc94c010910 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 -- 172.21.15.129:0/212034662 shutdown_connections 2024-05-20T17:58:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f59480a2df0 0x7f59480a5300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5960105c10 0x7f59601997e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5960108780 0x7f59601a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 --2- 172.21.15.129:0/212034662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5960100e10 0x7f59601992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 -- 172.21.15.129:0/212034662 >> 172.21.15.129:0/212034662 conn(0x7f59600fc340 msgr2=0x7f5960106930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 -- 172.21.15.129:0/212034662 shutdown_connections 2024-05-20T17:58:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.047+0000 7f59662e2640 1 -- 172.21.15.129:0/212034662 wait complete. 2024-05-20T17:58:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.048+0000 7fc93dffb640 1 -- 172.21.15.129:0/142064657 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7fc94404a460 con 0x55bf4c5bb4d0 2024-05-20T17:58:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.048+0000 7fc96356a740 10 client.25156 fetch_fsmap finished waiting for FSMap version 20 2024-05-20T17:58:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.048+0000 7fc96356a740 10 client.25156 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:58:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.048+0000 7fc96356a740 20 client.25156 populate_metadata read hostname 'smithi129' 2024-05-20T17:58:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.048+0000 7fc96356a740 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bf4c5e4390 0x55bf4ca12b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.048+0000 7fc96356a740 4 client.25156 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T17:58:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.048+0000 7fc96356a740 1 -- 172.21.15.129:0/142064657 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bf4c8f0780 con 0x55bf4c5e4390 2024-05-20T17:58:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.048+0000 7fc95a386640 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bf4c5e4390 0x55bf4ca12b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc95a386640 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bf4c5e4390 0x55bf4ca12b50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e367f98f70c60bbb server_cookie=7feeff0bc79aa8f1 in_seq=0 out_seq=0 2024-05-20T17:58:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc93dffb640 10 client.25156 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc93dffb640 1 -- 172.21.15.129:0/142064657 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bf4c8f0780 con 0x55bf4c5e4390 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc93dffb640 10 client.25156 handle_command_reply: tid=0 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc96356a740 1 client.25156 shutdown 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc96356a740 1 -- 172.21.15.129:0/142064657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5bb4d0 msgr2=0x55bf4c99fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc96356a740 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99fac0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55bf4c97fde0 tx=0x7fc94402a410 comp rx=0 tx=0).stop 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc96356a740 1 -- 172.21.15.129:0/142064657 shutdown_connections 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc96356a740 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bf4c5e4390 0x55bf4ca12b50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc96356a740 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf4c5c36b0 0x55bf4c9a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc96356a740 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bf4c5bb4d0 0x55bf4c99fac0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc96356a740 1 --2- 172.21.15.129:0/142064657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bf4c5a2f60 0x55bf4c9a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc96356a740 1 -- 172.21.15.129:0/142064657 >> 172.21.15.129:0/142064657 conn(0x55bf4c5d84e0 msgr2=0x55bf4c97e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.049+0000 7fc93dffb640 0 client.25156 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:58:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc96356a740 1 -- 172.21.15.129:0/142064657 shutdown_connections 2024-05-20T17:58:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc96356a740 1 -- 172.21.15.129:0/142064657 wait complete. 2024-05-20T17:58:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc96356a740 20 client.25156 trim_cache size 0 max 16384 2024-05-20T17:58:48.051 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T17:58:48.051 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T17:58:48.051 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T17:58:48.051 INFO:teuthology.orchestra.run.smithi129.stdout: "6f9d389c-bfb1-4100-8078-5efc77296dbe": { 2024-05-20T17:58:48.051 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T17:58:48.051 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "6f9d389c-bfb1-4100-8078-5efc77296dbe", 2024-05-20T17:58:48.051 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T17:58:48.051 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T17:58:48.052 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T17:58:48.052 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T17:58:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2265420106 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc9400a2d80 msgr2=0x7fc9400a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc95c5ea640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc9400a2d80 0x7fc9400a5290 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc94402a930 tx=0x7fc944000bc0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2265420106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc954101ac0 msgr2=0x7fc954199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc95c5ea640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc954101ac0 0x7fc954199630 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc95000e9c0 tx=0x7fc95000ee80 comp rx=0 tx=0).stop 2024-05-20T17:58:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2265420106 shutdown_connections 2024-05-20T17:58:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc95c5ea640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc9400a2d80 0x7fc9400a5290 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc95c5ea640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc95410c350 0x7fc95419de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc95c5ea640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc9541079f0 0x7fc954199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.050+0000 7fc95c5ea640 1 --2- 172.21.15.129:0/2265420106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc954101ac0 0x7fc954199630 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.051+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2265420106 >> 172.21.15.129:0/2265420106 conn(0x7fc954077130 msgr2=0x7fc954103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.051+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2265420106 shutdown_connections 2024-05-20T17:58:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.051+0000 7fc95c5ea640 1 -- 172.21.15.129:0/2265420106 wait complete. 2024-05-20T17:58:48.063 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.063 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4022254263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Unable to set osd_memory_target on smithi154 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.064 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Unable to set osd_memory_target on smithi129 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Updating smithi129:/etc/ceph/ceph.conf 2024-05-20T17:58:48.065 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Updating smithi154:/etc/ceph/ceph.conf 2024-05-20T17:58:48.066 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Updating smithi204:/etc/ceph/ceph.conf 2024-05-20T17:58:48.066 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4084710359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:48.066 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Updating smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:58:48.066 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:58:48.066 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:58:48.066 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:47 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/147904422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:48.067 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-20T17:58:48.068 INFO:tasks.cephfs.filesystem:scrub status for tag:6f9d389c-bfb1-4100-8078-5efc77296dbe - {'path': '/', 'tag': '6f9d389c-bfb1-4100-8078-5efc77296dbe', 'options': 'recursive,force'} 2024-05-20T17:58:48.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/4022254263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:48.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.144 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-05-20T17:58:48.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Unable to set osd_memory_target on smithi154 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-20T17:58:48.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-05-20T17:58:48.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-05-20T17:58:48.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.147 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-05-20T17:58:48.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Unable to set osd_memory_target on smithi129 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-05-20T17:58:48.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:48.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:48.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Updating smithi129:/etc/ceph/ceph.conf 2024-05-20T17:58:48.148 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Updating smithi154:/etc/ceph/ceph.conf 2024-05-20T17:58:48.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Updating smithi204:/etc/ceph/ceph.conf 2024-05-20T17:58:48.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/4084710359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:48.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Updating smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:58:48.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:58:48.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:58:48.149 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:47 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/147904422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:48.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/4022254263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T17:58:48.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Unable to set osd_memory_target on smithi154 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Unable to set osd_memory_target on smithi129 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Updating smithi129:/etc/ceph/ceph.conf 2024-05-20T17:58:48.181 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Updating smithi154:/etc/ceph/ceph.conf 2024-05-20T17:58:48.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Updating smithi204:/etc/ceph/ceph.conf 2024-05-20T17:58:48.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/4084710359' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:58:48.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Updating smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:58:48.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:58:48.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-20T17:58:48.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:47 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/147904422' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T17:58:48.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.247+0000 7fbd4459d640 1 -- 172.21.15.129:0/4047451549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd3c107a10 msgr2=0x7fbd3c1015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.247+0000 7fbd4459d640 1 --2- 172.21.15.129:0/4047451549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd3c107a10 0x7fbd3c1015c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009ca0 tx=0x7fbd3802a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.248+0000 7fbd4459d640 1 -- 172.21.15.129:0/4047451549 shutdown_connections 2024-05-20T17:58:48.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.248+0000 7fbd4459d640 1 --2- 172.21.15.129:0/4047451549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd3c107eb0 0x7fbd3c101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.248+0000 7fbd4459d640 1 --2- 172.21.15.129:0/4047451549 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd3c102170 0x7fbd3c10e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.248+0000 7fbd4459d640 1 --2- 172.21.15.129:0/4047451549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd3c107a10 0x7fbd3c1015c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.248+0000 7fbd4459d640 1 -- 172.21.15.129:0/4047451549 >> 172.21.15.129:0/4047451549 conn(0x7fbd3c077160 msgr2=0x7fbd3c0775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:48.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.248+0000 7fbd4459d640 1 -- 172.21.15.129:0/4047451549 shutdown_connections 2024-05-20T17:58:48.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.248+0000 7fbd4459d640 1 -- 172.21.15.129:0/4047451549 wait complete. 2024-05-20T17:58:48.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.248+0000 7fbd4459d640 1 Processor -- start 2024-05-20T17:58:48.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.248+0000 7fbd4459d640 1 -- start start 2024-05-20T17:58:48.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd4459d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd3c102170 0x7fbd3c19baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd4459d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd3c107a10 0x7fbd3c19c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd4459d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd3c107eb0 0x7fbd3c1a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd4459d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c111db0 con 0x7fbd3c107eb0 2024-05-20T17:58:48.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd4459d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c111c30 con 0x7fbd3c102170 2024-05-20T17:58:48.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd4459d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c111f30 con 0x7fbd3c107a10 2024-05-20T17:58:48.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42339640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd3c102170 0x7fbd3c19baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42b3a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd3c107eb0 0x7fbd3c1a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42b3a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd3c107eb0 0x7fbd3c1a0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42778/0 (socket says 172.21.15.129:42778) 2024-05-20T17:58:48.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42b3a640 1 -- 172.21.15.129:0/3042665404 learned_addr learned my addr 172.21.15.129:0/3042665404 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:48.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42339640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd3c102170 0x7fbd3c19baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:54352/0 (socket says 172.21.15.129:54352) 2024-05-20T17:58:48.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd41b38640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd3c107a10 0x7fbd3c19c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42b3a640 1 -- 172.21.15.129:0/3042665404 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd3c107a10 msgr2=0x7fbd3c19c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42b3a640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd3c107a10 0x7fbd3c19c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42b3a640 1 -- 172.21.15.129:0/3042665404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd3c102170 msgr2=0x7fbd3c19baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42b3a640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd3c102170 0x7fbd3c19baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42b3a640 1 -- 172.21.15.129:0/3042665404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3c1a0930 con 0x7fbd3c107eb0 2024-05-20T17:58:48.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42339640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd3c102170 0x7fbd3c19baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:48.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.249+0000 7fbd42b3a640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd3c107eb0 0x7fbd3c1a0320 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400e970 tx=0x7fbd3400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:48.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.252+0000 7fbd337fe640 1 -- 172.21.15.129:0/3042665404 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3400ccc0 con 0x7fbd3c107eb0 2024-05-20T17:58:48.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.252+0000 7fbd4459d640 1 -- 172.21.15.129:0/3042665404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd3c1a0c20 con 0x7fbd3c107eb0 2024-05-20T17:58:48.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.252+0000 7fbd337fe640 1 -- 172.21.15.129:0/3042665404 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd3400ce60 con 0x7fbd3c107eb0 2024-05-20T17:58:48.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.252+0000 7fbd4459d640 1 -- 172.21.15.129:0/3042665404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd3c1a85d0 con 0x7fbd3c107eb0 2024-05-20T17:58:48.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.252+0000 7fbd337fe640 1 -- 172.21.15.129:0/3042665404 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd34010450 con 0x7fbd3c107eb0 2024-05-20T17:58:48.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.253+0000 7fbd4459d640 1 -- 172.21.15.129:0/3042665404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd3c0690c0 con 0x7fbd3c107eb0 2024-05-20T17:58:48.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.255+0000 7fbd337fe640 1 -- 172.21.15.129:0/3042665404 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 23) ==== 138010+0+0 (secure 0 0 0) 0x7fbd34005000 con 0x7fbd3c107eb0 2024-05-20T17:58:48.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.256+0000 7fbd337fe640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd280a2d80 0x7fbd280a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.256+0000 7fbd42339640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd280a2d80 0x7fbd280a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.256+0000 7fbd337fe640 1 -- 172.21.15.129:0/3042665404 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7fbd340cfc40 con 0x7fbd3c107eb0 2024-05-20T17:58:48.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.257+0000 7fbd42339640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd280a2d80 0x7fbd280a5290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbd38000f80 tx=0x7fbd38005fd0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:48.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.260+0000 7fbd337fe640 1 -- 172.21.15.129:0/3042665404 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbd34053850 con 0x7fbd3c107eb0 2024-05-20T17:58:48.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.462+0000 7fbd4459d640 1 -- 172.21.15.129:0/3042665404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fbd3c19ce20 con 0x7fbd3c107eb0 2024-05-20T17:58:48.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.466+0000 7fbd337fe640 1 -- 172.21.15.129:0/3042665404 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7fbd3409c700 con 0x7fbd3c107eb0 2024-05-20T17:58:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 -- 172.21.15.129:0/3042665404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd280a2d80 msgr2=0x7fbd280a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd280a2d80 0x7fbd280a5290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbd38000f80 tx=0x7fbd38005fd0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 -- 172.21.15.129:0/3042665404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd3c107eb0 msgr2=0x7fbd3c1a0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd3c107eb0 0x7fbd3c1a0320 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400e970 tx=0x7fbd3400ed70 comp rx=0 tx=0).stop 2024-05-20T17:58:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 -- 172.21.15.129:0/3042665404 shutdown_connections 2024-05-20T17:58:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd280a2d80 0x7fbd280a5290 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd3c107eb0 0x7fbd3c1a0320 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd3c107a10 0x7fbd3c19c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 --2- 172.21.15.129:0/3042665404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd3c102170 0x7fbd3c19baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 -- 172.21.15.129:0/3042665404 >> 172.21.15.129:0/3042665404 conn(0x7fbd3c077160 msgr2=0x7fbd3c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 -- 172.21.15.129:0/3042665404 shutdown_connections 2024-05-20T17:58:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.469+0000 7fbd4459d640 1 -- 172.21.15.129:0/3042665404 wait complete. 2024-05-20T17:58:48.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-20T17:58:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.598+0000 7f4b56365640 1 -- 172.21.15.129:0/1008941232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b50107a30 msgr2=0x7f4b501015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.598+0000 7f4b56365640 1 --2- 172.21.15.129:0/1008941232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b50107a30 0x7f4b501015e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009900 tx=0x7f4b4002a410 comp rx=0 tx=0).stop 2024-05-20T17:58:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.598+0000 7f4b56365640 1 -- 172.21.15.129:0/1008941232 shutdown_connections 2024-05-20T17:58:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.598+0000 7f4b56365640 1 --2- 172.21.15.129:0/1008941232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b50107ed0 0x7f4b50101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.598+0000 7f4b56365640 1 --2- 172.21.15.129:0/1008941232 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b50102190 0x7f4b5010e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.598+0000 7f4b56365640 1 --2- 172.21.15.129:0/1008941232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b50107a30 0x7f4b501015e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.598+0000 7f4b56365640 1 -- 172.21.15.129:0/1008941232 >> 172.21.15.129:0/1008941232 conn(0x7f4b50077030 msgr2=0x7f4b50077490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.598+0000 7f4b56365640 1 -- 172.21.15.129:0/1008941232 shutdown_connections 2024-05-20T17:58:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.598+0000 7f4b56365640 1 -- 172.21.15.129:0/1008941232 wait complete. 2024-05-20T17:58:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b56365640 1 Processor -- start 2024-05-20T17:58:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b56365640 1 -- start start 2024-05-20T17:58:48.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b56365640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b50102190 0x7f4b5019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b56365640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b50107a30 0x7f4b5019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b56365640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b50107ed0 0x7f4b5019b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b56365640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4b50111d80 con 0x7f4b50107a30 2024-05-20T17:58:48.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b56365640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b50111c00 con 0x7f4b50102190 2024-05-20T17:58:48.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b56365640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4b50111f00 con 0x7f4b50107ed0 2024-05-20T17:58:48.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b47fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b50107a30 0x7f4b5019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b47fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b50107a30 0x7f4b5019aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42794/0 (socket says 172.21.15.129:42794) 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b47fff640 1 -- 172.21.15.129:0/1891430306 learned_addr learned my addr 172.21.15.129:0/1891430306 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.599+0000 7f4b54902640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b50107ed0 0x7f4b5019b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b4ffff640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b50102190 0x7f4b5019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b47fff640 1 -- 172.21.15.129:0/1891430306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b50107ed0 msgr2=0x7f4b5019b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b47fff640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b50107ed0 0x7f4b5019b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b47fff640 1 -- 172.21.15.129:0/1891430306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b50102190 msgr2=0x7f4b5019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b47fff640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b50102190 0x7f4b5019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b47fff640 1 -- 172.21.15.129:0/1891430306 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b501a5970 con 0x7f4b50107a30 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b4ffff640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b50102190 0x7f4b5019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b54902640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b50107ed0 0x7f4b5019b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b47fff640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b50107a30 0x7f4b5019aba0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4b38005fd0 tx=0x7f4b3800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b4dffb640 1 -- 172.21.15.129:0/1891430306 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b38017070 con 0x7f4b50107a30 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b4dffb640 1 -- 172.21.15.129:0/1891430306 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b38009c30 con 0x7f4b50107a30 2024-05-20T17:58:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b56365640 1 -- 172.21.15.129:0/1891430306 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b501a5c60 con 0x7f4b50107a30 2024-05-20T17:58:48.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b4dffb640 1 -- 172.21.15.129:0/1891430306 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b38003e40 con 0x7f4b50107a30 2024-05-20T17:58:48.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.600+0000 7f4b56365640 1 -- 172.21.15.129:0/1891430306 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b501a61b0 con 0x7f4b50107a30 2024-05-20T17:58:48.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.601+0000 7f4b56365640 1 -- 172.21.15.129:0/1891430306 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b5004f720 con 0x7f4b50107a30 2024-05-20T17:58:48.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.602+0000 7f4b4dffb640 1 -- 172.21.15.129:0/1891430306 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 23) ==== 138010+0+0 (secure 0 0 0) 0x7f4b38004000 con 0x7f4b50107a30 2024-05-20T17:58:48.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.603+0000 7f4b4dffb640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b3c0a2d80 0x7f4b3c0a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.603+0000 7f4b4dffb640 1 -- 172.21.15.129:0/1891430306 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f4b380cf200 con 0x7f4b50107a30 2024-05-20T17:58:48.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.603+0000 7f4b4ffff640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b3c0a2d80 0x7f4b3c0a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.604+0000 7f4b4ffff640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b3c0a2d80 0x7f4b3c0a5290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009230 tx=0x7f4b400095f0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.608+0000 7f4b4dffb640 1 -- 172.21.15.129:0/1891430306 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4b38020030 con 0x7f4b50107a30 2024-05-20T17:58:48.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.794+0000 7f4b56365640 1 -- 172.21.15.129:0/1891430306 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f4b501059b0 con 0x7f4b50107a30 2024-05-20T17:58:48.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.799+0000 7f4b4dffb640 1 -- 172.21.15.129:0/1891430306 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) ==== 133+0+0 (secure 0 0 0) 0x7f4b3809bcc0 con 0x7f4b50107a30 2024-05-20T17:58:48.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 -- 172.21.15.129:0/1891430306 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b3c0a2d80 msgr2=0x7f4b3c0a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b3c0a2d80 0x7f4b3c0a5290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009230 tx=0x7f4b400095f0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 -- 172.21.15.129:0/1891430306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b50107a30 msgr2=0x7f4b5019aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b50107a30 0x7f4b5019aba0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4b38005fd0 tx=0x7f4b3800dde0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 -- 172.21.15.129:0/1891430306 shutdown_connections 2024-05-20T17:58:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b3c0a2d80 0x7f4b3c0a5290 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4b50107ed0 0x7f4b5019b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b50107a30 0x7f4b5019aba0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 --2- 172.21.15.129:0/1891430306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b50102190 0x7f4b5019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 -- 172.21.15.129:0/1891430306 >> 172.21.15.129:0/1891430306 conn(0x7f4b50077030 msgr2=0x7f4b50103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 -- 172.21.15.129:0/1891430306 shutdown_connections 2024-05-20T17:58:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.801+0000 7f4b56365640 1 -- 172.21.15.129:0/1891430306 wait complete. 2024-05-20T17:58:48.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-20T17:58:48.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.930+0000 7f97f1a26640 1 -- 172.21.15.129:0/141061049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97ec10a700 msgr2=0x7f97ec10cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.930+0000 7f97f1a26640 1 --2- 172.21.15.129:0/141061049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97ec10a700 0x7f97ec10cb20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f97d4009900 tx=0x7f97d402a480 comp rx=0 tx=0).stop 2024-05-20T17:58:48.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.930+0000 7f97f1a26640 1 -- 172.21.15.129:0/141061049 shutdown_connections 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.930+0000 7f97f1a26640 1 --2- 172.21.15.129:0/141061049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97ec10d0b0 0x7f97ec10f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.930+0000 7f97f1a26640 1 --2- 172.21.15.129:0/141061049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97ec10a700 0x7f97ec10cb20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.930+0000 7f97f1a26640 1 --2- 172.21.15.129:0/141061049 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97ec103ca0 0x7f97ec1040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.930+0000 7f97f1a26640 1 -- 172.21.15.129:0/141061049 >> 172.21.15.129:0/141061049 conn(0x7f97ec077160 msgr2=0x7f97ec0775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.930+0000 7f97f1a26640 1 -- 172.21.15.129:0/141061049 shutdown_connections 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.930+0000 7f97f1a26640 1 -- 172.21.15.129:0/141061049 wait complete. 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.930+0000 7f97f1a26640 1 Processor -- start 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.930+0000 7f97f1a26640 1 -- start start 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97f1a26640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97ec103ca0 0x7f97ec19f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97f1a26640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97ec10a700 0x7f97ec19fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97f1a26640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97ec10d0b0 0x7f97ec199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97f1a26640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec112e40 con 0x7f97ec10a700 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97f1a26640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec112cc0 con 0x7f97ec10d0b0 2024-05-20T17:58:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97f1a26640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec112fc0 con 0x7f97ec103ca0 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97e3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97ec10a700 0x7f97ec19fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97e3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97ec10a700 0x7f97ec19fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42812/0 (socket says 172.21.15.129:42812) 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97e3fff640 1 -- 172.21.15.129:0/1766866112 learned_addr learned my addr 172.21.15.129:0/1766866112 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97eb7fe640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97ec10d0b0 0x7f97ec199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97eaffd640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97ec103ca0 0x7f97ec19f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97e3fff640 1 -- 172.21.15.129:0/1766866112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97ec103ca0 msgr2=0x7f97ec19f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97e3fff640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97ec103ca0 0x7f97ec19f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97e3fff640 1 -- 172.21.15.129:0/1766866112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97ec10d0b0 msgr2=0x7f97ec199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97e3fff640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97ec10d0b0 0x7f97ec199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97e3fff640 1 -- 172.21.15.129:0/1766866112 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ec199db0 con 0x7f97ec10a700 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97e3fff640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97ec10a700 0x7f97ec19fa40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f97d4006010 tx=0x7f97d40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:48.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97f0a24640 1 -- 172.21.15.129:0/1766866112 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d403f030 con 0x7f97ec10a700 2024-05-20T17:58:48.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97f0a24640 1 -- 172.21.15.129:0/1766866112 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97d4003ce0 con 0x7f97ec10a700 2024-05-20T17:58:48.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.931+0000 7f97f0a24640 1 -- 172.21.15.129:0/1766866112 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4035070 con 0x7f97ec10a700 2024-05-20T17:58:48.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.932+0000 7f97f1a26640 1 -- 172.21.15.129:0/1766866112 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ec199fe0 con 0x7f97ec10a700 2024-05-20T17:58:48.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.932+0000 7f97f1a26640 1 -- 172.21.15.129:0/1766866112 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ec19a4d0 con 0x7f97ec10a700 2024-05-20T17:58:48.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.933+0000 7f97f1a26640 1 -- 172.21.15.129:0/1766866112 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97ec069bd0 con 0x7f97ec10a700 2024-05-20T17:58:48.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.934+0000 7f97f0a24640 1 -- 172.21.15.129:0/1766866112 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 23) ==== 138010+0+0 (secure 0 0 0) 0x7f97d4042030 con 0x7f97ec10a700 2024-05-20T17:58:48.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.934+0000 7f97f0a24640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f97d80a2d80 0x7f97d80a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:48.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.934+0000 7f97eaffd640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f97d80a2d80 0x7f97d80a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:48.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.935+0000 7f97f0a24640 1 -- 172.21.15.129:0/1766866112 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f97d40efac0 con 0x7f97ec10a700 2024-05-20T17:58:48.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.935+0000 7f97eaffd640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f97d80a2d80 0x7f97d80a5290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97dc005fd0 tx=0x7f97dc005850 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:48.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:48.939+0000 7f97f0a24640 1 -- 172.21.15.129:0/1766866112 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f97d40bc640 con 0x7f97ec10a700 2024-05-20T17:58:49.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.144+0000 7f97f1a26640 1 -- 172.21.15.129:0/1766866112 --> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f97ec100850 con 0x7f97d80a2d80 2024-05-20T17:58:49.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:58:49.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:58:49.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: Updating smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='client.? ' entity='client.admin' 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3042665404' entity='client.admin' 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:49.164 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1891430306' entity='client.admin' 2024-05-20T17:58:49.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:49.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:49.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:49.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.203+0000 7f97f0a24640 1 -- 172.21.15.129:0/1766866112 <== mgr.15228 v2:172.21.15.129:6800/3515662734 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f97ec100850 con 0x7f97d80a2d80 2024-05-20T17:58:49.202 INFO:teuthology.orchestra.run.smithi129.stdout:Schedule set for path / 2024-05-20T17:58:49.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.205+0000 7f97f1a26640 1 -- 172.21.15.129:0/1766866112 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f97d80a2d80 msgr2=0x7f97d80a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.205+0000 7f97f1a26640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f97d80a2d80 0x7f97d80a5290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97dc005fd0 tx=0x7f97dc005850 comp rx=0 tx=0).stop 2024-05-20T17:58:49.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.205+0000 7f97f1a26640 1 -- 172.21.15.129:0/1766866112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97ec10a700 msgr2=0x7f97ec19fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.205+0000 7f97f1a26640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97ec10a700 0x7f97ec19fa40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f97d4006010 tx=0x7f97d40042f0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.205+0000 7f97f1a26640 1 -- 172.21.15.129:0/1766866112 shutdown_connections 2024-05-20T17:58:49.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.205+0000 7f97f1a26640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f97d80a2d80 0x7f97d80a5290 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.205+0000 7f97f1a26640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97ec10d0b0 0x7f97ec199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.205+0000 7f97f1a26640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97ec10a700 0x7f97ec19fa40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.205+0000 7f97f1a26640 1 --2- 172.21.15.129:0/1766866112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97ec103ca0 0x7f97ec19f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.205+0000 7f97f1a26640 1 -- 172.21.15.129:0/1766866112 >> 172.21.15.129:0/1766866112 conn(0x7f97ec077160 msgr2=0x7f97ec100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:49.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.206+0000 7f97f1a26640 1 -- 172.21.15.129:0/1766866112 shutdown_connections 2024-05-20T17:58:49.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.206+0000 7f97f1a26640 1 -- 172.21.15.129:0/1766866112 wait complete. 2024-05-20T17:58:49.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-20T17:58:49.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.331+0000 7f2ae20fe640 1 -- 172.21.15.129:0/2246663894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2adc106420 msgr2=0x7f2adc10dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.331+0000 7f2ae20fe640 1 --2- 172.21.15.129:0/2246663894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2adc106420 0x7f2adc10dbc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4009ca0 tx=0x7f2ad402a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.331+0000 7f2ae20fe640 1 -- 172.21.15.129:0/2246663894 shutdown_connections 2024-05-20T17:58:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.331+0000 7f2ae20fe640 1 --2- 172.21.15.129:0/2246663894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2adc105960 0x7f2adc105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.331+0000 7f2ae20fe640 1 --2- 172.21.15.129:0/2246663894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2adc106420 0x7f2adc10dbc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.331+0000 7f2ae20fe640 1 --2- 172.21.15.129:0/2246663894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2adc1013d0 0x7f2adc1017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.331+0000 7f2ae20fe640 1 -- 172.21.15.129:0/2246663894 >> 172.21.15.129:0/2246663894 conn(0x7f2adc0fc830 msgr2=0x7f2adc0fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.331+0000 7f2ae20fe640 1 -- 172.21.15.129:0/2246663894 shutdown_connections 2024-05-20T17:58:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.331+0000 7f2ae20fe640 1 -- 172.21.15.129:0/2246663894 wait complete. 2024-05-20T17:58:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.332+0000 7f2ae20fe640 1 Processor -- start 2024-05-20T17:58:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.332+0000 7f2ae20fe640 1 -- start start 2024-05-20T17:58:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.332+0000 7f2ae20fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2adc1013d0 0x7f2adc19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.332+0000 7f2ae20fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2adc105960 0x7f2adc19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.332+0000 7f2ae20fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2adc106420 0x7f2adc19b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2ae20fe640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc111d00 con 0x7f2adc106420 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2ae20fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc111b80 con 0x7f2adc1013d0 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2ae20fe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc111e80 con 0x7f2adc105960 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2adbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2adc106420 0x7f2adc19b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2adbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2adc106420 0x7f2adc19b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42824/0 (socket says 172.21.15.129:42824) 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2adb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2adc1013d0 0x7f2adc19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2adaffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2adc105960 0x7f2adc19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2adbfff640 1 -- 172.21.15.129:0/4104578855 learned_addr learned my addr 172.21.15.129:0/4104578855 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2adbfff640 1 -- 172.21.15.129:0/4104578855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2adc105960 msgr2=0x7f2adc19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2adbfff640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2adc105960 0x7f2adc19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2adbfff640 1 -- 172.21.15.129:0/4104578855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2adc1013d0 msgr2=0x7f2adc19a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2adbfff640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2adc1013d0 0x7f2adc19a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2adbfff640 1 -- 172.21.15.129:0/4104578855 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc1a5900 con 0x7f2adc106420 2024-05-20T17:58:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2adbfff640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2adc106420 0x7f2adc19b060 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4031a20 tx=0x7f2ad4031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2ad8ff9640 1 -- 172.21.15.129:0/4104578855 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4003c30 con 0x7f2adc106420 2024-05-20T17:58:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2ae20fe640 1 -- 172.21.15.129:0/4104578855 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2adc1a5b90 con 0x7f2adc106420 2024-05-20T17:58:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2ad8ff9640 1 -- 172.21.15.129:0/4104578855 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ad4003dd0 con 0x7f2adc106420 2024-05-20T17:58:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2ad8ff9640 1 -- 172.21.15.129:0/4104578855 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad402fc40 con 0x7f2adc106420 2024-05-20T17:58:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.333+0000 7f2ae20fe640 1 -- 172.21.15.129:0/4104578855 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2adc1a6130 con 0x7f2adc106420 2024-05-20T17:58:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.335+0000 7f2ae20fe640 1 -- 172.21.15.129:0/4104578855 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2adc04f720 con 0x7f2adc106420 2024-05-20T17:58:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.336+0000 7f2ad8ff9640 1 -- 172.21.15.129:0/4104578855 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 23) ==== 138010+0+0 (secure 0 0 0) 0x7f2ad4004050 con 0x7f2adc106420 2024-05-20T17:58:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.336+0000 7f2ad8ff9640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2ac40a2d80 0x7f2ac40a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.336+0000 7f2adb7fe640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2ac40a2d80 0x7f2ac40a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.336+0000 7f2ad8ff9640 1 -- 172.21.15.129:0/4104578855 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f2ad40f06a0 con 0x7f2adc106420 2024-05-20T17:58:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.337+0000 7f2adb7fe640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2ac40a2d80 0x7f2ac40a5290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0058c0 tx=0x7f2acc00a250 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:49.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.341+0000 7f2ad8ff9640 1 -- 172.21.15.129:0/4104578855 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2ad40bd300 con 0x7f2adc106420 2024-05-20T17:58:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:58:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:58:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:58:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: Updating smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='client.? ' entity='client.admin' 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3042665404' entity='client.admin' 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:49.395 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:49.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1891430306' entity='client.admin' 2024-05-20T17:58:49.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:49.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:49.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:49.396 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:58:49.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:58:49.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: Updating smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: Updating smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: Updating smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='client.? ' entity='client.admin' 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3042665404' entity='client.admin' 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:49.431 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1891430306' entity='client.admin' 2024-05-20T17:58:49.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:58:49.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:58:49.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:58:49.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:58:49.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.539+0000 7f2ae20fe640 1 -- 172.21.15.129:0/4104578855 --> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f2adc061f90 con 0x7f2ac40a2d80 2024-05-20T17:58:49.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.599+0000 7f2ad8ff9640 1 -- 172.21.15.129:0/4104578855 <== mgr.15228 v2:172.21.15.129:6800/3515662734 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f2adc061f90 con 0x7f2ac40a2d80 2024-05-20T17:58:49.598 INFO:teuthology.orchestra.run.smithi129.stdout:Retention added to path / 2024-05-20T17:58:49.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.601+0000 7f2ae20fe640 1 -- 172.21.15.129:0/4104578855 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2ac40a2d80 msgr2=0x7f2ac40a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.601+0000 7f2ae20fe640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2ac40a2d80 0x7f2ac40a5290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0058c0 tx=0x7f2acc00a250 comp rx=0 tx=0).stop 2024-05-20T17:58:49.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.601+0000 7f2ae20fe640 1 -- 172.21.15.129:0/4104578855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2adc106420 msgr2=0x7f2adc19b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.601+0000 7f2ae20fe640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2adc106420 0x7f2adc19b060 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4031a20 tx=0x7f2ad4031a50 comp rx=0 tx=0).stop 2024-05-20T17:58:49.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.601+0000 7f2ae20fe640 1 -- 172.21.15.129:0/4104578855 shutdown_connections 2024-05-20T17:58:49.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.601+0000 7f2ae20fe640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2ac40a2d80 0x7f2ac40a5290 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.601+0000 7f2ae20fe640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2adc106420 0x7f2adc19b060 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.601+0000 7f2ae20fe640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2adc105960 0x7f2adc19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.601+0000 7f2ae20fe640 1 --2- 172.21.15.129:0/4104578855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2adc1013d0 0x7f2adc19a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.601+0000 7f2ae20fe640 1 -- 172.21.15.129:0/4104578855 >> 172.21.15.129:0/4104578855 conn(0x7f2adc0fc830 msgr2=0x7f2adc103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:49.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.602+0000 7f2ae20fe640 1 -- 172.21.15.129:0/4104578855 shutdown_connections 2024-05-20T17:58:49.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.602+0000 7f2ae20fe640 1 -- 172.21.15.129:0/4104578855 wait complete. 2024-05-20T17:58:49.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-20T17:58:49.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.724+0000 7fcaedfc3640 1 -- 172.21.15.129:0/4203214322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae8102190 msgr2=0x7fcae810e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.724+0000 7fcaedfc3640 1 --2- 172.21.15.129:0/4203214322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae8102190 0x7fcae810e4c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fcae0009ca0 tx=0x7fcae002a8c0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.724+0000 7fcaedfc3640 1 -- 172.21.15.129:0/4203214322 shutdown_connections 2024-05-20T17:58:49.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.724+0000 7fcaedfc3640 1 --2- 172.21.15.129:0/4203214322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcae8107ed0 0x7fcae8101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.725+0000 7fcaedfc3640 1 --2- 172.21.15.129:0/4203214322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae8102190 0x7fcae810e4c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.725+0000 7fcaedfc3640 1 --2- 172.21.15.129:0/4203214322 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae8107a30 0x7fcae81015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.725+0000 7fcaedfc3640 1 -- 172.21.15.129:0/4203214322 >> 172.21.15.129:0/4203214322 conn(0x7fcae80fd210 msgr2=0x7fcae80ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:49.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.725+0000 7fcaedfc3640 1 -- 172.21.15.129:0/4203214322 shutdown_connections 2024-05-20T17:58:49.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.725+0000 7fcaedfc3640 1 -- 172.21.15.129:0/4203214322 wait complete. 2024-05-20T17:58:49.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.725+0000 7fcaedfc3640 1 Processor -- start 2024-05-20T17:58:49.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.725+0000 7fcaedfc3640 1 -- start start 2024-05-20T17:58:49.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcaedfc3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae8102190 0x7fcae819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:49.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcaedfc3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcae8107a30 0x7fcae819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:49.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcaedfc3640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae8107ed0 0x7fcae819b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:49.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcaedfc3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcae8111dc0 con 0x7fcae8102190 2024-05-20T17:58:49.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcaedfc3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcae8111c40 con 0x7fcae8107a30 2024-05-20T17:58:49.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcaedfc3640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcae8111f40 con 0x7fcae8107ed0 2024-05-20T17:58:49.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae8102190 0x7fcae819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:49.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae8102190 0x7fcae819a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33898/0 (socket says 172.21.15.129:33898) 2024-05-20T17:58:49.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae77fe640 1 -- 172.21.15.129:0/3418139687 learned_addr learned my addr 172.21.15.129:0/3418139687 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae6ffd640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcae8107a30 0x7fcae819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae7fff640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae8107ed0 0x7fcae819b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae77fe640 1 -- 172.21.15.129:0/3418139687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae8107ed0 msgr2=0x7fcae819b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae77fe640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae8107ed0 0x7fcae819b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae77fe640 1 -- 172.21.15.129:0/3418139687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcae8107a30 msgr2=0x7fcae819ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae77fe640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcae8107a30 0x7fcae819ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae77fe640 1 -- 172.21.15.129:0/3418139687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae81a5920 con 0x7fcae8102190 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae77fe640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae8102190 0x7fcae819a5e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcad800e9c0 tx=0x7fcad800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.726+0000 7fcae4ff9640 1 -- 172.21.15.129:0/3418139687 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad800ccc0 con 0x7fcae8102190 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.727+0000 7fcaedfc3640 1 -- 172.21.15.129:0/3418139687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae81a5c10 con 0x7fcae8102190 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.727+0000 7fcae4ff9640 1 -- 172.21.15.129:0/3418139687 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad8005500 con 0x7fcae8102190 2024-05-20T17:58:49.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.727+0000 7fcae4ff9640 1 -- 172.21.15.129:0/3418139687 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad8010690 con 0x7fcae8102190 2024-05-20T17:58:49.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.727+0000 7fcaedfc3640 1 -- 172.21.15.129:0/3418139687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae81a6110 con 0x7fcae8102190 2024-05-20T17:58:49.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.728+0000 7fcaedfc3640 1 -- 172.21.15.129:0/3418139687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcae804f720 con 0x7fcae8102190 2024-05-20T17:58:49.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.729+0000 7fcae4ff9640 1 -- 172.21.15.129:0/3418139687 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 23) ==== 138010+0+0 (secure 0 0 0) 0x7fcad8004e80 con 0x7fcae8102190 2024-05-20T17:58:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.729+0000 7fcae4ff9640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcad00a2d80 0x7fcad00a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.730+0000 7fcae4ff9640 1 -- 172.21.15.129:0/3418139687 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7fcad80cfc50 con 0x7fcae8102190 2024-05-20T17:58:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.730+0000 7fcae6ffd640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcad00a2d80 0x7fcad00a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.730+0000 7fcae6ffd640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcad00a2d80 0x7fcad00a5290 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcae8106bb0 tx=0x7fcadc005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:49.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.734+0000 7fcae4ff9640 1 -- 172.21.15.129:0/3418139687 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcad8053850 con 0x7fcae8102190 2024-05-20T17:58:49.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.934+0000 7fcaedfc3640 1 -- 172.21.15.129:0/3418139687 --> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fcae8061f90 con 0x7fcad00a2d80 2024-05-20T17:58:49.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.941+0000 7fcae4ff9640 1 -- 172.21.15.129:0/3418139687 <== mgr.15228 v2:172.21.15.129:6800/3515662734 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fcae8061f90 con 0x7fcad00a2d80 2024-05-20T17:58:49.940 INFO:teuthology.orchestra.run.smithi129.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-20T00:00:00", "created": "2024-05-20T17:58:49", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-20T17:58:49.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 -- 172.21.15.129:0/3418139687 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcad00a2d80 msgr2=0x7fcad00a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcad00a2d80 0x7fcad00a5290 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcae8106bb0 tx=0x7fcadc005f90 comp rx=0 tx=0).stop 2024-05-20T17:58:49.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 -- 172.21.15.129:0/3418139687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae8102190 msgr2=0x7fcae819a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:49.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae8102190 0x7fcae819a5e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcad800e9c0 tx=0x7fcad800ee80 comp rx=0 tx=0).stop 2024-05-20T17:58:49.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 -- 172.21.15.129:0/3418139687 shutdown_connections 2024-05-20T17:58:49.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcad00a2d80 0x7fcad00a5290 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcae8107a30 0x7fcae819ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcae8107ed0 0x7fcae819b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 --2- 172.21.15.129:0/3418139687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcae8102190 0x7fcae819a5e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:49.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 -- 172.21.15.129:0/3418139687 >> 172.21.15.129:0/3418139687 conn(0x7fcae80fd210 msgr2=0x7fcae80ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:49.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 -- 172.21.15.129:0/3418139687 shutdown_connections 2024-05-20T17:58:49.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:49.944+0000 7fcaedfc3640 1 -- 172.21.15.129:0/3418139687 wait complete. 2024-05-20T17:58:49.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-20T17:58:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.070+0000 7fd78754c640 1 -- 172.21.15.129:0/1369840833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd780107a30 msgr2=0x7fd7801015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.070+0000 7fd78754c640 1 --2- 172.21.15.129:0/1369840833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd780107a30 0x7fd7801015e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd770009900 tx=0x7fd77002a410 comp rx=0 tx=0).stop 2024-05-20T17:58:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.070+0000 7fd78754c640 1 -- 172.21.15.129:0/1369840833 shutdown_connections 2024-05-20T17:58:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.070+0000 7fd78754c640 1 --2- 172.21.15.129:0/1369840833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd780107ed0 0x7fd780101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.070+0000 7fd78754c640 1 --2- 172.21.15.129:0/1369840833 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd780102190 0x7fd78010e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.070+0000 7fd78754c640 1 --2- 172.21.15.129:0/1369840833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd780107a30 0x7fd7801015e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.070+0000 7fd78754c640 1 -- 172.21.15.129:0/1369840833 >> 172.21.15.129:0/1369840833 conn(0x7fd780077030 msgr2=0x7fd780077490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.070+0000 7fd78754c640 1 -- 172.21.15.129:0/1369840833 shutdown_connections 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.070+0000 7fd78754c640 1 -- 172.21.15.129:0/1369840833 wait complete. 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd78754c640 1 Processor -- start 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd78754c640 1 -- start start 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd78754c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd780102190 0x7fd780198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd78754c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd780107a30 0x7fd780198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd78754c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd780107ed0 0x7fd78019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd78754c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd780111d80 con 0x7fd780107a30 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd78754c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd780111c00 con 0x7fd780102190 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd78754c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd780111f00 con 0x7fd780107ed0 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd784ae7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd780107a30 0x7fd780198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd784ae7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd780107a30 0x7fd780198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33916/0 (socket says 172.21.15.129:33916) 2024-05-20T17:58:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd784ae7640 1 -- 172.21.15.129:0/1520551004 learned_addr learned my addr 172.21.15.129:0/1520551004 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:58:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd785ae9640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd780107ed0 0x7fd78019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd784ae7640 1 -- 172.21.15.129:0/1520551004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd780107ed0 msgr2=0x7fd78019cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd7852e8640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd780102190 0x7fd780198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd784ae7640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd780107ed0 0x7fd78019cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd784ae7640 1 -- 172.21.15.129:0/1520551004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd780102190 msgr2=0x7fd780198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd784ae7640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd780102190 0x7fd780198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.071+0000 7fd784ae7640 1 -- 172.21.15.129:0/1520551004 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78019d560 con 0x7fd780107a30 2024-05-20T17:58:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.072+0000 7fd784ae7640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd780107a30 0x7fd780198c60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd77400e9f0 tx=0x7fd77400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.072+0000 7fd77e7fc640 1 -- 172.21.15.129:0/1520551004 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77400ccc0 con 0x7fd780107a30 2024-05-20T17:58:50.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.072+0000 7fd77e7fc640 1 -- 172.21.15.129:0/1520551004 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7740035f0 con 0x7fd780107a30 2024-05-20T17:58:50.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.072+0000 7fd78754c640 1 -- 172.21.15.129:0/1520551004 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78019d850 con 0x7fd780107a30 2024-05-20T17:58:50.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.072+0000 7fd77e7fc640 1 -- 172.21.15.129:0/1520551004 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77400ce60 con 0x7fd780107a30 2024-05-20T17:58:50.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.072+0000 7fd78754c640 1 -- 172.21.15.129:0/1520551004 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7801a51d0 con 0x7fd780107a30 2024-05-20T17:58:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.073+0000 7fd78754c640 1 -- 172.21.15.129:0/1520551004 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78004f720 con 0x7fd780107a30 2024-05-20T17:58:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.074+0000 7fd77e7fc640 1 -- 172.21.15.129:0/1520551004 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 23) ==== 138010+0+0 (secure 0 0 0) 0x7fd7740037a0 con 0x7fd780107a30 2024-05-20T17:58:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.075+0000 7fd77e7fc640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd7680a2d80 0x7fd7680a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:58:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.075+0000 7fd7852e8640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd7680a2d80 0x7fd7680a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:58:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.075+0000 7fd77e7fc640 1 -- 172.21.15.129:0/1520551004 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7fd7740cfa60 con 0x7fd780107a30 2024-05-20T17:58:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.075+0000 7fd7852e8640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd7680a2d80 0x7fd7680a5290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd77002a930 tx=0x7fd770000bc0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:58:50.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.079+0000 7fd77e7fc640 1 -- 172.21.15.129:0/1520551004 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd77409c520 con 0x7fd780107a30 2024-05-20T17:58:50.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.275+0000 7fd78754c640 1 -- 172.21.15.129:0/1520551004 --> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fd780061f90 con 0x7fd7680a2d80 2024-05-20T17:58:50.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.282+0000 7fd77e7fc640 1 -- 172.21.15.129:0/1520551004 <== mgr.15228 v2:172.21.15.129:6800/3515662734 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fd780061f90 con 0x7fd7680a2d80 2024-05-20T17:58:50.281 INFO:teuthology.orchestra.run.smithi129.stdout:/ 1m 6m3h 2024-05-20T17:58:50.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 -- 172.21.15.129:0/1520551004 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd7680a2d80 msgr2=0x7fd7680a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:50.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd7680a2d80 0x7fd7680a5290 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd77002a930 tx=0x7fd770000bc0 comp rx=0 tx=0).stop 2024-05-20T17:58:50.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 -- 172.21.15.129:0/1520551004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd780107a30 msgr2=0x7fd780198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:58:50.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd780107a30 0x7fd780198c60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd77400e9f0 tx=0x7fd77400eeb0 comp rx=0 tx=0).stop 2024-05-20T17:58:50.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 -- 172.21.15.129:0/1520551004 shutdown_connections 2024-05-20T17:58:50.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd7680a2d80 0x7fd7680a5290 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:50.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd780107ed0 0x7fd78019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:50.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd780107a30 0x7fd780198c60 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:50.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 --2- 172.21.15.129:0/1520551004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd780102190 0x7fd780198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:58:50.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 -- 172.21.15.129:0/1520551004 >> 172.21.15.129:0/1520551004 conn(0x7fd780077030 msgr2=0x7fd780103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:58:50.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 -- 172.21.15.129:0/1520551004 shutdown_connections 2024-05-20T17:58:50.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:58:50.285+0000 7fd78754c640 1 -- 172.21.15.129:0/1520551004 wait complete. 2024-05-20T17:58:50.293 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:50 smithi129 ceph-mon[32029]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:50.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-20T17:58:50.320 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-20T17:58:50.331 INFO:teuthology.run_tasks:Running task background_exec... 2024-05-20T17:58:50.378 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi129.front.sepia.ceph.com 2024-05-20T17:58:50.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-05-20T17:58:50.381 INFO:teuthology.run_tasks:Running task workunit... 2024-05-20T17:58:50.413 INFO:tasks.workunit:Pulling workunits from ref d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:58:50.414 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-20T17:58:50.414 INFO:tasks.workunit:timeout=3h 2024-05-20T17:58:50.414 INFO:tasks.workunit:cleanup=True 2024-05-20T17:58:50.414 DEBUG:teuthology.orchestra.run.smithi129:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-20T17:58:50.415 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:50 smithi154 ceph-mon[36482]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:50.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:50 smithi204 ceph-mon[37076]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:50.430 INFO:teuthology.orchestra.run.smithi129.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-20T17:58:50.431 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-20T17:58:50.431 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-05-20T17:58:50.431 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-20T17:58:50.431 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-20T17:58:50.431 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-05-20 17:56:51.428959012 +0000 2024-05-20T17:58:50.431 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-05-20 17:56:51.428959012 +0000 2024-05-20T17:58:50.431 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-05-20 17:58:40.871118895 +0000 2024-05-20T17:58:50.431 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-05-20 17:56:51.428959012 +0000 2024-05-20T17:58:50.431 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-20T17:58:50.431 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-20T17:58:50.502 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d2b507c724ef6c8295ee0a35d185cdab167fd61d 2024-05-20T17:58:50.558 INFO:tasks.workunit.client.0.smithi129.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-20T17:58:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:51 smithi154 ceph-mon[36482]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:51 smithi154 ceph-mon[36482]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 93 KiB/s wr, 8 op/s 2024-05-20T17:58:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:51 smithi154 ceph-mon[36482]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:51 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T17:58:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:51 smithi154 ceph-mon[36482]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:51 smithi129 ceph-mon[32029]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:51 smithi129 ceph-mon[32029]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 93 KiB/s wr, 8 op/s 2024-05-20T17:58:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:51 smithi129 ceph-mon[32029]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:51 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T17:58:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:51 smithi129 ceph-mon[32029]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:51 smithi204 ceph-mon[37076]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:51 smithi204 ceph-mon[37076]: pgmap v6: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 93 KiB/s wr, 8 op/s 2024-05-20T17:58:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:51 smithi204 ceph-mon[37076]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:51 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T17:58:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:51 smithi204 ceph-mon[37076]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T17:58:53.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:53 smithi154 ceph-mon[36482]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 131 KiB/s wr, 63 op/s 2024-05-20T17:58:53.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:53 smithi129 ceph-mon[32029]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 131 KiB/s wr, 63 op/s 2024-05-20T17:58:53.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:53 smithi204 ceph-mon[37076]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 131 KiB/s wr, 63 op/s 2024-05-20T17:58:55.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:55 smithi154 ceph-mon[36482]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 102 KiB/s wr, 49 op/s 2024-05-20T17:58:55.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:55 smithi129 ceph-mon[32029]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 102 KiB/s wr, 49 op/s 2024-05-20T17:58:55.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:55 smithi204 ceph-mon[37076]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 102 KiB/s wr, 49 op/s 2024-05-20T17:58:57.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:57 smithi154 ceph-mon[36482]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-05-20T17:58:57.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:57 smithi129 ceph-mon[32029]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-05-20T17:58:57.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:57 smithi204 ceph-mon[37076]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-05-20T17:58:59.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:58:59 smithi154 ceph-mon[36482]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-05-20T17:58:59.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:58:59 smithi129 ceph-mon[32029]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-05-20T17:58:59.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:58:59 smithi204 ceph-mon[37076]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-05-20T17:59:00.912 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:59:00 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:59:00.611+0000 7fdd315a2700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:59:00.912 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:59:00 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:59:00.611+0000 7fdd315a2700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:59:00.912 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:59:00 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:59:00.611+0000 7fdd315a2700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:59:00.913 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:59:00 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:59:00.611+0000 7fdd315a2700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:59:00.913 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 17:59:00 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T17:59:00.611+0000 7fdd315a2700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-20T17:59:01.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:01 smithi154 ceph-mon[36482]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-05-20T17:59:01.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:01 smithi129 ceph-mon[32029]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-05-20T17:59:01.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:01 smithi204 ceph-mon[37076]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-05-20T17:59:03.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:03 smithi154 ceph-mon[36482]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 38 KiB/s wr, 40 op/s 2024-05-20T17:59:03.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:03 smithi154 ceph-mon[36482]: mgrmap e24: x(active, since 18s), standbys: z, y 2024-05-20T17:59:03.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:03 smithi129 ceph-mon[32029]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 38 KiB/s wr, 40 op/s 2024-05-20T17:59:03.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:03 smithi129 ceph-mon[32029]: mgrmap e24: x(active, since 18s), standbys: z, y 2024-05-20T17:59:03.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:03 smithi204 ceph-mon[37076]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 38 KiB/s wr, 40 op/s 2024-05-20T17:59:03.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:03 smithi204 ceph-mon[37076]: mgrmap e24: x(active, since 18s), standbys: z, y 2024-05-20T17:59:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.533+0000 7f4725691640 1 -- 172.21.15.129:0/2501040236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4720108bb0 msgr2=0x7f4720100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.533+0000 7f4725691640 1 --2- 172.21.15.129:0/2501040236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4720108bb0 0x7f4720100b20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4710009900 tx=0x7f471002a410 comp rx=0 tx=0).stop 2024-05-20T17:59:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.533+0000 7f4725691640 1 -- 172.21.15.129:0/2501040236 shutdown_connections 2024-05-20T17:59:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.533+0000 7f4725691640 1 --2- 172.21.15.129:0/2501040236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720101060 0x7f4720101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.533+0000 7f4725691640 1 --2- 172.21.15.129:0/2501040236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f472010d560 0x7f472010f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.533+0000 7f4725691640 1 --2- 172.21.15.129:0/2501040236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4720108bb0 0x7f4720100b20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.533+0000 7f4725691640 1 -- 172.21.15.129:0/2501040236 >> 172.21.15.129:0/2501040236 conn(0x7f47200fc850 msgr2=0x7f47200feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.533+0000 7f4725691640 1 -- 172.21.15.129:0/2501040236 shutdown_connections 2024-05-20T17:59:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.533+0000 7f4725691640 1 -- 172.21.15.129:0/2501040236 wait complete. 2024-05-20T17:59:03.533 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f4725691640 1 Processor -- start 2024-05-20T17:59:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f4725691640 1 -- start start 2024-05-20T17:59:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f4725691640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4720101060 0x7f47201997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f4725691640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720108bb0 0x7f4720199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f4725691640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f472010d560 0x7f472019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f4725691640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4720112e70 con 0x7f4720101060 2024-05-20T17:59:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f4725691640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4720112cf0 con 0x7f4720108bb0 2024-05-20T17:59:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f4725691640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4720112ff0 con 0x7f472010d560 2024-05-20T17:59:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f471effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4720101060 0x7f47201997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f471effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4720101060 0x7f47201997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52022/0 (socket says 172.21.15.129:52022) 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f471effd640 1 -- 172.21.15.129:0/72721099 learned_addr learned my addr 172.21.15.129:0/72721099 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f471f7fe640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f472010d560 0x7f472019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.534+0000 7f47167fc640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720108bb0 0x7f4720199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.535+0000 7f471effd640 1 -- 172.21.15.129:0/72721099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f472010d560 msgr2=0x7f472019e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.535+0000 7f471effd640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f472010d560 0x7f472019e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.535+0000 7f471effd640 1 -- 172.21.15.129:0/72721099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720108bb0 msgr2=0x7f4720199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.535+0000 7f471effd640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720108bb0 0x7f4720199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.535+0000 7f471effd640 1 -- 172.21.15.129:0/72721099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f472019e650 con 0x7f4720101060 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.535+0000 7f471effd640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4720101060 0x7f47201997f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4710005940 tx=0x7f47100040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.535+0000 7f471cff9640 1 -- 172.21.15.129:0/72721099 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471003f030 con 0x7f4720101060 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.535+0000 7f4725691640 1 -- 172.21.15.129:0/72721099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f472019e8e0 con 0x7f4720101060 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.535+0000 7f471cff9640 1 -- 172.21.15.129:0/72721099 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47100044c0 con 0x7f4720101060 2024-05-20T17:59:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.535+0000 7f471cff9640 1 -- 172.21.15.129:0/72721099 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4710035070 con 0x7f4720101060 2024-05-20T17:59:03.537 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.535+0000 7f4725691640 1 -- 172.21.15.129:0/72721099 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f472019ee20 con 0x7f4720101060 2024-05-20T17:59:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.538+0000 7f471cff9640 1 -- 172.21.15.129:0/72721099 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f4710042030 con 0x7f4720101060 2024-05-20T17:59:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.538+0000 7f472c60f740 1 Processor -- start 2024-05-20T17:59:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.538+0000 7f472c60f740 1 -- start start 2024-05-20T17:59:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f472c60f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 0x558d9c8ff080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f472c60f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c8ff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f472c60f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c53f4d0 0x558d9c8ffb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f472c60f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558d9c864340 con 0x558d9c5372f0 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f472c60f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558d9c80c290 con 0x558d9c5601b0 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f472c60f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558d9c860300 con 0x558d9c53f4d0 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c8ff5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c8ff5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52032/0 (socket says 172.21.15.129:52032) 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471effd640 1 -- 172.21.15.129:0/274213420 learned_addr learned my addr 172.21.15.129:0/274213420 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f47167fc640 1 --2- 172.21.15.129:0/274213420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c53f4d0 0x558d9c8ffb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471f7fe640 1 --2- 172.21.15.129:0/274213420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 0x558d9c8ff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471cff9640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f470c0a2e10 0x7f470c0a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471effd640 1 -- 172.21.15.129:0/274213420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c53f4d0 msgr2=0x558d9c8ffb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471effd640 1 --2- 172.21.15.129:0/274213420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c53f4d0 0x558d9c8ffb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471effd640 1 -- 172.21.15.129:0/274213420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 msgr2=0x558d9c8ff080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471effd640 1 --2- 172.21.15.129:0/274213420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 0x558d9c8ff080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471effd640 1 -- 172.21.15.129:0/274213420 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d9c8308e0 con 0x558d9c5372f0 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f47167fc640 1 --2- 172.21.15.129:0/274213420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c53f4d0 0x558d9c8ffb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471f7fe640 1 --2- 172.21.15.129:0/274213420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 0x558d9c8ff080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471f7fe640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f470c0a2e10 0x7f470c0a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471cff9640 1 -- 172.21.15.129:0/72721099 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f47100efa80 con 0x7f4720101060 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.539+0000 7f471effd640 1 --2- 172.21.15.129:0/274213420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c8ff5c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f47100ed170 tx=0x7f47100b12c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e00f861ab8c612 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.540+0000 7f47157fa640 1 -- 172.21.15.129:0/274213420 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47100f7040 con 0x558d9c5372f0 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.540+0000 7f47157fa640 1 -- 172.21.15.129:0/274213420 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f471003b460 con 0x558d9c5372f0 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.540+0000 7f47157fa640 1 -- 172.21.15.129:0/274213420 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47100bbe30 con 0x558d9c5372f0 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.540+0000 7f471f7fe640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f470c0a2e10 0x7f470c0a5320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x558d9c8fc1f0 tx=0x7f4718004a40 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.541+0000 7f472c60f740 1 -- 172.21.15.129:0/274213420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 msgr2=0x558d9c8ff5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.541+0000 7f472c60f740 1 --2- 172.21.15.129:0/274213420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c8ff5c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f47100ed170 tx=0x7f47100b12c0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.541+0000 7f472c60f740 1 -- 172.21.15.129:0/274213420 shutdown_connections 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.541+0000 7f472c60f740 1 --2- 172.21.15.129:0/274213420 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c53f4d0 0x558d9c8ffb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.541+0000 7f472c60f740 1 --2- 172.21.15.129:0/274213420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c8ff5c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.541+0000 7f472c60f740 1 --2- 172.21.15.129:0/274213420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 0x558d9c8ff080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.541+0000 7f472c60f740 1 -- 172.21.15.129:0/274213420 >> 172.21.15.129:0/274213420 conn(0x558d9c554300 msgr2=0x558d9c8fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.541+0000 7f472c60f740 1 -- 172.21.15.129:0/274213420 shutdown_connections 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.541+0000 7f472c60f740 1 -- 172.21.15.129:0/274213420 wait complete. 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.541+0000 7f472c60f740 1 -- start start 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.541+0000 7f472c60f740 1 Processor -- start 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f472c60f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c90b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f472c60f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c53f4d0 0x558d9c90b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f472c60f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 0x558d9c909580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f472c60f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558d9c90bf70 con 0x558d9c5372f0 2024-05-20T17:59:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f472c60f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558d9c81f790 con 0x558d9c5601b0 2024-05-20T17:59:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f472c60f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558d9c90c0f0 con 0x558d9c53f4d0 2024-05-20T17:59:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f472c60f740 10 client.0 _ll_register_callbacks cb 0x558d9c85aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f471effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c90b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f471effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c90b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52038/0 (socket says 172.21.15.129:52038) 2024-05-20T17:59:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f471effd640 1 -- 172.21.15.129:0/2127785769 learned_addr learned my addr 172.21.15.129:0/2127785769 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f47167fc640 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c53f4d0 0x558d9c90b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f471f7fe640 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 0x558d9c909580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f471effd640 1 -- 172.21.15.129:0/2127785769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c53f4d0 msgr2=0x558d9c90b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:03.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f471effd640 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c53f4d0 0x558d9c90b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f471effd640 1 -- 172.21.15.129:0/2127785769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 msgr2=0x558d9c909580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f471effd640 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 0x558d9c909580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f471effd640 1 -- 172.21.15.129:0/2127785769 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d9c578210 con 0x558d9c5372f0 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f471f7fe640 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 0x558d9c909580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f471effd640 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c90b090 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f47100ed120 tx=0x7f47100ed000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd58d362700852b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f47067fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f47067fc640 1 -- 172.21.15.129:0/2127785769 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4710049040 con 0x558d9c5372f0 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f47067fc640 1 -- 172.21.15.129:0/2127785769 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f47100ba130 con 0x558d9c5372f0 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f472c60f740 1 -- 172.21.15.129:0/2127785769 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558d9c871a70 con 0x558d9c5372f0 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f47067fc640 1 -- 172.21.15.129:0/2127785769 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f47100bb790 con 0x558d9c5372f0 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f472c60f740 10 client.15486 fetch_fsmap learned FSMap version 20 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.542+0000 7f472c60f740 1 -- 172.21.15.129:0/2127785769 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558d9c644620 con 0x558d9c5372f0 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.543+0000 7f47067fc640 1 -- 172.21.15.129:0/2127785769 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558d9c871a70 con 0x558d9c5372f0 2024-05-20T17:59:03.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.543+0000 7f47067fc640 1 -- 172.21.15.129:0/2127785769 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f47100f30d0 con 0x558d9c5372f0 2024-05-20T17:59:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.543+0000 7f472c60f740 10 client.15486 fetch_fsmap finished waiting for FSMap version 20 2024-05-20T17:59:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.543+0000 7f472c60f740 10 client.15486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.543+0000 7f472c60f740 20 client.15486 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.543+0000 7f472c60f740 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558d9c51ed80 0x558d9c91b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.543+0000 7f472c60f740 4 client.15486 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.543+0000 7f472c60f740 1 -- 172.21.15.129:0/2127785769 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d9c86bdc0 con 0x558d9c51ed80 2024-05-20T17:59:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.543+0000 7f47167fc640 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558d9c51ed80 0x558d9c91b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.544+0000 7f47167fc640 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558d9c51ed80 0x558d9c91b450 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84e8ab00fac5452a server_cookie=6bb97d7f8ed83c19 in_seq=0 out_seq=0 2024-05-20T17:59:03.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.544+0000 7f47067fc640 10 client.15486 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:03.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.544+0000 7f47067fc640 1 -- 172.21.15.129:0/2127785769 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558d9c86bdc0 con 0x558d9c51ed80 2024-05-20T17:59:03.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.544+0000 7f47067fc640 10 client.15486 handle_command_reply: tid=0 2024-05-20T17:59:03.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.544+0000 7f472c60f740 1 client.15486 shutdown 2024-05-20T17:59:03.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f472c60f740 1 -- 172.21.15.129:0/2127785769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 msgr2=0x558d9c90b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:03.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f472c60f740 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c90b090 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f47100ed120 tx=0x7f47100ed000 comp rx=0 tx=0).stop 2024-05-20T17:59:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f472c60f740 1 -- 172.21.15.129:0/2127785769 shutdown_connections 2024-05-20T17:59:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f472c60f740 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558d9c51ed80 0x558d9c91b450 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f472c60f740 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5601b0 0x558d9c909580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f47067fc640 0 client.15486 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f472c60f740 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c53f4d0 0x558d9c90b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f472c60f740 1 --2- 172.21.15.129:0/2127785769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c90b090 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f472c60f740 1 -- 172.21.15.129:0/2127785769 >> 172.21.15.129:0/2127785769 conn(0x558d9c554300 msgr2=0x558d9c8fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f472c60f740 1 -- 172.21.15.129:0/2127785769 shutdown_connections 2024-05-20T17:59:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f472c60f740 1 -- 172.21.15.129:0/2127785769 wait complete. 2024-05-20T17:59:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.545+0000 7f472c60f740 20 client.15486 trim_cache size 0 max 16384 2024-05-20T17:59:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f472c60f740 1 Processor -- start 2024-05-20T17:59:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f472c60f740 1 -- start start 2024-05-20T17:59:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f472c60f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c51ed80 0x558d9c90b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f472c60f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c911230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f472c60f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c53f4d0 0x558d9c911770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f472c60f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47201acab0 con 0x558d9c5372f0 2024-05-20T17:59:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f472c60f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47201af330 con 0x558d9c53f4d0 2024-05-20T17:59:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f472c60f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f47201ad120 con 0x558d9c51ed80 2024-05-20T17:59:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f47167fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c911230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f47167fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c911230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52040/0 (socket says 172.21.15.129:52040) 2024-05-20T17:59:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f47167fc640 1 -- 172.21.15.129:0/2701565122 learned_addr learned my addr 172.21.15.129:0/2701565122 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.560+0000 7f471f7fe640 1 --2- 172.21.15.129:0/2701565122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c53f4d0 0x558d9c911770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.561+0000 7f471effd640 1 --2- 172.21.15.129:0/2701565122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c51ed80 0x558d9c90b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.561+0000 7f47167fc640 1 -- 172.21.15.129:0/2701565122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c51ed80 msgr2=0x558d9c90b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.561+0000 7f47167fc640 1 --2- 172.21.15.129:0/2701565122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c51ed80 0x558d9c90b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.561+0000 7f47167fc640 1 -- 172.21.15.129:0/2701565122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c53f4d0 msgr2=0x558d9c911770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.561+0000 7f47167fc640 1 --2- 172.21.15.129:0/2701565122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c53f4d0 0x558d9c911770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.561+0000 7f47167fc640 1 -- 172.21.15.129:0/2701565122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d9c8fc420 con 0x558d9c5372f0 2024-05-20T17:59:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.561+0000 7f47167fc640 1 --2- 172.21.15.129:0/2701565122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c911230 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f47200fe2b0 tx=0x7f4708005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149c740084962070 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.561+0000 7f4714ff9640 1 -- 172.21.15.129:0/2701565122 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708004030 con 0x558d9c5372f0 2024-05-20T17:59:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.561+0000 7f4714ff9640 1 -- 172.21.15.129:0/2701565122 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4708025020 con 0x558d9c5372f0 2024-05-20T17:59:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.561+0000 7f4714ff9640 1 -- 172.21.15.129:0/2701565122 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708021340 con 0x558d9c5372f0 2024-05-20T17:59:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.562+0000 7f472c60f740 1 -- 172.21.15.129:0/2701565122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 msgr2=0x558d9c911230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.562+0000 7f472c60f740 1 --2- 172.21.15.129:0/2701565122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c911230 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f47200fe2b0 tx=0x7f4708005f90 comp rx=0 tx=0).stop 2024-05-20T17:59:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.562+0000 7f472c60f740 1 -- 172.21.15.129:0/2701565122 shutdown_connections 2024-05-20T17:59:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.562+0000 7f472c60f740 1 --2- 172.21.15.129:0/2701565122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c53f4d0 0x558d9c911770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.564 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.562+0000 7f472c60f740 1 --2- 172.21.15.129:0/2701565122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c5372f0 0x558d9c911230 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.562+0000 7f472c60f740 1 --2- 172.21.15.129:0/2701565122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c51ed80 0x558d9c90b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.562+0000 7f472c60f740 1 -- 172.21.15.129:0/2701565122 >> 172.21.15.129:0/2701565122 conn(0x558d9c554300 msgr2=0x558d9c90aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.562+0000 7f472c60f740 1 -- 172.21.15.129:0/2701565122 shutdown_connections 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.562+0000 7f472c60f740 1 -- 172.21.15.129:0/2701565122 wait complete. 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.562+0000 7f472c60f740 1 -- start start 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.562+0000 7f472c60f740 1 Processor -- start 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f472c60f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c51ed80 0x558d9c97ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f472c60f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5372f0 0x558d9c97b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f472c60f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c53f4d0 0x558d9c9107f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f472c60f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47201ad7f0 con 0x558d9c53f4d0 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f472c60f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47201aec60 con 0x558d9c5372f0 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f472c60f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f47201adec0 con 0x558d9c51ed80 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f472c60f740 10 client.0 _ll_register_callbacks cb 0x7f4718009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:03.565 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f471f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c53f4d0 0x558d9c9107f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f471f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c53f4d0 0x558d9c9107f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52044/0 (socket says 172.21.15.129:52044) 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f471f7fe640 1 -- 172.21.15.129:0/1772620218 learned_addr learned my addr 172.21.15.129:0/1772620218 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f471effd640 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c51ed80 0x558d9c97ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f47167fc640 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5372f0 0x558d9c97b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f471f7fe640 1 -- 172.21.15.129:0/1772620218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c51ed80 msgr2=0x558d9c97ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f471f7fe640 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c51ed80 0x558d9c97ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f471f7fe640 1 -- 172.21.15.129:0/1772620218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5372f0 msgr2=0x558d9c97b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f471f7fe640 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5372f0 0x558d9c97b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f471f7fe640 1 -- 172.21.15.129:0/1772620218 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d9c910de0 con 0x558d9c53f4d0 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f47167fc640 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5372f0 0x558d9c97b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f471f7fe640 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c53f4d0 0x558d9c9107f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4718020a40 tx=0x7f4718004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9b4ce8a743bf9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f47067fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:03.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f47067fc640 1 -- 172.21.15.129:0/1772620218 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471802f060 con 0x558d9c53f4d0 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f47067fc640 1 -- 172.21.15.129:0/1772620218 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8506+0+0 (secure 0 0 0) 0x7f471801ebf0 con 0x558d9c53f4d0 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.563+0000 7f472c60f740 1 -- 172.21.15.129:0/1772620218 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4708025020 con 0x558d9c53f4d0 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f47067fc640 1 -- 172.21.15.129:0/1772620218 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f4718020320 con 0x558d9c53f4d0 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f472c60f740 10 client.15492 fetch_fsmap learned FSMap version 20 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f472c60f740 1 -- 172.21.15.129:0/1772620218 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x558d9c9832f0 con 0x558d9c53f4d0 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f47067fc640 1 -- 172.21.15.129:0/1772620218 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4708025020 con 0x558d9c53f4d0 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f47067fc640 1 -- 172.21.15.129:0/1772620218 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 20) ==== 8322+0+0 (secure 0 0 0) 0x7f4718029030 con 0x558d9c53f4d0 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f472c60f740 10 client.15492 fetch_fsmap finished waiting for FSMap version 20 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f472c60f740 10 client.15492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f472c60f740 20 client.15492 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f472c60f740 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558d9c5601b0 0x558d9c985ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f472c60f740 4 client.15492 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T17:59:03.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f472c60f740 1 -- 172.21.15.129:0/1772620218 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558d9c7e98e0 con 0x558d9c5601b0 2024-05-20T17:59:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.564+0000 7f471effd640 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558d9c5601b0 0x558d9c985ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.565+0000 7f47067fc640 10 client.15492 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:03.568 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:03.565+0000 7f471effd640 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558d9c5601b0 0x558d9c985ae0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a4d5832ae7efaf8 server_cookie=39e26dfb99bc7d85 in_seq=0 out_seq=0 2024-05-20T17:59:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f47067fc640 1 -- 172.21.15.129:0/1772620218 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558d9c7e98e0 con 0x558d9c5601b0 2024-05-20T17:59:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f47067fc640 10 client.15492 handle_command_reply: tid=0 2024-05-20T17:59:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f472c60f740 1 client.15492 shutdown 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f472c60f740 1 -- 172.21.15.129:0/1772620218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c53f4d0 msgr2=0x558d9c9107f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f472c60f740 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c53f4d0 0x558d9c9107f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4718020a40 tx=0x7f4718004ab0 comp rx=0 tx=0).stop 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f472c60f740 1 -- 172.21.15.129:0/1772620218 shutdown_connections 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f472c60f740 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558d9c5601b0 0x558d9c985ae0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f472c60f740 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d9c5372f0 0x558d9c97b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f472c60f740 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558d9c53f4d0 0x558d9c9107f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f472c60f740 1 --2- 172.21.15.129:0/1772620218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558d9c51ed80 0x558d9c97ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f47067fc640 0 client.15492 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f472c60f740 1 -- 172.21.15.129:0/1772620218 >> 172.21.15.129:0/1772620218 conn(0x558d9c554300 msgr2=0x558d9c908e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f472c60f740 1 -- 172.21.15.129:0/1772620218 shutdown_connections 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.174+0000 7f472c60f740 1 -- 172.21.15.129:0/1772620218 wait complete. 2024-05-20T17:59:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.175+0000 7f472c60f740 20 client.15492 trim_cache size 0 max 16384 2024-05-20T17:59:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T17:59:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T17:59:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T17:59:05.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T17:59:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.175+0000 7f4725691640 1 -- 172.21.15.129:0/72721099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f470c0a2e10 msgr2=0x7f470c0a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.175+0000 7f4725691640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f470c0a2e10 0x7f470c0a5320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x558d9c8fc1f0 tx=0x7f4718004a40 comp rx=0 tx=0).stop 2024-05-20T17:59:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.175+0000 7f4725691640 1 -- 172.21.15.129:0/72721099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4720101060 msgr2=0x7f47201997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.175+0000 7f4725691640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4720101060 0x7f47201997f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4710005940 tx=0x7f47100040c0 comp rx=0 tx=0).stop 2024-05-20T17:59:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.175+0000 7f4725691640 1 -- 172.21.15.129:0/72721099 shutdown_connections 2024-05-20T17:59:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.175+0000 7f4725691640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f470c0a2e10 0x7f470c0a5320 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.175+0000 7f4725691640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720108bb0 0x7f4720199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.175+0000 7f4725691640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f472010d560 0x7f472019e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.175+0000 7f4725691640 1 --2- 172.21.15.129:0/72721099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4720101060 0x7f47201997f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.175+0000 7f4725691640 1 -- 172.21.15.129:0/72721099 >> 172.21.15.129:0/72721099 conn(0x7f47200fc850 msgr2=0x7f47200fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.176+0000 7f4725691640 1 -- 172.21.15.129:0/72721099 shutdown_connections 2024-05-20T17:59:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:05.176+0000 7f4725691640 1 -- 172.21.15.129:0/72721099 wait complete. 2024-05-20T17:59:05.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:05 smithi154 ceph-mon[36482]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-20T17:59:05.393 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: *** Caught signal (Segmentation fault) ** 2024-05-20T17:59:05.393 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: in thread 7f4ba43c6700 thread_name:md_log_replay 2024-05-20T17:59:05.393 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T17:59:05.393 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bb5788d20] 2024-05-20T17:59:05.393 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x1a) [0x5591ab460b7e] 2024-05-20T17:59:05.393 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x5591ab46102e] 2024-05-20T17:59:05.393 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x5591ab45b6a5] 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x5591ab11d3c3] 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 6: (MDLog::_replay_thread()+0x1f38) [0x5591ab4c169c] 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 7: (MDLog::ReplayThread::entry()+0x11) [0x5591ab4caae7] 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 8: (Thread::entry_wrapper()+0x43) [0x7f4bb6bd96f1] 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 9: (Thread::_entry_func(void*)+0xd) [0x7f4bb6bd970d] 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4bb577e1ca] 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 11: clone() 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 2024-05-20T17:59:05.207+0000 7f4ba43c6700 -1 *** Caught signal (Segmentation fault) ** 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: in thread 7f4ba43c6700 thread_name:md_log_replay 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bb5788d20] 2024-05-20T17:59:05.394 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x1a) [0x5591ab460b7e] 2024-05-20T17:59:05.395 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x5591ab46102e] 2024-05-20T17:59:05.395 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x5591ab45b6a5] 2024-05-20T17:59:05.395 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x5591ab11d3c3] 2024-05-20T17:59:05.395 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 6: (MDLog::_replay_thread()+0x1f38) [0x5591ab4c169c] 2024-05-20T17:59:05.395 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 7: (MDLog::ReplayThread::entry()+0x11) [0x5591ab4caae7] 2024-05-20T17:59:05.395 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 8: (Thread::entry_wrapper()+0x43) [0x7f4bb6bd96f1] 2024-05-20T17:59:05.395 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 9: (Thread::_entry_func(void*)+0xd) [0x7f4bb6bd970d] 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4bb577e1ca] 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 11: clone() 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 0> 2024-05-20T17:59:05.207+0000 7f4ba43c6700 -1 *** Caught signal (Segmentation fault) ** 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: in thread 7f4ba43c6700 thread_name:md_log_replay 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bb5788d20] 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x1a) [0x5591ab460b7e] 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x5591ab46102e] 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x5591ab45b6a5] 2024-05-20T17:59:05.396 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x5591ab11d3c3] 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 6: (MDLog::_replay_thread()+0x1f38) [0x5591ab4c169c] 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 7: (MDLog::ReplayThread::entry()+0x11) [0x5591ab4caae7] 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 8: (Thread::entry_wrapper()+0x43) [0x7f4bb6bd96f1] 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 9: (Thread::_entry_func(void*)+0xd) [0x7f4bb6bd970d] 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4bb577e1ca] 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 11: clone() 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 0> 2024-05-20T17:59:05.207+0000 7f4ba43c6700 -1 *** Caught signal (Segmentation fault) ** 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: in thread 7f4ba43c6700 thread_name:md_log_replay 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T17:59:05.397 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bb5788d20] 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x1a) [0x5591ab460b7e] 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x5591ab46102e] 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x5591ab45b6a5] 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x5591ab11d3c3] 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 6: (MDLog::_replay_thread()+0x1f38) [0x5591ab4c169c] 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 7: (MDLog::ReplayThread::entry()+0x11) [0x5591ab4caae7] 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 8: (Thread::entry_wrapper()+0x43) [0x7f4bb6bd96f1] 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 9: (Thread::_entry_func(void*)+0xd) [0x7f4bb6bd970d] 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4bb577e1ca] 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 11: clone() 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[66379]: 2024-05-20T17:59:05.398 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:05 smithi154 podman[69613]: 2024-05-20 17:59:05.266641233 +0000 UTC m=+0.021156176 container died 7367402f2476aba1761901d4110bb6d8d52aae1a370250c954f6d90cae2dd05d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-20T17:59:05.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:05 smithi129 ceph-mon[32029]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-20T17:59:05.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:05 smithi204 ceph-mon[37076]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-20T17:59:06.315 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:06 smithi154 podman[69613]: 2024-05-20 17:59:06.067735346 +0000 UTC m=+0.822250291 container remove 7367402f2476aba1761901d4110bb6d8d52aae1a370250c954f6d90cae2dd05d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-20T17:59:06.316 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:06 smithi154 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.h.service: Main process exited, code=exited, status=139/n/a 2024-05-20T17:59:06.642 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:06 smithi154 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-05-20T17:59:07.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:07 smithi154 ceph-mon[36482]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-20T17:59:07.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:07 smithi129 ceph-mon[32029]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-20T17:59:07.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:07 smithi204 ceph-mon[37076]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-05-20T17:59:09.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:09 smithi154 ceph-mon[36482]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-05-20T17:59:09.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:09 smithi129 ceph-mon[32029]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-05-20T17:59:09.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:09 smithi204 ceph-mon[37076]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-05-20T17:59:11.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:11 smithi154 ceph-mon[36482]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-05-20T17:59:11.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:11 smithi129 ceph-mon[32029]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-05-20T17:59:11.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:11 smithi204 ceph-mon[37076]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-05-20T17:59:13.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:13 smithi154 ceph-mon[36482]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-05-20T17:59:13.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:13 smithi129 ceph-mon[32029]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-05-20T17:59:13.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:13 smithi204 ceph-mon[37076]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-05-20T17:59:15.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:15 smithi154 ceph-mon[36482]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 20 op/s 2024-05-20T17:59:15.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:15 smithi129 ceph-mon[32029]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 20 op/s 2024-05-20T17:59:15.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:15 smithi204 ceph-mon[37076]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 20 op/s 2024-05-20T17:59:16.658 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:16 smithi154 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-05-20T17:59:16.658 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:16 smithi154 systemd[1]: Stopped Ceph mds.h for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:59:16.658 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:16 smithi154 systemd[1]: Starting Ceph mds.h for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T17:59:16.658 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:16 smithi154 podman[69708]: 2024-05-20 17:59:16.531756058 +0000 UTC m=+0.017860152 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T17:59:16.658 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:16 smithi154 podman[69708]: 2024-05-20 17:59:16.660283738 +0000 UTC m=+0.146387826 container create c0c469bc41bf45e7f69bf0dbaa64d117d9ddfe94cf0da6ae36e8e5fc4ea0bc08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, ceph=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T17:59:16.938 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:16 smithi154 podman[69708]: 2024-05-20 17:59:16.905040144 +0000 UTC m=+0.391144216 container init c0c469bc41bf45e7f69bf0dbaa64d117d9ddfe94cf0da6ae36e8e5fc4ea0bc08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-05-20T17:59:16.939 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:16 smithi154 podman[69708]: 2024-05-20 17:59:16.907510462 +0000 UTC m=+0.393614543 container start c0c469bc41bf45e7f69bf0dbaa64d117d9ddfe94cf0da6ae36e8e5fc4ea0bc08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True) 2024-05-20T17:59:17.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:17 smithi154 ceph-mon[36482]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 20 op/s 2024-05-20T17:59:17.276 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:17 smithi154 ceph-mon[36482]: daemon mds.h restarted 2024-05-20T17:59:17.276 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:16 smithi154 ceph-mds[69727]: -- 172.21.15.154:0/1509773093 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55729f47fa40 con 0x55729f4f7600 2024-05-20T17:59:17.276 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:16 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: starting mds.h at 2024-05-20T17:59:17.276 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:17 smithi154 bash[69708]: c0c469bc41bf45e7f69bf0dbaa64d117d9ddfe94cf0da6ae36e8e5fc4ea0bc08 2024-05-20T17:59:17.276 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 17:59:17 smithi154 systemd[1]: Started Ceph mds.h for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T17:59:17.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:17 smithi129 ceph-mon[32029]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 20 op/s 2024-05-20T17:59:17.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:17 smithi129 ceph-mon[32029]: daemon mds.h restarted 2024-05-20T17:59:17.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:17 smithi204 ceph-mon[37076]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 KiB/s wr, 20 op/s 2024-05-20T17:59:17.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:17 smithi204 ceph-mon[37076]: daemon mds.h restarted 2024-05-20T17:59:18.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T17:59:18.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T17:59:18.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.198+0000 7f8534e8b640 1 -- 172.21.15.129:0/2353958690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853010c350 msgr2=0x7f853010e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.198+0000 7f8534e8b640 1 --2- 172.21.15.129:0/2353958690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853010c350 0x7f853010e770 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8528009ca0 tx=0x7f852802a8c0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.198+0000 7f8534e8b640 1 -- 172.21.15.129:0/2353958690 shutdown_connections 2024-05-20T17:59:18.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.198+0000 7f8534e8b640 1 --2- 172.21.15.129:0/2353958690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853010c350 0x7f853010e770 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.198+0000 7f8534e8b640 1 --2- 172.21.15.129:0/2353958690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8530101ac0 0x7f8530101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.198+0000 7f8534e8b640 1 --2- 172.21.15.129:0/2353958690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85301079f0 0x7f8530101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.198+0000 7f8534e8b640 1 -- 172.21.15.129:0/2353958690 >> 172.21.15.129:0/2353958690 conn(0x7f8530077130 msgr2=0x7f8530077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.198+0000 7f8534e8b640 1 -- 172.21.15.129:0/2353958690 shutdown_connections 2024-05-20T17:59:18.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.198+0000 7f8534e8b640 1 -- 172.21.15.129:0/2353958690 wait complete. 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f8534e8b640 1 Processor -- start 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f8534e8b640 1 -- start start 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f8534e8b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101ac0 0x7f85301996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f8534e8b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85301079f0 0x7f8530199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f8534e8b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853010c350 0x7f853019df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f8534e8b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8530111d00 con 0x7f853010c350 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f8534e8b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8530111b80 con 0x7f8530101ac0 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f8534e8b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8530111e80 con 0x7f85301079f0 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f852ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853010c350 0x7f853019df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f852ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853010c350 0x7f853019df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54040/0 (socket says 172.21.15.129:54040) 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f852ed9d640 1 -- 172.21.15.129:0/2318401442 learned_addr learned my addr 172.21.15.129:0/2318401442 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.199+0000 7f852e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101ac0 0x7f85301996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f8527fff640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85301079f0 0x7f8530199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f852ed9d640 1 -- 172.21.15.129:0/2318401442 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85301079f0 msgr2=0x7f8530199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f852ed9d640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85301079f0 0x7f8530199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f852ed9d640 1 -- 172.21.15.129:0/2318401442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101ac0 msgr2=0x7f85301996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f852ed9d640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101ac0 0x7f85301996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f852ed9d640 1 -- 172.21.15.129:0/2318401442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853019e550 con 0x7f853010c350 2024-05-20T17:59:18.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f852e59c640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101ac0 0x7f85301996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:18.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f852ed9d640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853010c350 0x7f853019df10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8528031a20 tx=0x7f8528031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f85277fe640 1 -- 172.21.15.129:0/2318401442 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8528003c30 con 0x7f853010c350 2024-05-20T17:59:18.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f85277fe640 1 -- 172.21.15.129:0/2318401442 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8528003dd0 con 0x7f853010c350 2024-05-20T17:59:18.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f8534e8b640 1 -- 172.21.15.129:0/2318401442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853019e7e0 con 0x7f853010c350 2024-05-20T17:59:18.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f85277fe640 1 -- 172.21.15.129:0/2318401442 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852802fc40 con 0x7f853010c350 2024-05-20T17:59:18.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.200+0000 7f8534e8b640 1 -- 172.21.15.129:0/2318401442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853019ed30 con 0x7f853010c350 2024-05-20T17:59:18.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.203+0000 7f85277fe640 1 -- 172.21.15.129:0/2318401442 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f8528004050 con 0x7f853010c350 2024-05-20T17:59:18.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.203+0000 7f853be12740 1 Processor -- start 2024-05-20T17:59:18.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.203+0000 7f853be12740 1 -- start start 2024-05-20T17:59:18.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f853be12740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31bc390 0x5593e355ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f853be12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31934d0 0x5593e355b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f853be12740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e355b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f853be12740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5593e34c0d00 con 0x5593e319b6b0 2024-05-20T17:59:18.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f853be12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5593e34684c0 con 0x5593e31934d0 2024-05-20T17:59:18.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f853be12740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5593e34bccc0 con 0x5593e31bc390 2024-05-20T17:59:18.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f8527fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e355b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f852e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31934d0 0x5593e355b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f8527fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e355b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54054/0 (socket says 172.21.15.129:54054) 2024-05-20T17:59:18.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f8527fff640 1 -- 172.21.15.129:0/1986016614 learned_addr learned my addr 172.21.15.129:0/1986016614 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f85277fe640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f851c0a2e20 0x7f851c0a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f8527fff640 1 --2- 172.21.15.129:0/1986016614 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31bc390 0x5593e355ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f852ed9d640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f851c0a2e20 0x7f851c0a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f852e59c640 1 -- 172.21.15.129:0/1986016614 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31bc390 msgr2=0x5593e355ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f852e59c640 1 --2- 172.21.15.129:0/1986016614 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31bc390 0x5593e355ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f852e59c640 1 -- 172.21.15.129:0/1986016614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 msgr2=0x5593e355b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f852e59c640 1 --2- 172.21.15.129:0/1986016614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e355b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f852e59c640 1 -- 172.21.15.129:0/1986016614 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593e329e7d0 con 0x5593e31934d0 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f85277fe640 1 -- 172.21.15.129:0/2318401442 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f85280f0820 con 0x7f853010c350 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.204+0000 7f852e59c640 1 --2- 172.21.15.129:0/1986016614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31934d0 0x5593e355b270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8520009f20 tx=0x7f85200118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=703473b946631b76 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.205+0000 7f8524ff9640 1 -- 172.21.15.129:0/1986016614 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520004020 con 0x5593e31934d0 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.205+0000 7f8524ff9640 1 -- 172.21.15.129:0/1986016614 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8520005500 con 0x5593e31934d0 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.205+0000 7f8524ff9640 1 -- 172.21.15.129:0/1986016614 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520004ca0 con 0x5593e31934d0 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.206+0000 7f853be12740 1 -- 172.21.15.129:0/1986016614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31934d0 msgr2=0x5593e355b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.206+0000 7f852ed9d640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f851c0a2e20 0x7f851c0a5330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f85280edf70 tx=0x7f852802a8c0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.206+0000 7f853be12740 1 --2- 172.21.15.129:0/1986016614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31934d0 0x5593e355b270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8520009f20 tx=0x7f85200118d0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.206+0000 7f853be12740 1 -- 172.21.15.129:0/1986016614 shutdown_connections 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.206+0000 7f853be12740 1 --2- 172.21.15.129:0/1986016614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31934d0 0x5593e355b270 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.206+0000 7f853be12740 1 --2- 172.21.15.129:0/1986016614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e355b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.206+0000 7f853be12740 1 --2- 172.21.15.129:0/1986016614 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31bc390 0x5593e355ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.206+0000 7f853be12740 1 -- 172.21.15.129:0/1986016614 >> 172.21.15.129:0/1986016614 conn(0x5593e31b04e0 msgr2=0x5593e35574e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.206+0000 7f853be12740 1 -- 172.21.15.129:0/1986016614 shutdown_connections 2024-05-20T17:59:18.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.206+0000 7f853be12740 1 -- 172.21.15.129:0/1986016614 wait complete. 2024-05-20T17:59:18.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f853be12740 1 -- start start 2024-05-20T17:59:18.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f853be12740 1 Processor -- start 2024-05-20T17:59:18.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f853be12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31934d0 0x5593e3566db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f853be12740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e35672f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f853be12740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31bc390 0x5593e35653a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f853be12740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5593e3567fc0 con 0x5593e319b6b0 2024-05-20T17:59:18.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f853be12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5593e347bb90 con 0x5593e31934d0 2024-05-20T17:59:18.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f853be12740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5593e3568140 con 0x5593e31bc390 2024-05-20T17:59:18.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f853be12740 10 client.0 _ll_register_callbacks cb 0x5593e34b7da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:18.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f852e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e35672f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f852e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e35672f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54062/0 (socket says 172.21.15.129:54062) 2024-05-20T17:59:18.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f852e59c640 1 -- 172.21.15.129:0/2661468769 learned_addr learned my addr 172.21.15.129:0/2661468769 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.207+0000 7f8527fff640 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31934d0 0x5593e3566db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f8527fff640 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31bc390 0x5593e35653a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f852e59c640 1 -- 172.21.15.129:0/2661468769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31bc390 msgr2=0x5593e35653a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f852e59c640 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31bc390 0x5593e35653a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f852e59c640 1 -- 172.21.15.129:0/2661468769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31934d0 msgr2=0x5593e3566db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f852e59c640 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31934d0 0x5593e3566db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f852e59c640 1 -- 172.21.15.129:0/2661468769 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593e31d43f0 con 0x5593e319b6b0 2024-05-20T17:59:18.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f8527fff640 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31bc390 0x5593e35653a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:18.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f852e59c640 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e35672f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8520000c00 tx=0x7f852000fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3281fe6e1ec68ce7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f8515ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:18.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f8515ffb640 1 -- 172.21.15.129:0/2661468769 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852001d040 con 0x5593e319b6b0 2024-05-20T17:59:18.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f8515ffb640 1 -- 172.21.15.129:0/2661468769 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f852000fdd0 con 0x5593e319b6b0 2024-05-20T17:59:18.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f853be12740 1 -- 172.21.15.129:0/2661468769 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8520005500 con 0x5593e319b6b0 2024-05-20T17:59:18.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f8515ffb640 1 -- 172.21.15.129:0/2661468769 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8520032050 con 0x5593e319b6b0 2024-05-20T17:59:18.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f853be12740 10 client.15510 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:18.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f853be12740 1 -- 172.21.15.129:0/2661468769 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5593e31940f0 con 0x5593e319b6b0 2024-05-20T17:59:18.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.208+0000 7f8515ffb640 1 -- 172.21.15.129:0/2661468769 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8520005500 con 0x5593e319b6b0 2024-05-20T17:59:18.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.209+0000 7f8515ffb640 1 -- 172.21.15.129:0/2661468769 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f8520024c60 con 0x5593e319b6b0 2024-05-20T17:59:18.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.209+0000 7f853be12740 10 client.15510 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:18.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.209+0000 7f853be12740 10 client.15510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:18.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.209+0000 7f853be12740 20 client.15510 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:18.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.209+0000 7f853be12740 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593e317af60 0x5593e3577b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.209+0000 7f853be12740 4 client.15510 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:18.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.209+0000 7f853be12740 1 -- 172.21.15.129:0/2661468769 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593e34ce430 con 0x5593e317af60 2024-05-20T17:59:18.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.209+0000 7f8527fff640 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593e317af60 0x5593e3577b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.209+0000 7f8527fff640 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593e317af60 0x5593e3577b50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=498977e71a192fcd server_cookie=5e6d19d39fc25abd in_seq=0 out_seq=0 2024-05-20T17:59:18.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.209+0000 7f8515ffb640 10 client.15510 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f8515ffb640 1 -- 172.21.15.129:0/2661468769 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5593e34ce430 con 0x5593e317af60 2024-05-20T17:59:18.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f8515ffb640 10 client.15510 handle_command_reply: tid=0 2024-05-20T17:59:18.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 1 client.15510 shutdown 2024-05-20T17:59:18.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 1 -- 172.21.15.129:0/2661468769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 msgr2=0x5593e35672f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e35672f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8520000c00 tx=0x7f852000fbe0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 1 -- 172.21.15.129:0/2661468769 shutdown_connections 2024-05-20T17:59:18.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593e317af60 0x5593e3577b50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31bc390 0x5593e35653a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e35672f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 1 --2- 172.21.15.129:0/2661468769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31934d0 0x5593e3566db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f8515ffb640 0 client.15510 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 1 -- 172.21.15.129:0/2661468769 >> 172.21.15.129:0/2661468769 conn(0x5593e31b04e0 msgr2=0x5593e355ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 1 -- 172.21.15.129:0/2661468769 shutdown_connections 2024-05-20T17:59:18.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 1 -- 172.21.15.129:0/2661468769 wait complete. 2024-05-20T17:59:18.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.210+0000 7f853be12740 20 client.15510 trim_cache size 0 max 16384 2024-05-20T17:59:18.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f853be12740 1 Processor -- start 2024-05-20T17:59:18.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f853be12740 1 -- start start 2024-05-20T17:59:18.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f853be12740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31934d0 0x5593e3557510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f853be12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e319b6b0 0x5593e35562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f853be12740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e31bc390 0x5593e35567e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f853be12740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f85301ac900 con 0x5593e31bc390 2024-05-20T17:59:18.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f853be12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85301af180 con 0x5593e319b6b0 2024-05-20T17:59:18.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f853be12740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f85301acf70 con 0x5593e31934d0 2024-05-20T17:59:18.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f8527fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e31bc390 0x5593e35567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f8527fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e31bc390 0x5593e35567e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54070/0 (socket says 172.21.15.129:54070) 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f8527fff640 1 -- 172.21.15.129:0/2531091935 learned_addr learned my addr 172.21.15.129:0/2531091935 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f852e59c640 1 --2- 172.21.15.129:0/2531091935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e319b6b0 0x5593e35562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f8527fff640 1 --2- 172.21.15.129:0/2531091935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31934d0 0x5593e3557510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f8527fff640 1 -- 172.21.15.129:0/2531091935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31934d0 msgr2=0x5593e3557510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f8527fff640 1 --2- 172.21.15.129:0/2531091935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31934d0 0x5593e3557510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f8527fff640 1 -- 172.21.15.129:0/2531091935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e319b6b0 msgr2=0x5593e35562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f8527fff640 1 --2- 172.21.15.129:0/2531091935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e319b6b0 0x5593e35562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f8527fff640 1 -- 172.21.15.129:0/2531091935 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593e3577fb0 con 0x5593e31bc390 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.239+0000 7f852e59c640 1 --2- 172.21.15.129:0/2531091935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e319b6b0 0x5593e35562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.240+0000 7f8527fff640 1 --2- 172.21.15.129:0/2531091935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e31bc390 0x5593e35567e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8518005fd0 tx=0x7f85180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2938bd28f333566f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.240+0000 7f8517fff640 1 -- 172.21.15.129:0/2531091935 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518021a50 con 0x5593e31bc390 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.240+0000 7f8517fff640 1 -- 172.21.15.129:0/2531091935 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8518004c80 con 0x5593e31bc390 2024-05-20T17:59:18.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.240+0000 7f8517fff640 1 -- 172.21.15.129:0/2531091935 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518005580 con 0x5593e31bc390 2024-05-20T17:59:18.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.241+0000 7f853be12740 1 -- 172.21.15.129:0/2531091935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e31bc390 msgr2=0x5593e35567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.241+0000 7f853be12740 1 --2- 172.21.15.129:0/2531091935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e31bc390 0x5593e35567e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8518005fd0 tx=0x7f85180046c0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.241+0000 7f853be12740 1 -- 172.21.15.129:0/2531091935 shutdown_connections 2024-05-20T17:59:18.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.241+0000 7f853be12740 1 --2- 172.21.15.129:0/2531091935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e319b6b0 0x5593e35562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.241+0000 7f853be12740 1 --2- 172.21.15.129:0/2531091935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e31bc390 0x5593e35567e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.241+0000 7f853be12740 1 --2- 172.21.15.129:0/2531091935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31934d0 0x5593e3557510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.241+0000 7f853be12740 1 -- 172.21.15.129:0/2531091935 >> 172.21.15.129:0/2531091935 conn(0x5593e31b04e0 msgr2=0x5593e3555b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.241+0000 7f853be12740 1 -- 172.21.15.129:0/2531091935 shutdown_connections 2024-05-20T17:59:18.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.241+0000 7f853be12740 1 -- 172.21.15.129:0/2531091935 wait complete. 2024-05-20T17:59:18.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.241+0000 7f853be12740 1 -- start start 2024-05-20T17:59:18.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.241+0000 7f853be12740 1 Processor -- start 2024-05-20T17:59:18.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f853be12740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31934d0 0x5593e356c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f853be12740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e356cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f853be12740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31bc390 0x5593e356d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f853be12740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f85301ad640 con 0x5593e319b6b0 2024-05-20T17:59:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f853be12740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85301aeab0 con 0x5593e31bc390 2024-05-20T17:59:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f853be12740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f85301add10 con 0x5593e31934d0 2024-05-20T17:59:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f853be12740 10 client.0 _ll_register_callbacks cb 0x7f851800ca70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f852e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e356cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f852e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e356cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54086/0 (socket says 172.21.15.129:54086) 2024-05-20T17:59:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f852e59c640 1 -- 172.21.15.129:0/1683861553 learned_addr learned my addr 172.21.15.129:0/1683861553 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f8527fff640 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31bc390 0x5593e356d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f8527fff640 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31934d0 0x5593e356c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f852e59c640 1 -- 172.21.15.129:0/1683861553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31934d0 msgr2=0x5593e356c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f852e59c640 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31934d0 0x5593e356c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f852e59c640 1 -- 172.21.15.129:0/1683861553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31bc390 msgr2=0x5593e356d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f852e59c640 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31bc390 0x5593e356d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f852e59c640 1 -- 172.21.15.129:0/1683861553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593e35ddde0 con 0x5593e319b6b0 2024-05-20T17:59:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f8527fff640 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31934d0 0x5593e356c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f852e59c640 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e356cbd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8520022680 tx=0x7f8520009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de270c7e32d4ee5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f8515ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:18.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f8515ffb640 1 -- 172.21.15.129:0/1683861553 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852001d040 con 0x5593e319b6b0 2024-05-20T17:59:18.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f8515ffb640 1 -- 172.21.15.129:0/1683861553 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f8520022b80 con 0x5593e319b6b0 2024-05-20T17:59:18.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f853be12740 1 -- 172.21.15.129:0/1683861553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8518004c80 con 0x5593e319b6b0 2024-05-20T17:59:18.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f8515ffb640 1 -- 172.21.15.129:0/1683861553 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8520022910 con 0x5593e319b6b0 2024-05-20T17:59:18.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f853be12740 10 client.15516 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:18.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.242+0000 7f853be12740 1 -- 172.21.15.129:0/1683861553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5593e35df000 con 0x5593e319b6b0 2024-05-20T17:59:18.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.243+0000 7f8515ffb640 1 -- 172.21.15.129:0/1683861553 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8518004c80 con 0x5593e319b6b0 2024-05-20T17:59:18.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.243+0000 7f8515ffb640 1 -- 172.21.15.129:0/1683861553 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f852002e4e0 con 0x5593e319b6b0 2024-05-20T17:59:18.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.243+0000 7f853be12740 10 client.15516 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:18.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.243+0000 7f853be12740 10 client.15516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:18.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.243+0000 7f853be12740 20 client.15516 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:18.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.243+0000 7f853be12740 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593e31503a0 0x5593e35e1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.243+0000 7f853be12740 4 client.15516 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T17:59:18.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.243+0000 7f853be12740 1 -- 172.21.15.129:0/1683861553 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5593e34c8780 con 0x5593e31503a0 2024-05-20T17:59:18.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.243+0000 7f8527fff640 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593e31503a0 0x5593e35e1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f8515ffb640 10 client.15516 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f8527fff640 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593e31503a0 0x5593e35e1840 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e85c453ae21cf489 server_cookie=972a013fb0ca831a in_seq=0 out_seq=0 2024-05-20T17:59:18.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f8515ffb640 1 -- 172.21.15.129:0/1683861553 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5593e34c8780 con 0x5593e31503a0 2024-05-20T17:59:18.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f8515ffb640 10 client.15516 handle_command_reply: tid=0 2024-05-20T17:59:18.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 1 client.15516 shutdown 2024-05-20T17:59:18.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 1 -- 172.21.15.129:0/1683861553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 msgr2=0x5593e356cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e356cbd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8520022680 tx=0x7f8520009c80 comp rx=0 tx=0).stop 2024-05-20T17:59:18.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 1 -- 172.21.15.129:0/1683861553 shutdown_connections 2024-05-20T17:59:18.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593e31503a0 0x5593e35e1840 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e31bc390 0x5593e356d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593e319b6b0 0x5593e356cbd0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f8515ffb640 0 client.15516 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 1 --2- 172.21.15.129:0/1683861553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593e31934d0 0x5593e356c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 1 -- 172.21.15.129:0/1683861553 >> 172.21.15.129:0/1683861553 conn(0x5593e31b04e0 msgr2=0x5593e35565c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 1 -- 172.21.15.129:0/1683861553 shutdown_connections 2024-05-20T17:59:18.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 1 -- 172.21.15.129:0/1683861553 wait complete. 2024-05-20T17:59:18.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.244+0000 7f853be12740 20 client.15516 trim_cache size 0 max 16384 2024-05-20T17:59:18.258 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T17:59:18.259 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T17:59:18.259 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T17:59:18.259 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T17:59:18.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 -- 172.21.15.129:0/2318401442 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f851c0a2e20 msgr2=0x7f851c0a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f851c0a2e20 0x7f851c0a5330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f85280edf70 tx=0x7f852802a8c0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 -- 172.21.15.129:0/2318401442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853010c350 msgr2=0x7f853019df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853010c350 0x7f853019df10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8528031a20 tx=0x7f8528031a50 comp rx=0 tx=0).stop 2024-05-20T17:59:18.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 -- 172.21.15.129:0/2318401442 shutdown_connections 2024-05-20T17:59:18.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f851c0a2e20 0x7f851c0a5330 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853010c350 0x7f853019df10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85301079f0 0x7f8530199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 --2- 172.21.15.129:0/2318401442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101ac0 0x7f85301996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 -- 172.21.15.129:0/2318401442 >> 172.21.15.129:0/2318401442 conn(0x7f8530077130 msgr2=0x7f853006a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 -- 172.21.15.129:0/2318401442 shutdown_connections 2024-05-20T17:59:18.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.245+0000 7f8534e8b640 1 -- 172.21.15.129:0/2318401442 wait complete. 2024-05-20T17:59:18.261 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T17:59:18.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:59:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.293+0000 7f9d658bc640 1 -- 172.21.15.129:0/4275947843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60101b10 msgr2=0x7f9d60101f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.293+0000 7f9d658bc640 1 --2- 172.21.15.129:0/4275947843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60101b10 0x7f9d60101f20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009900 tx=0x7f9d4802a3f0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.293+0000 7f9d658bc640 1 -- 172.21.15.129:0/4275947843 shutdown_connections 2024-05-20T17:59:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.293+0000 7f9d658bc640 1 --2- 172.21.15.129:0/4275947843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60101b10 0x7f9d60101f20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.293+0000 7f9d658bc640 1 --2- 172.21.15.129:0/4275947843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d6010c3a0 0x7f9d6010e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.293+0000 7f9d658bc640 1 --2- 172.21.15.129:0/4275947843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d60107a40 0x7f9d601015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.293+0000 7f9d658bc640 1 -- 172.21.15.129:0/4275947843 >> 172.21.15.129:0/4275947843 conn(0x7f9d600fd210 msgr2=0x7f9d600ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.293+0000 7f9d658bc640 1 -- 172.21.15.129:0/4275947843 shutdown_connections 2024-05-20T17:59:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.293+0000 7f9d658bc640 1 -- 172.21.15.129:0/4275947843 wait complete. 2024-05-20T17:59:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.294+0000 7f9d658bc640 1 Processor -- start 2024-05-20T17:59:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.294+0000 7f9d658bc640 1 -- start start 2024-05-20T17:59:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.294+0000 7f9d658bc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60101b10 0x7f9d600fd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.294+0000 7f9d658bc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d60107a40 0x7f9d600fdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.294+0000 7f9d658bc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d6010c3a0 0x7f9d600fe170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.294+0000 7f9d658bc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d60111d50 con 0x7f9d6010c3a0 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d658bc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d60111bd0 con 0x7f9d60101b10 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d658bc640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d60111ed0 con 0x7f9d60107a40 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d5f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d6010c3a0 0x7f9d600fe170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d5f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d6010c3a0 0x7f9d600fe170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54102/0 (socket says 172.21.15.129:54102) 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d5f7fe640 1 -- 172.21.15.129:0/2861213559 learned_addr learned my addr 172.21.15.129:0/2861213559 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d60107a40 0x7f9d600fdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d5effd640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60101b10 0x7f9d600fd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d5f7fe640 1 -- 172.21.15.129:0/2861213559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d60107a40 msgr2=0x7f9d600fdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d5f7fe640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d60107a40 0x7f9d600fdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d5f7fe640 1 -- 172.21.15.129:0/2861213559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60101b10 msgr2=0x7f9d600fd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d5f7fe640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60101b10 0x7f9d600fd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d5f7fe640 1 -- 172.21.15.129:0/2861213559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d601a7ba0 con 0x7f9d6010c3a0 2024-05-20T17:59:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d5f7fe640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d6010c3a0 0x7f9d600fe170 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800eaa0 tx=0x7f9d5800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d648ba640 1 -- 172.21.15.129:0/2861213559 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5800cdb0 con 0x7f9d6010c3a0 2024-05-20T17:59:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d648ba640 1 -- 172.21.15.129:0/2861213559 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d58005020 con 0x7f9d6010c3a0 2024-05-20T17:59:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d658bc640 1 -- 172.21.15.129:0/2861213559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d601a7e90 con 0x7f9d6010c3a0 2024-05-20T17:59:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d648ba640 1 -- 172.21.15.129:0/2861213559 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d580116b0 con 0x7f9d6010c3a0 2024-05-20T17:59:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.295+0000 7f9d658bc640 1 -- 172.21.15.129:0/2861213559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d601a82f0 con 0x7f9d6010c3a0 2024-05-20T17:59:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.298+0000 7f9d648ba640 1 -- 172.21.15.129:0/2861213559 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f9d58005560 con 0x7f9d6010c3a0 2024-05-20T17:59:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.298+0000 7f9d6c3fd740 1 Processor -- start 2024-05-20T17:59:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.298+0000 7f9d6c3fd740 1 -- start start 2024-05-20T17:59:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.298+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 0x55cea8641080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.298+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 0x55cea86415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.298+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82814d0 0x55cea8641b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.298+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cea85a6340 con 0x55cea82a21b0 2024-05-20T17:59:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.298+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cea854e290 con 0x55cea82792f0 2024-05-20T17:59:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.298+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55cea85a2300 con 0x55cea82814d0 2024-05-20T17:59:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 0x55cea8641080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 0x55cea8641080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54104/0 (socket says 172.21.15.129:54104) 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5e7fc640 1 -- 172.21.15.129:0/792715352 learned_addr learned my addr 172.21.15.129:0/792715352 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5effd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 0x55cea86415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d648ba640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d4c0a2e10 0x7f9d4c0a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/792715352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82814d0 0x55cea8641b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5f7fe640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d4c0a2e10 0x7f9d4c0a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5effd640 1 -- 172.21.15.129:0/792715352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82814d0 msgr2=0x55cea8641b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5effd640 1 --2- 172.21.15.129:0/792715352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82814d0 0x55cea8641b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5effd640 1 -- 172.21.15.129:0/792715352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 msgr2=0x55cea8641080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5effd640 1 --2- 172.21.15.129:0/792715352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 0x55cea8641080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5effd640 1 -- 172.21.15.129:0/792715352 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cea85728e0 con 0x55cea82792f0 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d648ba640 1 -- 172.21.15.129:0/2861213559 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f9d580cf830 con 0x7f9d6010c3a0 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5effd640 1 --2- 172.21.15.129:0/792715352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 0x55cea86415c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009bf0 tx=0x7f9d50011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c50ce1ec0d586cd5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d55ffb640 1 -- 172.21.15.129:0/792715352 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5001e070 con 0x55cea82792f0 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d5f7fe640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d4c0a2e10 0x7f9d4c0a5320 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9d580cce70 tx=0x7f9d5801c040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d55ffb640 1 -- 172.21.15.129:0/792715352 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d50009da0 con 0x55cea82792f0 2024-05-20T17:59:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.299+0000 7f9d55ffb640 1 -- 172.21.15.129:0/792715352 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d50018870 con 0x55cea82792f0 2024-05-20T17:59:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.300+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/792715352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 msgr2=0x55cea86415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.300+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/792715352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 0x55cea86415c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009bf0 tx=0x7f9d50011810 comp rx=0 tx=0).stop 2024-05-20T17:59:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.300+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/792715352 shutdown_connections 2024-05-20T17:59:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.300+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/792715352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 0x55cea86415c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.300+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/792715352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82814d0 0x55cea8641b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.300+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/792715352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 0x55cea8641080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.300+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/792715352 >> 172.21.15.129:0/792715352 conn(0x55cea8296300 msgr2=0x55cea863d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.300+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/792715352 shutdown_connections 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.300+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/792715352 wait complete. 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d6c3fd740 1 -- start start 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d6c3fd740 1 Processor -- start 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82792f0 0x55cea864d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82814d0 0x55cea864d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82a21b0 0x55cea864b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cea864e1b0 con 0x55cea82792f0 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cea8561790 con 0x55cea82814d0 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55cea864e330 con 0x55cea82a21b0 2024-05-20T17:59:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d6c3fd740 10 client.0 _ll_register_callbacks cb 0x55cea859ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d5e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82792f0 0x55cea864d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d5e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82792f0 0x55cea864d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54110/0 (socket says 172.21.15.129:54110) 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d5e7fc640 1 -- 172.21.15.129:0/1433814540 learned_addr learned my addr 172.21.15.129:0/1433814540 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.301+0000 7f9d5effd640 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82814d0 0x55cea864d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82a21b0 0x55cea864b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d5effd640 1 -- 172.21.15.129:0/1433814540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82a21b0 msgr2=0x55cea864b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d5effd640 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82a21b0 0x55cea864b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d5effd640 1 -- 172.21.15.129:0/1433814540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82792f0 msgr2=0x55cea864d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d5effd640 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82792f0 0x55cea864d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d5effd640 1 -- 172.21.15.129:0/1433814540 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cea82ba210 con 0x55cea82814d0 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82792f0 0x55cea864d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d5effd640 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82814d0 0x55cea864d620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x55cea8642430 tx=0x7f9d500056a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f13e65a8aa136c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d3effd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d3effd640 1 -- 172.21.15.129:0/1433814540 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d50027020 con 0x55cea82814d0 2024-05-20T17:59:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/1433814540 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d50009da0 con 0x55cea82814d0 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d3effd640 1 -- 172.21.15.129:0/1433814540 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f9d50003800 con 0x55cea82814d0 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d3effd640 1 -- 172.21.15.129:0/1433814540 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9d5001e070 con 0x55cea82814d0 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d6c3fd740 10 client.25259 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.302+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/1433814540 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55cea8386620 con 0x55cea82814d0 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.303+0000 7f9d3effd640 1 -- 172.21.15.129:0/1433814540 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d50009da0 con 0x55cea82814d0 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.303+0000 7f9d3effd640 1 -- 172.21.15.129:0/1433814540 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f9d50018680 con 0x55cea82814d0 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.303+0000 7f9d6c3fd740 10 client.25259 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.303+0000 7f9d6c3fd740 10 client.25259 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.303+0000 7f9d6c3fd740 20 client.25259 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.303+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cea8260d80 0x55cea865d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.303+0000 7f9d6c3fd740 4 client.25259 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.303+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/1433814540 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cea85b3a70 con 0x55cea8260d80 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.303+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cea8260d80 0x55cea865d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.303+0000 7f9d3effd640 10 client.25259 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.303+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cea8260d80 0x55cea865d860 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b1f7b2a93210951 server_cookie=c4ed5753604e117a in_seq=0 out_seq=0 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d3effd640 1 -- 172.21.15.129:0/1433814540 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cea85b3a70 con 0x55cea8260d80 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d3effd640 10 client.25259 handle_command_reply: tid=0 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d6c3fd740 1 client.25259 shutdown 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/1433814540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82814d0 msgr2=0x55cea864d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82814d0 0x55cea864d620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x55cea8642430 tx=0x7f9d500056a0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/1433814540 shutdown_connections 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cea8260d80 0x55cea865d860 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82814d0 0x55cea864d620 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82a21b0 0x55cea864b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/1433814540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82792f0 0x55cea864d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d3effd640 0 client.25259 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/1433814540 >> 172.21.15.129:0/1433814540 conn(0x55cea8296300 msgr2=0x55cea8641020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/1433814540 shutdown_connections 2024-05-20T17:59:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.304+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/1433814540 wait complete. 2024-05-20T17:59:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.305+0000 7f9d6c3fd740 20 client.25259 trim_cache size 0 max 16384 2024-05-20T17:59:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.319+0000 7f9d6c3fd740 1 Processor -- start 2024-05-20T17:59:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.319+0000 7f9d6c3fd740 1 -- start start 2024-05-20T17:59:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82792f0 0x55cea8645450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82814d0 0x55cea8662e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 0x55cea86633a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d601aed50 con 0x55cea82a21b0 2024-05-20T17:59:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d601b15d0 con 0x55cea82814d0 2024-05-20T17:59:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d601af3c0 con 0x55cea82792f0 2024-05-20T17:59:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d5e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 0x55cea86633a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d5e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 0x55cea86633a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54122/0 (socket says 172.21.15.129:54122) 2024-05-20T17:59:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d5e7fc640 1 -- 172.21.15.129:0/4146772509 learned_addr learned my addr 172.21.15.129:0/4146772509 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/4146772509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82792f0 0x55cea8645450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d5effd640 1 --2- 172.21.15.129:0/4146772509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82814d0 0x55cea8662e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d5e7fc640 1 -- 172.21.15.129:0/4146772509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82792f0 msgr2=0x55cea8645450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/4146772509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82792f0 0x55cea8645450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d5e7fc640 1 -- 172.21.15.129:0/4146772509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82814d0 msgr2=0x55cea8662e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/4146772509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82814d0 0x55cea8662e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d5e7fc640 1 -- 172.21.15.129:0/4146772509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cea8642460 con 0x55cea82a21b0 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/4146772509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 0x55cea86633a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9d4802a910 tx=0x7f9d48000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf17716d7bff143 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.320+0000 7f9d557fa640 1 -- 172.21.15.129:0/4146772509 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d48052070 con 0x55cea82a21b0 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.321+0000 7f9d557fa640 1 -- 172.21.15.129:0/4146772509 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d48031db0 con 0x55cea82a21b0 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.321+0000 7f9d557fa640 1 -- 172.21.15.129:0/4146772509 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d48047040 con 0x55cea82a21b0 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.321+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/4146772509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 msgr2=0x55cea86633a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.321+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/4146772509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 0x55cea86633a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9d4802a910 tx=0x7f9d48000bc0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.321+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/4146772509 shutdown_connections 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.321+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/4146772509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82814d0 0x55cea8662e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.321+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/4146772509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82a21b0 0x55cea86633a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.321+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/4146772509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82792f0 0x55cea8645450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.321+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/4146772509 >> 172.21.15.129:0/4146772509 conn(0x55cea8296300 msgr2=0x55cea8644d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.322+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/4146772509 shutdown_connections 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.322+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/4146772509 wait complete. 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.322+0000 7f9d6c3fd740 1 -- start start 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.322+0000 7f9d6c3fd740 1 Processor -- start 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.322+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 0x55cea86bce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.322+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82814d0 0x55cea86bd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.322+0000 7f9d6c3fd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82a21b0 0x55cea8650a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.322+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d601afa90 con 0x55cea82814d0 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.322+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d601b0f00 con 0x55cea82792f0 2024-05-20T17:59:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.322+0000 7f9d6c3fd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d601b0160 con 0x55cea82a21b0 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.322+0000 7f9d6c3fd740 10 client.0 _ll_register_callbacks cb 0x7f9d5001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82814d0 0x55cea86bd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82814d0 0x55cea86bd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54126/0 (socket says 172.21.15.129:54126) 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5effd640 1 -- 172.21.15.129:0/3672261542 learned_addr learned my addr 172.21.15.129:0/3672261542 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 0x55cea86bce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82a21b0 0x55cea8650a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5effd640 1 -- 172.21.15.129:0/3672261542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82a21b0 msgr2=0x55cea8650a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5effd640 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82a21b0 0x55cea8650a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5effd640 1 -- 172.21.15.129:0/3672261542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 msgr2=0x55cea86bce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5effd640 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 0x55cea86bce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5effd640 1 -- 172.21.15.129:0/3672261542 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cea8651130 con 0x55cea82814d0 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 0x55cea86bce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d5effd640 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82814d0 0x55cea86bd360 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9d500058c0 tx=0x7f9d50005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947b1ad1ea519a8d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d3effd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d3effd640 1 -- 172.21.15.129:0/3672261542 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d50027070 con 0x55cea82814d0 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d3effd640 1 -- 172.21.15.129:0/3672261542 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f9d50003800 con 0x55cea82814d0 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/3672261542 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d48031db0 con 0x55cea82814d0 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d3effd640 1 -- 172.21.15.129:0/3672261542 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f9d50009a70 con 0x55cea82814d0 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d6c3fd740 10 client.15534 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.323+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/3672261542 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55cea86c5440 con 0x55cea82814d0 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.324+0000 7f9d3effd640 1 -- 172.21.15.129:0/3672261542 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d48031db0 con 0x55cea82814d0 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.324+0000 7f9d3effd640 1 -- 172.21.15.129:0/3672261542 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f9d50032070 con 0x55cea82814d0 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.324+0000 7f9d6c3fd740 10 client.15534 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.324+0000 7f9d6c3fd740 10 client.15534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.324+0000 7f9d6c3fd740 20 client.15534 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.324+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cea82361c0 0x55cea86c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.324+0000 7f9d6c3fd740 4 client.15534 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T17:59:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.324+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/3672261542 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cea85addc0 con 0x55cea82361c0 2024-05-20T17:59:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.324+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cea82361c0 0x55cea86c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.324+0000 7f9d3effd640 10 client.15534 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.324+0000 7f9d5e7fc640 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cea82361c0 0x55cea86c7c50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28ca2dc5bfddf8ba server_cookie=2892c059d2da1349 in_seq=0 out_seq=0 2024-05-20T17:59:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d3effd640 1 -- 172.21.15.129:0/3672261542 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cea85addc0 con 0x55cea82361c0 2024-05-20T17:59:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d3effd640 10 client.15534 handle_command_reply: tid=0 2024-05-20T17:59:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d6c3fd740 1 client.15534 shutdown 2024-05-20T17:59:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/3672261542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82814d0 msgr2=0x55cea86bd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82814d0 0x55cea86bd360 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9d500058c0 tx=0x7f9d50005780 comp rx=0 tx=0).stop 2024-05-20T17:59:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/3672261542 shutdown_connections 2024-05-20T17:59:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cea82361c0 0x55cea86c7c50 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cea82a21b0 0x55cea8650a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cea82814d0 0x55cea86bd360 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d3effd640 0 client.15534 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d6c3fd740 1 --2- 172.21.15.129:0/3672261542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cea82792f0 0x55cea86bce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.328+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/3672261542 >> 172.21.15.129:0/3672261542 conn(0x55cea8296300 msgr2=0x55cea8640fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/3672261542 shutdown_connections 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d6c3fd740 1 -- 172.21.15.129:0/3672261542 wait complete. 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d6c3fd740 20 client.15534 trim_cache size 0 max 16384 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T17:59:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T17:59:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 -- 172.21.15.129:0/2861213559 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d4c0a2e10 msgr2=0x7f9d4c0a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d4c0a2e10 0x7f9d4c0a5320 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9d580cce70 tx=0x7f9d5801c040 comp rx=0 tx=0).stop 2024-05-20T17:59:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 -- 172.21.15.129:0/2861213559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d6010c3a0 msgr2=0x7f9d600fe170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d6010c3a0 0x7f9d600fe170 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800eaa0 tx=0x7f9d5800ef60 comp rx=0 tx=0).stop 2024-05-20T17:59:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 -- 172.21.15.129:0/2861213559 shutdown_connections 2024-05-20T17:59:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d4c0a2e10 0x7f9d4c0a5320 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d6010c3a0 0x7f9d600fe170 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d60107a40 0x7f9d600fdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 --2- 172.21.15.129:0/2861213559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d60101b10 0x7f9d600fd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 -- 172.21.15.129:0/2861213559 >> 172.21.15.129:0/2861213559 conn(0x7f9d600fd210 msgr2=0x7f9d600ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 -- 172.21.15.129:0/2861213559 shutdown_connections 2024-05-20T17:59:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:18.329+0000 7f9d658bc640 1 -- 172.21.15.129:0/2861213559 wait complete. 2024-05-20T17:59:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.381+0000 7f3d20921640 1 -- 172.21.15.129:0/136675787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1c10c370 msgr2=0x7f3d1c10e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.381+0000 7f3d20921640 1 --2- 172.21.15.129:0/136675787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1c10c370 0x7f3d1c10e790 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009ca0 tx=0x7f3d1402a8c0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.381+0000 7f3d20921640 1 -- 172.21.15.129:0/136675787 shutdown_connections 2024-05-20T17:59:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.381+0000 7f3d20921640 1 --2- 172.21.15.129:0/136675787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1c101b30 0x7f3d1c101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.381+0000 7f3d20921640 1 --2- 172.21.15.129:0/136675787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1c10c370 0x7f3d1c10e790 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.381+0000 7f3d20921640 1 --2- 172.21.15.129:0/136675787 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d1c107a10 0x7f3d1c1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.381+0000 7f3d20921640 1 -- 172.21.15.129:0/136675787 >> 172.21.15.129:0/136675787 conn(0x7f3d1c077130 msgr2=0x7f3d1c077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.382+0000 7f3d20921640 1 -- 172.21.15.129:0/136675787 shutdown_connections 2024-05-20T17:59:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.382+0000 7f3d20921640 1 -- 172.21.15.129:0/136675787 wait complete. 2024-05-20T17:59:18.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.382+0000 7f3d20921640 1 Processor -- start 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.382+0000 7f3d20921640 1 -- start start 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d20921640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1c101b30 0x7f3d1c198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d20921640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d1c107a10 0x7f3d1c198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d20921640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1c10c370 0x7f3d1c19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d20921640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c111d40 con 0x7f3d1c10c370 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d20921640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c111bc0 con 0x7f3d1c101b30 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d20921640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c111ec0 con 0x7f3d1c107a10 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d1ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1c10c370 0x7f3d1c19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d1ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1c10c370 0x7f3d1c19ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54148/0 (socket says 172.21.15.129:54148) 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d1ad9d640 1 -- 172.21.15.129:0/1156984315 learned_addr learned my addr 172.21.15.129:0/1156984315 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d1a59c640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1c101b30 0x7f3d1c198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d19d9b640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d1c107a10 0x7f3d1c198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d1ad9d640 1 -- 172.21.15.129:0/1156984315 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d1c107a10 msgr2=0x7f3d1c198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d1ad9d640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d1c107a10 0x7f3d1c198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d1ad9d640 1 -- 172.21.15.129:0/1156984315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1c101b30 msgr2=0x7f3d1c198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d1ad9d640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1c101b30 0x7f3d1c198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d1ad9d640 1 -- 172.21.15.129:0/1156984315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1c19d4a0 con 0x7f3d1c10c370 2024-05-20T17:59:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d19d9b640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d1c107a10 0x7f3d1c198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d1ad9d640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1c10c370 0x7f3d1c19ce60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3d14031a20 tx=0x7f3d14031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d0b7fe640 1 -- 172.21.15.129:0/1156984315 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d14003c30 con 0x7f3d1c10c370 2024-05-20T17:59:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d0b7fe640 1 -- 172.21.15.129:0/1156984315 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d14003dd0 con 0x7f3d1c10c370 2024-05-20T17:59:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d20921640 1 -- 172.21.15.129:0/1156984315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d1c19d730 con 0x7f3d1c10c370 2024-05-20T17:59:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d0b7fe640 1 -- 172.21.15.129:0/1156984315 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1402fcf0 con 0x7f3d1c10c370 2024-05-20T17:59:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.383+0000 7f3d20921640 1 -- 172.21.15.129:0/1156984315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d1c19dc80 con 0x7f3d1c10c370 2024-05-20T17:59:18.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.385+0000 7f3d20921640 1 -- 172.21.15.129:0/1156984315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d1c069140 con 0x7f3d1c10c370 2024-05-20T17:59:18.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.386+0000 7f3d0b7fe640 1 -- 172.21.15.129:0/1156984315 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f3d14004050 con 0x7f3d1c10c370 2024-05-20T17:59:18.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.386+0000 7f3d0b7fe640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3d040a2e20 0x7f3d040a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.387+0000 7f3d1a59c640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3d040a2e20 0x7f3d040a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.387+0000 7f3d0b7fe640 1 -- 172.21.15.129:0/1156984315 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f3d140f09c0 con 0x7f3d1c10c370 2024-05-20T17:59:18.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.387+0000 7f3d1a59c640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3d040a2e20 0x7f3d040a5330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c005fd0 tx=0x7f3d0c00b390 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.391+0000 7f3d0b7fe640 1 -- 172.21.15.129:0/1156984315 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3d140f2e20 con 0x7f3d1c10c370 2024-05-20T17:59:18.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:18 smithi154 ceph-mon[36482]: osdmap e90: 12 total, 12 up, 12 in 2024-05-20T17:59:18.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:18 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.154:6836/1860679091,v1:172.21.15.154:6837/1860679091] up:boot 2024-05-20T17:59:18.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:18 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:59:18.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:18 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T17:59:18.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:18 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T17:59:18.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:18 smithi129 ceph-mon[32029]: osdmap e90: 12 total, 12 up, 12 in 2024-05-20T17:59:18.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:18 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.154:6836/1860679091,v1:172.21.15.154:6837/1860679091] up:boot 2024-05-20T17:59:18.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:18 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:59:18.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:18 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T17:59:18.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:18 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T17:59:18.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:18 smithi204 ceph-mon[37076]: osdmap e90: 12 total, 12 up, 12 in 2024-05-20T17:59:18.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:18 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.154:6836/1860679091,v1:172.21.15.154:6837/1860679091] up:boot 2024-05-20T17:59:18.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:18 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T17:59:18.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:18 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T17:59:18.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:18 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T17:59:18.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.605+0000 7f3d20921640 1 -- 172.21.15.129:0/1156984315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d1c199990 con 0x7f3d1c10c370 2024-05-20T17:59:18.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.606+0000 7f3d0b7fe640 1 -- 172.21.15.129:0/1156984315 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11829 (secure 0 0 0) 0x7f3d140bd660 con 0x7f3d1c10c370 2024-05-20T17:59:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:59:18.605 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:59:17.074606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25229":{"gid":25229,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1860679091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1860679091},{"type":"v1","addr":"172.21.15.154:6837","nonce":1860679091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T17:59:18.605 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-05-20T17:59:18.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 -- 172.21.15.129:0/1156984315 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3d040a2e20 msgr2=0x7f3d040a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3d040a2e20 0x7f3d040a5330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c005fd0 tx=0x7f3d0c00b390 comp rx=0 tx=0).stop 2024-05-20T17:59:18.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 -- 172.21.15.129:0/1156984315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1c10c370 msgr2=0x7f3d1c19ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1c10c370 0x7f3d1c19ce60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3d14031a20 tx=0x7f3d14031a50 comp rx=0 tx=0).stop 2024-05-20T17:59:18.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 -- 172.21.15.129:0/1156984315 shutdown_connections 2024-05-20T17:59:18.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3d040a2e20 0x7f3d040a5330 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1c10c370 0x7f3d1c19ce60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d1c107a10 0x7f3d1c198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 --2- 172.21.15.129:0/1156984315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1c101b30 0x7f3d1c198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 -- 172.21.15.129:0/1156984315 >> 172.21.15.129:0/1156984315 conn(0x7f3d1c077130 msgr2=0x7f3d1c1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 -- 172.21.15.129:0/1156984315 shutdown_connections 2024-05-20T17:59:18.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.608+0000 7f3d20921640 1 -- 172.21.15.129:0/1156984315 wait complete. 2024-05-20T17:59:18.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.774+0000 7f5362137640 1 -- 172.21.15.129:0/1412834070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f535c101ab0 msgr2=0x7f535c101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.774+0000 7f5362137640 1 --2- 172.21.15.129:0/1412834070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f535c101ab0 0x7f535c101ec0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5344009900 tx=0x7f534402a410 comp rx=0 tx=0).stop 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.774+0000 7f5362137640 1 -- 172.21.15.129:0/1412834070 shutdown_connections 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.774+0000 7f5362137640 1 --2- 172.21.15.129:0/1412834070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f535c10c340 0x7f535c10e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.774+0000 7f5362137640 1 --2- 172.21.15.129:0/1412834070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f535c101ab0 0x7f535c101ec0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.774+0000 7f5362137640 1 --2- 172.21.15.129:0/1412834070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f535c1079e0 0x7f535c101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.774+0000 7f5362137640 1 -- 172.21.15.129:0/1412834070 >> 172.21.15.129:0/1412834070 conn(0x7f535c0fd210 msgr2=0x7f535c0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.774+0000 7f5362137640 1 -- 172.21.15.129:0/1412834070 shutdown_connections 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.774+0000 7f5362137640 1 -- 172.21.15.129:0/1412834070 wait complete. 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f5362137640 1 Processor -- start 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f5362137640 1 -- start start 2024-05-20T17:59:18.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f5362137640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f535c101ab0 0x7f535c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f5362137640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f535c1079e0 0x7f535c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f5362137640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f535c10c340 0x7f535c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f5362137640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f535c111cf0 con 0x7f535c101ab0 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f5362137640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f535c111b70 con 0x7f535c10c340 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f5362137640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f535c111e70 con 0x7f535c1079e0 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f535c101ab0 0x7f535c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f535c101ab0 0x7f535c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54160/0 (socket says 172.21.15.129:54160) 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f535b7fe640 1 -- 172.21.15.129:0/3014324361 learned_addr learned my addr 172.21.15.129:0/3014324361 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f535bfff640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f535c10c340 0x7f535c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.775+0000 7f5353fff640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f535c1079e0 0x7f535c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.776+0000 7f535b7fe640 1 -- 172.21.15.129:0/3014324361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f535c1079e0 msgr2=0x7f535c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.776+0000 7f535b7fe640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f535c1079e0 0x7f535c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.776+0000 7f535b7fe640 1 -- 172.21.15.129:0/3014324361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f535c10c340 msgr2=0x7f535c19cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.776+0000 7f535b7fe640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f535c10c340 0x7f535c19cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.776+0000 7f535b7fe640 1 -- 172.21.15.129:0/3014324361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f535c19d590 con 0x7f535c101ab0 2024-05-20T17:59:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.776+0000 7f535b7fe640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f535c101ab0 0x7f535c198730 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f534c00d740 tx=0x7f534c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.776+0000 7f53597fa640 1 -- 172.21.15.129:0/3014324361 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c017070 con 0x7f535c101ab0 2024-05-20T17:59:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.776+0000 7f5362137640 1 -- 172.21.15.129:0/3014324361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f535c19d880 con 0x7f535c101ab0 2024-05-20T17:59:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.776+0000 7f53597fa640 1 -- 172.21.15.129:0/3014324361 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f534c00de50 con 0x7f535c101ab0 2024-05-20T17:59:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.776+0000 7f53597fa640 1 -- 172.21.15.129:0/3014324361 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c004e00 con 0x7f535c101ab0 2024-05-20T17:59:18.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.776+0000 7f5362137640 1 -- 172.21.15.129:0/3014324361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f535c1a5200 con 0x7f535c101ab0 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f53597fa640 1 -- 172.21.15.129:0/3014324361 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f534c004fa0 con 0x7f535c101ab0 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f5368fda740 1 Processor -- start 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f5368fda740 1 -- start start 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f5368fda740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b10391b390 0x55b103cb9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f5368fda740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cba0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f5368fda740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cba610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f5368fda740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b103c1fc40 con 0x55b1038f24d0 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f5368fda740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b103bc7580 con 0x55b10391b390 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f5368fda740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b103c1bc00 con 0x55b1038fa6b0 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cba0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cba0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54174/0 (socket says 172.21.15.129:54174) 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f535b7fe640 1 -- 172.21.15.129:0/3693847191 learned_addr learned my addr 172.21.15.129:0/3693847191 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f535bfff640 1 --2- 172.21.15.129:0/3693847191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b10391b390 0x55b103cb9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f5353fff640 1 --2- 172.21.15.129:0/3693847191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cba610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f53597fa640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f53480a2e20 0x7f53480a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f535b7fe640 1 -- 172.21.15.129:0/3693847191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 msgr2=0x55b103cba610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f535b7fe640 1 --2- 172.21.15.129:0/3693847191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cba610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f535b7fe640 1 -- 172.21.15.129:0/3693847191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b10391b390 msgr2=0x55b103cb9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.779+0000 7f535b7fe640 1 --2- 172.21.15.129:0/3693847191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b10391b390 0x55b103cb9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.780+0000 7f535b7fe640 1 -- 172.21.15.129:0/3693847191 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1039333f0 con 0x55b1038f24d0 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.780+0000 7f535bfff640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f53480a2e20 0x7f53480a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.780+0000 7f53597fa640 1 -- 172.21.15.129:0/3014324361 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f534c0cf0f0 con 0x7f535c101ab0 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.780+0000 7f535b7fe640 1 --2- 172.21.15.129:0/3693847191 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cba0d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f534c0cc7e0 tx=0x7f534c066000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7519f2d0a0c2a221 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.780+0000 7f5351ffb640 1 -- 172.21.15.129:0/3693847191 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c09af10 con 0x55b1038f24d0 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.780+0000 7f535bfff640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f53480a2e20 0x7f53480a5330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55b103cb73f0 tx=0x7f5354004a40 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.780+0000 7f5351ffb640 1 -- 172.21.15.129:0/3693847191 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f534c0046c0 con 0x55b1038f24d0 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.780+0000 7f5351ffb640 1 -- 172.21.15.129:0/3693847191 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c0d2d30 con 0x55b1038f24d0 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.781+0000 7f5368fda740 1 -- 172.21.15.129:0/3693847191 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 msgr2=0x55b103cba0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.781+0000 7f5368fda740 1 --2- 172.21.15.129:0/3693847191 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cba0d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f534c0cc7e0 tx=0x7f534c066000 comp rx=0 tx=0).stop 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.781+0000 7f5368fda740 1 -- 172.21.15.129:0/3693847191 shutdown_connections 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.781+0000 7f5368fda740 1 --2- 172.21.15.129:0/3693847191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cba610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.781+0000 7f5368fda740 1 --2- 172.21.15.129:0/3693847191 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cba0d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.781+0000 7f5368fda740 1 --2- 172.21.15.129:0/3693847191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b10391b390 0x55b103cb9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.781+0000 7f5368fda740 1 -- 172.21.15.129:0/3693847191 >> 172.21.15.129:0/3693847191 conn(0x55b10390f4e0 msgr2=0x55b103cb64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.781+0000 7f5368fda740 1 -- 172.21.15.129:0/3693847191 shutdown_connections 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.781+0000 7f5368fda740 1 -- 172.21.15.129:0/3693847191 wait complete. 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.781+0000 7f5368fda740 1 -- start start 2024-05-20T17:59:18.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f5368fda740 1 Processor -- start 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f5368fda740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b1038f24d0 0x55b103cc6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f5368fda740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cc6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f5368fda740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b10391b390 0x55b103cd26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f5368fda740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b103cc9220 con 0x55b10391b390 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f5368fda740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b103bdac40 con 0x55b1038f24d0 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f5368fda740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b103cc93a0 con 0x55b1038fa6b0 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f5368fda740 10 client.0 _ll_register_callbacks cb 0x55b103932be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f535bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b10391b390 0x55b103cd26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f535bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b10391b390 0x55b103cd26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54190/0 (socket says 172.21.15.129:54190) 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f535bfff640 1 -- 172.21.15.129:0/823694838 learned_addr learned my addr 172.21.15.129:0/823694838 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f5353fff640 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cc6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f535b7fe640 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b1038f24d0 0x55b103cc6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f535bfff640 1 -- 172.21.15.129:0/823694838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 msgr2=0x55b103cc6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f535bfff640 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cc6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f535bfff640 1 -- 172.21.15.129:0/823694838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b1038f24d0 msgr2=0x55b103cc6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f535bfff640 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b1038f24d0 0x55b103cc6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f535bfff640 1 -- 172.21.15.129:0/823694838 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1039fd7d0 con 0x55b10391b390 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f535bfff640 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b10391b390 0x55b103cd26d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5354010a00 tx=0x7f5354004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3386c70cdcd1e8e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.782+0000 7f533affd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f533affd640 1 -- 172.21.15.129:0/823694838 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535402b060 con 0x55b10391b390 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f533affd640 1 -- 172.21.15.129:0/823694838 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f5354018070 con 0x55b10391b390 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f5368fda740 1 -- 172.21.15.129:0/823694838 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b103c2d370 con 0x55b10391b390 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f533affd640 1 -- 172.21.15.129:0/823694838 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f535401d940 con 0x55b10391b390 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f5368fda740 10 client.15552 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f5368fda740 1 -- 172.21.15.129:0/823694838 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b103cd59a0 con 0x55b10391b390 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f533affd640 1 -- 172.21.15.129:0/823694838 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b103c2d370 con 0x55b10391b390 2024-05-20T17:59:18.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f533affd640 1 -- 172.21.15.129:0/823694838 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f535400f040 con 0x55b10391b390 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f5368fda740 10 client.15552 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f5368fda740 10 client.15552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f5368fda740 20 client.15552 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f5368fda740 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b1038d9f60 0x55b103cd8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f5368fda740 4 client.15552 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.783+0000 7f5368fda740 1 -- 172.21.15.129:0/823694838 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b103c276c0 con 0x55b1038d9f60 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.784+0000 7f535b7fe640 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b1038d9f60 0x55b103cd8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.784+0000 7f533affd640 10 client.15552 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.784+0000 7f535b7fe640 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b1038d9f60 0x55b103cd8ed0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf75182ec33bdc2a server_cookie=ccfba38d0571d082 in_seq=0 out_seq=0 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f533affd640 1 -- 172.21.15.129:0/823694838 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b103c276c0 con 0x55b1038d9f60 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f533affd640 10 client.15552 handle_command_reply: tid=0 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 1 client.15552 shutdown 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 1 -- 172.21.15.129:0/823694838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b10391b390 msgr2=0x55b103cd26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b10391b390 0x55b103cd26d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5354010a00 tx=0x7f5354004ab0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 1 -- 172.21.15.129:0/823694838 shutdown_connections 2024-05-20T17:59:18.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b1038d9f60 0x55b103cd8ed0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b10391b390 0x55b103cd26d0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cc6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 1 --2- 172.21.15.129:0/823694838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b1038f24d0 0x55b103cc6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f533affd640 0 client.15552 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 1 -- 172.21.15.129:0/823694838 >> 172.21.15.129:0/823694838 conn(0x55b10390f4e0 msgr2=0x55b103cba4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 1 -- 172.21.15.129:0/823694838 shutdown_connections 2024-05-20T17:59:18.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 1 -- 172.21.15.129:0/823694838 wait complete. 2024-05-20T17:59:18.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.785+0000 7f5368fda740 20 client.15552 trim_cache size 0 max 16384 2024-05-20T17:59:18.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5368fda740 1 Processor -- start 2024-05-20T17:59:18.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5368fda740 1 -- start start 2024-05-20T17:59:18.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5368fda740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b1038f24d0 0x55b103cc52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5368fda740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038fa6b0 0x55b103cd22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5368fda740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b10391b390 0x55b103cbfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5368fda740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1aa7e0 con 0x55b1038fa6b0 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5368fda740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1ad060 con 0x55b1038f24d0 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5368fda740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1aae50 con 0x55b10391b390 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5353fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038fa6b0 0x55b103cd22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5353fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038fa6b0 0x55b103cd22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54196/0 (socket says 172.21.15.129:54196) 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5353fff640 1 -- 172.21.15.129:0/4131208285 learned_addr learned my addr 172.21.15.129:0/4131208285 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f535bfff640 1 --2- 172.21.15.129:0/4131208285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b10391b390 0x55b103cbfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f535b7fe640 1 --2- 172.21.15.129:0/4131208285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b1038f24d0 0x55b103cc52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5353fff640 1 -- 172.21.15.129:0/4131208285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b10391b390 msgr2=0x55b103cbfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5353fff640 1 --2- 172.21.15.129:0/4131208285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b10391b390 0x55b103cbfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5353fff640 1 -- 172.21.15.129:0/4131208285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b1038f24d0 msgr2=0x55b103cc52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5353fff640 1 --2- 172.21.15.129:0/4131208285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b1038f24d0 0x55b103cc52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5353fff640 1 -- 172.21.15.129:0/4131208285 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b103cbb2f0 con 0x55b1038fa6b0 2024-05-20T17:59:18.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f535b7fe640 1 --2- 172.21.15.129:0/4131208285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b1038f24d0 0x55b103cc52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:18.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.799+0000 7f5353fff640 1 --2- 172.21.15.129:0/4131208285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038fa6b0 0x55b103cd22b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5344031550 tx=0x7f5344009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f028894429f94222 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f53517fa640 1 -- 172.21.15.129:0/4131208285 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344035070 con 0x55b1038fa6b0 2024-05-20T17:59:18.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f53517fa640 1 -- 172.21.15.129:0/4131208285 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5344004030 con 0x55b1038fa6b0 2024-05-20T17:59:18.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f53517fa640 1 -- 172.21.15.129:0/4131208285 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53440055b0 con 0x55b1038fa6b0 2024-05-20T17:59:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f5368fda740 1 -- 172.21.15.129:0/4131208285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038fa6b0 msgr2=0x55b103cd22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f5368fda740 1 --2- 172.21.15.129:0/4131208285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038fa6b0 0x55b103cd22b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5344031550 tx=0x7f5344009680 comp rx=0 tx=0).stop 2024-05-20T17:59:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f5368fda740 1 -- 172.21.15.129:0/4131208285 shutdown_connections 2024-05-20T17:59:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f5368fda740 1 --2- 172.21.15.129:0/4131208285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b10391b390 0x55b103cbfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f5368fda740 1 --2- 172.21.15.129:0/4131208285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038fa6b0 0x55b103cd22b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f5368fda740 1 --2- 172.21.15.129:0/4131208285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b1038f24d0 0x55b103cc52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f5368fda740 1 -- 172.21.15.129:0/4131208285 >> 172.21.15.129:0/4131208285 conn(0x55b10390f4e0 msgr2=0x55b103cc4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f5368fda740 1 -- 172.21.15.129:0/4131208285 shutdown_connections 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.800+0000 7f5368fda740 1 -- 172.21.15.129:0/4131208285 wait complete. 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f5368fda740 1 -- start start 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f5368fda740 1 Processor -- start 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f5368fda740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cdd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f5368fda740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cddbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f5368fda740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b10391b390 0x55b103ccd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f5368fda740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1ab520 con 0x55b1038f24d0 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f5368fda740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1ac990 con 0x55b10391b390 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f5368fda740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1abbf0 con 0x55b1038fa6b0 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f5368fda740 10 client.0 _ll_register_callbacks cb 0x7f5354009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cdd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cdd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54200/0 (socket says 172.21.15.129:54200) 2024-05-20T17:59:18.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f535b7fe640 1 -- 172.21.15.129:0/2571339103 learned_addr learned my addr 172.21.15.129:0/2571339103 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.801+0000 7f535bfff640 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b10391b390 0x55b103ccd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f5353fff640 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cddbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f535b7fe640 1 -- 172.21.15.129:0/2571339103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 msgr2=0x55b103cddbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f535b7fe640 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cddbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f535b7fe640 1 -- 172.21.15.129:0/2571339103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b10391b390 msgr2=0x55b103ccd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f535b7fe640 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b10391b390 0x55b103ccd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f535b7fe640 1 -- 172.21.15.129:0/2571339103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b103cde4c0 con 0x55b1038f24d0 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f535b7fe640 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cdd680 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f534c0970b0 tx=0x7f534c0d2db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193433de7fa23552 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f533affd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f533affd640 1 -- 172.21.15.129:0/2571339103 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c020040 con 0x55b1038f24d0 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f533affd640 1 -- 172.21.15.129:0/2571339103 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f534c009cc0 con 0x55b1038f24d0 2024-05-20T17:59:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f5368fda740 1 -- 172.21.15.129:0/2571339103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5344004030 con 0x55b1038f24d0 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f533affd640 1 -- 172.21.15.129:0/2571339103 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f534c09ad30 con 0x55b1038f24d0 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f5368fda740 10 client.15558 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f5368fda740 1 -- 172.21.15.129:0/2571339103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b103ccec20 con 0x55b1038f24d0 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f533affd640 1 -- 172.21.15.129:0/2571339103 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5344004030 con 0x55b1038f24d0 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f533affd640 1 -- 172.21.15.129:0/2571339103 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f534c097a40 con 0x55b1038f24d0 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f5368fda740 10 client.15558 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f5368fda740 10 client.15558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f5368fda740 20 client.15558 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f5368fda740 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b1038af3a0 0x55b103d43250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f5368fda740 4 client.15558 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.802+0000 7f5368fda740 1 -- 172.21.15.129:0/2571339103 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b103cb67a0 con 0x55b1038af3a0 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f5353fff640 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b1038af3a0 0x55b103d43250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f533affd640 10 client.15558 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f5353fff640 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b1038af3a0 0x55b103d43250 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=379b468687f9f2b7 server_cookie=db749dfb1c90cecc in_seq=0 out_seq=0 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f533affd640 1 -- 172.21.15.129:0/2571339103 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b103cb67a0 con 0x55b1038af3a0 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f533affd640 10 client.15558 handle_command_reply: tid=0 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f5368fda740 1 client.15558 shutdown 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f5368fda740 1 -- 172.21.15.129:0/2571339103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 msgr2=0x55b103cdd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f5368fda740 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cdd680 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f534c0970b0 tx=0x7f534c0d2db0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f5368fda740 1 -- 172.21.15.129:0/2571339103 shutdown_connections 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f5368fda740 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b1038af3a0 0x55b103d43250 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f5368fda740 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b10391b390 0x55b103ccd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f5368fda740 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b1038fa6b0 0x55b103cddbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f5368fda740 1 --2- 172.21.15.129:0/2571339103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b1038f24d0 0x55b103cdd680 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f533affd640 0 client.15558 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.803+0000 7f5368fda740 1 -- 172.21.15.129:0/2571339103 >> 172.21.15.129:0/2571339103 conn(0x55b10390f4e0 msgr2=0x55b103cbb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5368fda740 1 -- 172.21.15.129:0/2571339103 shutdown_connections 2024-05-20T17:59:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5368fda740 1 -- 172.21.15.129:0/2571339103 wait complete. 2024-05-20T17:59:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5368fda740 20 client.15558 trim_cache size 0 max 16384 2024-05-20T17:59:18.809 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 -- 172.21.15.129:0/3014324361 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f53480a2e20 msgr2=0x7f53480a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f53480a2e20 0x7f53480a5330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55b103cb73f0 tx=0x7f5354004a40 comp rx=0 tx=0).stop 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 -- 172.21.15.129:0/3014324361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f535c101ab0 msgr2=0x7f535c198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f535c101ab0 0x7f535c198730 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f534c00d740 tx=0x7f534c00dc00 comp rx=0 tx=0).stop 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 -- 172.21.15.129:0/3014324361 shutdown_connections 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f53480a2e20 0x7f53480a5330 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f535c10c340 0x7f535c19cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f535c1079e0 0x7f535c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 --2- 172.21.15.129:0/3014324361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f535c101ab0 0x7f535c198730 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 -- 172.21.15.129:0/3014324361 >> 172.21.15.129:0/3014324361 conn(0x7f535c0fd210 msgr2=0x7f535c0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 -- 172.21.15.129:0/3014324361 shutdown_connections 2024-05-20T17:59:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:18.804+0000 7f5362137640 1 -- 172.21.15.129:0/3014324361 wait complete. 2024-05-20T17:59:19.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:19 smithi154 ceph-mon[36482]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 204 B/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-20T17:59:19.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:19 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1156984315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:59:19.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:19 smithi129 ceph-mon[32029]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 204 B/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-20T17:59:19.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:19 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1156984315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:59:19.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:19 smithi204 ceph-mon[37076]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 204 B/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-20T17:59:19.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:19 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1156984315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:59:19.814 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T17:59:19.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:59:19.815 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T17:59:19.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.947+0000 7f1e037aa640 1 -- 172.21.15.129:0/2710483053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dfc10d060 msgr2=0x7f1dfc10f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:19.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.947+0000 7f1e037aa640 1 --2- 172.21.15.129:0/2710483053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dfc10d060 0x7f1dfc10f510 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009900 tx=0x7f1df402a480 comp rx=0 tx=0).stop 2024-05-20T17:59:19.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.947+0000 7f1e037aa640 1 -- 172.21.15.129:0/2710483053 shutdown_connections 2024-05-20T17:59:19.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.947+0000 7f1e037aa640 1 --2- 172.21.15.129:0/2710483053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10a6b0 0x7f1dfc10cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.947+0000 7f1e037aa640 1 --2- 172.21.15.129:0/2710483053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dfc10d060 0x7f1dfc10f510 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.947+0000 7f1e037aa640 1 --2- 172.21.15.129:0/2710483053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dfc103c50 0x7f1dfc104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.947+0000 7f1e037aa640 1 -- 172.21.15.129:0/2710483053 >> 172.21.15.129:0/2710483053 conn(0x7f1dfc077130 msgr2=0x7f1dfc077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:19.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.947+0000 7f1e037aa640 1 -- 172.21.15.129:0/2710483053 shutdown_connections 2024-05-20T17:59:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.947+0000 7f1e037aa640 1 -- 172.21.15.129:0/2710483053 wait complete. 2024-05-20T17:59:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e037aa640 1 Processor -- start 2024-05-20T17:59:19.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e037aa640 1 -- start start 2024-05-20T17:59:19.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e037aa640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dfc103c50 0x7f1dfc199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e037aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dfc10a6b0 0x7f1dfc199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e037aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10d060 0x7f1dfc19df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e037aa640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc112dd0 con 0x7f1dfc10a6b0 2024-05-20T17:59:19.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e037aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc112c50 con 0x7f1dfc10d060 2024-05-20T17:59:19.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e037aa640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc112f50 con 0x7f1dfc103c50 2024-05-20T17:59:19.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e00d45640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dfc10a6b0 0x7f1dfc199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e00d45640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dfc10a6b0 0x7f1dfc199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44684/0 (socket says 172.21.15.129:44684) 2024-05-20T17:59:19.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e00d45640 1 -- 172.21.15.129:0/3737847613 learned_addr learned my addr 172.21.15.129:0/3737847613 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:19.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e01546640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dfc103c50 0x7f1dfc199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e01d47640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10d060 0x7f1dfc19df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e00d45640 1 -- 172.21.15.129:0/3737847613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dfc103c50 msgr2=0x7f1dfc199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e00d45640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dfc103c50 0x7f1dfc199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e00d45640 1 -- 172.21.15.129:0/3737847613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10d060 msgr2=0x7f1dfc19df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e00d45640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10d060 0x7f1dfc19df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.948+0000 7f1e00d45640 1 -- 172.21.15.129:0/3737847613 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dfc19e570 con 0x7f1dfc10a6b0 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.949+0000 7f1e00d45640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dfc10a6b0 0x7f1dfc199c70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1df000c890 tx=0x7f1df000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.949+0000 7f1dfa7fc640 1 -- 172.21.15.129:0/3737847613 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0004220 con 0x7f1dfc10a6b0 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.949+0000 7f1dfa7fc640 1 -- 172.21.15.129:0/3737847613 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df00043c0 con 0x7f1dfc10a6b0 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.949+0000 7f1e037aa640 1 -- 172.21.15.129:0/3737847613 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dfc19e860 con 0x7f1dfc10a6b0 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.949+0000 7f1dfa7fc640 1 -- 172.21.15.129:0/3737847613 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df00049e0 con 0x7f1dfc10a6b0 2024-05-20T17:59:19.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.949+0000 7f1e037aa640 1 -- 172.21.15.129:0/3737847613 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dfc1a61e0 con 0x7f1dfc10a6b0 2024-05-20T17:59:19.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.951+0000 7f1dfa7fc640 1 -- 172.21.15.129:0/3737847613 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f1df00079b0 con 0x7f1dfc10a6b0 2024-05-20T17:59:19.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e0a64e740 1 Processor -- start 2024-05-20T17:59:19.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e0a64e740 1 -- start start 2024-05-20T17:59:19.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a170c410 0x5645a1aab620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1aabb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1aac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5645a1a10990 con 0x5645a16eb730 2024-05-20T17:59:19.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5645a19b85b0 con 0x5645a16e3550 2024-05-20T17:59:19.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5645a1a0c950 con 0x5645a170c410 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e00d45640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1aac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e00d45640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1aac0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44686/0 (socket says 172.21.15.129:44686) 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e00d45640 1 -- 172.21.15.129:0/1934329273 learned_addr learned my addr 172.21.15.129:0/1934329273 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1dfa7fc640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1de40a2e20 0x7f1de40a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e01546640 1 --2- 172.21.15.129:0/1934329273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1aabb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.952+0000 7f1e01d47640 1 --2- 172.21.15.129:0/1934329273 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a170c410 0x5645a1aab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1e01d47640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1de40a2e20 0x7f1de40a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1e00d45640 1 -- 172.21.15.129:0/1934329273 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a170c410 msgr2=0x5645a1aab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1e00d45640 1 --2- 172.21.15.129:0/1934329273 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a170c410 0x5645a1aab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1e00d45640 1 -- 172.21.15.129:0/1934329273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 msgr2=0x5645a1aabb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1e00d45640 1 --2- 172.21.15.129:0/1934329273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1aabb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1e00d45640 1 -- 172.21.15.129:0/1934329273 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645a17ec7e0 con 0x5645a16eb730 2024-05-20T17:59:19.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1e01546640 1 --2- 172.21.15.129:0/1934329273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1aabb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1dfa7fc640 1 -- 172.21.15.129:0/3737847613 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f1df00cf7b0 con 0x7f1dfc10a6b0 2024-05-20T17:59:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1e00d45640 1 --2- 172.21.15.129:0/1934329273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1aac0a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1df00cce20 tx=0x7f1df0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923e9d6750a7d7f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1e01d47640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1de40a2e20 0x7f1de40a5330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1df4005f70 tx=0x7f1df4005fd0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1de37fe640 1 -- 172.21.15.129:0/1934329273 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0021c30 con 0x5645a16eb730 2024-05-20T17:59:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1de37fe640 1 -- 172.21.15.129:0/1934329273 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df009c190 con 0x5645a16eb730 2024-05-20T17:59:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.953+0000 7f1de37fe640 1 -- 172.21.15.129:0/1934329273 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df00139e0 con 0x5645a16eb730 2024-05-20T17:59:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.954+0000 7f1e0a64e740 1 -- 172.21.15.129:0/1934329273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 msgr2=0x5645a1aac0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.954+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/1934329273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1aac0a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1df00cce20 tx=0x7f1df0020000 comp rx=0 tx=0).stop 2024-05-20T17:59:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.954+0000 7f1e0a64e740 1 -- 172.21.15.129:0/1934329273 shutdown_connections 2024-05-20T17:59:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.954+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/1934329273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1aabb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.954+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/1934329273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1aac0a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.954+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/1934329273 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a170c410 0x5645a1aab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.954+0000 7f1e0a64e740 1 -- 172.21.15.129:0/1934329273 >> 172.21.15.129:0/1934329273 conn(0x5645a1700560 msgr2=0x5645a1aa7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.954+0000 7f1e0a64e740 1 -- 172.21.15.129:0/1934329273 shutdown_connections 2024-05-20T17:59:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.954+0000 7f1e0a64e740 1 -- 172.21.15.129:0/1934329273 wait complete. 2024-05-20T17:59:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.954+0000 7f1e0a64e740 1 -- start start 2024-05-20T17:59:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e0a64e740 1 Processor -- start 2024-05-20T17:59:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16e3550 0x5645a1ab7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16eb730 0x5645a1ab7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a170c410 0x5645a1ab5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5645a1ab8910 con 0x5645a170c410 2024-05-20T17:59:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5645a19cbb40 con 0x5645a16eb730 2024-05-20T17:59:19.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5645a1ab8a90 con 0x5645a16e3550 2024-05-20T17:59:19.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e0a64e740 10 client.0 _ll_register_callbacks cb 0x5645a19c00f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:19.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e01d47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a170c410 0x5645a1ab5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e01d47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a170c410 0x5645a1ab5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44690/0 (socket says 172.21.15.129:44690) 2024-05-20T17:59:19.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e01d47640 1 -- 172.21.15.129:0/2100185056 learned_addr learned my addr 172.21.15.129:0/2100185056 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:19.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e01546640 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16e3550 0x5645a1ab7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.955+0000 7f1e00d45640 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16eb730 0x5645a1ab7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e01d47640 1 -- 172.21.15.129:0/2100185056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16e3550 msgr2=0x5645a1ab7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:19.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e01d47640 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16e3550 0x5645a1ab7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e01d47640 1 -- 172.21.15.129:0/2100185056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16eb730 msgr2=0x5645a1ab7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:19.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e01d47640 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16eb730 0x5645a1ab7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e01d47640 1 -- 172.21.15.129:0/2100185056 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645a184cf70 con 0x5645a170c410 2024-05-20T17:59:19.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e00d45640 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16eb730 0x5645a1ab7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:19.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e01546640 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16e3550 0x5645a1ab7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:19.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e01d47640 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a170c410 0x5645a1ab5ba0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1df4033000 tx=0x7f1df4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff20b9db7e9a881 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:19.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1de0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:19.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2100185056 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df404c020 con 0x5645a170c410 2024-05-20T17:59:19.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2100185056 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f1df4048620 con 0x5645a170c410 2024-05-20T17:59:19.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2100185056 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1df009c190 con 0x5645a170c410 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2100185056 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1df4057050 con 0x5645a170c410 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e0a64e740 10 client.15570 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2100185056 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5645a16e4170 con 0x5645a170c410 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2100185056 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df009c190 con 0x5645a170c410 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2100185056 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f1df403a070 con 0x5645a170c410 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e0a64e740 10 client.15570 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e0a64e740 10 client.15570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.956+0000 7f1e0a64e740 20 client.15570 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.957+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5645a16cb1f0 0x5645a1ac8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.957+0000 7f1e0a64e740 4 client.15570 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.957+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2100185056 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645a1a1e0c0 con 0x5645a16cb1f0 2024-05-20T17:59:19.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.957+0000 7f1e01546640 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5645a16cb1f0 0x5645a1ac8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.957+0000 7f1e01546640 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5645a16cb1f0 0x5645a1ac8200 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=185b9baaeb23db11 server_cookie=e32e48ef5ab71d40 in_seq=0 out_seq=0 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.957+0000 7f1de0ff9640 10 client.15570 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2100185056 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5645a1a1e0c0 con 0x5645a16cb1f0 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1de0ff9640 10 client.15570 handle_command_reply: tid=0 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 1 client.15570 shutdown 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2100185056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a170c410 msgr2=0x5645a1ab5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a170c410 0x5645a1ab5ba0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1df4033000 tx=0x7f1df4009260 comp rx=0 tx=0).stop 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2100185056 shutdown_connections 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5645a16cb1f0 0x5645a1ac8200 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16eb730 0x5645a1ab7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a170c410 0x5645a1ab5ba0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1de0ff9640 0 client.15570 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:19.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2100185056 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16e3550 0x5645a1ab7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2100185056 >> 172.21.15.129:0/2100185056 conn(0x5645a1700560 msgr2=0x5645a1aad340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:19.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2100185056 shutdown_connections 2024-05-20T17:59:19.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2100185056 wait complete. 2024-05-20T17:59:19.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.958+0000 7f1e0a64e740 20 client.15570 trim_cache size 0 max 16384 2024-05-20T17:59:19.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e0a64e740 1 Processor -- start 2024-05-20T17:59:19.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e0a64e740 1 -- start start 2024-05-20T17:59:19.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16cb1f0 0x5645a1aabf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1acd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1acdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1ac9f0 con 0x5645a16eb730 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1af270 con 0x5645a16e3550 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1ad060 con 0x5645a16cb1f0 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e01d47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1acdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e01d47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1acdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44694/0 (socket says 172.21.15.129:44694) 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e01d47640 1 -- 172.21.15.129:0/1377090931 learned_addr learned my addr 172.21.15.129:0/1377090931 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e00d45640 1 --2- 172.21.15.129:0/1377090931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1acd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e01546640 1 --2- 172.21.15.129:0/1377090931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16cb1f0 0x5645a1aabf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e01d47640 1 -- 172.21.15.129:0/1377090931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16cb1f0 msgr2=0x5645a1aabf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e01d47640 1 --2- 172.21.15.129:0/1377090931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16cb1f0 0x5645a1aabf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e01d47640 1 -- 172.21.15.129:0/1377090931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 msgr2=0x5645a1acd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:19.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e01d47640 1 --2- 172.21.15.129:0/1377090931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1acd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.977+0000 7f1e01d47640 1 -- 172.21.15.129:0/1377090931 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645a1aa2ea0 con 0x5645a16eb730 2024-05-20T17:59:19.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.978+0000 7f1e01d47640 1 --2- 172.21.15.129:0/1377090931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1acdf00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1df402a9a0 tx=0x7f1df40092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393490a8ef3909d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:19.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.978+0000 7f1de2ffd640 1 -- 172.21.15.129:0/1377090931 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df404c070 con 0x5645a16eb730 2024-05-20T17:59:19.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.978+0000 7f1de2ffd640 1 -- 172.21.15.129:0/1377090931 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df4043040 con 0x5645a16eb730 2024-05-20T17:59:19.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.978+0000 7f1de2ffd640 1 -- 172.21.15.129:0/1377090931 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4042030 con 0x5645a16eb730 2024-05-20T17:59:19.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.978+0000 7f1e0a64e740 1 -- 172.21.15.129:0/1377090931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 msgr2=0x5645a1acdf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:19.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.978+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/1377090931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1acdf00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1df402a9a0 tx=0x7f1df40092d0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.978+0000 7f1e0a64e740 1 -- 172.21.15.129:0/1377090931 shutdown_connections 2024-05-20T17:59:19.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.979+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/1377090931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1acd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.979+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/1377090931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1acdf00 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.979+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/1377090931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16cb1f0 0x5645a1aabf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.979+0000 7f1e0a64e740 1 -- 172.21.15.129:0/1377090931 >> 172.21.15.129:0/1377090931 conn(0x5645a1700560 msgr2=0x5645a1aad7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.979+0000 7f1e0a64e740 1 -- 172.21.15.129:0/1377090931 shutdown_connections 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.979+0000 7f1e0a64e740 1 -- 172.21.15.129:0/1377090931 wait complete. 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.979+0000 7f1e0a64e740 1 -- start start 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.979+0000 7f1e0a64e740 1 Processor -- start 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.979+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16cb1f0 0x5645a1abb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.979+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1abb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.979+0000 7f1e0a64e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1abd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1ad730 con 0x5645a16eb730 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1aeba0 con 0x5645a16e3550 2024-05-20T17:59:19.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e0a64e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc1ade00 con 0x5645a16cb1f0 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e0a64e740 10 client.0 _ll_register_callbacks cb 0x7f1df4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e01d47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1abd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e01d47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1abd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44712/0 (socket says 172.21.15.129:44712) 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e01d47640 1 -- 172.21.15.129:0/2884709007 learned_addr learned my addr 172.21.15.129:0/2884709007 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e01546640 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16cb1f0 0x5645a1abb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e00d45640 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1abb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e01d47640 1 -- 172.21.15.129:0/2884709007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16cb1f0 msgr2=0x5645a1abb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e01d47640 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16cb1f0 0x5645a1abb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e01d47640 1 -- 172.21.15.129:0/2884709007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 msgr2=0x5645a1abb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e01d47640 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1abb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e01d47640 1 -- 172.21.15.129:0/2884709007 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645a1abde50 con 0x5645a16eb730 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e01d47640 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1abd8e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1df403e840 tx=0x7f1df4048420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc458af8a253974 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:19.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1de0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2884709007 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df404c070 con 0x5645a16eb730 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2884709007 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f1df403ed20 con 0x5645a16eb730 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2884709007 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1df4043040 con 0x5645a16eb730 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2884709007 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1df406a050 con 0x5645a16eb730 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e0a64e740 10 client.15576 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.980+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2884709007 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5645a1b2ffc0 con 0x5645a16eb730 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.981+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2884709007 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1df4043040 con 0x5645a16eb730 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.981+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2884709007 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f1df403a070 con 0x5645a16eb730 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.981+0000 7f1e0a64e740 10 client.15576 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.981+0000 7f1e0a64e740 10 client.15576 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.981+0000 7f1e0a64e740 20 client.15576 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.981+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5645a170c410 0x5645a1b32620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.981+0000 7f1e0a64e740 4 client.15576 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.981+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2884709007 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5645a1a18410 con 0x5645a170c410 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.981+0000 7f1e01546640 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5645a170c410 0x5645a1b32620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.982+0000 7f1e01546640 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5645a170c410 0x5645a1b32620 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=377c00f57e56a3ad server_cookie=557d27d3c1905d92 in_seq=0 out_seq=0 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.982+0000 7f1de0ff9640 10 client.15576 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1de0ff9640 1 -- 172.21.15.129:0/2884709007 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5645a1a18410 con 0x5645a170c410 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1de0ff9640 10 client.15576 handle_command_reply: tid=0 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 1 client.15576 shutdown 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2884709007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 msgr2=0x5645a1abd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1abd8e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1df403e840 tx=0x7f1df4048420 comp rx=0 tx=0).stop 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2884709007 shutdown_connections 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5645a170c410 0x5645a1b32620 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645a16e3550 0x5645a1abb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645a16eb730 0x5645a1abd8e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 1 --2- 172.21.15.129:0/2884709007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5645a16cb1f0 0x5645a1abb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2884709007 >> 172.21.15.129:0/2884709007 conn(0x5645a1700560 msgr2=0x5645a1aad7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:19.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1de0ff9640 0 client.15576 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:19.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2884709007 shutdown_connections 2024-05-20T17:59:19.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 1 -- 172.21.15.129:0/2884709007 wait complete. 2024-05-20T17:59:19.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.983+0000 7f1e0a64e740 20 client.15576 trim_cache size 0 max 16384 2024-05-20T17:59:19.987 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T17:59:19.987 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T17:59:19.987 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "73300e76-f3c0-496f-98fd-df16c103d353", 2024-05-20T17:59:19.988 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T17:59:19.988 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T17:59:19.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 -- 172.21.15.129:0/3737847613 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1de40a2e20 msgr2=0x7f1de40a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:19.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1de40a2e20 0x7f1de40a5330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1df4005f70 tx=0x7f1df4005fd0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 -- 172.21.15.129:0/3737847613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dfc10a6b0 msgr2=0x7f1dfc199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:19.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dfc10a6b0 0x7f1dfc199c70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1df000c890 tx=0x7f1df000cd50 comp rx=0 tx=0).stop 2024-05-20T17:59:19.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 -- 172.21.15.129:0/3737847613 shutdown_connections 2024-05-20T17:59:19.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1de40a2e20 0x7f1de40a5330 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dfc10d060 0x7f1dfc19df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dfc10a6b0 0x7f1dfc199c70 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 --2- 172.21.15.129:0/3737847613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1dfc103c50 0x7f1dfc199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:19.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 -- 172.21.15.129:0/3737847613 >> 172.21.15.129:0/3737847613 conn(0x7f1dfc077130 msgr2=0x7f1dfc100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:19.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 -- 172.21.15.129:0/3737847613 shutdown_connections 2024-05-20T17:59:19.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:19.984+0000 7f1e037aa640 1 -- 172.21.15.129:0/3737847613 wait complete. 2024-05-20T17:59:19.994 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T17:59:19.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T17:59:20.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.109+0000 7f29b5284640 1 -- 172.21.15.129:0/2677605068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b0101ab0 msgr2=0x7f29b0101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:20.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.109+0000 7f29b5284640 1 --2- 172.21.15.129:0/2677605068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b0101ab0 0x7f29b0101ec0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009900 tx=0x7f29a402a410 comp rx=0 tx=0).stop 2024-05-20T17:59:20.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.109+0000 7f29b5284640 1 -- 172.21.15.129:0/2677605068 shutdown_connections 2024-05-20T17:59:20.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.109+0000 7f29b5284640 1 --2- 172.21.15.129:0/2677605068 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29b010c340 0x7f29b010e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.109+0000 7f29b5284640 1 --2- 172.21.15.129:0/2677605068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b0101ab0 0x7f29b0101ec0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.109+0000 7f29b5284640 1 --2- 172.21.15.129:0/2677605068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b01079e0 0x7f29b0101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.109+0000 7f29b5284640 1 -- 172.21.15.129:0/2677605068 >> 172.21.15.129:0/2677605068 conn(0x7f29b00fd210 msgr2=0x7f29b00ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:20.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.109+0000 7f29b5284640 1 -- 172.21.15.129:0/2677605068 shutdown_connections 2024-05-20T17:59:20.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.109+0000 7f29b5284640 1 -- 172.21.15.129:0/2677605068 wait complete. 2024-05-20T17:59:20.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.110+0000 7f29b5284640 1 Processor -- start 2024-05-20T17:59:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.110+0000 7f29b5284640 1 -- start start 2024-05-20T17:59:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.110+0000 7f29b5284640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29b0101ab0 0x7f29b019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.110+0000 7f29b5284640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b01079e0 0x7f29b019ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29b5284640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b010c340 0x7f29b019b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29b5284640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f29b0111cf0 con 0x7f29b01079e0 2024-05-20T17:59:20.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29b5284640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f29b0111b70 con 0x7f29b010c340 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29b5284640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f29b0111e70 con 0x7f29b0101ab0 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29ae7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b01079e0 0x7f29b019ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29ae7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b01079e0 0x7f29b019ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44730/0 (socket says 172.21.15.129:44730) 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29ae7fc640 1 -- 172.21.15.129:0/1156890994 learned_addr learned my addr 172.21.15.129:0/1156890994 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29aeffd640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29b0101ab0 0x7f29b019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29af7fe640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b010c340 0x7f29b019b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29ae7fc640 1 -- 172.21.15.129:0/1156890994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29b0101ab0 msgr2=0x7f29b019a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29b0101ab0 0x7f29b019a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29ae7fc640 1 -- 172.21.15.129:0/1156890994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b010c340 msgr2=0x7f29b019b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b010c340 0x7f29b019b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29ae7fc640 1 -- 172.21.15.129:0/1156890994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b01a5940 con 0x7f29b01079e0 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29af7fe640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b010c340 0x7f29b019b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:20.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b01079e0 0x7f29b019ab70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f29a4005940 tx=0x7f29a40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f2997fff640 1 -- 172.21.15.129:0/1156890994 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a403f030 con 0x7f29b01079e0 2024-05-20T17:59:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29b5284640 1 -- 172.21.15.129:0/1156890994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b01a5bd0 con 0x7f29b01079e0 2024-05-20T17:59:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f2997fff640 1 -- 172.21.15.129:0/1156890994 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29a4003ce0 con 0x7f29b01079e0 2024-05-20T17:59:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f29b5284640 1 -- 172.21.15.129:0/1156890994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b01a6110 con 0x7f29b01079e0 2024-05-20T17:59:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.111+0000 7f2997fff640 1 -- 172.21.15.129:0/1156890994 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4035070 con 0x7f29b01079e0 2024-05-20T17:59:20.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.114+0000 7f2997fff640 1 -- 172.21.15.129:0/1156890994 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f29a4042030 con 0x7f29b01079e0 2024-05-20T17:59:20.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.114+0000 7f29bc20a740 1 Processor -- start 2024-05-20T17:59:20.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.114+0000 7f29bc20a740 1 -- start start 2024-05-20T17:59:20.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.114+0000 7f2997fff640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29980a2e20 0x7f29980a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68c18390 0x55ec68fb6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 0x55ec68fb7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fb77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29bc20a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ec68f1cd00 con 0x55ec68bf76b0 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29bc20a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ec68ec44c0 con 0x55ec68c18390 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29bc20a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ec68f18cc0 con 0x55ec68bef4d0 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29af7fe640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29980a2e20 0x7f29980a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f2997fff640 1 -- 172.21.15.129:0/1156890994 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f29a40ef830 con 0x7f29b01079e0 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29af7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fb77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29af7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fb77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44742/0 (socket says 172.21.15.129:44742) 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29af7fe640 1 -- 172.21.15.129:0/1230811527 learned_addr learned my addr 172.21.15.129:0/1230811527 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/1230811527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 0x55ec68fb7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29aeffd640 1 --2- 172.21.15.129:0/1230811527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68c18390 0x55ec68fb6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29af7fe640 1 -- 172.21.15.129:0/1230811527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 msgr2=0x55ec68fb7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29af7fe640 1 --2- 172.21.15.129:0/1230811527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 0x55ec68fb7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29af7fe640 1 -- 172.21.15.129:0/1230811527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68c18390 msgr2=0x55ec68fb6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29af7fe640 1 --2- 172.21.15.129:0/1230811527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68c18390 0x55ec68fb6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29af7fe640 1 -- 172.21.15.129:0/1230811527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec68cfa7d0 con 0x55ec68bf76b0 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/1230811527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 0x55ec68fb7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29af7fe640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29980a2e20 0x7f29980a5330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f29a8005c40 tx=0x7f29a8013d50 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29af7fe640 1 --2- 172.21.15.129:0/1230811527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fb77b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f29a800f540 tx=0x7f29a8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fde4ef618ac701a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.115+0000 7f29957fa640 1 -- 172.21.15.129:0/1230811527 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a8004b20 con 0x55ec68bf76b0 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.116+0000 7f29957fa640 1 -- 172.21.15.129:0/1230811527 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29a8004cc0 con 0x55ec68bf76b0 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.116+0000 7f29957fa640 1 -- 172.21.15.129:0/1230811527 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a8029680 con 0x55ec68bf76b0 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.116+0000 7f29bc20a740 1 -- 172.21.15.129:0/1230811527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 msgr2=0x55ec68fb77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.116+0000 7f29bc20a740 1 --2- 172.21.15.129:0/1230811527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fb77b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f29a800f540 tx=0x7f29a8009f90 comp rx=0 tx=0).stop 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.116+0000 7f29bc20a740 1 -- 172.21.15.129:0/1230811527 shutdown_connections 2024-05-20T17:59:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.116+0000 7f29bc20a740 1 --2- 172.21.15.129:0/1230811527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fb77b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.116+0000 7f29bc20a740 1 --2- 172.21.15.129:0/1230811527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 0x55ec68fb7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.116+0000 7f29bc20a740 1 --2- 172.21.15.129:0/1230811527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68c18390 0x55ec68fb6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.116+0000 7f29bc20a740 1 -- 172.21.15.129:0/1230811527 >> 172.21.15.129:0/1230811527 conn(0x55ec68c0c4e0 msgr2=0x55ec68fb34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.116+0000 7f29bc20a740 1 -- 172.21.15.129:0/1230811527 shutdown_connections 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29bc20a740 1 -- 172.21.15.129:0/1230811527 wait complete. 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29bc20a740 1 -- start start 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29bc20a740 1 Processor -- start 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bef4d0 0x55ec68fc2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bf76b0 0x55ec68fc32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68c18390 0x55ec68fc1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29bc20a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ec68fc3fc0 con 0x55ec68bef4d0 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29bc20a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ec68ed7b90 con 0x55ec68c18390 2024-05-20T17:59:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29bc20a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ec68fc4140 con 0x55ec68bf76b0 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29bc20a740 10 client.0 _ll_register_callbacks cb 0x55ec68f13da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29aeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bef4d0 0x55ec68fc2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29aeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bef4d0 0x55ec68fc2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44758/0 (socket says 172.21.15.129:44758) 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.117+0000 7f29aeffd640 1 -- 172.21.15.129:0/233082066 learned_addr learned my addr 172.21.15.129:0/233082066 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bf76b0 0x55ec68fc32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29af7fe640 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68c18390 0x55ec68fc1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29aeffd640 1 -- 172.21.15.129:0/233082066 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bf76b0 msgr2=0x55ec68fc32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29aeffd640 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bf76b0 0x55ec68fc32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29aeffd640 1 -- 172.21.15.129:0/233082066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68c18390 msgr2=0x55ec68fc1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29aeffd640 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68c18390 0x55ec68fc1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29aeffd640 1 -- 172.21.15.129:0/233082066 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec68c303f0 con 0x55ec68bef4d0 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29af7fe640 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68c18390 0x55ec68fc1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29aeffd640 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bef4d0 0x55ec68fc2d90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f29a000bad0 tx=0x7f29a000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46b7145a20814d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f2992ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f2992ff5640 1 -- 172.21.15.129:0/233082066 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a0011440 con 0x55ec68bef4d0 2024-05-20T17:59:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f2992ff5640 1 -- 172.21.15.129:0/233082066 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f29a00115e0 con 0x55ec68bef4d0 2024-05-20T17:59:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29bc20a740 1 -- 172.21.15.129:0/233082066 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29a8004cc0 con 0x55ec68bef4d0 2024-05-20T17:59:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f2992ff5640 1 -- 172.21.15.129:0/233082066 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f29a00117e0 con 0x55ec68bef4d0 2024-05-20T17:59:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29bc20a740 10 client.15588 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29bc20a740 1 -- 172.21.15.129:0/233082066 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ec68bf00f0 con 0x55ec68bef4d0 2024-05-20T17:59:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f2992ff5640 1 -- 172.21.15.129:0/233082066 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29a8004cc0 con 0x55ec68bef4d0 2024-05-20T17:59:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f2992ff5640 1 -- 172.21.15.129:0/233082066 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f29a001bad0 con 0x55ec68bef4d0 2024-05-20T17:59:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29bc20a740 10 client.15588 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:20.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.118+0000 7f29bc20a740 10 client.15588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.119+0000 7f29bc20a740 20 client.15588 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.119+0000 7f29bc20a740 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec68bd6f60 0x55ec68fd3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.119+0000 7f29bc20a740 4 client.15588 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.119+0000 7f29bc20a740 1 -- 172.21.15.129:0/233082066 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec68f2a430 con 0x55ec68bd6f60 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.119+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec68bd6f60 0x55ec68fd3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.119+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec68bd6f60 0x55ec68fd3c20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4a7aa4f04a174ed server_cookie=486d16a4c22009f0 in_seq=0 out_seq=0 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.119+0000 7f2992ff5640 10 client.15588 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f2992ff5640 1 -- 172.21.15.129:0/233082066 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ec68f2a430 con 0x55ec68bd6f60 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f2992ff5640 10 client.15588 handle_command_reply: tid=0 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 1 client.15588 shutdown 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 1 -- 172.21.15.129:0/233082066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bef4d0 msgr2=0x55ec68fc2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bef4d0 0x55ec68fc2d90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f29a000bad0 tx=0x7f29a000bf90 comp rx=0 tx=0).stop 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 1 -- 172.21.15.129:0/233082066 shutdown_connections 2024-05-20T17:59:20.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec68bd6f60 0x55ec68fd3c20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68c18390 0x55ec68fc1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bf76b0 0x55ec68fc32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 1 --2- 172.21.15.129:0/233082066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bef4d0 0x55ec68fc2d90 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f2992ff5640 0 client.15588 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 1 -- 172.21.15.129:0/233082066 >> 172.21.15.129:0/233082066 conn(0x55ec68c0c4e0 msgr2=0x55ec68fb8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 1 -- 172.21.15.129:0/233082066 shutdown_connections 2024-05-20T17:59:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 1 -- 172.21.15.129:0/233082066 wait complete. 2024-05-20T17:59:20.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.120+0000 7f29bc20a740 20 client.15588 trim_cache size 0 max 16384 2024-05-20T17:59:20.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29bc20a740 1 Processor -- start 2024-05-20T17:59:20.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29bc20a740 1 -- start start 2024-05-20T17:59:20.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bef4d0 0x55ec68fc0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fbaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68c18390 0x55ec68fb2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29bc20a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f29b01aa7a0 con 0x55ec68bf76b0 2024-05-20T17:59:20.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29bc20a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f29b01ad000 con 0x55ec68bef4d0 2024-05-20T17:59:20.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29bc20a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f29b01aadf0 con 0x55ec68c18390 2024-05-20T17:59:20.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29ae7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fbaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29ae7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fbaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44774/0 (socket says 172.21.15.129:44774) 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29ae7fc640 1 -- 172.21.15.129:0/143029617 learned_addr learned my addr 172.21.15.129:0/143029617 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29af7fe640 1 --2- 172.21.15.129:0/143029617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68c18390 0x55ec68fb2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29aeffd640 1 --2- 172.21.15.129:0/143029617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bef4d0 0x55ec68fc0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29ae7fc640 1 -- 172.21.15.129:0/143029617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68c18390 msgr2=0x55ec68fb2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/143029617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68c18390 0x55ec68fb2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29ae7fc640 1 -- 172.21.15.129:0/143029617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bef4d0 msgr2=0x55ec68fc0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/143029617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bef4d0 0x55ec68fc0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29ae7fc640 1 -- 172.21.15.129:0/143029617 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec68fbb4c0 con 0x55ec68bf76b0 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.138+0000 7f29aeffd640 1 --2- 172.21.15.129:0/143029617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bef4d0 0x55ec68fc0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.139+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/143029617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fbaf80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f29a40ecf20 tx=0x7f29a40044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db95ce47a1dfea8b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.139+0000 7f2994ff9640 1 -- 172.21.15.129:0/143029617 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4034170 con 0x55ec68bf76b0 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.139+0000 7f2994ff9640 1 -- 172.21.15.129:0/143029617 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29a40bc2e0 con 0x55ec68bf76b0 2024-05-20T17:59:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.139+0000 7f2994ff9640 1 -- 172.21.15.129:0/143029617 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4047d10 con 0x55ec68bf76b0 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.139+0000 7f29bc20a740 1 -- 172.21.15.129:0/143029617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 msgr2=0x55ec68fbaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.139+0000 7f29bc20a740 1 --2- 172.21.15.129:0/143029617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fbaf80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f29a40ecf20 tx=0x7f29a40044e0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.139+0000 7f29bc20a740 1 -- 172.21.15.129:0/143029617 shutdown_connections 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.139+0000 7f29bc20a740 1 --2- 172.21.15.129:0/143029617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68c18390 0x55ec68fb2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.139+0000 7f29bc20a740 1 --2- 172.21.15.129:0/143029617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68bf76b0 0x55ec68fbaf80 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.139+0000 7f29bc20a740 1 --2- 172.21.15.129:0/143029617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bef4d0 0x55ec68fc0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.139+0000 7f29bc20a740 1 -- 172.21.15.129:0/143029617 >> 172.21.15.129:0/143029617 conn(0x55ec68c0c4e0 msgr2=0x55ec68fba870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29bc20a740 1 -- 172.21.15.129:0/143029617 shutdown_connections 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29bc20a740 1 -- 172.21.15.129:0/143029617 wait complete. 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29bc20a740 1 -- start start 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29bc20a740 1 Processor -- start 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 0x55ec68fc7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bf76b0 0x55ec68fc7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29bc20a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68c18390 0x55ec6903afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29bc20a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f29b01ab4c0 con 0x55ec68c18390 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29bc20a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f29b01ac930 con 0x55ec68bf76b0 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29bc20a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f29b01abb90 con 0x55ec68bef4d0 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29bc20a740 10 client.0 _ll_register_callbacks cb 0x7f29a0004060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29af7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68c18390 0x55ec6903afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29af7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68c18390 0x55ec6903afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44776/0 (socket says 172.21.15.129:44776) 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29af7fe640 1 -- 172.21.15.129:0/2973069564 learned_addr learned my addr 172.21.15.129:0/2973069564 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.140+0000 7f29aeffd640 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 0x55ec68fc7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bf76b0 0x55ec68fc7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29af7fe640 1 -- 172.21.15.129:0/2973069564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 msgr2=0x55ec68fc7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29af7fe640 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 0x55ec68fc7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29af7fe640 1 -- 172.21.15.129:0/2973069564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bf76b0 msgr2=0x55ec68fc7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:20.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29af7fe640 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bf76b0 0x55ec68fc7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29af7fe640 1 -- 172.21.15.129:0/2973069564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec6903b520 con 0x55ec68c18390 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29ae7fc640 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bf76b0 0x55ec68fc7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29aeffd640 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 0x55ec68fc7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29af7fe640 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68c18390 0x55ec6903afe0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f29a800f540 tx=0x7f29a80138e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553eee08d800c7a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f2992ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f2992ff5640 1 -- 172.21.15.129:0/2973069564 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a8036020 con 0x55ec68c18390 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f2992ff5640 1 -- 172.21.15.129:0/2973069564 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7f29a8004bd0 con 0x55ec68c18390 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29bc20a740 1 -- 172.21.15.129:0/2973069564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29a40bc2e0 con 0x55ec68c18390 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f2992ff5640 1 -- 172.21.15.129:0/2973069564 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f29a8041050 con 0x55ec68c18390 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29bc20a740 10 client.15594 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f29bc20a740 1 -- 172.21.15.129:0/2973069564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ec6903c5f0 con 0x55ec68c18390 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.141+0000 7f2992ff5640 1 -- 172.21.15.129:0/2973069564 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29a40bc2e0 con 0x55ec68c18390 2024-05-20T17:59:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.142+0000 7f2992ff5640 1 -- 172.21.15.129:0/2973069564 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f29a8012790 con 0x55ec68c18390 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.142+0000 7f29bc20a740 10 client.15594 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.142+0000 7f29bc20a740 10 client.15594 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.142+0000 7f29bc20a740 20 client.15594 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.142+0000 7f29bc20a740 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec68bac3a0 0x55ec6903eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.142+0000 7f29bc20a740 4 client.15594 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.142+0000 7f29bc20a740 1 -- 172.21.15.129:0/2973069564 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ec68f24780 con 0x55ec68bac3a0 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.142+0000 7f29aeffd640 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec68bac3a0 0x55ec6903eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.142+0000 7f2992ff5640 10 client.15594 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.142+0000 7f29aeffd640 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec68bac3a0 0x55ec6903eac0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=730e63809f1542e8 server_cookie=76f4fe31c0e89e5 in_seq=0 out_seq=0 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f2992ff5640 1 -- 172.21.15.129:0/2973069564 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ec68f24780 con 0x55ec68bac3a0 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f2992ff5640 10 client.15594 handle_command_reply: tid=0 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 1 client.15594 shutdown 2024-05-20T17:59:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 1 -- 172.21.15.129:0/2973069564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68c18390 msgr2=0x55ec6903afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68c18390 0x55ec6903afe0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f29a800f540 tx=0x7f29a80138e0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 1 -- 172.21.15.129:0/2973069564 shutdown_connections 2024-05-20T17:59:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec68bac3a0 0x55ec6903eac0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec68bf76b0 0x55ec68fc7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f2992ff5640 0 client.15594 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec68c18390 0x55ec6903afe0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 1 --2- 172.21.15.129:0/2973069564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec68bef4d0 0x55ec68fc7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 1 -- 172.21.15.129:0/2973069564 >> 172.21.15.129:0/2973069564 conn(0x55ec68c0c4e0 msgr2=0x55ec68fb6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 1 -- 172.21.15.129:0/2973069564 shutdown_connections 2024-05-20T17:59:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 1 -- 172.21.15.129:0/2973069564 wait complete. 2024-05-20T17:59:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29bc20a740 20 client.15594 trim_cache size 0 max 16384 2024-05-20T17:59:20.149 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T17:59:20.149 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T17:59:20.149 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T17:59:20.149 INFO:teuthology.orchestra.run.smithi129.stdout: "73300e76-f3c0-496f-98fd-df16c103d353": { 2024-05-20T17:59:20.149 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T17:59:20.149 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "73300e76-f3c0-496f-98fd-df16c103d353", 2024-05-20T17:59:20.149 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T17:59:20.149 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T17:59:20.149 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T17:59:20.149 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T17:59:20.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29b5284640 1 -- 172.21.15.129:0/1156890994 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29980a2e20 msgr2=0x7f29980a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:20.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29b5284640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29980a2e20 0x7f29980a5330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f29a8005c40 tx=0x7f29a8013d50 comp rx=0 tx=0).stop 2024-05-20T17:59:20.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29b5284640 1 -- 172.21.15.129:0/1156890994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b01079e0 msgr2=0x7f29b019ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:20.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29b5284640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b01079e0 0x7f29b019ab70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f29a4005940 tx=0x7f29a40042f0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29b5284640 1 -- 172.21.15.129:0/1156890994 shutdown_connections 2024-05-20T17:59:20.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29b5284640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29980a2e20 0x7f29980a5330 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29b5284640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29b010c340 0x7f29b019b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29b5284640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b01079e0 0x7f29b019ab70 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29b5284640 1 --2- 172.21.15.129:0/1156890994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29b0101ab0 0x7f29b019a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:20.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.143+0000 7f29b5284640 1 -- 172.21.15.129:0/1156890994 >> 172.21.15.129:0/1156890994 conn(0x7f29b00fd210 msgr2=0x7f29b00fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:20.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.144+0000 7f29b5284640 1 -- 172.21.15.129:0/1156890994 shutdown_connections 2024-05-20T17:59:20.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:20.144+0000 7f29b5284640 1 -- 172.21.15.129:0/1156890994 wait complete. 2024-05-20T17:59:20.153 INFO:tasks.cephfs.filesystem:scrub status for tag:73300e76-f3c0-496f-98fd-df16c103d353 - {'path': '/', 'tag': '73300e76-f3c0-496f-98fd-df16c103d353', 'options': 'recursive,force'} 2024-05-20T17:59:21.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:21 smithi154 ceph-mon[36482]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s rd, 1023 B/s wr, 3 op/s 2024-05-20T17:59:21.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:21 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T17:59:21.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:21 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:21.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:21 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T17:59:21.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:21 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:21.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:21 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T17:59:21.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:21 smithi129 ceph-mon[32029]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s rd, 1023 B/s wr, 3 op/s 2024-05-20T17:59:21.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:21 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T17:59:21.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:21 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:21.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:21 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T17:59:21.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:21 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:21.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:21 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T17:59:21.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:21 smithi204 ceph-mon[37076]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s rd, 1023 B/s wr, 3 op/s 2024-05-20T17:59:21.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:21 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T17:59:21.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:21 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:21.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:21 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T17:59:21.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:21 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:21.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:21 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T17:59:23.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:23 smithi154 ceph-mon[36482]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 307 B/s wr, 1 op/s 2024-05-20T17:59:23.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:23 smithi129 ceph-mon[32029]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 307 B/s wr, 1 op/s 2024-05-20T17:59:23.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:23 smithi204 ceph-mon[37076]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 307 B/s wr, 1 op/s 2024-05-20T17:59:25.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:25 smithi154 ceph-mon[36482]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 307 B/s wr, 1 op/s 2024-05-20T17:59:25.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:25 smithi129 ceph-mon[32029]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 307 B/s wr, 1 op/s 2024-05-20T17:59:25.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:25 smithi204 ceph-mon[37076]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 307 B/s wr, 1 op/s 2024-05-20T17:59:27.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:27 smithi154 ceph-mon[36482]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-05-20T17:59:27.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:27 smithi129 ceph-mon[32029]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-05-20T17:59:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:27 smithi204 ceph-mon[37076]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-05-20T17:59:29.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:29 smithi154 ceph-mon[36482]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-05-20T17:59:29.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:29 smithi129 ceph-mon[32029]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-05-20T17:59:29.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:29 smithi204 ceph-mon[37076]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-05-20T17:59:31.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:31 smithi154 ceph-mon[36482]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-20T17:59:31.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:31 smithi129 ceph-mon[32029]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-20T17:59:31.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:31 smithi204 ceph-mon[37076]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-05-20T17:59:31.439 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.440+0000 7fd1fc299640 1 -- 172.21.15.129:0/4084536541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f410c3a0 msgr2=0x7fd1f410e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:31.439 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.440+0000 7fd1fc299640 1 --2- 172.21.15.129:0/4084536541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f410c3a0 0x7fd1f410e7c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec009ca0 tx=0x7fd1ec02a8c0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.441+0000 7fd1fc299640 1 -- 172.21.15.129:0/4084536541 shutdown_connections 2024-05-20T17:59:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.441+0000 7fd1fc299640 1 --2- 172.21.15.129:0/4084536541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1f4101b10 0x7fd1f4101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.441+0000 7fd1fc299640 1 --2- 172.21.15.129:0/4084536541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f410c3a0 0x7fd1f410e7c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.441+0000 7fd1fc299640 1 --2- 172.21.15.129:0/4084536541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1f4107a40 0x7fd1f41015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.441+0000 7fd1fc299640 1 -- 172.21.15.129:0/4084536541 >> 172.21.15.129:0/4084536541 conn(0x7fd1f40fd210 msgr2=0x7fd1f40ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.441+0000 7fd1fc299640 1 -- 172.21.15.129:0/4084536541 shutdown_connections 2024-05-20T17:59:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.441+0000 7fd1fc299640 1 -- 172.21.15.129:0/4084536541 wait complete. 2024-05-20T17:59:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.441+0000 7fd1fc299640 1 Processor -- start 2024-05-20T17:59:31.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.441+0000 7fd1fc299640 1 -- start start 2024-05-20T17:59:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fc299640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1f4101b10 0x7fd1f41986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fc299640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1f4107a40 0x7fd1f4198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fc299640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f410c3a0 0x7fd1f419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fc299640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f4111d70 con 0x7fd1f410c3a0 2024-05-20T17:59:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fc299640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f4111bf0 con 0x7fd1f4101b10 2024-05-20T17:59:31.442 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fc299640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f4111ef0 con 0x7fd1f4107a40 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fa836640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f410c3a0 0x7fd1f419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fa836640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f410c3a0 0x7fd1f419ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45628/0 (socket says 172.21.15.129:45628) 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fa836640 1 -- 172.21.15.129:0/1649338321 learned_addr learned my addr 172.21.15.129:0/1649338321 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1f9834640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1f4107a40 0x7fd1f4198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fa035640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1f4101b10 0x7fd1f41986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fa836640 1 -- 172.21.15.129:0/1649338321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1f4107a40 msgr2=0x7fd1f4198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fa836640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1f4107a40 0x7fd1f4198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fa836640 1 -- 172.21.15.129:0/1649338321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1f4101b10 msgr2=0x7fd1f41986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fa836640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1f4101b10 0x7fd1f41986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fa836640 1 -- 172.21.15.129:0/1649338321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f419d490 con 0x7fd1f410c3a0 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1fa836640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f410c3a0 0x7fd1f419ced0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec031a50 tx=0x7fd1ec031a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.442+0000 7fd1eb7fe640 1 -- 172.21.15.129:0/1649338321 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec003c30 con 0x7fd1f410c3a0 2024-05-20T17:59:31.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.443+0000 7fd1eb7fe640 1 -- 172.21.15.129:0/1649338321 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1ec003dd0 con 0x7fd1f410c3a0 2024-05-20T17:59:31.444 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.443+0000 7fd1fc299640 1 -- 172.21.15.129:0/1649338321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f419d720 con 0x7fd1f410c3a0 2024-05-20T17:59:31.444 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.443+0000 7fd1eb7fe640 1 -- 172.21.15.129:0/1649338321 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec004760 con 0x7fd1f410c3a0 2024-05-20T17:59:31.444 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.443+0000 7fd1fc299640 1 -- 172.21.15.129:0/1649338321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f419dc70 con 0x7fd1f410c3a0 2024-05-20T17:59:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.445+0000 7fd1eb7fe640 1 -- 172.21.15.129:0/1649338321 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7fd1ec02f710 con 0x7fd1f410c3a0 2024-05-20T17:59:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.445+0000 7fd20321d740 1 Processor -- start 2024-05-20T17:59:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd20321d740 1 -- start start 2024-05-20T17:59:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd20321d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a894b1b0 0x5639a8ce9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd20321d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a89222f0 0x5639a8cea450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd20321d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cea990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd20321d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5639a8c4f340 con 0x5639a892a4d0 2024-05-20T17:59:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd20321d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5639a8bf7290 con 0x5639a894b1b0 2024-05-20T17:59:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd20321d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5639a8c4b300 con 0x5639a89222f0 2024-05-20T17:59:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd1eb7fe640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd1e00a2e10 0x7fd1e00a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd1f9834640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cea990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd1f9834640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cea990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45632/0 (socket says 172.21.15.129:45632) 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd1f9834640 1 -- 172.21.15.129:0/1188200658 learned_addr learned my addr 172.21.15.129:0/1188200658 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd1fa035640 1 --2- 172.21.15.129:0/1188200658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a89222f0 0x5639a8cea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd1fa836640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd1e00a2e10 0x7fd1e00a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd1f9834640 1 --2- 172.21.15.129:0/1188200658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a894b1b0 0x5639a8ce9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.446+0000 7fd1eb7fe640 1 -- 172.21.15.129:0/1649338321 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7fd1ec0f0910 con 0x7fd1f410c3a0 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.447+0000 7fd1fa035640 1 -- 172.21.15.129:0/1188200658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a894b1b0 msgr2=0x5639a8ce9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.447+0000 7fd1fa035640 1 --2- 172.21.15.129:0/1188200658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a894b1b0 0x5639a8ce9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.447+0000 7fd1fa035640 1 -- 172.21.15.129:0/1188200658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 msgr2=0x5639a8cea990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.447+0000 7fd1fa035640 1 --2- 172.21.15.129:0/1188200658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cea990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.447+0000 7fd1fa035640 1 -- 172.21.15.129:0/1188200658 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a8c1b8e0 con 0x5639a89222f0 2024-05-20T17:59:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.447+0000 7fd1f9834640 1 --2- 172.21.15.129:0/1188200658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cea990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.447+0000 7fd1fa836640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd1e00a2e10 0x7fd1e00a5320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0ee060 tx=0x7fd1ec08ee70 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.447+0000 7fd1fa035640 1 --2- 172.21.15.129:0/1188200658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a89222f0 0x5639a8cea450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000bbb0 tx=0x7fd1f000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f9091515a277a8f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.447+0000 7fd1e8bfe640 1 -- 172.21.15.129:0/1188200658 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0015070 con 0x5639a89222f0 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.447+0000 7fd1e8bfe640 1 -- 172.21.15.129:0/1188200658 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1f0005500 con 0x5639a89222f0 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.447+0000 7fd1e8bfe640 1 -- 172.21.15.129:0/1188200658 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0011440 con 0x5639a89222f0 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.448+0000 7fd20321d740 1 -- 172.21.15.129:0/1188200658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a89222f0 msgr2=0x5639a8cea450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.448+0000 7fd20321d740 1 --2- 172.21.15.129:0/1188200658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a89222f0 0x5639a8cea450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000bbb0 tx=0x7fd1f000f730 comp rx=0 tx=0).stop 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.448+0000 7fd20321d740 1 -- 172.21.15.129:0/1188200658 shutdown_connections 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.448+0000 7fd20321d740 1 --2- 172.21.15.129:0/1188200658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cea990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.448+0000 7fd20321d740 1 --2- 172.21.15.129:0/1188200658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a89222f0 0x5639a8cea450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.448+0000 7fd20321d740 1 --2- 172.21.15.129:0/1188200658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a894b1b0 0x5639a8ce9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.448+0000 7fd20321d740 1 -- 172.21.15.129:0/1188200658 >> 172.21.15.129:0/1188200658 conn(0x5639a893f300 msgr2=0x5639a8ce6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:31.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.448+0000 7fd20321d740 1 -- 172.21.15.129:0/1188200658 shutdown_connections 2024-05-20T17:59:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.448+0000 7fd20321d740 1 -- 172.21.15.129:0/1188200658 wait complete. 2024-05-20T17:59:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd20321d740 1 -- start start 2024-05-20T17:59:31.450 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd20321d740 1 Processor -- start 2024-05-20T17:59:31.453 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd20321d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a89222f0 0x5639a8cfa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd20321d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a892a4d0 0x5639a8cfaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd20321d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a894b1b0 0x5639a8cf89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd20321d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5639a8cfb620 con 0x5639a894b1b0 2024-05-20T17:59:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd20321d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5639a8c0a790 con 0x5639a89222f0 2024-05-20T17:59:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd20321d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5639a8cfb7a0 con 0x5639a892a4d0 2024-05-20T17:59:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd20321d740 10 client.0 _ll_register_callbacks cb 0x5639a8c45a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd1f9834640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a894b1b0 0x5639a8cf89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd1f9834640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a894b1b0 0x5639a8cf89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45642/0 (socket says 172.21.15.129:45642) 2024-05-20T17:59:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd1f9834640 1 -- 172.21.15.129:0/3425752600 learned_addr learned my addr 172.21.15.129:0/3425752600 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd1f9834640 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a89222f0 0x5639a8cfa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd1fa035640 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a892a4d0 0x5639a8cfaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd1f9834640 1 -- 172.21.15.129:0/3425752600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a892a4d0 msgr2=0x5639a8cfaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd1f9834640 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a892a4d0 0x5639a8cfaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd1f9834640 1 -- 172.21.15.129:0/3425752600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a89222f0 msgr2=0x5639a8cfa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd1f9834640 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a89222f0 0x5639a8cfa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.449+0000 7fd1f9834640 1 -- 172.21.15.129:0/3425752600 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639a8963210 con 0x5639a894b1b0 2024-05-20T17:59:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd1f9834640 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a894b1b0 0x5639a8cf89a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4016ee0 tx=0x7fd1e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bdc79733fe34f3f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd1ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3425752600 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4021030 con 0x5639a894b1b0 2024-05-20T17:59:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3425752600 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7fd1e4015040 con 0x5639a894b1b0 2024-05-20T17:59:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd20321d740 1 -- 172.21.15.129:0/3425752600 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1f0005500 con 0x5639a894b1b0 2024-05-20T17:59:31.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3425752600 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd1e402b050 con 0x5639a894b1b0 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd20321d740 10 client.15606 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd20321d740 1 -- 172.21.15.129:0/3425752600 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5639a8a2f620 con 0x5639a894b1b0 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3425752600 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1f0005500 con 0x5639a894b1b0 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3425752600 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fd1e401b4e0 con 0x5639a894b1b0 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd20321d740 10 client.15606 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd20321d740 10 client.15606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd20321d740 20 client.15606 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.450+0000 7fd20321d740 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639a8909d80 0x5639a8d0adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.451+0000 7fd20321d740 4 client.15606 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.451+0000 7fd20321d740 1 -- 172.21.15.129:0/3425752600 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639a8c5ca70 con 0x5639a8909d80 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.451+0000 7fd1fa035640 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639a8909d80 0x5639a8d0adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.451+0000 7fd1ddffb640 10 client.15606 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:31.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.451+0000 7fd1fa035640 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639a8909d80 0x5639a8d0adc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52fc071ef483ded server_cookie=a906a352c18127ea in_seq=0 out_seq=0 2024-05-20T17:59:31.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3425752600 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5639a8c5ca70 con 0x5639a8909d80 2024-05-20T17:59:31.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd1ddffb640 10 client.15606 handle_command_reply: tid=0 2024-05-20T17:59:31.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 1 client.15606 shutdown 2024-05-20T17:59:31.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 1 -- 172.21.15.129:0/3425752600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a894b1b0 msgr2=0x5639a8cf89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:31.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a894b1b0 0x5639a8cf89a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4016ee0 tx=0x7fd1e4004730 comp rx=0 tx=0).stop 2024-05-20T17:59:31.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 1 -- 172.21.15.129:0/3425752600 shutdown_connections 2024-05-20T17:59:31.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639a8909d80 0x5639a8d0adc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a894b1b0 0x5639a8cf89a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a892a4d0 0x5639a8cfaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd1ddffb640 0 client.15606 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 1 --2- 172.21.15.129:0/3425752600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a89222f0 0x5639a8cfa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 1 -- 172.21.15.129:0/3425752600 >> 172.21.15.129:0/3425752600 conn(0x5639a893f300 msgr2=0x5639a8cebda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 1 -- 172.21.15.129:0/3425752600 shutdown_connections 2024-05-20T17:59:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 1 -- 172.21.15.129:0/3425752600 wait complete. 2024-05-20T17:59:31.458 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.452+0000 7fd20321d740 20 client.15606 trim_cache size 0 max 16384 2024-05-20T17:59:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.467+0000 7fd20321d740 1 Processor -- start 2024-05-20T17:59:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.467+0000 7fd20321d740 1 -- start start 2024-05-20T17:59:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd20321d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a89222f0 0x5639a8cf22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd20321d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cf2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd20321d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a894b1b0 0x5639a8d105c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd20321d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f41aa750 con 0x5639a892a4d0 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd20321d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f41acfd0 con 0x5639a89222f0 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd20321d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f41aadc0 con 0x5639a894b1b0 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1fa035640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cf2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1fa035640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cf2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45652/0 (socket says 172.21.15.129:45652) 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1fa035640 1 -- 172.21.15.129:0/1657458471 learned_addr learned my addr 172.21.15.129:0/1657458471 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1f9834640 1 --2- 172.21.15.129:0/1657458471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a894b1b0 0x5639a8d105c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1f9834640 1 --2- 172.21.15.129:0/1657458471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a89222f0 0x5639a8cf22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1fa035640 1 -- 172.21.15.129:0/1657458471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a894b1b0 msgr2=0x5639a8d105c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1fa035640 1 --2- 172.21.15.129:0/1657458471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a894b1b0 0x5639a8d105c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1fa035640 1 -- 172.21.15.129:0/1657458471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a89222f0 msgr2=0x5639a8cf22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1fa035640 1 --2- 172.21.15.129:0/1657458471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a89222f0 0x5639a8cf22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1fa035640 1 -- 172.21.15.129:0/1657458471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a8ce7420 con 0x5639a892a4d0 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1f9834640 1 --2- 172.21.15.129:0/1657458471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a894b1b0 0x5639a8d105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1fa035640 1 --2- 172.21.15.129:0/1657458471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cf2800 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000bbb0 tx=0x7fd1f0004c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7416d53a3db851bc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1dffff640 1 -- 172.21.15.129:0/1657458471 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f002b070 con 0x5639a892a4d0 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1dffff640 1 -- 172.21.15.129:0/1657458471 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1f0026e50 con 0x5639a892a4d0 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.468+0000 7fd1dffff640 1 -- 172.21.15.129:0/1657458471 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0015040 con 0x5639a892a4d0 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.469+0000 7fd20321d740 1 -- 172.21.15.129:0/1657458471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 msgr2=0x5639a8cf2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.469+0000 7fd20321d740 1 --2- 172.21.15.129:0/1657458471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cf2800 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000bbb0 tx=0x7fd1f0004c00 comp rx=0 tx=0).stop 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.469+0000 7fd20321d740 1 -- 172.21.15.129:0/1657458471 shutdown_connections 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.469+0000 7fd20321d740 1 --2- 172.21.15.129:0/1657458471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a894b1b0 0x5639a8d105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.469+0000 7fd20321d740 1 --2- 172.21.15.129:0/1657458471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a892a4d0 0x5639a8cf2800 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.469+0000 7fd20321d740 1 --2- 172.21.15.129:0/1657458471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a89222f0 0x5639a8cf22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.469+0000 7fd20321d740 1 -- 172.21.15.129:0/1657458471 >> 172.21.15.129:0/1657458471 conn(0x5639a893f300 msgr2=0x5639a8cfa700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:31.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.469+0000 7fd20321d740 1 -- 172.21.15.129:0/1657458471 shutdown_connections 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.469+0000 7fd20321d740 1 -- 172.21.15.129:0/1657458471 wait complete. 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.469+0000 7fd20321d740 1 -- start start 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd20321d740 1 Processor -- start 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd20321d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a89222f0 0x5639a8d00c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd20321d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a892a4d0 0x5639a8cfbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd20321d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a894b1b0 0x5639a8cfc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd20321d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f41ab490 con 0x5639a89222f0 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd20321d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f41ac900 con 0x5639a892a4d0 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd20321d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f41abb60 con 0x5639a894b1b0 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd20321d740 10 client.0 _ll_register_callbacks cb 0x7fd1e4016b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd1f9834640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a89222f0 0x5639a8d00c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd1f9834640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a89222f0 0x5639a8d00c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45656/0 (socket says 172.21.15.129:45656) 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd1f9834640 1 -- 172.21.15.129:0/3111250506 learned_addr learned my addr 172.21.15.129:0/3111250506 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:31.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd1fa035640 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a892a4d0 0x5639a8cfbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd1f9834640 1 -- 172.21.15.129:0/3111250506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a894b1b0 msgr2=0x5639a8cfc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd1f9834640 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a894b1b0 0x5639a8cfc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd1f9834640 1 -- 172.21.15.129:0/3111250506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a892a4d0 msgr2=0x5639a8cfbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd1f9834640 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a892a4d0 0x5639a8cfbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd1f9834640 1 -- 172.21.15.129:0/3111250506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639a8cfca70 con 0x5639a89222f0 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd1f9834640 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a89222f0 0x5639a8d00c50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4019010 tx=0x7fd1e4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1148584b04428a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.470+0000 7fd1ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3111250506 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e4035030 con 0x5639a89222f0 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd20321d740 1 -- 172.21.15.129:0/3111250506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1f0026e50 con 0x5639a89222f0 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3111250506 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8506+0+0 (secure 0 0 0) 0x7fd1e401bd00 con 0x5639a89222f0 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3111250506 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd1e403f050 con 0x5639a89222f0 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd20321d740 10 client.15612 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:31.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd20321d740 1 -- 172.21.15.129:0/3111250506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5639a8d72be0 con 0x5639a89222f0 2024-05-20T17:59:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3111250506 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1f0026e50 con 0x5639a89222f0 2024-05-20T17:59:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3111250506 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fd1e40054a0 con 0x5639a89222f0 2024-05-20T17:59:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd20321d740 10 client.15612 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd20321d740 10 client.15612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd20321d740 20 client.15612 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd20321d740 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639a88df1c0 0x5639a8d75340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd20321d740 4 client.15612 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T17:59:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.471+0000 7fd20321d740 1 -- 172.21.15.129:0/3111250506 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5639a8c56dc0 con 0x5639a88df1c0 2024-05-20T17:59:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.472+0000 7fd1fa035640 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639a88df1c0 0x5639a8d75340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.472+0000 7fd1ddffb640 10 client.15612 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:31.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.472+0000 7fd1fa035640 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639a88df1c0 0x5639a8d75340 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64f27208be70ffbf server_cookie=eeeb988a1f2c7d19 in_seq=0 out_seq=0 2024-05-20T17:59:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd1ddffb640 1 -- 172.21.15.129:0/3111250506 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5639a8c56dc0 con 0x5639a88df1c0 2024-05-20T17:59:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd1ddffb640 10 client.15612 handle_command_reply: tid=0 2024-05-20T17:59:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 1 client.15612 shutdown 2024-05-20T17:59:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 1 -- 172.21.15.129:0/3111250506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a89222f0 msgr2=0x5639a8d00c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a89222f0 0x5639a8d00c50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4019010 tx=0x7fd1e4005f90 comp rx=0 tx=0).stop 2024-05-20T17:59:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 1 -- 172.21.15.129:0/3111250506 shutdown_connections 2024-05-20T17:59:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639a88df1c0 0x5639a8d75340 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639a892a4d0 0x5639a8cfbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd1ddffb640 0 client.15612 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639a894b1b0 0x5639a8cfc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 1 --2- 172.21.15.129:0/3111250506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639a89222f0 0x5639a8d00c50 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 1 -- 172.21.15.129:0/3111250506 >> 172.21.15.129:0/3111250506 conn(0x5639a893f300 msgr2=0x5639a8cfa700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 1 -- 172.21.15.129:0/3111250506 shutdown_connections 2024-05-20T17:59:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 1 -- 172.21.15.129:0/3111250506 wait complete. 2024-05-20T17:59:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.480+0000 7fd20321d740 20 client.15612 trim_cache size 0 max 16384 2024-05-20T17:59:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T17:59:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T17:59:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T17:59:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 -- 172.21.15.129:0/1649338321 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd1e00a2e10 msgr2=0x7fd1e00a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd1e00a2e10 0x7fd1e00a5320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0ee060 tx=0x7fd1ec08ee70 comp rx=0 tx=0).stop 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 -- 172.21.15.129:0/1649338321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f410c3a0 msgr2=0x7fd1f419ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f410c3a0 0x7fd1f419ced0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec031a50 tx=0x7fd1ec031a80 comp rx=0 tx=0).stop 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 -- 172.21.15.129:0/1649338321 shutdown_connections 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd1e00a2e10 0x7fd1e00a5320 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f410c3a0 0x7fd1f419ced0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1f4107a40 0x7fd1f4198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 --2- 172.21.15.129:0/1649338321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1f4101b10 0x7fd1f41986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 -- 172.21.15.129:0/1649338321 >> 172.21.15.129:0/1649338321 conn(0x7fd1f40fd210 msgr2=0x7fd1f40fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 -- 172.21.15.129:0/1649338321 shutdown_connections 2024-05-20T17:59:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:31.481+0000 7fd1fc299640 1 -- 172.21.15.129:0/1649338321 wait complete. 2024-05-20T17:59:33.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:33 smithi154 ceph-mon[36482]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-05-20T17:59:33.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:33 smithi129 ceph-mon[32029]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-05-20T17:59:33.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:33 smithi204 ceph-mon[37076]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-05-20T17:59:35.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:35 smithi154 ceph-mon[36482]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-05-20T17:59:35.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:35 smithi129 ceph-mon[32029]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-05-20T17:59:35.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:35 smithi204 ceph-mon[37076]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-05-20T17:59:37.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:37 smithi154 ceph-mon[36482]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-05-20T17:59:37.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:37 smithi129 ceph-mon[32029]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-05-20T17:59:37.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:37 smithi204 ceph-mon[37076]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-05-20T17:59:39.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:39 smithi154 ceph-mon[36482]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-20T17:59:39.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:39 smithi129 ceph-mon[32029]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-20T17:59:39.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:39 smithi204 ceph-mon[37076]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-20T17:59:41.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:41 smithi154 ceph-mon[36482]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-20T17:59:41.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:41 smithi129 ceph-mon[32029]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-20T17:59:41.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:41 smithi204 ceph-mon[37076]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-20T17:59:43.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:43 smithi154 ceph-mon[36482]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-20T17:59:43.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:43 smithi129 ceph-mon[32029]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-20T17:59:43.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:43 smithi204 ceph-mon[37076]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-20T17:59:44.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:44 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-05-20T17:59:44.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:44 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]: dispatch 2024-05-20T17:59:44.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:44 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-05-20T17:59:44.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:44 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]: dispatch 2024-05-20T17:59:44.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:44 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-05-20T17:59:44.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:44 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]: dispatch 2024-05-20T17:59:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.581+0000 7ff29c6de640 1 -- 172.21.15.129:0/1617243374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29410d0a0 msgr2=0x7ff29410f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.581+0000 7ff29c6de640 1 --2- 172.21.15.129:0/1617243374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29410d0a0 0x7ff29410f590 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff28c009ca0 tx=0x7ff28c02a9e0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.581+0000 7ff29c6de640 1 -- 172.21.15.129:0/1617243374 shutdown_connections 2024-05-20T17:59:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.581+0000 7ff29c6de640 1 --2- 172.21.15.129:0/1617243374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29410a6f0 0x7ff29410cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.581+0000 7ff29c6de640 1 --2- 172.21.15.129:0/1617243374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29410d0a0 0x7ff29410f590 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.581+0000 7ff29c6de640 1 --2- 172.21.15.129:0/1617243374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff294103c90 0x7ff2941040a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.581+0000 7ff29c6de640 1 -- 172.21.15.129:0/1617243374 >> 172.21.15.129:0/1617243374 conn(0x7ff294078120 msgr2=0x7ff294078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.582+0000 7ff29c6de640 1 -- 172.21.15.129:0/1617243374 shutdown_connections 2024-05-20T17:59:44.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.582+0000 7ff29c6de640 1 -- 172.21.15.129:0/1617243374 wait complete. 2024-05-20T17:59:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.582+0000 7ff29c6de640 1 Processor -- start 2024-05-20T17:59:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.582+0000 7ff29c6de640 1 -- start start 2024-05-20T17:59:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff29c6de640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff294103c90 0x7ff2941997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff29c6de640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29410a6f0 0x7ff294199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff29c6de640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29410d0a0 0x7ff29419dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff29c6de640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff294112e50 con 0x7ff29410a6f0 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff29c6de640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff294112cd0 con 0x7ff29410d0a0 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff29c6de640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff294112fd0 con 0x7ff294103c90 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff299c79640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29410a6f0 0x7ff294199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff299c79640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29410a6f0 0x7ff294199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56820/0 (socket says 172.21.15.129:56820) 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff299c79640 1 -- 172.21.15.129:0/2411622862 learned_addr learned my addr 172.21.15.129:0/2411622862 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff29a47a640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff294103c90 0x7ff2941997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff29ac7b640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29410d0a0 0x7ff29419dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff299c79640 1 -- 172.21.15.129:0/2411622862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff294103c90 msgr2=0x7ff2941997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff299c79640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff294103c90 0x7ff2941997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff299c79640 1 -- 172.21.15.129:0/2411622862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29410d0a0 msgr2=0x7ff29419dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff299c79640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29410d0a0 0x7ff29419dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff299c79640 1 -- 172.21.15.129:0/2411622862 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29419e610 con 0x7ff29410a6f0 2024-05-20T17:59:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff29ac7b640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29410d0a0 0x7ff29419dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.583+0000 7ff299c79640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29410a6f0 0x7ff294199d10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff28400e9f0 tx=0x7ff28400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.584+0000 7ff28b7fe640 1 -- 172.21.15.129:0/2411622862 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28400cd80 con 0x7ff29410a6f0 2024-05-20T17:59:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.584+0000 7ff29c6de640 1 -- 172.21.15.129:0/2411622862 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29419e900 con 0x7ff29410a6f0 2024-05-20T17:59:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.584+0000 7ff29c6de640 1 -- 172.21.15.129:0/2411622862 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2941a6280 con 0x7ff29410a6f0 2024-05-20T17:59:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.584+0000 7ff28b7fe640 1 -- 172.21.15.129:0/2411622862 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff284010910 con 0x7ff29410a6f0 2024-05-20T17:59:44.585 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.584+0000 7ff28b7fe640 1 -- 172.21.15.129:0/2411622862 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2840188f0 con 0x7ff29410a6f0 2024-05-20T17:59:44.587 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.586+0000 7ff2a357e740 1 Processor -- start 2024-05-20T17:59:44.587 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff2a357e740 1 -- start start 2024-05-20T17:59:44.587 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9e31b0 0x55c10bd81f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd82450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9c24d0 0x55c10bd82990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff2a357e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c10bce7340 con 0x55c10b9ba2f0 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff2a357e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c10bc8f290 con 0x55c10b9c24d0 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff2a357e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c10bce3300 con 0x55c10b9e31b0 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff28b7fe640 1 -- 172.21.15.129:0/2411622862 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7ff284018bd0 con 0x7ff29410a6f0 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff29ac7b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9e31b0 0x55c10bd81f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff29ac7b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9e31b0 0x55c10bd81f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:47086/0 (socket says 172.21.15.129:47086) 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff29ac7b640 1 -- 172.21.15.129:0/1805359931 learned_addr learned my addr 172.21.15.129:0/1805359931 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.587+0000 7ff29a47a640 1 --2- 172.21.15.129:0/1805359931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd82450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.588+0000 7ff299c79640 1 --2- 172.21.15.129:0/1805359931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9c24d0 0x55c10bd82990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.588+0000 7ff29ac7b640 1 -- 172.21.15.129:0/1805359931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9c24d0 msgr2=0x55c10bd82990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.588+0000 7ff29ac7b640 1 --2- 172.21.15.129:0/1805359931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9c24d0 0x55c10bd82990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.588+0000 7ff29ac7b640 1 -- 172.21.15.129:0/1805359931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 msgr2=0x55c10bd82450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.588+0000 7ff29ac7b640 1 --2- 172.21.15.129:0/1805359931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd82450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.588+0000 7ff29ac7b640 1 -- 172.21.15.129:0/1805359931 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c10bcb38e0 con 0x55c10b9e31b0 2024-05-20T17:59:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.588+0000 7ff29ac7b640 1 --2- 172.21.15.129:0/1805359931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9e31b0 0x55c10bd81f10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0098a0 tx=0x7ff28c0098d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99bb0597a3eecf2c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.588+0000 7ff28b7fe640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2800a2e90 0x7ff2800a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.588+0000 7ff29a47a640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2800a2e90 0x7ff2800a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.589+0000 7ff28b7fe640 1 -- 172.21.15.129:0/2411622862 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8543+0+0 (secure 0 0 0) 0x7ff2840cf9f0 con 0x7ff29410a6f0 2024-05-20T17:59:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.589+0000 7ff288ff9640 1 -- 172.21.15.129:0/1805359931 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c047070 con 0x55c10b9e31b0 2024-05-20T17:59:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.589+0000 7ff288ff9640 1 -- 172.21.15.129:0/1805359931 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff28c003970 con 0x55c10b9e31b0 2024-05-20T17:59:44.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.589+0000 7ff288ff9640 1 -- 172.21.15.129:0/1805359931 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c033050 con 0x55c10b9e31b0 2024-05-20T17:59:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.590+0000 7ff2a357e740 1 -- 172.21.15.129:0/1805359931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9e31b0 msgr2=0x55c10bd81f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.590+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1805359931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9e31b0 0x55c10bd81f10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0098a0 tx=0x7ff28c0098d0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.590+0000 7ff29a47a640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2800a2e90 0x7ff2800a53a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55c10bd7f8e0 tx=0x7ff29000b4d0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.590+0000 7ff2a357e740 1 -- 172.21.15.129:0/1805359931 shutdown_connections 2024-05-20T17:59:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.590+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1805359931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9c24d0 0x55c10bd82990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.590+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1805359931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd82450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.590+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1805359931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9e31b0 0x55c10bd81f10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.590+0000 7ff2a357e740 1 -- 172.21.15.129:0/1805359931 >> 172.21.15.129:0/1805359931 conn(0x55c10b9d7300 msgr2=0x55c10bd7e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:44.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.590+0000 7ff2a357e740 1 -- 172.21.15.129:0/1805359931 shutdown_connections 2024-05-20T17:59:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.590+0000 7ff2a357e740 1 -- 172.21.15.129:0/1805359931 wait complete. 2024-05-20T17:59:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.591+0000 7ff2a357e740 1 -- start start 2024-05-20T17:59:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.591+0000 7ff2a357e740 1 Processor -- start 2024-05-20T17:59:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.591+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd92580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.591+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9c24d0 0x55c10bd92ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.591+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd90a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.591+0000 7ff2a357e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c10bd935c0 con 0x55c10b9ba2f0 2024-05-20T17:59:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.591+0000 7ff2a357e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c10bca2790 con 0x55c10b9e31b0 2024-05-20T17:59:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.591+0000 7ff2a357e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c10bd93740 con 0x55c10b9c24d0 2024-05-20T17:59:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff2a357e740 10 client.0 _ll_register_callbacks cb 0x55c10bcdda20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:44.594 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff29a47a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd92580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff29a47a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd92580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56836/0 (socket says 172.21.15.129:56836) 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff29a47a640 1 -- 172.21.15.129:0/1638064984 learned_addr learned my addr 172.21.15.129:0/1638064984 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff29ac7b640 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd90a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff299c79640 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9c24d0 0x55c10bd92ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff29a47a640 1 -- 172.21.15.129:0/1638064984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9c24d0 msgr2=0x55c10bd92ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff29a47a640 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9c24d0 0x55c10bd92ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff29a47a640 1 -- 172.21.15.129:0/1638064984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 msgr2=0x55c10bd90a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff29a47a640 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd90a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff29a47a640 1 -- 172.21.15.129:0/1638064984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c10b9fb210 con 0x55c10b9ba2f0 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff29a47a640 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd92580 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff290005ce0 tx=0x7ff2900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9b3cae88f0eee3 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff27dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:44.595 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.592+0000 7ff27dffb640 1 -- 172.21.15.129:0/1638064984 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29000f910 con 0x55c10b9ba2f0 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.593+0000 7ff27dffb640 1 -- 172.21.15.129:0/1638064984 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8543+0+0 (secure 0 0 0) 0x7ff29000fab0 con 0x55c10b9ba2f0 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.593+0000 7ff2a357e740 1 -- 172.21.15.129:0/1638064984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff28c003970 con 0x55c10b9ba2f0 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.593+0000 7ff27dffb640 1 -- 172.21.15.129:0/1638064984 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff29000fcb0 con 0x55c10b9ba2f0 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.593+0000 7ff2a357e740 10 client.15624 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.593+0000 7ff2a357e740 1 -- 172.21.15.129:0/1638064984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c10bac7620 con 0x55c10b9ba2f0 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.593+0000 7ff27dffb640 1 -- 172.21.15.129:0/1638064984 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff28c003970 con 0x55c10b9ba2f0 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.593+0000 7ff27dffb640 1 -- 172.21.15.129:0/1638064984 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7ff29001f480 con 0x55c10b9ba2f0 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.593+0000 7ff2a357e740 10 client.15624 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.593+0000 7ff2a357e740 10 client.15624 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.593+0000 7ff2a357e740 20 client.15624 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.593+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10b9a1d80 0x55c10bda2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.594+0000 7ff2a357e740 4 client.15624 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:44.596 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.594+0000 7ff2a357e740 1 -- 172.21.15.129:0/1638064984 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c10bcf4a70 con 0x55c10b9a1d80 2024-05-20T17:59:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.594+0000 7ff299c79640 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10b9a1d80 0x55c10bda2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.594+0000 7ff27dffb640 10 client.15624 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:44.597 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.594+0000 7ff299c79640 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10b9a1d80 0x55c10bda2f20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c00979bc5c0bb9d0 server_cookie=e350c57f0a6ccf04 in_seq=0 out_seq=0 2024-05-20T17:59:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.595+0000 7ff27dffb640 1 -- 172.21.15.129:0/1638064984 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c10bcf4a70 con 0x55c10b9a1d80 2024-05-20T17:59:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.595+0000 7ff27dffb640 10 client.15624 handle_command_reply: tid=0 2024-05-20T17:59:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.595+0000 7ff2a357e740 1 client.15624 shutdown 2024-05-20T17:59:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.595+0000 7ff2a357e740 1 -- 172.21.15.129:0/1638064984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 msgr2=0x55c10bd92580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.595+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd92580 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff290005ce0 tx=0x7ff2900046c0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.598 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.595+0000 7ff2a357e740 1 -- 172.21.15.129:0/1638064984 shutdown_connections 2024-05-20T17:59:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.595+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10b9a1d80 0x55c10bda2f20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.595+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd90a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.595+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9c24d0 0x55c10bd92ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.595+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1638064984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd92580 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.595+0000 7ff2a357e740 1 -- 172.21.15.129:0/1638064984 >> 172.21.15.129:0/1638064984 conn(0x55c10b9d7300 msgr2=0x55c10bd83d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.596+0000 7ff27dffb640 0 client.15624 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.596+0000 7ff2a357e740 1 -- 172.21.15.129:0/1638064984 shutdown_connections 2024-05-20T17:59:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.596+0000 7ff2a357e740 1 -- 172.21.15.129:0/1638064984 wait complete. 2024-05-20T17:59:44.599 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.596+0000 7ff2a357e740 20 client.15624 trim_cache size 0 max 16384 2024-05-20T17:59:44.611 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.611+0000 7ff2a357e740 1 Processor -- start 2024-05-20T17:59:44.611 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.611+0000 7ff2a357e740 1 -- start start 2024-05-20T17:59:44.611 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.611+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9ba2f0 0x55c10bd92b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.611 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.611+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9c24d0 0x55c10bd82520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.611 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.611+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd82a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.611 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff2a357e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff27800c7b0 con 0x55c10b9c24d0 2024-05-20T17:59:44.611 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff2a357e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff27800f030 con 0x55c10b9e31b0 2024-05-20T17:59:44.611 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff2a357e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff27800ce20 con 0x55c10b9ba2f0 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff299c79640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9c24d0 0x55c10bd82520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff299c79640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9c24d0 0x55c10bd82520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56844/0 (socket says 172.21.15.129:56844) 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff299c79640 1 -- 172.21.15.129:0/978541141 learned_addr learned my addr 172.21.15.129:0/978541141 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff29a47a640 1 --2- 172.21.15.129:0/978541141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9ba2f0 0x55c10bd92b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff29ac7b640 1 --2- 172.21.15.129:0/978541141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd82a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff299c79640 1 -- 172.21.15.129:0/978541141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9ba2f0 msgr2=0x55c10bd92b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff299c79640 1 --2- 172.21.15.129:0/978541141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9ba2f0 0x55c10bd92b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff299c79640 1 -- 172.21.15.129:0/978541141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 msgr2=0x55c10bd82a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff299c79640 1 --2- 172.21.15.129:0/978541141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd82a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff299c79640 1 -- 172.21.15.129:0/978541141 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c10b9a1fb0 con 0x55c10b9c24d0 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff29ac7b640 1 --2- 172.21.15.129:0/978541141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd82a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:44.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff299c79640 1 --2- 172.21.15.129:0/978541141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9c24d0 0x55c10bd82520 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff284018500 tx=0x7ff2840d1f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a7d4170c0bd216 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:44.613 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.612+0000 7ff27ffff640 1 -- 172.21.15.129:0/978541141 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2840a0060 con 0x55c10b9c24d0 2024-05-20T17:59:44.613 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.613+0000 7ff27ffff640 1 -- 172.21.15.129:0/978541141 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2840a4920 con 0x55c10b9c24d0 2024-05-20T17:59:44.613 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.613+0000 7ff27ffff640 1 -- 172.21.15.129:0/978541141 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28409b100 con 0x55c10b9c24d0 2024-05-20T17:59:44.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.614+0000 7ff2a357e740 1 -- 172.21.15.129:0/978541141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9c24d0 msgr2=0x55c10bd82520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:44.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.614+0000 7ff2a357e740 1 --2- 172.21.15.129:0/978541141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9c24d0 0x55c10bd82520 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff284018500 tx=0x7ff2840d1f00 comp rx=0 tx=0).stop 2024-05-20T17:59:44.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.614+0000 7ff2a357e740 1 -- 172.21.15.129:0/978541141 shutdown_connections 2024-05-20T17:59:44.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.614+0000 7ff2a357e740 1 --2- 172.21.15.129:0/978541141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd82a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.614+0000 7ff2a357e740 1 --2- 172.21.15.129:0/978541141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9c24d0 0x55c10bd82520 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.614+0000 7ff2a357e740 1 --2- 172.21.15.129:0/978541141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9ba2f0 0x55c10bd92b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.614+0000 7ff2a357e740 1 -- 172.21.15.129:0/978541141 >> 172.21.15.129:0/978541141 conn(0x55c10b9d7300 msgr2=0x55c10bd81e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:44.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.614+0000 7ff2a357e740 1 -- 172.21.15.129:0/978541141 shutdown_connections 2024-05-20T17:59:44.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.614+0000 7ff2a357e740 1 -- 172.21.15.129:0/978541141 wait complete. 2024-05-20T17:59:44.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.614+0000 7ff2a357e740 1 -- start start 2024-05-20T17:59:44.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.615+0000 7ff2a357e740 1 Processor -- start 2024-05-20T17:59:44.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.615+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd98ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.615+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9c24d0 0x55c10bd96280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.615+0000 7ff2a357e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd967c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.615+0000 7ff2a357e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff27800d4f0 con 0x55c10b9ba2f0 2024-05-20T17:59:44.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.615+0000 7ff29a47a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd98ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.615+0000 7ff29a47a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd98ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56848/0 (socket says 172.21.15.129:56848) 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.615+0000 7ff29a47a640 1 -- 172.21.15.129:0/1845943608 learned_addr learned my addr 172.21.15.129:0/1845943608 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.615+0000 7ff2a357e740 1 -- 172.21.15.129:0/1845943608 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff27800e960 con 0x55c10b9e31b0 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.615+0000 7ff2a357e740 1 -- 172.21.15.129:0/1845943608 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff27800dbc0 con 0x55c10b9c24d0 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.615+0000 7ff29ac7b640 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff299c79640 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9c24d0 0x55c10bd96280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff2a357e740 10 client.0 _ll_register_callbacks cb 0x7ff29000b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff29a47a640 1 -- 172.21.15.129:0/1845943608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9c24d0 msgr2=0x55c10bd96280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff29a47a640 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9c24d0 0x55c10bd96280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff29a47a640 1 -- 172.21.15.129:0/1845943608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 msgr2=0x55c10bd967c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff29a47a640 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd967c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff29a47a640 1 -- 172.21.15.129:0/1845943608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c10bd96dd0 con 0x55c10b9ba2f0 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff29ac7b640 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd967c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff299c79640 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9c24d0 0x55c10bd96280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff29a47a640 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd98ec0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff290005ce0 tx=0x7ff29001f620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20f8c392d431b6f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff27dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff27dffb640 1 -- 172.21.15.129:0/1845943608 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff290034050 con 0x55c10b9ba2f0 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff27dffb640 1 -- 172.21.15.129:0/1845943608 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8543+0+0 (secure 0 0 0) 0x7ff29003b450 con 0x55c10b9ba2f0 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.616+0000 7ff2a357e740 1 -- 172.21.15.129:0/1845943608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2840a4920 con 0x55c10b9ba2f0 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.617+0000 7ff27dffb640 1 -- 172.21.15.129:0/1845943608 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff290048050 con 0x55c10b9ba2f0 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.617+0000 7ff2a357e740 10 client.15630 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.617+0000 7ff2a357e740 1 -- 172.21.15.129:0/1845943608 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55c10be0b040 con 0x55c10b9ba2f0 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.617+0000 7ff27dffb640 1 -- 172.21.15.129:0/1845943608 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2840a4920 con 0x55c10b9ba2f0 2024-05-20T17:59:44.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.618+0000 7ff27dffb640 1 -- 172.21.15.129:0/1845943608 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7ff290043030 con 0x55c10b9ba2f0 2024-05-20T17:59:44.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.618+0000 7ff2a357e740 10 client.15630 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:44.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.618+0000 7ff2a357e740 10 client.15630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:44.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.618+0000 7ff2a357e740 20 client.15630 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:44.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.618+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10b9771c0 0x55c10be0d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:44.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.618+0000 7ff2a357e740 4 client.15630 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T17:59:44.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.618+0000 7ff2a357e740 1 -- 172.21.15.129:0/1845943608 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c10bceedc0 con 0x55c10b9771c0 2024-05-20T17:59:44.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.618+0000 7ff299c79640 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10b9771c0 0x55c10be0d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:44.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.619+0000 7ff27dffb640 10 client.15630 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:44.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.619+0000 7ff299c79640 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10b9771c0 0x55c10be0d820 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ae9ef8c7d3587f7 server_cookie=41a6b7e34a4b9a8e in_seq=0 out_seq=0 2024-05-20T17:59:44.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff27dffb640 1 -- 172.21.15.129:0/1845943608 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c10bceedc0 con 0x55c10b9771c0 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff27dffb640 10 client.15630 handle_command_reply: tid=0 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 1 client.15630 shutdown 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 1 -- 172.21.15.129:0/1845943608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 msgr2=0x55c10bd98ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd98ec0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff290005ce0 tx=0x7ff29001f620 comp rx=0 tx=0).stop 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 1 -- 172.21.15.129:0/1845943608 shutdown_connections 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10b9771c0 0x55c10be0d820 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10b9e31b0 0x55c10bd967c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10b9c24d0 0x55c10bd96280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 1 --2- 172.21.15.129:0/1845943608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10b9ba2f0 0x55c10bd98ec0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 1 -- 172.21.15.129:0/1845943608 >> 172.21.15.129:0/1845943608 conn(0x55c10b9d7300 msgr2=0x55c10bd928f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff27dffb640 0 client.15630 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 1 -- 172.21.15.129:0/1845943608 shutdown_connections 2024-05-20T17:59:44.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 1 -- 172.21.15.129:0/1845943608 wait complete. 2024-05-20T17:59:44.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.623+0000 7ff2a357e740 20 client.15630 trim_cache size 0 max 16384 2024-05-20T17:59:44.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T17:59:44.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T17:59:44.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T17:59:44.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T17:59:44.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 -- 172.21.15.129:0/2411622862 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2800a2e90 msgr2=0x7ff2800a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:44.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2800a2e90 0x7ff2800a53a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55c10bd7f8e0 tx=0x7ff29000b4d0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 -- 172.21.15.129:0/2411622862 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29410a6f0 msgr2=0x7ff294199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:44.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29410a6f0 0x7ff294199d10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff28400e9f0 tx=0x7ff28400eeb0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 -- 172.21.15.129:0/2411622862 shutdown_connections 2024-05-20T17:59:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2800a2e90 0x7ff2800a53a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff29410d0a0 0x7ff29419dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff29410a6f0 0x7ff294199d10 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 --2- 172.21.15.129:0/2411622862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff294103c90 0x7ff2941997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 -- 172.21.15.129:0/2411622862 >> 172.21.15.129:0/2411622862 conn(0x7ff294078120 msgr2=0x7ff2941000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 -- 172.21.15.129:0/2411622862 shutdown_connections 2024-05-20T17:59:44.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:44.624+0000 7ff29c6de640 1 -- 172.21.15.129:0/2411622862 wait complete. 2024-05-20T17:59:45.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:45 smithi154 ceph-mon[36482]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-20T17:59:45.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:45 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-05-20T17:59:45.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:45 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]': finished 2024-05-20T17:59:45.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:45 smithi154 ceph-mon[36482]: osdmap e91: 12 total, 12 up, 12 in 2024-05-20T17:59:45.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:45 smithi154 ceph-mon[36482]: osdmap e92: 12 total, 12 up, 12 in 2024-05-20T17:59:45.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:45 smithi129 ceph-mon[32029]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-20T17:59:45.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:45 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-05-20T17:59:45.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:45 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]': finished 2024-05-20T17:59:45.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:45 smithi129 ceph-mon[32029]: osdmap e91: 12 total, 12 up, 12 in 2024-05-20T17:59:45.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:45 smithi129 ceph-mon[32029]: osdmap e92: 12 total, 12 up, 12 in 2024-05-20T17:59:45.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:45 smithi204 ceph-mon[37076]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-20T17:59:45.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:45 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-05-20T17:59:45.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:45 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]': finished 2024-05-20T17:59:45.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:45 smithi204 ceph-mon[37076]: osdmap e91: 12 total, 12 up, 12 in 2024-05-20T17:59:45.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:45 smithi204 ceph-mon[37076]: osdmap e92: 12 total, 12 up, 12 in 2024-05-20T17:59:47.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:46 smithi129 ceph-mon[32029]: pgmap v37: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:59:47.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:46 smithi129 ceph-mon[32029]: osdmap e93: 12 total, 12 up, 12 in 2024-05-20T17:59:47.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:46 smithi129 ceph-mon[32029]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-20T17:59:47.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:46 smithi204 ceph-mon[37076]: pgmap v37: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:59:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:46 smithi204 ceph-mon[37076]: osdmap e93: 12 total, 12 up, 12 in 2024-05-20T17:59:47.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:46 smithi204 ceph-mon[37076]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-20T17:59:47.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:46 smithi154 ceph-mon[36482]: pgmap v37: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:59:47.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:46 smithi154 ceph-mon[36482]: osdmap e93: 12 total, 12 up, 12 in 2024-05-20T17:59:47.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:46 smithi154 ceph-mon[36482]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-20T17:59:48.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:47 smithi204 ceph-mon[37076]: osdmap e94: 12 total, 12 up, 12 in 2024-05-20T17:59:48.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:47 smithi154 ceph-mon[36482]: osdmap e94: 12 total, 12 up, 12 in 2024-05-20T17:59:48.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:47 smithi129 ceph-mon[32029]: osdmap e94: 12 total, 12 up, 12 in 2024-05-20T17:59:49.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:48 smithi204 ceph-mon[37076]: pgmap v40: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:59:49.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:48 smithi204 ceph-mon[37076]: osdmap e95: 12 total, 12 up, 12 in 2024-05-20T17:59:49.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:48 smithi154 ceph-mon[36482]: pgmap v40: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:59:49.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:48 smithi154 ceph-mon[36482]: osdmap e95: 12 total, 12 up, 12 in 2024-05-20T17:59:49.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:48 smithi129 ceph-mon[32029]: pgmap v40: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-05-20T17:59:49.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:48 smithi129 ceph-mon[32029]: osdmap e95: 12 total, 12 up, 12 in 2024-05-20T17:59:50.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T17:59:50.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T17:59:50.180 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:59:50.180 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:59:50.180 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:49 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:59:50.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.297+0000 7f056d691640 1 -- 172.21.15.129:0/1858972833 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05681013b0 msgr2=0x7f05681017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.297+0000 7f056d691640 1 --2- 172.21.15.129:0/1858972833 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05681013b0 0x7f05681017c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0558009900 tx=0x7f055802b500 comp rx=0 tx=0).stop 2024-05-20T17:59:50.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.297+0000 7f056d691640 1 -- 172.21.15.129:0/1858972833 shutdown_connections 2024-05-20T17:59:50.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.297+0000 7f056d691640 1 --2- 172.21.15.129:0/1858972833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0568105940 0x7f0568105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.297+0000 7f056d691640 1 --2- 172.21.15.129:0/1858972833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0568106400 0x7f056810dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.297+0000 7f056d691640 1 --2- 172.21.15.129:0/1858972833 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f05681013b0 0x7f05681017c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.297+0000 7f056d691640 1 -- 172.21.15.129:0/1858972833 >> 172.21.15.129:0/1858972833 conn(0x7f0568076830 msgr2=0x7f0568076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:50.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.297+0000 7f056d691640 1 -- 172.21.15.129:0/1858972833 shutdown_connections 2024-05-20T17:59:50.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.297+0000 7f056d691640 1 -- 172.21.15.129:0/1858972833 wait complete. 2024-05-20T17:59:50.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f056d691640 1 Processor -- start 2024-05-20T17:59:50.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f056d691640 1 -- start start 2024-05-20T17:59:50.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f056d691640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05681013b0 0x7f05680fd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f056d691640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0568105940 0x7f05680fde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f056d691640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0568106400 0x7f05680fe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f056d691640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0568110be0 con 0x7f05681013b0 2024-05-20T17:59:50.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f056d691640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0568110a60 con 0x7f0568106400 2024-05-20T17:59:50.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f056d691640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0568110d60 con 0x7f0568105940 2024-05-20T17:59:50.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f0566ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05681013b0 0x7f05680fd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f0566ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05681013b0 0x7f05680fd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52670/0 (socket says 172.21.15.129:52670) 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f0566ffd640 1 -- 172.21.15.129:0/1878458668 learned_addr learned my addr 172.21.15.129:0/1878458668 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f05677fe640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0568106400 0x7f05680fe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.298+0000 7f05667fc640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0568105940 0x7f05680fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f0566ffd640 1 -- 172.21.15.129:0/1878458668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0568105940 msgr2=0x7f05680fde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f0566ffd640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0568105940 0x7f05680fde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f0566ffd640 1 -- 172.21.15.129:0/1878458668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0568106400 msgr2=0x7f05680fe3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f0566ffd640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0568106400 0x7f05680fe3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f0566ffd640 1 -- 172.21.15.129:0/1878458668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05681a7c00 con 0x7f05681013b0 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f05677fe640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0568106400 0x7f05680fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f0566ffd640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05681013b0 0x7f05680fd930 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0558006010 tx=0x7f0558005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f054ffff640 1 -- 172.21.15.129:0/1878458668 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558041070 con 0x7f05681013b0 2024-05-20T17:59:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f056d691640 1 -- 172.21.15.129:0/1878458668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05681a7e90 con 0x7f05681013b0 2024-05-20T17:59:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f056d691640 1 -- 172.21.15.129:0/1878458668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05681a82f0 con 0x7f05681013b0 2024-05-20T17:59:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f054ffff640 1 -- 172.21.15.129:0/1878458668 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0558003ad0 con 0x7f05681013b0 2024-05-20T17:59:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.299+0000 7f054ffff640 1 -- 172.21.15.129:0/1878458668 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558036040 con 0x7f05681013b0 2024-05-20T17:59:50.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f0574610740 1 Processor -- start 2024-05-20T17:59:50.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f0574610740 1 -- start start 2024-05-20T17:59:50.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f054ffff640 1 -- 172.21.15.129:0/1878458668 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f0558044070 con 0x7f05681013b0 2024-05-20T17:59:50.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f0574610740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2aa390 0x55c10c651540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f0574610740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2814d0 0x55c10c651a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f0574610740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2896b0 0x55c10c651fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f0574610740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c10c5aed00 con 0x55c10c2814d0 2024-05-20T17:59:50.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f0574610740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c10c5564c0 con 0x55c10c2aa390 2024-05-20T17:59:50.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f0574610740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c10c5aacc0 con 0x55c10c2896b0 2024-05-20T17:59:50.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f05667fc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2896b0 0x55c10c651fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f0566ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2814d0 0x55c10c651a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f05667fc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2896b0 0x55c10c651fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50194/0 (socket says 172.21.15.129:50194) 2024-05-20T17:59:50.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f05667fc640 1 -- 172.21.15.129:0/4223806940 learned_addr learned my addr 172.21.15.129:0/4223806940 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:50.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.303+0000 7f05677fe640 1 --2- 172.21.15.129:0/4223806940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2aa390 0x55c10c651540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.304+0000 7f05677fe640 1 -- 172.21.15.129:0/4223806940 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2896b0 msgr2=0x55c10c651fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:50.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.304+0000 7f05677fe640 1 --2- 172.21.15.129:0/4223806940 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2896b0 0x55c10c651fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.304+0000 7f05677fe640 1 -- 172.21.15.129:0/4223806940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2814d0 msgr2=0x55c10c651a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:50.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.304+0000 7f05677fe640 1 --2- 172.21.15.129:0/4223806940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2814d0 0x55c10c651a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.304+0000 7f05677fe640 1 -- 172.21.15.129:0/4223806940 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c10c38c7d0 con 0x55c10c2aa390 2024-05-20T17:59:50.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.304+0000 7f05667fc640 1 --2- 172.21.15.129:0/4223806940 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2896b0 0x55c10c651fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:50.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.304+0000 7f0566ffd640 1 --2- 172.21.15.129:0/4223806940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2814d0 0x55c10c651a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:50.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.304+0000 7f054ffff640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f05500a2f20 0x7f05500a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.304+0000 7f05677fe640 1 --2- 172.21.15.129:0/4223806940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2aa390 0x55c10c651540 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0560009f10 tx=0x7f05600107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d5f953026faba5e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:50.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.304+0000 7f054ffff640 1 -- 172.21.15.129:0/1878458668 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f05580ef860 con 0x7f05681013b0 2024-05-20T17:59:50.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.304+0000 7f0566ffd640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f05500a2f20 0x7f05500a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.305+0000 7f054d7fa640 1 -- 172.21.15.129:0/4223806940 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056001c680 con 0x55c10c2aa390 2024-05-20T17:59:50.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.305+0000 7f054d7fa640 1 -- 172.21.15.129:0/4223806940 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f056001cd00 con 0x55c10c2aa390 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.305+0000 7f054d7fa640 1 -- 172.21.15.129:0/4223806940 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056001a380 con 0x55c10c2aa390 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.306+0000 7f0574610740 1 -- 172.21.15.129:0/4223806940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2aa390 msgr2=0x55c10c651540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.306+0000 7f0566ffd640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f05500a2f20 0x7f05500a5430 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f05580ece20 tx=0x7f05580f5530 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.306+0000 7f0574610740 1 --2- 172.21.15.129:0/4223806940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2aa390 0x55c10c651540 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0560009f10 tx=0x7f05600107a0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.306+0000 7f0574610740 1 -- 172.21.15.129:0/4223806940 shutdown_connections 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.306+0000 7f0574610740 1 --2- 172.21.15.129:0/4223806940 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2896b0 0x55c10c651fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.306+0000 7f0574610740 1 --2- 172.21.15.129:0/4223806940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2814d0 0x55c10c651a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.306+0000 7f0574610740 1 --2- 172.21.15.129:0/4223806940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2aa390 0x55c10c651540 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.306+0000 7f0574610740 1 -- 172.21.15.129:0/4223806940 >> 172.21.15.129:0/4223806940 conn(0x55c10c29e4e0 msgr2=0x55c10c64dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.306+0000 7f0574610740 1 -- 172.21.15.129:0/4223806940 shutdown_connections 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.306+0000 7f0574610740 1 -- 172.21.15.129:0/4223806940 wait complete. 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.306+0000 7f0574610740 1 -- start start 2024-05-20T17:59:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.307+0000 7f0574610740 1 Processor -- start 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.307+0000 7f0574610740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 0x55c10c65bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.307+0000 7f0574610740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 0x55c10c65c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.307+0000 7f05667fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 0x55c10c65c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.307+0000 7f05677fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 0x55c10c65bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f05677fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 0x55c10c65bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50202/0 (socket says 172.21.15.129:50202) 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f05667fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 0x55c10c65c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52684/0 (socket says 172.21.15.129:52684) 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f05677fe640 1 -- 172.21.15.129:0/3744083627 learned_addr learned my addr 172.21.15.129:0/3744083627 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f0574610740 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2aa390 0x55c10c667de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f0574610740 1 -- 172.21.15.129:0/3744083627 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c10c65e8e0 con 0x55c10c2896b0 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f0574610740 1 -- 172.21.15.129:0/3744083627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c10c569b90 con 0x55c10c2aa390 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f0574610740 1 -- 172.21.15.129:0/3744083627 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c10c65ea60 con 0x55c10c2814d0 2024-05-20T17:59:50.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f05677fe640 1 -- 172.21.15.129:0/3744083627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2aa390 msgr2=0x55c10c667de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f05677fe640 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2aa390 0x55c10c667de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f05677fe640 1 -- 172.21.15.129:0/3744083627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 msgr2=0x55c10c65c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f05677fe640 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 0x55c10c65c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f05677fe640 1 -- 172.21.15.129:0/3744083627 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c10c2c23f0 con 0x55c10c2814d0 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f05667fc640 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 0x55c10c65c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f0574610740 10 client.0 _ll_register_callbacks cb 0x55c10c5a5da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f0574610740 1 -- 172.21.15.129:0/3744083627 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f056001cd00 con 0x55c10c2814d0 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f05677fe640 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 0x55c10c65bb00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0560000f80 tx=0x7f0560009b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=865f9d3cc51bbe5d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.308+0000 7f054aff5640 10 client.25342 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.309+0000 7f054aff5640 1 -- 172.21.15.129:0/3744083627 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0560004e50 con 0x55c10c2814d0 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.309+0000 7f054aff5640 1 -- 172.21.15.129:0/3744083627 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f056001c480 con 0x55c10c2814d0 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.309+0000 7f054aff5640 1 -- 172.21.15.129:0/3744083627 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f056001c6c0 con 0x55c10c2814d0 2024-05-20T17:59:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.309+0000 7f0574610740 10 client.25342 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.309+0000 7f0574610740 1 -- 172.21.15.129:0/3744083627 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({fsmap=22}) -- 0x55c10c2820f0 con 0x55c10c2814d0 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.309+0000 7f054aff5640 1 -- 172.21.15.129:0/3744083627 <== mon.1 v2:172.21.15.204:3300/0 4 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f056002f030 con 0x55c10c2814d0 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.309+0000 7f0574610740 10 client.25342 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.309+0000 7f0574610740 10 client.25342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.310+0000 7f0574610740 20 client.25342 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.310+0000 7f0574610740 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10c268f60 0x55c10c66e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.310+0000 7f0574610740 4 client.25342 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.310+0000 7f0574610740 1 -- 172.21.15.129:0/3744083627 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c10c5bc430 con 0x55c10c268f60 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.310+0000 7f05667fc640 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10c268f60 0x55c10c66e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.310+0000 7f054aff5640 10 client.25342 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.310+0000 7f05667fc640 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10c268f60 0x55c10c66e020 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bca07c1d25e9c7b4 server_cookie=b02dfc05f544ada4 in_seq=0 out_seq=0 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f054aff5640 1 -- 172.21.15.129:0/3744083627 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c10c5bc430 con 0x55c10c268f60 2024-05-20T17:59:50.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f054aff5640 10 client.25342 handle_command_reply: tid=0 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f0574610740 1 client.25342 shutdown 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f0574610740 1 -- 172.21.15.129:0/3744083627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 msgr2=0x55c10c65bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f0574610740 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 0x55c10c65bb00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0560000f80 tx=0x7f0560009b70 comp rx=0 tx=0).stop 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f0574610740 1 -- 172.21.15.129:0/3744083627 shutdown_connections 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f0574610740 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10c268f60 0x55c10c66e020 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f054aff5640 0 client.25342 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f0574610740 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2aa390 0x55c10c667de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f0574610740 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 0x55c10c65c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f0574610740 1 --2- 172.21.15.129:0/3744083627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 0x55c10c65bb00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.311+0000 7f0574610740 1 -- 172.21.15.129:0/3744083627 >> 172.21.15.129:0/3744083627 conn(0x55c10c29e4e0 msgr2=0x55c10c651370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.312+0000 7f0574610740 1 -- 172.21.15.129:0/3744083627 shutdown_connections 2024-05-20T17:59:50.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.312+0000 7f0574610740 1 -- 172.21.15.129:0/3744083627 wait complete. 2024-05-20T17:59:50.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.312+0000 7f0574610740 20 client.25342 trim_cache size 0 max 16384 2024-05-20T17:59:50.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.333+0000 7f0574610740 1 Processor -- start 2024-05-20T17:59:50.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.333+0000 7f0574610740 1 -- start start 2024-05-20T17:59:50.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.333+0000 7f0574610740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c268f60 0x55c10c65c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.333+0000 7f0574610740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2814d0 0x55c10c65d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.333+0000 7f0574610740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 0x55c10c6679c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.333+0000 7f0574610740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f055400c720 con 0x55c10c2896b0 2024-05-20T17:59:50.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.333+0000 7f0574610740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f055400efa0 con 0x55c10c2814d0 2024-05-20T17:59:50.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.333+0000 7f0574610740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f055400cd90 con 0x55c10c268f60 2024-05-20T17:59:50.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05677fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c268f60 0x55c10c65c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05677fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c268f60 0x55c10c65c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50210/0 (socket says 172.21.15.129:50210) 2024-05-20T17:59:50.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05677fe640 1 -- 172.21.15.129:0/80890650 learned_addr learned my addr 172.21.15.129:0/80890650 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:50.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05677fe640 1 --2- 172.21.15.129:0/80890650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 0x55c10c6679c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05667fc640 1 --2- 172.21.15.129:0/80890650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2814d0 0x55c10c65d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05667fc640 1 -- 172.21.15.129:0/80890650 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c268f60 msgr2=0x55c10c65c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:50.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05667fc640 1 --2- 172.21.15.129:0/80890650 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c268f60 0x55c10c65c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05667fc640 1 -- 172.21.15.129:0/80890650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 msgr2=0x55c10c6679c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:50.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05667fc640 1 --2- 172.21.15.129:0/80890650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 0x55c10c6679c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05667fc640 1 -- 172.21.15.129:0/80890650 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c10c668dc0 con 0x55c10c2814d0 2024-05-20T17:59:50.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05677fe640 1 --2- 172.21.15.129:0/80890650 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c268f60 0x55c10c65c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:50.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f05667fc640 1 --2- 172.21.15.129:0/80890650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2814d0 0x55c10c65d720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x55c10c66e760 tx=0x7f055c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=432bb63f48e636b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:50.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f054cff9640 1 -- 172.21.15.129:0/80890650 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c005220 con 0x55c10c2814d0 2024-05-20T17:59:50.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f054cff9640 1 -- 172.21.15.129:0/80890650 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f055c025030 con 0x55c10c2814d0 2024-05-20T17:59:50.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.334+0000 7f054cff9640 1 -- 172.21.15.129:0/80890650 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0214c0 con 0x55c10c2814d0 2024-05-20T17:59:50.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.335+0000 7f0574610740 1 -- 172.21.15.129:0/80890650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2814d0 msgr2=0x55c10c65d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.335+0000 7f0574610740 1 --2- 172.21.15.129:0/80890650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2814d0 0x55c10c65d720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x55c10c66e760 tx=0x7f055c005f90 comp rx=0 tx=0).stop 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.335+0000 7f0574610740 1 -- 172.21.15.129:0/80890650 shutdown_connections 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.335+0000 7f0574610740 1 --2- 172.21.15.129:0/80890650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2814d0 0x55c10c65d720 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.335+0000 7f0574610740 1 --2- 172.21.15.129:0/80890650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c2896b0 0x55c10c6679c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.335+0000 7f0574610740 1 --2- 172.21.15.129:0/80890650 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c268f60 0x55c10c65c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.335+0000 7f0574610740 1 -- 172.21.15.129:0/80890650 >> 172.21.15.129:0/80890650 conn(0x55c10c29e4e0 msgr2=0x55c10c65d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.335+0000 7f0574610740 1 -- 172.21.15.129:0/80890650 shutdown_connections 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.336+0000 7f0574610740 1 -- 172.21.15.129:0/80890650 wait complete. 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.336+0000 7f0574610740 1 -- start start 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.336+0000 7f0574610740 1 Processor -- start 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.336+0000 7f0574610740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c268f60 0x55c10c65ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.336+0000 7f0574610740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 0x55c10c65f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f05677fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c268f60 0x55c10c65ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f05677fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c268f60 0x55c10c65ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52694/0 (socket says 172.21.15.129:52694) 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f05677fe640 1 -- 172.21.15.129:0/2873482486 learned_addr learned my addr 172.21.15.129:0/2873482486 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f0574610740 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2896b0 0x55c10c6d43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f0574610740 1 -- 172.21.15.129:0/2873482486 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f055400d460 con 0x55c10c268f60 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f0574610740 1 -- 172.21.15.129:0/2873482486 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f055400e8d0 con 0x55c10c2896b0 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f0574610740 1 -- 172.21.15.129:0/2873482486 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f055400db30 con 0x55c10c2814d0 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f05667fc640 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 0x55c10c65f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f0574610740 10 client.0 _ll_register_callbacks cb 0x7f05580f5400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f05677fe640 1 -- 172.21.15.129:0/2873482486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 msgr2=0x55c10c65f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f05677fe640 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 0x55c10c65f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f05677fe640 1 -- 172.21.15.129:0/2873482486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2896b0 msgr2=0x55c10c6d43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f05677fe640 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2896b0 0x55c10c6d43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f05677fe640 1 -- 172.21.15.129:0/2873482486 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c10c65fb00 con 0x55c10c268f60 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f0574610740 1 -- 172.21.15.129:0/2873482486 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f055c025030 con 0x55c10c268f60 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f05677fe640 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c268f60 0x55c10c65ed00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0560005cc0 tx=0x7f0560009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793d6eb57b6aa5a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.337+0000 7f054aff5640 10 client.15648 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f054aff5640 1 -- 172.21.15.129:0/2873482486 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0560041020 con 0x55c10c268f60 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f054aff5640 1 -- 172.21.15.129:0/2873482486 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f056001c480 con 0x55c10c268f60 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f054aff5640 1 -- 172.21.15.129:0/2873482486 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0560009b30 con 0x55c10c268f60 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f0574610740 10 client.15648 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f0574610740 1 -- 172.21.15.129:0/2873482486 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({fsmap=22}) -- 0x55c10c6d58b0 con 0x55c10c268f60 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f054aff5640 1 -- 172.21.15.129:0/2873482486 <== mon.0 v2:172.21.15.129:3300/0 4 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f056002f040 con 0x55c10c268f60 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f0574610740 10 client.15648 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f0574610740 10 client.15648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f0574610740 20 client.15648 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f0574610740 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10c2aa390 0x55c10c6d7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f0574610740 4 client.15648 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f0574610740 1 -- 172.21.15.129:0/2873482486 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c10c5b6780 con 0x55c10c2aa390 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.338+0000 7f05667fc640 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10c2aa390 0x55c10c6d7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f054aff5640 10 client.15648 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f05667fc640 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10c2aa390 0x55c10c6d7f00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a394e57cfa7c021e server_cookie=ec96a921fc1d0ea5 in_seq=0 out_seq=0 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f054aff5640 1 -- 172.21.15.129:0/2873482486 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c10c5b6780 con 0x55c10c2aa390 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f054aff5640 10 client.15648 handle_command_reply: tid=0 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f0574610740 1 client.15648 shutdown 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f0574610740 1 -- 172.21.15.129:0/2873482486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c268f60 msgr2=0x55c10c65ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f0574610740 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c268f60 0x55c10c65ed00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0560005cc0 tx=0x7f0560009500 comp rx=0 tx=0).stop 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f0574610740 1 -- 172.21.15.129:0/2873482486 shutdown_connections 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f0574610740 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c10c2aa390 0x55c10c6d7f00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f0574610740 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c10c2896b0 0x55c10c6d43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f0574610740 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c10c2814d0 0x55c10c65f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f0574610740 1 --2- 172.21.15.129:0/2873482486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c10c268f60 0x55c10c65ed00 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f0574610740 1 -- 172.21.15.129:0/2873482486 >> 172.21.15.129:0/2873482486 conn(0x55c10c29e4e0 msgr2=0x55c10c659d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:50.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.339+0000 7f054aff5640 0 client.15648 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:50.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f0574610740 1 -- 172.21.15.129:0/2873482486 shutdown_connections 2024-05-20T17:59:50.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f0574610740 1 -- 172.21.15.129:0/2873482486 wait complete. 2024-05-20T17:59:50.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f0574610740 20 client.15648 trim_cache size 0 max 16384 2024-05-20T17:59:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T17:59:50.343 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T17:59:50.344 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T17:59:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T17:59:50.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f056d691640 1 -- 172.21.15.129:0/1878458668 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f05500a2f20 msgr2=0x7f05500a5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f056d691640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f05500a2f20 0x7f05500a5430 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f05580ece20 tx=0x7f05580f5530 comp rx=0 tx=0).stop 2024-05-20T17:59:50.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f056d691640 1 -- 172.21.15.129:0/1878458668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05681013b0 msgr2=0x7f05680fd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f056d691640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05681013b0 0x7f05680fd930 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0558006010 tx=0x7f0558005dc0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f056d691640 1 -- 172.21.15.129:0/1878458668 shutdown_connections 2024-05-20T17:59:50.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f056d691640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f05500a2f20 0x7f05500a5430 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f056d691640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0568106400 0x7f05680fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f056d691640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0568105940 0x7f05680fde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f056d691640 1 --2- 172.21.15.129:0/1878458668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05681013b0 0x7f05680fd930 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.340+0000 7f056d691640 1 -- 172.21.15.129:0/1878458668 >> 172.21.15.129:0/1878458668 conn(0x7f0568076830 msgr2=0x7f056810cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:50.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.341+0000 7f056d691640 1 -- 172.21.15.129:0/1878458668 shutdown_connections 2024-05-20T17:59:50.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.341+0000 7f056d691640 1 -- 172.21.15.129:0/1878458668 wait complete. 2024-05-20T17:59:50.354 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T17:59:50.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T17:59:50.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:59:50.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:59:50.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:49 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:59:50.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T17:59:50.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:59:50.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:49 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:59:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.558+0000 7f887b59e640 1 -- 172.21.15.129:0/2915860977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f887c100ea0 msgr2=0x7f887c1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.558+0000 7f887b59e640 1 --2- 172.21.15.129:0/2915860977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f887c100ea0 0x7f887c1012b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8870009860 tx=0x7f887002a340 comp rx=0 tx=0).stop 2024-05-20T17:59:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.558+0000 7f887b59e640 1 -- 172.21.15.129:0/2915860977 shutdown_connections 2024-05-20T17:59:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.558+0000 7f887b59e640 1 --2- 172.21.15.129:0/2915860977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f887c10d260 0x7f887c10f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.558+0000 7f887b59e640 1 --2- 172.21.15.129:0/2915860977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f887c100ea0 0x7f887c1012b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.558+0000 7f887b59e640 1 --2- 172.21.15.129:0/2915860977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f887c108960 0x7f887c100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.558+0000 7f887b59e640 1 -- 172.21.15.129:0/2915860977 >> 172.21.15.129:0/2915860977 conn(0x7f887c076730 msgr2=0x7f887c076b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.558+0000 7f887b59e640 1 -- 172.21.15.129:0/2915860977 shutdown_connections 2024-05-20T17:59:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.558+0000 7f887b59e640 1 -- 172.21.15.129:0/2915860977 wait complete. 2024-05-20T17:59:50.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.559+0000 7f887b59e640 1 Processor -- start 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.559+0000 7f887b59e640 1 -- start start 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.559+0000 7f887b59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f887c100ea0 0x7f887c199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f887b59e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f887c108960 0x7f887c199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f887b59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f887c10d260 0x7f887c19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f887b59e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f887c112c60 con 0x7f887c100ea0 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f887b59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f887c112ae0 con 0x7f887c10d260 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f887b59e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f887c112de0 con 0x7f887c108960 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f8879d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f887c108960 0x7f887c199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f887a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f887c100ea0 0x7f887c199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f887a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f887c100ea0 0x7f887c199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52706/0 (socket says 172.21.15.129:52706) 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f8879d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f887c108960 0x7f887c199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50236/0 (socket says 172.21.15.129:50236) 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f887a59c640 1 -- 172.21.15.129:0/957683547 learned_addr learned my addr 172.21.15.129:0/957683547 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:50.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f887ad9d640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f887c10d260 0x7f887c19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f8879d9b640 1 -- 172.21.15.129:0/957683547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f887c10d260 msgr2=0x7f887c19de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f8879d9b640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f887c10d260 0x7f887c19de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f8879d9b640 1 -- 172.21.15.129:0/957683547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f887c100ea0 msgr2=0x7f887c199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f8879d9b640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f887c100ea0 0x7f887c199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.560+0000 7f8879d9b640 1 -- 172.21.15.129:0/957683547 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887c19e4a0 con 0x7f887c108960 2024-05-20T17:59:50.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.561+0000 7f8879d9b640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f887c108960 0x7f887c199ba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8870009780 tx=0x7f8870033b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:50.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.561+0000 7f886b7fe640 1 -- 172.21.15.129:0/957683547 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8870004250 con 0x7f887c108960 2024-05-20T17:59:50.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.561+0000 7f887b59e640 1 -- 172.21.15.129:0/957683547 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f887c19e730 con 0x7f887c108960 2024-05-20T17:59:50.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.561+0000 7f887b59e640 1 -- 172.21.15.129:0/957683547 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f887c19ec40 con 0x7f887c108960 2024-05-20T17:59:50.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.561+0000 7f886b7fe640 1 -- 172.21.15.129:0/957683547 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88700043f0 con 0x7f887c108960 2024-05-20T17:59:50.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.561+0000 7f886b7fe640 1 -- 172.21.15.129:0/957683547 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8870003de0 con 0x7f887c108960 2024-05-20T17:59:50.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.563+0000 7f887b59e640 1 -- 172.21.15.129:0/957683547 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f887c068fe0 con 0x7f887c108960 2024-05-20T17:59:50.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.564+0000 7f886b7fe640 1 -- 172.21.15.129:0/957683547 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f8870030550 con 0x7f887c108960 2024-05-20T17:59:50.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.564+0000 7f886b7fe640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f88640a2e90 0x7f88640a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:50.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.564+0000 7f886b7fe640 1 -- 172.21.15.129:0/957683547 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f88700f3610 con 0x7f887c108960 2024-05-20T17:59:50.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.564+0000 7f887a59c640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f88640a2e90 0x7f88640a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:50.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.565+0000 7f887a59c640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f88640a2e90 0x7f88640a53a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f886c0096c0 tx=0x7f886c009260 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:50.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.570+0000 7f886b7fe640 1 -- 172.21.15.129:0/957683547 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f88700c0080 con 0x7f887c108960 2024-05-20T17:59:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.972+0000 7f887b59e640 1 -- 172.21.15.129:0/957683547 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f887c02d2a0 con 0x7f887c108960 2024-05-20T17:59:50.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.973+0000 7f886b7fe640 1 -- 172.21.15.129:0/957683547 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11829 (secure 0 0 0) 0x7f88700bf980 con 0x7f887c108960 2024-05-20T17:59:50.972 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T17:59:50.972 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:59:17.074606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25229":{"gid":25229,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1860679091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1860679091},{"type":"v1","addr":"172.21.15.154:6837","nonce":1860679091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T17:59:50.972 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-05-20T17:59:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.976+0000 7f887b59e640 1 -- 172.21.15.129:0/957683547 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f88640a2e90 msgr2=0x7f88640a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.976+0000 7f887b59e640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f88640a2e90 0x7f88640a53a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f886c0096c0 tx=0x7f886c009260 comp rx=0 tx=0).stop 2024-05-20T17:59:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.976+0000 7f887b59e640 1 -- 172.21.15.129:0/957683547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f887c108960 msgr2=0x7f887c199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.976+0000 7f887b59e640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f887c108960 0x7f887c199ba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8870009780 tx=0x7f8870033b60 comp rx=0 tx=0).stop 2024-05-20T17:59:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.977+0000 7f887b59e640 1 -- 172.21.15.129:0/957683547 shutdown_connections 2024-05-20T17:59:50.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.977+0000 7f887b59e640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f88640a2e90 0x7f88640a53a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.977+0000 7f887b59e640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f887c10d260 0x7f887c19de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.977+0000 7f887b59e640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f887c108960 0x7f887c199ba0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.977+0000 7f887b59e640 1 --2- 172.21.15.129:0/957683547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f887c100ea0 0x7f887c199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:50.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.977+0000 7f887b59e640 1 -- 172.21.15.129:0/957683547 >> 172.21.15.129:0/957683547 conn(0x7f887c076730 msgr2=0x7f887c103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:50.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.977+0000 7f887b59e640 1 -- 172.21.15.129:0/957683547 shutdown_connections 2024-05-20T17:59:50.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:50.977+0000 7f887b59e640 1 -- 172.21.15.129:0/957683547 wait complete. 2024-05-20T17:59:50.990 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T17:59:51.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.120+0000 7f5d50977640 1 -- 172.21.15.129:0/2659928268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c108b50 msgr2=0x7f5d4c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:51.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.120+0000 7f5d50977640 1 --2- 172.21.15.129:0/2659928268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c108b50 0x7f5d4c100ac0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009900 tx=0x7f5d3c02a410 comp rx=0 tx=0).stop 2024-05-20T17:59:51.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.121+0000 7f5d50977640 1 -- 172.21.15.129:0/2659928268 shutdown_connections 2024-05-20T17:59:51.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.121+0000 7f5d50977640 1 --2- 172.21.15.129:0/2659928268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d4c101000 0x7f5d4c101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.121+0000 7f5d50977640 1 --2- 172.21.15.129:0/2659928268 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d4c10d500 0x7f5d4c10f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.121+0000 7f5d50977640 1 --2- 172.21.15.129:0/2659928268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c108b50 0x7f5d4c100ac0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.121+0000 7f5d50977640 1 -- 172.21.15.129:0/2659928268 >> 172.21.15.129:0/2659928268 conn(0x7f5d4c076830 msgr2=0x7f5d4c076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:51.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.121+0000 7f5d50977640 1 -- 172.21.15.129:0/2659928268 shutdown_connections 2024-05-20T17:59:51.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.121+0000 7f5d50977640 1 -- 172.21.15.129:0/2659928268 wait complete. 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d50977640 1 Processor -- start 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d50977640 1 -- start start 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d50977640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d4c101000 0x7f5d4c19f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d50977640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c108b50 0x7f5d4c19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d50977640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d4c10d500 0x7f5d4c199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d50977640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c112e10 con 0x7f5d4c108b50 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d50977640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c112c90 con 0x7f5d4c10d500 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d50977640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c112f90 con 0x7f5d4c101000 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d49d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c108b50 0x7f5d4c19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d49d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c108b50 0x7f5d4c19fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52728/0 (socket says 172.21.15.129:52728) 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d49d9b640 1 -- 172.21.15.129:0/3535097710 learned_addr learned my addr 172.21.15.129:0/3535097710 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:51.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d4a59c640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d4c101000 0x7f5d4c19f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.122+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d4c10d500 0x7f5d4c199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.123+0000 7f5d49d9b640 1 -- 172.21.15.129:0/3535097710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d4c101000 msgr2=0x7f5d4c19f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.123+0000 7f5d49d9b640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d4c101000 0x7f5d4c19f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.123+0000 7f5d49d9b640 1 -- 172.21.15.129:0/3535097710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d4c10d500 msgr2=0x7f5d4c199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.123+0000 7f5d49d9b640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d4c10d500 0x7f5d4c199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.123+0000 7f5d49d9b640 1 -- 172.21.15.129:0/3535097710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4c199dc0 con 0x7f5d4c108b50 2024-05-20T17:59:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.123+0000 7f5d49d9b640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c108b50 0x7f5d4c19fa50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5d4000e9c0 tx=0x7f5d4000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.123+0000 7f5d3b7fe640 1 -- 172.21.15.129:0/3535097710 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4000cd80 con 0x7f5d4c108b50 2024-05-20T17:59:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.123+0000 7f5d3b7fe640 1 -- 172.21.15.129:0/3535097710 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d40005500 con 0x7f5d4c108b50 2024-05-20T17:59:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.123+0000 7f5d50977640 1 -- 172.21.15.129:0/3535097710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d4c19a050 con 0x7f5d4c108b50 2024-05-20T17:59:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.123+0000 7f5d3b7fe640 1 -- 172.21.15.129:0/3535097710 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d40010690 con 0x7f5d4c108b50 2024-05-20T17:59:51.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.124+0000 7f5d50977640 1 -- 172.21.15.129:0/3535097710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d4c1a62b0 con 0x7f5d4c108b50 2024-05-20T17:59:51.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.127+0000 7f5d57824740 1 Processor -- start 2024-05-20T17:59:51.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.127+0000 7f5d3b7fe640 1 -- 172.21.15.129:0/3535097710 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f5d400046c0 con 0x7f5d4c108b50 2024-05-20T17:59:51.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.128+0000 7f5d57824740 1 -- start start 2024-05-20T17:59:51.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.128+0000 7f5d57824740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 0x560f287b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.128+0000 7f5d57824740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287b5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.128+0000 7f5d57824740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283f56b0 0x560f287b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.128+0000 7f5d57824740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560f2871ac40 con 0x560f283ed4d0 2024-05-20T17:59:51.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.128+0000 7f5d57824740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f286c2580 con 0x560f28416390 2024-05-20T17:59:51.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.128+0000 7f5d57824740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560f28716c00 con 0x560f283f56b0 2024-05-20T17:59:51.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.128+0000 7f5d4a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287b5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.128+0000 7f5d4a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287b5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52744/0 (socket says 172.21.15.129:52744) 2024-05-20T17:59:51.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.128+0000 7f5d4a59c640 1 -- 172.21.15.129:0/1204231771 learned_addr learned my addr 172.21.15.129:0/1204231771 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:51.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.128+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/1204231771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 0x560f287b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d49d9b640 1 --2- 172.21.15.129:0/1204231771 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283f56b0 0x560f287b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d4ad9d640 1 -- 172.21.15.129:0/1204231771 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283f56b0 msgr2=0x560f287b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/1204231771 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283f56b0 0x560f287b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d4ad9d640 1 -- 172.21.15.129:0/1204231771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 msgr2=0x560f287b5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/1204231771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287b5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d4ad9d640 1 -- 172.21.15.129:0/1204231771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f2842e3f0 con 0x560f28416390 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d49d9b640 1 --2- 172.21.15.129:0/1204231771 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283f56b0 0x560f287b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d4a59c640 1 --2- 172.21.15.129:0/1204231771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d3b7fe640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d340a2ea0 0x7f5d340a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/1204231771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 0x560f287b4d20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5d44005fd0 tx=0x7f5d44011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a09eaa239f06dd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d4a59c640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d340a2ea0 0x7f5d340a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.129+0000 7f5d38ff9640 1 -- 172.21.15.129:0/1204231771 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44004d20 con 0x560f28416390 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.130+0000 7f5d38ff9640 1 -- 172.21.15.129:0/1204231771 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d44004050 con 0x560f28416390 2024-05-20T17:59:51.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.130+0000 7f5d38ff9640 1 -- 172.21.15.129:0/1204231771 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44005280 con 0x560f28416390 2024-05-20T17:59:51.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.130+0000 7f5d3b7fe640 1 -- 172.21.15.129:0/3535097710 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5d400cfac0 con 0x7f5d4c108b50 2024-05-20T17:59:51.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d57824740 1 -- 172.21.15.129:0/1204231771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 msgr2=0x560f287b4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:51.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d4a59c640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d340a2ea0 0x7f5d340a53b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x560f287b2460 tx=0x7f5d3c0058d0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:51.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d57824740 1 --2- 172.21.15.129:0/1204231771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 0x560f287b4d20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5d44005fd0 tx=0x7f5d44011810 comp rx=0 tx=0).stop 2024-05-20T17:59:51.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d57824740 1 -- 172.21.15.129:0/1204231771 shutdown_connections 2024-05-20T17:59:51.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d57824740 1 --2- 172.21.15.129:0/1204231771 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283f56b0 0x560f287b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d57824740 1 --2- 172.21.15.129:0/1204231771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287b5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d57824740 1 --2- 172.21.15.129:0/1204231771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 0x560f287b4d20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d57824740 1 -- 172.21.15.129:0/1204231771 >> 172.21.15.129:0/1204231771 conn(0x560f2840a4e0 msgr2=0x560f287b14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d57824740 1 -- 172.21.15.129:0/1204231771 shutdown_connections 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d57824740 1 -- 172.21.15.129:0/1204231771 wait complete. 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d57824740 1 -- start start 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.131+0000 7f5d57824740 1 Processor -- start 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d57824740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283ed4d0 0x560f287bf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d57824740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287bf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d57824740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 0x560f287c5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d57824740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560f287c2010 con 0x560f283f56b0 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d57824740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f286d5c40 con 0x560f28416390 2024-05-20T17:59:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d57824740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560f287c2190 con 0x560f283ed4d0 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d57824740 10 client.0 _ll_register_callbacks cb 0x560f2842dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d49d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287bf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d4ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 0x560f287c5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d4ad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 0x560f287c5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:48374/0 (socket says 172.21.15.129:48374) 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d49d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287bf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52760/0 (socket says 172.21.15.129:52760) 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d4ad9d640 1 -- 172.21.15.129:0/3844679794 learned_addr learned my addr 172.21.15.129:0/3844679794 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d4a59c640 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283ed4d0 0x560f287bf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d4ad9d640 1 -- 172.21.15.129:0/3844679794 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283ed4d0 msgr2=0x560f287bf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283ed4d0 0x560f287bf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d4ad9d640 1 -- 172.21.15.129:0/3844679794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 msgr2=0x560f287bf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287bf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d4ad9d640 1 -- 172.21.15.129:0/3844679794 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f284f87d0 con 0x560f28416390 2024-05-20T17:59:51.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.132+0000 7f5d49d9b640 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287bf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.133+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 0x560f287c5450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5d44003c90 tx=0x7f5d44003cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ea23e8ce0e89cd5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.133+0000 7f5d31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.133+0000 7f5d31ffb640 1 -- 172.21.15.129:0/3844679794 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44029070 con 0x560f28416390 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.133+0000 7f5d57824740 1 -- 172.21.15.129:0/3844679794 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d44004050 con 0x560f28416390 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.133+0000 7f5d31ffb640 1 -- 172.21.15.129:0/3844679794 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5d44005020 con 0x560f28416390 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.133+0000 7f5d31ffb640 1 -- 172.21.15.129:0/3844679794 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5d44035050 con 0x560f28416390 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.133+0000 7f5d57824740 10 client.25355 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.133+0000 7f5d57824740 1 -- 172.21.15.129:0/3844679794 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x560f287ce740 con 0x560f28416390 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.134+0000 7f5d31ffb640 1 -- 172.21.15.129:0/3844679794 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d44004050 con 0x560f28416390 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.134+0000 7f5d31ffb640 1 -- 172.21.15.129:0/3844679794 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f5d440235a0 con 0x560f28416390 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.134+0000 7f5d57824740 10 client.25355 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.134+0000 7f5d57824740 10 client.25355 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.134+0000 7f5d57824740 20 client.25355 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.134+0000 7f5d57824740 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560f283d4f60 0x560f287d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.134+0000 7f5d57824740 4 client.25355 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:51.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.134+0000 7f5d57824740 1 -- 172.21.15.129:0/3844679794 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f28728370 con 0x560f283d4f60 2024-05-20T17:59:51.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.135+0000 7f5d4a59c640 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560f283d4f60 0x560f287d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.135+0000 7f5d31ffb640 10 client.25355 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:51.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.135+0000 7f5d4a59c640 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560f283d4f60 0x560f287d1f20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ff31d5bc44d4514 server_cookie=b35cea3664b85c6d in_seq=0 out_seq=0 2024-05-20T17:59:51.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d31ffb640 1 -- 172.21.15.129:0/3844679794 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560f28728370 con 0x560f283d4f60 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d31ffb640 10 client.25355 handle_command_reply: tid=0 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 1 client.25355 shutdown 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 1 -- 172.21.15.129:0/3844679794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 msgr2=0x560f287c5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 0x560f287c5450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5d44003c90 tx=0x7f5d44003cc0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 1 -- 172.21.15.129:0/3844679794 shutdown_connections 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560f283d4f60 0x560f287d1f20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f28416390 0x560f287c5450 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d31ffb640 0 client.25355 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287bf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 1 --2- 172.21.15.129:0/3844679794 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283ed4d0 0x560f287bf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 1 -- 172.21.15.129:0/3844679794 >> 172.21.15.129:0/3844679794 conn(0x560f2840a4e0 msgr2=0x560f287b6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 1 -- 172.21.15.129:0/3844679794 shutdown_connections 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 1 -- 172.21.15.129:0/3844679794 wait complete. 2024-05-20T17:59:51.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.136+0000 7f5d57824740 20 client.25355 trim_cache size 0 max 16384 2024-05-20T17:59:51.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d57824740 1 Processor -- start 2024-05-20T17:59:51.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d57824740 1 -- start start 2024-05-20T17:59:51.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d57824740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f283d4f60 0x560f287bf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d57824740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283ed4d0 0x560f287c12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d57824740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d57824740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c1acb70 con 0x560f283f56b0 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d57824740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c1af3f0 con 0x560f283d4f60 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d57824740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c1ad1e0 con 0x560f283ed4d0 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d4ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d4ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287b51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52776/0 (socket says 172.21.15.129:52776) 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d4ad9d640 1 -- 172.21.15.129:0/3038346002 learned_addr learned my addr 172.21.15.129:0/3038346002 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d49d9b640 1 --2- 172.21.15.129:0/3038346002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283ed4d0 0x560f287c12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.150+0000 7f5d4a59c640 1 --2- 172.21.15.129:0/3038346002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f283d4f60 0x560f287bf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.151+0000 7f5d4ad9d640 1 -- 172.21.15.129:0/3038346002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283ed4d0 msgr2=0x560f287c12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.151+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/3038346002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283ed4d0 0x560f287c12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.151+0000 7f5d4ad9d640 1 -- 172.21.15.129:0/3038346002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f283d4f60 msgr2=0x560f287bf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.151+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/3038346002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f283d4f60 0x560f287bf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.151+0000 7f5d4ad9d640 1 -- 172.21.15.129:0/3038346002 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f287c0190 con 0x560f283f56b0 2024-05-20T17:59:51.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.151+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/3038346002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287b51c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5d44023970 tx=0x7f5d44005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358f1d9d3786e244 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:51.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.151+0000 7f5d33fff640 1 -- 172.21.15.129:0/3038346002 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44029040 con 0x560f283f56b0 2024-05-20T17:59:51.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.151+0000 7f5d33fff640 1 -- 172.21.15.129:0/3038346002 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d44023be0 con 0x560f283f56b0 2024-05-20T17:59:51.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.151+0000 7f5d33fff640 1 -- 172.21.15.129:0/3038346002 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d440198b0 con 0x560f283f56b0 2024-05-20T17:59:51.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.152+0000 7f5d57824740 1 -- 172.21.15.129:0/3038346002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 msgr2=0x560f287b51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.152+0000 7f5d57824740 1 --2- 172.21.15.129:0/3038346002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287b51c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5d44023970 tx=0x7f5d44005f10 comp rx=0 tx=0).stop 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.152+0000 7f5d57824740 1 -- 172.21.15.129:0/3038346002 shutdown_connections 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.152+0000 7f5d57824740 1 --2- 172.21.15.129:0/3038346002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283f56b0 0x560f287b51c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.152+0000 7f5d57824740 1 --2- 172.21.15.129:0/3038346002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283ed4d0 0x560f287c12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.152+0000 7f5d57824740 1 --2- 172.21.15.129:0/3038346002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f283d4f60 0x560f287bf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.152+0000 7f5d57824740 1 -- 172.21.15.129:0/3038346002 >> 172.21.15.129:0/3038346002 conn(0x560f2840a4e0 msgr2=0x560f287b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.152+0000 7f5d57824740 1 -- 172.21.15.129:0/3038346002 shutdown_connections 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.152+0000 7f5d57824740 1 -- 172.21.15.129:0/3038346002 wait complete. 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.152+0000 7f5d57824740 1 -- start start 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.152+0000 7f5d57824740 1 Processor -- start 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d57824740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283d4f60 0x560f287c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d57824740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d49d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d49d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287c89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52780/0 (socket says 172.21.15.129:52780) 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d49d9b640 1 -- 172.21.15.129:0/3776950827 learned_addr learned my addr 172.21.15.129:0/3776950827 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d57824740 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f283f56b0 0x560f287c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d57824740 1 -- 172.21.15.129:0/3776950827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c1ad8b0 con 0x560f283ed4d0 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d57824740 1 -- 172.21.15.129:0/3776950827 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c1aed20 con 0x560f283f56b0 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d57824740 1 -- 172.21.15.129:0/3776950827 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c1adf80 con 0x560f283d4f60 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d4a59c640 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283d4f60 0x560f287c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f283f56b0 0x560f287c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d49d9b640 1 -- 172.21.15.129:0/3776950827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283d4f60 msgr2=0x560f287c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d49d9b640 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283d4f60 0x560f287c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d49d9b640 1 -- 172.21.15.129:0/3776950827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f283f56b0 msgr2=0x560f287c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d49d9b640 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f283f56b0 0x560f287c8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d49d9b640 1 -- 172.21.15.129:0/3776950827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f288381d0 con 0x560f283ed4d0 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.153+0000 7f5d4ad9d640 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f283f56b0 0x560f287c8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d49d9b640 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287c89f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5d40098180 tx=0x7f5d40009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e1d630c1e66c1f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d57824740 10 client.0 _ll_register_callbacks cb 0x7f5d440093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d57824740 1 -- 172.21.15.129:0/3776950827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d44023be0 con 0x560f283ed4d0 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d31ffb640 1 -- 172.21.15.129:0/3776950827 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4009b680 con 0x560f283ed4d0 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d31ffb640 1 -- 172.21.15.129:0/3776950827 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5d40097a70 con 0x560f283ed4d0 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d31ffb640 1 -- 172.21.15.129:0/3776950827 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5d4009c420 con 0x560f283ed4d0 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d57824740 10 client.15672 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d57824740 1 -- 172.21.15.129:0/3776950827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({fsmap=22}) -- 0x560f28839130 con 0x560f283ed4d0 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d31ffb640 1 -- 172.21.15.129:0/3776950827 <== mon.0 v2:172.21.15.129:3300/0 4 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f5d40097c70 con 0x560f283ed4d0 2024-05-20T17:59:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d57824740 10 client.15672 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d57824740 10 client.15672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d57824740 20 client.15672 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d57824740 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560f28416390 0x560f2883b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.154+0000 7f5d57824740 4 client.15672 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.155+0000 7f5d57824740 1 -- 172.21.15.129:0/3776950827 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f5d44023be0 con 0x560f28416390 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.155+0000 7f5d4a59c640 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560f28416390 0x560f2883b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.155+0000 7f5d4a59c640 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560f28416390 0x560f2883b7d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=639de26b96822400 server_cookie=1ae6b920d76dc3c9 in_seq=0 out_seq=0 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.155+0000 7f5d31ffb640 10 client.15672 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.155+0000 7f5d31ffb640 1 -- 172.21.15.129:0/3776950827 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f5d44023be0 con 0x560f28416390 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.155+0000 7f5d31ffb640 10 client.15672 handle_command_reply: tid=0 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.155+0000 7f5d57824740 1 client.15672 shutdown 2024-05-20T17:59:51.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.155+0000 7f5d57824740 1 -- 172.21.15.129:0/3776950827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 msgr2=0x560f287c89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d57824740 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287c89f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5d40098180 tx=0x7f5d40009700 comp rx=0 tx=0).stop 2024-05-20T17:59:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d57824740 1 -- 172.21.15.129:0/3776950827 shutdown_connections 2024-05-20T17:59:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d57824740 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560f28416390 0x560f2883b7d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d31ffb640 0 client.15672 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d57824740 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f283f56b0 0x560f287c8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d57824740 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560f283ed4d0 0x560f287c89f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d57824740 1 --2- 172.21.15.129:0/3776950827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560f283d4f60 0x560f287c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d57824740 1 -- 172.21.15.129:0/3776950827 >> 172.21.15.129:0/3776950827 conn(0x560f2840a4e0 msgr2=0x560f287b54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d57824740 1 -- 172.21.15.129:0/3776950827 shutdown_connections 2024-05-20T17:59:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d57824740 1 -- 172.21.15.129:0/3776950827 wait complete. 2024-05-20T17:59:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d57824740 20 client.15672 trim_cache size 0 max 16384 2024-05-20T17:59:51.160 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T17:59:51.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d50977640 1 -- 172.21.15.129:0/3535097710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d340a2ea0 msgr2=0x7f5d340a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:51.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d50977640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d340a2ea0 0x7f5d340a53b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x560f287b2460 tx=0x7f5d3c0058d0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d50977640 1 -- 172.21.15.129:0/3535097710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c108b50 msgr2=0x7f5d4c19fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:51.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d50977640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c108b50 0x7f5d4c19fa50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5d4000e9c0 tx=0x7f5d4000ee80 comp rx=0 tx=0).stop 2024-05-20T17:59:51.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d50977640 1 -- 172.21.15.129:0/3535097710 shutdown_connections 2024-05-20T17:59:51.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d50977640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d340a2ea0 0x7f5d340a53b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d50977640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d4c10d500 0x7f5d4c199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d50977640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4c108b50 0x7f5d4c19fa50 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d50977640 1 --2- 172.21.15.129:0/3535097710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d4c101000 0x7f5d4c19f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:51.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.156+0000 7f5d50977640 1 -- 172.21.15.129:0/3535097710 >> 172.21.15.129:0/3535097710 conn(0x7f5d4c076830 msgr2=0x7f5d4c1094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:51.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.157+0000 7f5d50977640 1 -- 172.21.15.129:0/3535097710 shutdown_connections 2024-05-20T17:59:51.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:51.157+0000 7f5d50977640 1 -- 172.21.15.129:0/3535097710 wait complete. 2024-05-20T17:59:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:50 smithi154 ceph-mon[36482]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 654 B/s wr, 3 op/s; 82 KiB/s, 1 keys/s, 0 objects/s recovering 2024-05-20T17:59:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:50 smithi154 ceph-mon[36482]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-20T17:59:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:50 smithi154 ceph-mon[36482]: Cluster is now healthy 2024-05-20T17:59:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:50 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:59:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:50 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:59:51.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:50 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:59:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:50 smithi129 ceph-mon[32029]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 654 B/s wr, 3 op/s; 82 KiB/s, 1 keys/s, 0 objects/s recovering 2024-05-20T17:59:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:50 smithi129 ceph-mon[32029]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-20T17:59:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:50 smithi129 ceph-mon[32029]: Cluster is now healthy 2024-05-20T17:59:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:50 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:59:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:50 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:59:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:50 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:59:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:50 smithi204 ceph-mon[37076]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 654 B/s wr, 3 op/s; 82 KiB/s, 1 keys/s, 0 objects/s recovering 2024-05-20T17:59:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:50 smithi204 ceph-mon[37076]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-20T17:59:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:50 smithi204 ceph-mon[37076]: Cluster is now healthy 2024-05-20T17:59:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:50 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T17:59:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:50 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T17:59:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:50 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T17:59:52.168 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T17:59:52.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T17:59:52.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T17:59:52.190 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:51 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/957683547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:59:52.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.326+0000 7f3aa30e2640 1 -- 172.21.15.129:0/4044903102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a9c101790 msgr2=0x7f3a9c10f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.326+0000 7f3aa30e2640 1 --2- 172.21.15.129:0/4044903102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a9c101790 0x7f3a9c10f610 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009ca0 tx=0x7f3a9402a8c0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.326+0000 7f3aa30e2640 1 -- 172.21.15.129:0/4044903102 shutdown_connections 2024-05-20T17:59:52.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.326+0000 7f3aa30e2640 1 --2- 172.21.15.129:0/4044903102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a9c101790 0x7f3a9c10f610 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.326+0000 7f3aa30e2640 1 --2- 172.21.15.129:0/4044903102 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a9c108f90 0x7f3a9c101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.326+0000 7f3aa30e2640 1 --2- 172.21.15.129:0/4044903102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a9c108af0 0x7f3a9c100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.326+0000 7f3aa30e2640 1 -- 172.21.15.129:0/4044903102 >> 172.21.15.129:0/4044903102 conn(0x7f3a9c0768d0 msgr2=0x7f3a9c076d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.326+0000 7f3aa30e2640 1 -- 172.21.15.129:0/4044903102 shutdown_connections 2024-05-20T17:59:52.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.326+0000 7f3aa30e2640 1 -- 172.21.15.129:0/4044903102 wait complete. 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.327+0000 7f3aa30e2640 1 Processor -- start 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.327+0000 7f3aa30e2640 1 -- start start 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.327+0000 7f3aa30e2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a9c108af0 0x7f3a9c198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.327+0000 7f3aa30e2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a9c1994b0 0x7f3a9c19d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.327+0000 7f3aa30e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a9c19dcd0 0x7f3a9c19e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.327+0000 7f3aa30e2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c112e50 con 0x7f3a9c19dcd0 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.327+0000 7f3aa30e2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c112cd0 con 0x7f3a9c1994b0 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.327+0000 7f3aa30e2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c112fd0 con 0x7f3a9c108af0 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3aa167f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a9c19dcd0 0x7f3a9c19e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3aa0e7e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a9c108af0 0x7f3a9c198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3aa167f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a9c19dcd0 0x7f3a9c19e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52790/0 (socket says 172.21.15.129:52790) 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3aa167f640 1 -- 172.21.15.129:0/3461696532 learned_addr learned my addr 172.21.15.129:0/3461696532 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3a9bfff640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a9c1994b0 0x7f3a9c19d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3aa167f640 1 -- 172.21.15.129:0/3461696532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a9c108af0 msgr2=0x7f3a9c198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3aa167f640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a9c108af0 0x7f3a9c198f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3aa167f640 1 -- 172.21.15.129:0/3461696532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a9c1994b0 msgr2=0x7f3a9c19d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3aa167f640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a9c1994b0 0x7f3a9c19d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3aa167f640 1 -- 172.21.15.129:0/3461696532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9c1a5a40 con 0x7f3a9c19dcd0 2024-05-20T17:59:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3aa167f640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a9c19dcd0 0x7f3a9c19e0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3a94031a20 tx=0x7f3a94031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3a99ffb640 1 -- 172.21.15.129:0/3461696532 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a94003b70 con 0x7f3a9c19dcd0 2024-05-20T17:59:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3aa30e2640 1 -- 172.21.15.129:0/3461696532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a9c1a5c10 con 0x7f3a9c19dcd0 2024-05-20T17:59:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3a99ffb640 1 -- 172.21.15.129:0/3461696532 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a94003d10 con 0x7f3a9c19dcd0 2024-05-20T17:59:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.328+0000 7f3a99ffb640 1 -- 172.21.15.129:0/3461696532 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9402fc40 con 0x7f3a9c19dcd0 2024-05-20T17:59:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.329+0000 7f3aa30e2640 1 -- 172.21.15.129:0/3461696532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9c1a60d0 con 0x7f3a9c19dcd0 2024-05-20T17:59:52.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.332+0000 7f3a99ffb640 1 -- 172.21.15.129:0/3461696532 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f3a94004050 con 0x7f3a9c19dcd0 2024-05-20T17:59:52.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.332+0000 7f3aa9f86740 1 Processor -- start 2024-05-20T17:59:52.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3aa9f86740 1 -- start start 2024-05-20T17:59:52.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900ba410 0x561f90459620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90091550 0x561f90459b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90099730 0x561f9045a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3a9bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900ba410 0x561f90459620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3a99ffb640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a8c0a2ea0 0x7f3a8c0a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3a9bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900ba410 0x561f90459620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52800/0 (socket says 172.21.15.129:52800) 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3a9bfff640 1 -- 172.21.15.129:0/1618065933 learned_addr learned my addr 172.21.15.129:0/1618065933 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3aa9f86740 1 -- 172.21.15.129:0/1618065933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561f903be990 con 0x561f900ba410 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/1618065933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90091550 0x561f90459b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3aa9f86740 1 -- 172.21.15.129:0/1618065933 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561f903665b0 con 0x561f90091550 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.333+0000 7f3a99ffb640 1 -- 172.21.15.129:0/3461696532 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f3a940f0800 con 0x7f3a9c19dcd0 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.334+0000 7f3aa167f640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a8c0a2ea0 0x7f3a8c0a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.334+0000 7f3aa9f86740 1 -- 172.21.15.129:0/1618065933 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561f903ba950 con 0x561f90099730 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.334+0000 7f3aa167f640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a8c0a2ea0 0x7f3a8c0a53b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a940edf30 tx=0x7f3a940095c0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.334+0000 7f3a9bfff640 1 -- 172.21.15.129:0/1618065933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90099730 msgr2=0x561f9045a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.334+0000 7f3a9bfff640 1 --2- 172.21.15.129:0/1618065933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90099730 0x561f9045a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.334+0000 7f3a9bfff640 1 -- 172.21.15.129:0/1618065933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90091550 msgr2=0x561f90459b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.334+0000 7f3a9bfff640 1 --2- 172.21.15.129:0/1618065933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90091550 0x561f90459b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.334+0000 7f3a9bfff640 1 -- 172.21.15.129:0/1618065933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f9019a7e0 con 0x561f900ba410 2024-05-20T17:59:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.334+0000 7f3a9bfff640 1 --2- 172.21.15.129:0/1618065933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900ba410 0x561f90459620 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3a88009ca0 tx=0x7f3a88011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a61a6ae9886184d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.335+0000 7f3a86ffd640 1 -- 172.21.15.129:0/1618065933 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88004030 con 0x561f900ba410 2024-05-20T17:59:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.335+0000 7f3a86ffd640 1 -- 172.21.15.129:0/1618065933 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a88005500 con 0x561f900ba410 2024-05-20T17:59:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.335+0000 7f3a86ffd640 1 -- 172.21.15.129:0/1618065933 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88004c00 con 0x561f900ba410 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.336+0000 7f3aa9f86740 1 -- 172.21.15.129:0/1618065933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900ba410 msgr2=0x561f90459620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.336+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/1618065933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900ba410 0x561f90459620 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3a88009ca0 tx=0x7f3a88011810 comp rx=0 tx=0).stop 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.336+0000 7f3aa9f86740 1 -- 172.21.15.129:0/1618065933 shutdown_connections 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.336+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/1618065933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90091550 0x561f90459b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.336+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/1618065933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90099730 0x561f9045a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.336+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/1618065933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900ba410 0x561f90459620 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.336+0000 7f3aa9f86740 1 -- 172.21.15.129:0/1618065933 >> 172.21.15.129:0/1618065933 conn(0x561f900ae560 msgr2=0x561f90455dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.336+0000 7f3aa9f86740 1 -- 172.21.15.129:0/1618065933 shutdown_connections 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.336+0000 7f3aa9f86740 1 -- 172.21.15.129:0/1618065933 wait complete. 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.337+0000 7f3aa9f86740 1 -- start start 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.337+0000 7f3aa9f86740 1 Processor -- start 2024-05-20T17:59:52.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.337+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90091550 0x561f90465640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90099730 0x561f90465b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f900ba410 0x561f90463ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3a9bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90091550 0x561f90465640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3a9bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90091550 0x561f90465640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52812/0 (socket says 172.21.15.129:52812) 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3a9bfff640 1 -- 172.21.15.129:0/2327111078 learned_addr learned my addr 172.21.15.129:0/2327111078 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa9f86740 1 -- 172.21.15.129:0/2327111078 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561f90466970 con 0x561f90091550 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa9f86740 1 -- 172.21.15.129:0/2327111078 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561f90379b40 con 0x561f90099730 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa9f86740 1 -- 172.21.15.129:0/2327111078 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561f90466af0 con 0x561f900ba410 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90099730 0x561f90465b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa0e7e640 1 -- 172.21.15.129:0/2327111078 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f900ba410 msgr2=0x561f90463ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f900ba410 0x561f90463ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa0e7e640 1 -- 172.21.15.129:0/2327111078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90091550 msgr2=0x561f90465640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90091550 0x561f90465640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa0e7e640 1 -- 172.21.15.129:0/2327111078 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f901faf70 con 0x561f90099730 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3a9bfff640 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90091550 0x561f90465640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa9f86740 10 client.0 _ll_register_callbacks cb 0x561f9036e0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa9f86740 1 -- 172.21.15.129:0/2327111078 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a88005500 con 0x561f90099730 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90099730 0x561f90465b80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x561f90456d60 tx=0x7f3a90012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d38bcc87b015df26 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.338+0000 7f3a7bfff640 10 client.25373 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3a7bfff640 1 -- 172.21.15.129:0/2327111078 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9000f890 con 0x561f90099730 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3a7bfff640 1 -- 172.21.15.129:0/2327111078 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f3a9000fa30 con 0x561f90099730 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3a7bfff640 1 -- 172.21.15.129:0/2327111078 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3a9000fc70 con 0x561f90099730 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3aa9f86740 10 client.25373 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3aa9f86740 1 -- 172.21.15.129:0/2327111078 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({fsmap=22}) -- 0x561f90092170 con 0x561f90099730 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3a7bfff640 1 -- 172.21.15.129:0/2327111078 <== mon.2 v2:172.21.15.154:3300/0 4 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f3a90026470 con 0x561f90099730 2024-05-20T17:59:52.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3aa9f86740 10 client.25373 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:52.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3aa9f86740 10 client.25373 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:52.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3aa9f86740 20 client.25373 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:52.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561f900791f0 0x561f904760e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3aa9f86740 4 client.25373 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:52.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.339+0000 7f3aa9f86740 1 -- 172.21.15.129:0/2327111078 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f903cc0c0 con 0x561f900791f0 2024-05-20T17:59:52.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.340+0000 7f3a9bfff640 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561f900791f0 0x561f904760e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.340+0000 7f3a7bfff640 10 client.25373 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:52.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.340+0000 7f3a9bfff640 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561f900791f0 0x561f904760e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=241f723c888ca357 server_cookie=3b3f7abf1c2a39ee in_seq=0 out_seq=0 2024-05-20T17:59:52.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3a7bfff640 1 -- 172.21.15.129:0/2327111078 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561f903cc0c0 con 0x561f900791f0 2024-05-20T17:59:52.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3a7bfff640 10 client.25373 handle_command_reply: tid=0 2024-05-20T17:59:52.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3aa9f86740 1 client.25373 shutdown 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3aa9f86740 1 -- 172.21.15.129:0/2327111078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90099730 msgr2=0x561f90465b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90099730 0x561f90465b80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x561f90456d60 tx=0x7f3a90012b60 comp rx=0 tx=0).stop 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3aa9f86740 1 -- 172.21.15.129:0/2327111078 shutdown_connections 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561f900791f0 0x561f904760e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90099730 0x561f90465b80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3a7bfff640 0 client.25373 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f900ba410 0x561f90463ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/2327111078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90091550 0x561f90465640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3aa9f86740 1 -- 172.21.15.129:0/2327111078 >> 172.21.15.129:0/2327111078 conn(0x561f900ae560 msgr2=0x561f90459450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3aa9f86740 1 -- 172.21.15.129:0/2327111078 shutdown_connections 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.341+0000 7f3aa9f86740 1 -- 172.21.15.129:0/2327111078 wait complete. 2024-05-20T17:59:52.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.342+0000 7f3aa9f86740 20 client.25373 trim_cache size 0 max 16384 2024-05-20T17:59:52.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.377+0000 7f3aa9f86740 1 Processor -- start 2024-05-20T17:59:52.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.377+0000 7f3aa9f86740 1 -- start start 2024-05-20T17:59:52.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900791f0 0x561f9045ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90091550 0x561f9045f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90099730 0x561f9047b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa9f86740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c1aca60 con 0x561f900791f0 2024-05-20T17:59:52.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa9f86740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c1af2e0 con 0x561f90099730 2024-05-20T17:59:52.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa9f86740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c1ad0d0 con 0x561f90091550 2024-05-20T17:59:52.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3a9bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900791f0 0x561f9045ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3a9bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900791f0 0x561f9045ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52814/0 (socket says 172.21.15.129:52814) 2024-05-20T17:59:52.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3a9bfff640 1 -- 172.21.15.129:0/638546946 learned_addr learned my addr 172.21.15.129:0/638546946 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:52.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/638546946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90091550 0x561f9045f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa0e7e640 1 -- 172.21.15.129:0/638546946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90099730 msgr2=0x561f9047b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/638546946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90099730 0x561f9047b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa0e7e640 1 -- 172.21.15.129:0/638546946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900791f0 msgr2=0x561f9045ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/638546946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900791f0 0x561f9045ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa0e7e640 1 -- 172.21.15.129:0/638546946 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f90465430 con 0x561f90091550 2024-05-20T17:59:52.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.378+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/638546946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90091550 0x561f9045f190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3a90000c00 tx=0x7f3a90014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b30d9f1bba7eea0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.379+0000 7f3a867fc640 1 -- 172.21.15.129:0/638546946 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9000fe10 con 0x561f90091550 2024-05-20T17:59:52.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.379+0000 7f3a867fc640 1 -- 172.21.15.129:0/638546946 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a9002c020 con 0x561f90091550 2024-05-20T17:59:52.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.379+0000 7f3a867fc640 1 -- 172.21.15.129:0/638546946 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a90010850 con 0x561f90091550 2024-05-20T17:59:52.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.380+0000 7f3aa9f86740 1 -- 172.21.15.129:0/638546946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90091550 msgr2=0x561f9045f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.380+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/638546946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90091550 0x561f9045f190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3a90000c00 tx=0x7f3a90014000 comp rx=0 tx=0).stop 2024-05-20T17:59:52.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.380+0000 7f3aa9f86740 1 -- 172.21.15.129:0/638546946 shutdown_connections 2024-05-20T17:59:52.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.380+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/638546946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90099730 0x561f9047b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.380+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/638546946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90091550 0x561f9045f190 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.380+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/638546946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f900791f0 0x561f9045ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.380+0000 7f3aa9f86740 1 -- 172.21.15.129:0/638546946 >> 172.21.15.129:0/638546946 conn(0x561f900ae560 msgr2=0x561f9045d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.380+0000 7f3aa9f86740 1 -- 172.21.15.129:0/638546946 shutdown_connections 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.380+0000 7f3aa9f86740 1 -- 172.21.15.129:0/638546946 wait complete. 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.380+0000 7f3aa9f86740 1 -- start start 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.380+0000 7f3aa9f86740 1 Processor -- start 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f900791f0 0x561f904d8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90091550 0x561f9046b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3aa9f86740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90099730 0x561f9046b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3aa9f86740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c1ad7a0 con 0x561f90099730 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3aa9f86740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c1aec10 con 0x561f900791f0 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3aa9f86740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c1ade70 con 0x561f90091550 2024-05-20T17:59:52.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3aa9f86740 10 client.0 _ll_register_callbacks cb 0x7f3a90009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:52.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3a9bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90099730 0x561f9046b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3a9bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90099730 0x561f9046b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52820/0 (socket says 172.21.15.129:52820) 2024-05-20T17:59:52.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3a9bfff640 1 -- 172.21.15.129:0/3400524175 learned_addr learned my addr 172.21.15.129:0/3400524175 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:52.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3a9bfff640 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f900791f0 0x561f904d8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90091550 0x561f9046b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3a9bfff640 1 -- 172.21.15.129:0/3400524175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90091550 msgr2=0x561f9046b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3a9bfff640 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90091550 0x561f9046b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3a9bfff640 1 -- 172.21.15.129:0/3400524175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f900791f0 msgr2=0x561f904d8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3a9bfff640 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f900791f0 0x561f904d8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.381+0000 7f3a9bfff640 1 -- 172.21.15.129:0/3400524175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f9046be10 con 0x561f90099730 2024-05-20T17:59:52.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.382+0000 7f3a9bfff640 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90099730 0x561f9046b8d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3a88005e00 tx=0x7f3a88011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33af2af1939cb69f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.382+0000 7f3a7bfff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:52.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.382+0000 7f3a7bfff640 1 -- 172.21.15.129:0/3400524175 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88037060 con 0x561f90099730 2024-05-20T17:59:52.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.382+0000 7f3a7bfff640 1 -- 172.21.15.129:0/3400524175 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f3a88004b70 con 0x561f90099730 2024-05-20T17:59:52.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.382+0000 7f3aa9f86740 1 -- 172.21.15.129:0/3400524175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a9002c020 con 0x561f90099730 2024-05-20T17:59:52.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.382+0000 7f3a7bfff640 1 -- 172.21.15.129:0/3400524175 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3a88004d70 con 0x561f90099730 2024-05-20T17:59:52.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.382+0000 7f3aa9f86740 10 client.15690 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:52.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.382+0000 7f3aa9f86740 1 -- 172.21.15.129:0/3400524175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561f904e2490 con 0x561f90099730 2024-05-20T17:59:52.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.382+0000 7f3a7bfff640 1 -- 172.21.15.129:0/3400524175 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a8802c6a0 con 0x561f90099730 2024-05-20T17:59:52.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.383+0000 7f3a7bfff640 1 -- 172.21.15.129:0/3400524175 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f3a8802cce0 con 0x561f90099730 2024-05-20T17:59:52.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.383+0000 7f3aa9f86740 10 client.15690 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:52.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.383+0000 7f3aa9f86740 10 client.15690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:52.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.383+0000 7f3aa9f86740 20 client.15690 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:52.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.383+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561f900ba410 0x561f904e4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.383+0000 7f3aa9f86740 4 client.15690 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T17:59:52.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.383+0000 7f3aa9f86740 1 -- 172.21.15.129:0/3400524175 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f3a9002c020 con 0x561f900ba410 2024-05-20T17:59:52.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.383+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561f900ba410 0x561f904e4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.383+0000 7f3a7bfff640 10 client.15690 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:52.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.383+0000 7f3aa0e7e640 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561f900ba410 0x561f904e4970 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddfd0d41a8f20f46 server_cookie=7ba883fcfa426538 in_seq=0 out_seq=0 2024-05-20T17:59:52.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3a7bfff640 1 -- 172.21.15.129:0/3400524175 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f3a9002c020 con 0x561f900ba410 2024-05-20T17:59:52.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3a7bfff640 10 client.15690 handle_command_reply: tid=0 2024-05-20T17:59:52.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3aa9f86740 1 client.15690 shutdown 2024-05-20T17:59:52.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3aa9f86740 1 -- 172.21.15.129:0/3400524175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90099730 msgr2=0x561f9046b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90099730 0x561f9046b8d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3a88005e00 tx=0x7f3a88011810 comp rx=0 tx=0).stop 2024-05-20T17:59:52.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3aa9f86740 1 -- 172.21.15.129:0/3400524175 shutdown_connections 2024-05-20T17:59:52.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561f900ba410 0x561f904e4970 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3a7bfff640 0 client.15690 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f90099730 0x561f9046b8d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561f90091550 0x561f9046b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3aa9f86740 1 --2- 172.21.15.129:0/3400524175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f900791f0 0x561f904d8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.385+0000 7f3aa9f86740 1 -- 172.21.15.129:0/3400524175 >> 172.21.15.129:0/3400524175 conn(0x561f900ae560 msgr2=0x561f9045d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.386+0000 7f3aa9f86740 1 -- 172.21.15.129:0/3400524175 shutdown_connections 2024-05-20T17:59:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.386+0000 7f3aa9f86740 1 -- 172.21.15.129:0/3400524175 wait complete. 2024-05-20T17:59:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.386+0000 7f3aa9f86740 20 client.15690 trim_cache size 0 max 16384 2024-05-20T17:59:52.388 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T17:59:52.388 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T17:59:52.388 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "9098578c-c6ea-4243-a3ff-3c2849c1db3b", 2024-05-20T17:59:52.389 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T17:59:52.389 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T17:59:52.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.386+0000 7f3aa30e2640 1 -- 172.21.15.129:0/3461696532 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a8c0a2ea0 msgr2=0x7f3a8c0a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.386+0000 7f3aa30e2640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a8c0a2ea0 0x7f3a8c0a53b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a940edf30 tx=0x7f3a940095c0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.386+0000 7f3aa30e2640 1 -- 172.21.15.129:0/3461696532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a9c19dcd0 msgr2=0x7f3a9c19e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.386+0000 7f3aa30e2640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a9c19dcd0 0x7f3a9c19e0e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3a94031a20 tx=0x7f3a94031a50 comp rx=0 tx=0).stop 2024-05-20T17:59:52.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.387+0000 7f3aa30e2640 1 -- 172.21.15.129:0/3461696532 shutdown_connections 2024-05-20T17:59:52.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.387+0000 7f3aa30e2640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a8c0a2ea0 0x7f3a8c0a53b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.387+0000 7f3aa30e2640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a9c1994b0 0x7f3a9c19d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.387+0000 7f3aa30e2640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a9c19dcd0 0x7f3a9c19e0e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.387+0000 7f3aa30e2640 1 --2- 172.21.15.129:0/3461696532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a9c108af0 0x7f3a9c198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.387+0000 7f3aa30e2640 1 -- 172.21.15.129:0/3461696532 >> 172.21.15.129:0/3461696532 conn(0x7f3a9c0768d0 msgr2=0x7f3a9c0fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.387+0000 7f3aa30e2640 1 -- 172.21.15.129:0/3461696532 shutdown_connections 2024-05-20T17:59:52.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.387+0000 7f3aa30e2640 1 -- 172.21.15.129:0/3461696532 wait complete. 2024-05-20T17:59:52.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:51 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/957683547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:59:52.397 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T17:59:52.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T17:59:52.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:51 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/957683547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T17:59:52.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.557+0000 7fe405884640 1 -- 172.21.15.129:0/3024544672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe400105940 msgr2=0x7fe400105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.557+0000 7fe405884640 1 --2- 172.21.15.129:0/3024544672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe400105940 0x7fe400105d70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009900 tx=0x7fe3f402a410 comp rx=0 tx=0).stop 2024-05-20T17:59:52.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.557+0000 7fe405884640 1 -- 172.21.15.129:0/3024544672 shutdown_connections 2024-05-20T17:59:52.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.557+0000 7fe405884640 1 --2- 172.21.15.129:0/3024544672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe400106400 0x7fe40010dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.557+0000 7fe405884640 1 --2- 172.21.15.129:0/3024544672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe400105940 0x7fe400105d70 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.557+0000 7fe405884640 1 --2- 172.21.15.129:0/3024544672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4001013b0 0x7fe4001017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.557+0000 7fe405884640 1 -- 172.21.15.129:0/3024544672 >> 172.21.15.129:0/3024544672 conn(0x7fe400076830 msgr2=0x7fe400076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.557+0000 7fe405884640 1 -- 172.21.15.129:0/3024544672 shutdown_connections 2024-05-20T17:59:52.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.557+0000 7fe405884640 1 -- 172.21.15.129:0/3024544672 wait complete. 2024-05-20T17:59:52.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.558+0000 7fe405884640 1 Processor -- start 2024-05-20T17:59:52.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.558+0000 7fe405884640 1 -- start start 2024-05-20T17:59:52.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.558+0000 7fe405884640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4001013b0 0x7fe4001a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.558+0000 7fe405884640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe400106400 0x7fe4001a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.558+0000 7fe405884640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe40019b8d0 0x7fe40019bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.558+0000 7fe405884640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe400110be0 con 0x7fe4001013b0 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.558+0000 7fe405884640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe400110a60 con 0x7fe40019b8d0 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.558+0000 7fe405884640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe400110d60 con 0x7fe400106400 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.558+0000 7fe3feffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4001013b0 0x7fe4001a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.558+0000 7fe3feffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4001013b0 0x7fe4001a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52834/0 (socket says 172.21.15.129:52834) 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.558+0000 7fe3feffd640 1 -- 172.21.15.129:0/623083661 learned_addr learned my addr 172.21.15.129:0/623083661 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.559+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe400106400 0x7fe4001a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.559+0000 7fe3ff7fe640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe40019b8d0 0x7fe40019bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.559+0000 7fe3feffd640 1 -- 172.21.15.129:0/623083661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe400106400 msgr2=0x7fe4001a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.559+0000 7fe3feffd640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe400106400 0x7fe4001a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.559+0000 7fe3feffd640 1 -- 172.21.15.129:0/623083661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe40019b8d0 msgr2=0x7fe40019bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.559+0000 7fe3feffd640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe40019b8d0 0x7fe40019bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.559+0000 7fe3feffd640 1 -- 172.21.15.129:0/623083661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40019c460 con 0x7fe4001013b0 2024-05-20T17:59:52.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.559+0000 7fe3feffd640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4001013b0 0x7fe4001a1780 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000ea40 tx=0x7fe3f000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.559+0000 7fe404882640 1 -- 172.21.15.129:0/623083661 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f000cd40 con 0x7fe4001013b0 2024-05-20T17:59:52.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.559+0000 7fe404882640 1 -- 172.21.15.129:0/623083661 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3f0005500 con 0x7fe4001013b0 2024-05-20T17:59:52.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.559+0000 7fe404882640 1 -- 172.21.15.129:0/623083661 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0016670 con 0x7fe4001013b0 2024-05-20T17:59:52.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.560+0000 7fe405884640 1 -- 172.21.15.129:0/623083661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4001a8540 con 0x7fe4001013b0 2024-05-20T17:59:52.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.560+0000 7fe405884640 1 -- 172.21.15.129:0/623083661 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4001a88f0 con 0x7fe4001013b0 2024-05-20T17:59:52.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.563+0000 7fe40c80a740 1 Processor -- start 2024-05-20T17:59:52.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.563+0000 7fe40c80a740 1 -- start start 2024-05-20T17:59:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64c4390 0x5598c6862d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c649b4d0 0x5598c6863270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe40c80a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5598c67c8d00 con 0x5598c64a36b0 2024-05-20T17:59:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe40c80a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5598c67704c0 con 0x5598c64c4390 2024-05-20T17:59:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe40c80a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5598c67c4cc0 con 0x5598c649b4d0 2024-05-20T17:59:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe3fe7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe3fe7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52848/0 (socket says 172.21.15.129:52848) 2024-05-20T17:59:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe3fe7fc640 1 -- 172.21.15.129:0/2532243721 learned_addr learned my addr 172.21.15.129:0/2532243721 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe3ff7fe640 1 --2- 172.21.15.129:0/2532243721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64c4390 0x5598c6862d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe3fe7fc640 1 -- 172.21.15.129:0/2532243721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c649b4d0 msgr2=0x5598c6863270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T17:59:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/2532243721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c649b4d0 0x5598c6863270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe3fe7fc640 1 -- 172.21.15.129:0/2532243721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64c4390 msgr2=0x5598c6862d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/2532243721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64c4390 0x5598c6862d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe3fe7fc640 1 -- 172.21.15.129:0/2532243721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598c65a67d0 con 0x5598c64a36b0 2024-05-20T17:59:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe3ff7fe640 1 --2- 172.21.15.129:0/2532243721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64c4390 0x5598c6862d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.564+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/2532243721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68637b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009230 tx=0x7fe3f4005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c166e4b9926541 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.565+0000 7fe3e5ffb640 1 -- 172.21.15.129:0/2532243721 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4040070 con 0x5598c64a36b0 2024-05-20T17:59:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.565+0000 7fe3e5ffb640 1 -- 172.21.15.129:0/2532243721 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3f4004030 con 0x5598c64a36b0 2024-05-20T17:59:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.565+0000 7fe3e5ffb640 1 -- 172.21.15.129:0/2532243721 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4035040 con 0x5598c64a36b0 2024-05-20T17:59:52.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.566+0000 7fe40c80a740 1 -- 172.21.15.129:0/2532243721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 msgr2=0x5598c68637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.566+0000 7fe40c80a740 1 --2- 172.21.15.129:0/2532243721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68637b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009230 tx=0x7fe3f4005e60 comp rx=0 tx=0).stop 2024-05-20T17:59:52.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.566+0000 7fe40c80a740 1 -- 172.21.15.129:0/2532243721 shutdown_connections 2024-05-20T17:59:52.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.566+0000 7fe404882640 1 -- 172.21.15.129:0/623083661 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7fe3f00046c0 con 0x7fe4001013b0 2024-05-20T17:59:52.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.566+0000 7fe40c80a740 1 --2- 172.21.15.129:0/2532243721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68637b0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.566+0000 7fe40c80a740 1 --2- 172.21.15.129:0/2532243721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c649b4d0 0x5598c6863270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.566+0000 7fe40c80a740 1 --2- 172.21.15.129:0/2532243721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64c4390 0x5598c6862d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.566+0000 7fe40c80a740 1 -- 172.21.15.129:0/2532243721 >> 172.21.15.129:0/2532243721 conn(0x5598c64b84e0 msgr2=0x5598c685f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.567+0000 7fe404882640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe3e80a2ea0 0x7fe3e80a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.567+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe3e80a2ea0 0x7fe3e80a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.567+0000 7fe404882640 1 -- 172.21.15.129:0/623083661 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fe3f00cf480 con 0x7fe4001013b0 2024-05-20T17:59:52.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.568+0000 7fe40c80a740 1 -- 172.21.15.129:0/2532243721 shutdown_connections 2024-05-20T17:59:52.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.568+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe3e80a2ea0 0x7fe3e80a53b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40096c0 tx=0x7fe3f4005040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.568+0000 7fe40c80a740 1 -- 172.21.15.129:0/2532243721 wait complete. 2024-05-20T17:59:52.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.569+0000 7fe40c80a740 1 -- start start 2024-05-20T17:59:52.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.569+0000 7fe40c80a740 1 Processor -- start 2024-05-20T17:59:52.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c686eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64c4390 0x5598c686f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 0x5598c686d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe40c80a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5598c686ffc0 con 0x5598c64a36b0 2024-05-20T17:59:52.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe40c80a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5598c6783b90 con 0x5598c64c4390 2024-05-20T17:59:52.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe40c80a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5598c6870140 con 0x5598c6482f60 2024-05-20T17:59:52.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe40c80a740 10 client.0 _ll_register_callbacks cb 0x5598c67bfda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:52.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe3ff7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 0x5598c686d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe3ff7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 0x5598c686d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50358/0 (socket says 172.21.15.129:50358) 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe3ff7fe640 1 -- 172.21.15.129:0/781274797 learned_addr learned my addr 172.21.15.129:0/781274797 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe3feffd640 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c686eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64c4390 0x5598c686f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe3feffd640 1 -- 172.21.15.129:0/781274797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 msgr2=0x5598c686d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe3feffd640 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 0x5598c686d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe3feffd640 1 -- 172.21.15.129:0/781274797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64c4390 msgr2=0x5598c686f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.570+0000 7fe3feffd640 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64c4390 0x5598c686f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.571+0000 7fe3feffd640 1 -- 172.21.15.129:0/781274797 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598c64dc3f0 con 0x5598c64a36b0 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.571+0000 7fe3ff7fe640 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 0x5598c686d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.571+0000 7fe3deffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.571+0000 7fe3feffd640 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c686eec0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe3f00cca40 tx=0x7fe3f009b240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f74bdd455059d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.571+0000 7fe3deffd640 1 -- 172.21.15.129:0/781274797 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f00a43b0 con 0x5598c64a36b0 2024-05-20T17:59:52.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.571+0000 7fe40c80a740 1 -- 172.21.15.129:0/781274797 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3f4004030 con 0x5598c64a36b0 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.571+0000 7fe3deffd640 1 -- 172.21.15.129:0/781274797 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fe3f00cf480 con 0x5598c64a36b0 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.572+0000 7fe3deffd640 1 -- 172.21.15.129:0/781274797 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe3f00a43b0 con 0x5598c64a36b0 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.572+0000 7fe40c80a740 10 client.15702 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.572+0000 7fe40c80a740 1 -- 172.21.15.129:0/781274797 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5598c649c0f0 con 0x5598c64a36b0 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.572+0000 7fe3deffd640 1 -- 172.21.15.129:0/781274797 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3f00b3030 con 0x5598c64a36b0 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.572+0000 7fe3deffd640 1 -- 172.21.15.129:0/781274797 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fe3f000b040 con 0x5598c64a36b0 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.572+0000 7fe40c80a740 10 client.15702 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.572+0000 7fe40c80a740 10 client.15702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.572+0000 7fe40c80a740 20 client.15702 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.572+0000 7fe40c80a740 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5598c64583a0 0x5598c687fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.572+0000 7fe40c80a740 4 client.15702 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.572+0000 7fe40c80a740 1 -- 172.21.15.129:0/781274797 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598c67d6430 con 0x5598c64583a0 2024-05-20T17:59:52.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.573+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5598c64583a0 0x5598c687fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.573+0000 7fe3deffd640 10 client.15702 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.573+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5598c64583a0 0x5598c687fe90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f163cf3bd9e8a03 server_cookie=a5b7ba24eb5f74df in_seq=0 out_seq=0 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe3deffd640 1 -- 172.21.15.129:0/781274797 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5598c67d6430 con 0x5598c64583a0 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe3deffd640 10 client.15702 handle_command_reply: tid=0 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe40c80a740 1 client.15702 shutdown 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe40c80a740 1 -- 172.21.15.129:0/781274797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 msgr2=0x5598c686eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe40c80a740 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c686eec0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe3f00cca40 tx=0x7fe3f009b240 comp rx=0 tx=0).stop 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe40c80a740 1 -- 172.21.15.129:0/781274797 shutdown_connections 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe40c80a740 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5598c64583a0 0x5598c687fe90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe3deffd640 0 client.15702 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe40c80a740 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64c4390 0x5598c686f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe40c80a740 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 0x5598c686d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe40c80a740 1 --2- 172.21.15.129:0/781274797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c686eec0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe40c80a740 1 -- 172.21.15.129:0/781274797 >> 172.21.15.129:0/781274797 conn(0x5598c64b84e0 msgr2=0x5598c6864a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe40c80a740 1 -- 172.21.15.129:0/781274797 shutdown_connections 2024-05-20T17:59:52.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.574+0000 7fe40c80a740 1 -- 172.21.15.129:0/781274797 wait complete. 2024-05-20T17:59:52.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.575+0000 7fe40c80a740 20 client.15702 trim_cache size 0 max 16384 2024-05-20T17:59:52.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.603+0000 7fe40c80a740 1 Processor -- start 2024-05-20T17:59:52.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.603+0000 7fe40c80a740 1 -- start start 2024-05-20T17:59:52.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.603+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64583a0 0x5598c686c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.603+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c6482f60 0x5598c6864ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.603+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c64a36b0 0x5598c6885690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.603+0000 7fe40c80a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe4001af3c0 con 0x5598c6482f60 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.603+0000 7fe40c80a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4001b1ca0 con 0x5598c64583a0 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.603+0000 7fe40c80a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe4001afa90 con 0x5598c64a36b0 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.603+0000 7fe3fe7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c6482f60 0x5598c6864ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.603+0000 7fe3fe7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c6482f60 0x5598c6864ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52864/0 (socket says 172.21.15.129:52864) 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.603+0000 7fe3fe7fc640 1 -- 172.21.15.129:0/1953343141 learned_addr learned my addr 172.21.15.129:0/1953343141 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3ff7fe640 1 --2- 172.21.15.129:0/1953343141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c64a36b0 0x5598c6885690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3feffd640 1 --2- 172.21.15.129:0/1953343141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64583a0 0x5598c686c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3fe7fc640 1 -- 172.21.15.129:0/1953343141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c64a36b0 msgr2=0x5598c6885690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/1953343141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c64a36b0 0x5598c6885690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3fe7fc640 1 -- 172.21.15.129:0/1953343141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64583a0 msgr2=0x5598c686c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/1953343141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64583a0 0x5598c686c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3fe7fc640 1 -- 172.21.15.129:0/1953343141 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598c6850990 con 0x5598c6482f60 2024-05-20T17:59:52.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3ff7fe640 1 --2- 172.21.15.129:0/1953343141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c64a36b0 0x5598c6885690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:52.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/1953343141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c6482f60 0x5598c6864ee0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009900 tx=0x7fe3f4005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90e27099c13dfd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3e57fa640 1 -- 172.21.15.129:0/1953343141 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4035030 con 0x5598c6482f60 2024-05-20T17:59:52.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3e57fa640 1 -- 172.21.15.129:0/1953343141 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3f40041e0 con 0x5598c6482f60 2024-05-20T17:59:52.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.604+0000 7fe3e57fa640 1 -- 172.21.15.129:0/1953343141 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4040070 con 0x5598c6482f60 2024-05-20T17:59:52.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.605+0000 7fe40c80a740 1 -- 172.21.15.129:0/1953343141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c6482f60 msgr2=0x5598c6864ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.605+0000 7fe40c80a740 1 --2- 172.21.15.129:0/1953343141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c6482f60 0x5598c6864ee0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009900 tx=0x7fe3f4005e60 comp rx=0 tx=0).stop 2024-05-20T17:59:52.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.605+0000 7fe40c80a740 1 -- 172.21.15.129:0/1953343141 shutdown_connections 2024-05-20T17:59:52.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.605+0000 7fe40c80a740 1 --2- 172.21.15.129:0/1953343141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c64a36b0 0x5598c6885690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.605+0000 7fe40c80a740 1 --2- 172.21.15.129:0/1953343141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c6482f60 0x5598c6864ee0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.605+0000 7fe40c80a740 1 --2- 172.21.15.129:0/1953343141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64583a0 0x5598c686c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.605+0000 7fe40c80a740 1 -- 172.21.15.129:0/1953343141 >> 172.21.15.129:0/1953343141 conn(0x5598c64b84e0 msgr2=0x5598c68647d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.605+0000 7fe40c80a740 1 -- 172.21.15.129:0/1953343141 shutdown_connections 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.605+0000 7fe40c80a740 1 -- 172.21.15.129:0/1953343141 wait complete. 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.606+0000 7fe40c80a740 1 -- start start 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.606+0000 7fe40c80a740 1 Processor -- start 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64583a0 0x5598c6872310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 0x5598c6872850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe40c80a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68e6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe40c80a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe4001b0160 con 0x5598c64a36b0 2024-05-20T17:59:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe40c80a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4001b15d0 con 0x5598c64583a0 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe40c80a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe4001b0830 con 0x5598c6482f60 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe40c80a740 10 client.0 _ll_register_callbacks cb 0x7fe3f000cee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3ff7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3ff7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68e6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52876/0 (socket says 172.21.15.129:52876) 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3ff7fe640 1 -- 172.21.15.129:0/1887920074 learned_addr learned my addr 172.21.15.129:0/1887920074 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3fe7fc640 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 0x5598c6872850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3feffd640 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64583a0 0x5598c6872310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3ff7fe640 1 -- 172.21.15.129:0/1887920074 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 msgr2=0x5598c6872850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3ff7fe640 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 0x5598c6872850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3ff7fe640 1 -- 172.21.15.129:0/1887920074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64583a0 msgr2=0x5598c6872310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3ff7fe640 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64583a0 0x5598c6872310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3ff7fe640 1 -- 172.21.15.129:0/1887920074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598c6873050 con 0x5598c64a36b0 2024-05-20T17:59:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3ff7fe640 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68e6240 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800f460 tx=0x7fe3f800f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78081173d3452ae6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.607+0000 7fe3deffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.608+0000 7fe3deffd640 1 -- 172.21.15.129:0/1887920074 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8022070 con 0x5598c64a36b0 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.608+0000 7fe40c80a740 1 -- 172.21.15.129:0/1887920074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3f40041e0 con 0x5598c64a36b0 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.608+0000 7fe3deffd640 1 -- 172.21.15.129:0/1887920074 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fe3f8004d20 con 0x5598c64a36b0 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.608+0000 7fe3deffd640 1 -- 172.21.15.129:0/1887920074 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe3f802b050 con 0x5598c64a36b0 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.608+0000 7fe40c80a740 10 client.15708 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.608+0000 7fe40c80a740 1 -- 172.21.15.129:0/1887920074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5598c68e77f0 con 0x5598c64a36b0 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.608+0000 7fe3deffd640 1 -- 172.21.15.129:0/1887920074 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3f40041e0 con 0x5598c64a36b0 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.609+0000 7fe3deffd640 1 -- 172.21.15.129:0/1887920074 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fe3f8025020 con 0x5598c64a36b0 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.609+0000 7fe40c80a740 10 client.15708 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.609+0000 7fe40c80a740 10 client.15708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.609+0000 7fe40c80a740 20 client.15708 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.609+0000 7fe40c80a740 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5598c64c4390 0x5598c68e9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.609+0000 7fe40c80a740 4 client.15708 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.609+0000 7fe40c80a740 1 -- 172.21.15.129:0/1887920074 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5598c67d0780 con 0x5598c64c4390 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.609+0000 7fe3feffd640 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5598c64c4390 0x5598c68e9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe3deffd640 10 client.15708 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe3feffd640 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5598c64c4390 0x5598c68e9fe0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d1ae66aadafbb70 server_cookie=9a56f0cf9a5e75fc in_seq=0 out_seq=0 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe3deffd640 1 -- 172.21.15.129:0/1887920074 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5598c67d0780 con 0x5598c64c4390 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe3deffd640 10 client.15708 handle_command_reply: tid=0 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 1 client.15708 shutdown 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 1 -- 172.21.15.129:0/1887920074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 msgr2=0x5598c68e6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68e6240 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe3f800f460 tx=0x7fe3f800f920 comp rx=0 tx=0).stop 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 1 -- 172.21.15.129:0/1887920074 shutdown_connections 2024-05-20T17:59:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5598c64c4390 0x5598c68e9fe0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598c64a36b0 0x5598c68e6240 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5598c6482f60 0x5598c6872850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 1 --2- 172.21.15.129:0/1887920074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5598c64583a0 0x5598c6872310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe3deffd640 0 client.15708 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:52.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 1 -- 172.21.15.129:0/1887920074 >> 172.21.15.129:0/1887920074 conn(0x5598c64b84e0 msgr2=0x5598c686d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 1 -- 172.21.15.129:0/1887920074 shutdown_connections 2024-05-20T17:59:52.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 1 -- 172.21.15.129:0/1887920074 wait complete. 2024-05-20T17:59:52.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.610+0000 7fe40c80a740 20 client.15708 trim_cache size 0 max 16384 2024-05-20T17:59:52.615 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T17:59:52.615 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T17:59:52.616 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T17:59:52.616 INFO:teuthology.orchestra.run.smithi129.stdout: "9098578c-c6ea-4243-a3ff-3c2849c1db3b": { 2024-05-20T17:59:52.616 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T17:59:52.616 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "9098578c-c6ea-4243-a3ff-3c2849c1db3b", 2024-05-20T17:59:52.616 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T17:59:52.616 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T17:59:52.616 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T17:59:52.616 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T17:59:52.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 -- 172.21.15.129:0/623083661 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe3e80a2ea0 msgr2=0x7fe3e80a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe3e80a2ea0 0x7fe3e80a53b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40096c0 tx=0x7fe3f4005040 comp rx=0 tx=0).stop 2024-05-20T17:59:52.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 -- 172.21.15.129:0/623083661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4001013b0 msgr2=0x7fe4001a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:52.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4001013b0 0x7fe4001a1780 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000ea40 tx=0x7fe3f000ef00 comp rx=0 tx=0).stop 2024-05-20T17:59:52.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 -- 172.21.15.129:0/623083661 shutdown_connections 2024-05-20T17:59:52.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe3e80a2ea0 0x7fe3e80a53b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe40019b8d0 0x7fe40019bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe400106400 0x7fe4001a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 --2- 172.21.15.129:0/623083661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4001013b0 0x7fe4001a1780 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:52.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 -- 172.21.15.129:0/623083661 >> 172.21.15.129:0/623083661 conn(0x7fe400076830 msgr2=0x7fe4000ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:52.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 -- 172.21.15.129:0/623083661 shutdown_connections 2024-05-20T17:59:52.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T17:59:52.611+0000 7fe405884640 1 -- 172.21.15.129:0/623083661 wait complete. 2024-05-20T17:59:52.624 INFO:tasks.cephfs.filesystem:scrub status for tag:9098578c-c6ea-4243-a3ff-3c2849c1db3b - {'path': '/', 'tag': '9098578c-c6ea-4243-a3ff-3c2849c1db3b', 'options': 'recursive,force'} 2024-05-20T17:59:53.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:52 smithi154 ceph-mon[36482]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s; 64 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T17:59:53.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:52 smithi129 ceph-mon[32029]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s; 64 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T17:59:53.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:52 smithi204 ceph-mon[37076]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s; 64 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T17:59:54.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:53 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T17:59:54.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:53 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:54.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:53 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T17:59:54.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:53 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:54.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:53 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T17:59:54.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:53 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:54.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:53 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T17:59:54.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:53 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:54.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:53 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T17:59:54.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:53 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:54.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:53 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T17:59:54.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:53 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T17:59:55.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:54 smithi154 ceph-mon[36482]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 798 B/s rd, 399 B/s wr, 1 op/s; 50 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T17:59:55.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:54 smithi129 ceph-mon[32029]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 798 B/s rd, 399 B/s wr, 1 op/s; 50 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T17:59:55.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:54 smithi204 ceph-mon[37076]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 798 B/s rd, 399 B/s wr, 1 op/s; 50 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T17:59:56.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:55 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T17:59:56.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:55 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T17:59:56.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:55 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T17:59:57.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:56 smithi154 ceph-mon[36482]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 353 B/s wr, 4 op/s; 44 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T17:59:57.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:56 smithi129 ceph-mon[32029]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 353 B/s wr, 4 op/s; 44 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T17:59:57.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:56 smithi204 ceph-mon[37076]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 353 B/s wr, 4 op/s; 44 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T17:59:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.770+0000 7f8bff1c3640 1 -- 172.21.15.129:0/1418375645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bf810a6a0 msgr2=0x7f8bf810cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.770+0000 7f8bff1c3640 1 --2- 172.21.15.129:0/1418375645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bf810a6a0 0x7f8bf810cac0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009900 tx=0x7f8be402a480 comp rx=0 tx=0).stop 2024-05-20T17:59:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.770+0000 7f8bff1c3640 1 -- 172.21.15.129:0/1418375645 shutdown_connections 2024-05-20T17:59:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.770+0000 7f8bff1c3640 1 --2- 172.21.15.129:0/1418375645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bf810d090 0x7f8bf810f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.770+0000 7f8bff1c3640 1 --2- 172.21.15.129:0/1418375645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bf810a6a0 0x7f8bf810cac0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.770+0000 7f8bff1c3640 1 --2- 172.21.15.129:0/1418375645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bf8103c40 0x7f8bf8104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.769 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.770+0000 7f8bff1c3640 1 -- 172.21.15.129:0/1418375645 >> 172.21.15.129:0/1418375645 conn(0x7f8bf8077160 msgr2=0x7f8bf80775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.770+0000 7f8bff1c3640 1 -- 172.21.15.129:0/1418375645 shutdown_connections 2024-05-20T17:59:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.770+0000 7f8bff1c3640 1 -- 172.21.15.129:0/1418375645 wait complete. 2024-05-20T17:59:57.772 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.771+0000 7f8bff1c3640 1 Processor -- start 2024-05-20T17:59:57.772 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.771+0000 7f8bff1c3640 1 -- start start 2024-05-20T17:59:57.772 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bff1c3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bf8103c40 0x7f8bf819f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.772 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bff1c3640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bf810a6a0 0x7f8bf819fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bff1c3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bf810d090 0x7f8bf8199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bfd760640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bf810d090 0x7f8bf8199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bfd760640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bf810d090 0x7f8bf8199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52890/0 (socket says 172.21.15.129:52890) 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bfd760640 1 -- 172.21.15.129:0/3706267664 learned_addr learned my addr 172.21.15.129:0/3706267664 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bff1c3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf8112e40 con 0x7f8bf810d090 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bff1c3640 1 -- 172.21.15.129:0/3706267664 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf8112cc0 con 0x7f8bf8103c40 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bff1c3640 1 -- 172.21.15.129:0/3706267664 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf8112fc0 con 0x7f8bf810a6a0 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bf7fff640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bf810a6a0 0x7f8bf819fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bfd760640 1 -- 172.21.15.129:0/3706267664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bf810a6a0 msgr2=0x7f8bf819fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bf8103c40 0x7f8bf819f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bfd760640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bf810a6a0 0x7f8bf819fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bfd760640 1 -- 172.21.15.129:0/3706267664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bf8103c40 msgr2=0x7f8bf819f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bfd760640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bf8103c40 0x7f8bf819f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.772+0000 7f8bfd760640 1 -- 172.21.15.129:0/3706267664 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf8199d90 con 0x7f8bf810d090 2024-05-20T17:59:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.773+0000 7f8bf7fff640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bf810a6a0 0x7f8bf819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.773+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bf8103c40 0x7f8bf819f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.773+0000 7f8bfd760640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bf810d090 0x7f8bf8199660 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000eb10 tx=0x7f8bf000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.773+0000 7f8bf5ffb640 1 -- 172.21.15.129:0/3706267664 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0004b20 con 0x7f8bf810d090 2024-05-20T17:59:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.773+0000 7f8bff1c3640 1 -- 172.21.15.129:0/3706267664 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf819a020 con 0x7f8bf810d090 2024-05-20T17:59:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.773+0000 7f8bf5ffb640 1 -- 172.21.15.129:0/3706267664 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf0004cc0 con 0x7f8bf810d090 2024-05-20T17:59:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.773+0000 7f8bf5ffb640 1 -- 172.21.15.129:0/3706267664 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0010760 con 0x7f8bf810d090 2024-05-20T17:59:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.773+0000 7f8bff1c3640 1 -- 172.21.15.129:0/3706267664 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf81a6280 con 0x7f8bf810d090 2024-05-20T17:59:57.775 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.776+0000 7f8c0606f740 1 Processor -- start 2024-05-20T17:59:57.775 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.777+0000 7f8bf5ffb640 1 -- 172.21.15.129:0/3706267664 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7f8bf0010900 con 0x7f8bf810d090 2024-05-20T17:59:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.777+0000 7f8c0606f740 1 -- start start 2024-05-20T17:59:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.777+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc4a41b0 0x5627cc843080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.777+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc47b2f0 0x5627cc8435c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.777+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 0x5627cc843b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.777+0000 7f8c0606f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5627cc7a8340 con 0x5627cc4a41b0 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.777+0000 7f8c0606f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5627cc750290 con 0x5627cc4834d0 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.777+0000 7f8c0606f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5627cc7a4300 con 0x5627cc47b2f0 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc4a41b0 0x5627cc843080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc4a41b0 0x5627cc843080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52892/0 (socket says 172.21.15.129:52892) 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf7fff640 1 -- 172.21.15.129:0/2710626852 learned_addr learned my addr 172.21.15.129:0/2710626852 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf7fff640 1 --2- 172.21.15.129:0/2710626852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 0x5627cc843b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/2710626852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc47b2f0 0x5627cc8435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf7fff640 1 -- 172.21.15.129:0/2710626852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc47b2f0 msgr2=0x5627cc8435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf7fff640 1 --2- 172.21.15.129:0/2710626852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc47b2f0 0x5627cc8435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf7fff640 1 -- 172.21.15.129:0/2710626852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 msgr2=0x5627cc843b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf7fff640 1 --2- 172.21.15.129:0/2710626852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 0x5627cc843b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf7fff640 1 -- 172.21.15.129:0/2710626852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627cc7748e0 con 0x5627cc4a41b0 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/2710626852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc47b2f0 0x5627cc8435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf5ffb640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8be80a2e90 0x7f8be80a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf7fff640 1 --2- 172.21.15.129:0/2710626852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc4a41b0 0x5627cc843080 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8be4000c00 tx=0x7f8be40095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b87536ab4f6c05 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8be80a2e90 0x7f8be80a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.778+0000 7f8bf5ffb640 1 -- 172.21.15.129:0/3706267664 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f8bf00cfdc0 con 0x7f8bf810d090 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.779+0000 7f8be2ffd640 1 -- 172.21.15.129:0/2710626852 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4044070 con 0x5627cc4a41b0 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.779+0000 7f8be2ffd640 1 -- 172.21.15.129:0/2710626852 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8be4005020 con 0x5627cc4a41b0 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.779+0000 7f8be2ffd640 1 -- 172.21.15.129:0/2710626852 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4035040 con 0x5627cc4a41b0 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.779+0000 7f8c0606f740 1 -- 172.21.15.129:0/2710626852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc4a41b0 msgr2=0x5627cc843080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.779+0000 7f8c0606f740 1 --2- 172.21.15.129:0/2710626852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc4a41b0 0x5627cc843080 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8be4000c00 tx=0x7f8be40095a0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.780+0000 7f8c0606f740 1 -- 172.21.15.129:0/2710626852 shutdown_connections 2024-05-20T17:59:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.780+0000 7f8c0606f740 1 --2- 172.21.15.129:0/2710626852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 0x5627cc843b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.780+0000 7f8c0606f740 1 --2- 172.21.15.129:0/2710626852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc47b2f0 0x5627cc8435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.780+0000 7f8c0606f740 1 --2- 172.21.15.129:0/2710626852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc4a41b0 0x5627cc843080 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.780+0000 7f8c0606f740 1 -- 172.21.15.129:0/2710626852 >> 172.21.15.129:0/2710626852 conn(0x5627cc498300 msgr2=0x5627cc83f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.780+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8be80a2e90 0x7f8be80a53a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x5627cc844430 tx=0x7f8bec005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.780+0000 7f8c0606f740 1 -- 172.21.15.129:0/2710626852 shutdown_connections 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.780+0000 7f8c0606f740 1 -- 172.21.15.129:0/2710626852 wait complete. 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.780+0000 7f8c0606f740 1 -- start start 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.780+0000 7f8c0606f740 1 Processor -- start 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc84f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 0x5627cc84f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 0x5627cc84d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8c0606f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5627cc850150 con 0x5627cc47b2f0 2024-05-20T17:59:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8c0606f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5627cc763790 con 0x5627cc462d80 2024-05-20T17:59:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8c0606f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5627cc8502d0 con 0x5627cc4834d0 2024-05-20T17:59:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8c0606f740 10 client.0 _ll_register_callbacks cb 0x5627cc79ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8bfcf5f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc84f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8bfd760640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 0x5627cc84d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8bf7fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 0x5627cc84f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8bf7fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 0x5627cc84f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50396/0 (socket says 172.21.15.129:50396) 2024-05-20T17:59:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8bf7fff640 1 -- 172.21.15.129:0/3444083431 learned_addr learned my addr 172.21.15.129:0/3444083431 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8bfd760640 1 -- 172.21.15.129:0/3444083431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 msgr2=0x5627cc84f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.781+0000 7f8bfd760640 1 --2- 172.21.15.129:0/3444083431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 0x5627cc84f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.782+0000 7f8bfd760640 1 -- 172.21.15.129:0/3444083431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 msgr2=0x5627cc84f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.782+0000 7f8bfd760640 1 --2- 172.21.15.129:0/3444083431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc84f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.782+0000 7f8bfd760640 1 -- 172.21.15.129:0/3444083431 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627cc4bc210 con 0x5627cc462d80 2024-05-20T17:59:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.782+0000 7f8bfd760640 1 --2- 172.21.15.129:0/3444083431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 0x5627cc84d710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00cd380 tx=0x7f8bf009bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58819e63e986564a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.782+0000 7f8bd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T17:59:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.782+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3444083431 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf009c080 con 0x5627cc462d80 2024-05-20T17:59:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.782+0000 7f8c0606f740 1 -- 172.21.15.129:0/3444083431 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8be4005020 con 0x5627cc462d80 2024-05-20T17:59:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.782+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3444083431 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f8bf00a4860 con 0x5627cc462d80 2024-05-20T17:59:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.782+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3444083431 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8bf00a0430 con 0x5627cc462d80 2024-05-20T17:59:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.782+0000 7f8c0606f740 10 client.25403 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:57.786 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.782+0000 7f8c0606f740 1 -- 172.21.15.129:0/3444083431 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5627cc588620 con 0x5627cc462d80 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.783+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3444083431 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8be4005020 con 0x5627cc462d80 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.783+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3444083431 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f8bf009b4d0 con 0x5627cc462d80 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.783+0000 7f8c0606f740 10 client.25403 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.783+0000 7f8c0606f740 10 client.25403 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.783+0000 7f8c0606f740 20 client.25403 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.783+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3444083431 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627cc4381c0 0x5627cc863db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.783+0000 7f8c0606f740 4 client.25403 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.783+0000 7f8c0606f740 1 -- 172.21.15.129:0/3444083431 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627cc7b5a70 con 0x5627cc4381c0 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.783+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/3444083431 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627cc4381c0 0x5627cc863db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.784+0000 7f8bd7fff640 10 client.25403 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.784+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/3444083431 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627cc4381c0 0x5627cc863db0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef657f03a3594935 server_cookie=a94f6a9894839471 in_seq=0 out_seq=0 2024-05-20T17:59:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.784+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3444083431 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5627cc7b5a70 con 0x5627cc4381c0 2024-05-20T17:59:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.784+0000 7f8bd7fff640 10 client.25403 handle_command_reply: tid=0 2024-05-20T17:59:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.784+0000 7f8c0606f740 1 client.25403 shutdown 2024-05-20T17:59:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8c0606f740 1 -- 172.21.15.129:0/3444083431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 msgr2=0x5627cc84d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3444083431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 0x5627cc84d710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00cd380 tx=0x7f8bf009bf50 comp rx=0 tx=0).stop 2024-05-20T17:59:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8c0606f740 1 -- 172.21.15.129:0/3444083431 shutdown_connections 2024-05-20T17:59:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3444083431 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627cc4381c0 0x5627cc863db0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3444083431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 0x5627cc84d710 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3444083431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 0x5627cc84f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8bd7fff640 0 client.25403 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3444083431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc84f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8c0606f740 1 -- 172.21.15.129:0/3444083431 >> 172.21.15.129:0/3444083431 conn(0x5627cc498300 msgr2=0x5627cc842fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8c0606f740 1 -- 172.21.15.129:0/3444083431 shutdown_connections 2024-05-20T17:59:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8c0606f740 1 -- 172.21.15.129:0/3444083431 wait complete. 2024-05-20T17:59:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.785+0000 7f8c0606f740 20 client.25403 trim_cache size 0 max 16384 2024-05-20T17:59:57.799 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.800+0000 7f8c0606f740 1 Processor -- start 2024-05-20T17:59:57.799 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.800+0000 7f8c0606f740 1 -- start start 2024-05-20T17:59:57.799 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.800+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 0x5627cc84c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.799 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.800+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc843670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.799 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.800+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 0x5627cc846cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.799 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.800+0000 7f8c0606f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81aa850 con 0x5627cc47b2f0 2024-05-20T17:59:57.799 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.800+0000 7f8c0606f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81ad0d0 con 0x5627cc462d80 2024-05-20T17:59:57.799 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.800+0000 7f8c0606f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81aaec0 con 0x5627cc4834d0 2024-05-20T17:59:57.799 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bf7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc843670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.799 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bfd760640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 0x5627cc846cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.799 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bfd760640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 0x5627cc846cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50408/0 (socket says 172.21.15.129:50408) 2024-05-20T17:59:57.800 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bfd760640 1 -- 172.21.15.129:0/194252188 learned_addr learned my addr 172.21.15.129:0/194252188 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:57.800 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bf7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc843670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52914/0 (socket says 172.21.15.129:52914) 2024-05-20T17:59:57.800 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/194252188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 0x5627cc84c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.801 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bfd760640 1 -- 172.21.15.129:0/194252188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 msgr2=0x5627cc84c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bfd760640 1 --2- 172.21.15.129:0/194252188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 0x5627cc84c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bfd760640 1 -- 172.21.15.129:0/194252188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 msgr2=0x5627cc843670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bfd760640 1 --2- 172.21.15.129:0/194252188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc843670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bfd760640 1 -- 172.21.15.129:0/194252188 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627cc843f40 con 0x5627cc4834d0 2024-05-20T17:59:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/194252188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 0x5627cc84c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T17:59:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8bfd760640 1 --2- 172.21.15.129:0/194252188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 0x5627cc846cd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00cd490 tx=0x7f8bf00bc050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7ebb3a947157bf0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8be27fc640 1 -- 172.21.15.129:0/194252188 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf009b9e0 con 0x5627cc4834d0 2024-05-20T17:59:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8be27fc640 1 -- 172.21.15.129:0/194252188 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf009c080 con 0x5627cc4834d0 2024-05-20T17:59:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.801+0000 7f8be27fc640 1 -- 172.21.15.129:0/194252188 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf009b4d0 con 0x5627cc4834d0 2024-05-20T17:59:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.802+0000 7f8c0606f740 1 -- 172.21.15.129:0/194252188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 msgr2=0x5627cc846cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:57.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.802+0000 7f8c0606f740 1 --2- 172.21.15.129:0/194252188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 0x5627cc846cd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00cd490 tx=0x7f8bf00bc050 comp rx=0 tx=0).stop 2024-05-20T17:59:57.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.802+0000 7f8c0606f740 1 -- 172.21.15.129:0/194252188 shutdown_connections 2024-05-20T17:59:57.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.802+0000 7f8c0606f740 1 --2- 172.21.15.129:0/194252188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc4834d0 0x5627cc846cd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.802+0000 7f8c0606f740 1 --2- 172.21.15.129:0/194252188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc843670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.802+0000 7f8c0606f740 1 --2- 172.21.15.129:0/194252188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc462d80 0x5627cc84c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.802+0000 7f8c0606f740 1 -- 172.21.15.129:0/194252188 >> 172.21.15.129:0/194252188 conn(0x5627cc498300 msgr2=0x5627cc842f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:57.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.803+0000 7f8c0606f740 1 -- 172.21.15.129:0/194252188 shutdown_connections 2024-05-20T17:59:57.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.803+0000 7f8c0606f740 1 -- 172.21.15.129:0/194252188 wait complete. 2024-05-20T17:59:57.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.803+0000 7f8c0606f740 1 -- start start 2024-05-20T17:59:57.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.803+0000 7f8c0606f740 1 Processor -- start 2024-05-20T17:59:57.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.803+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc462d80 0x5627cc859f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.803+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc8572e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.803+0000 7f8c0606f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 0x5627cc857820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8c0606f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81ab590 con 0x5627cc47b2f0 2024-05-20T17:59:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8c0606f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81aca00 con 0x5627cc4834d0 2024-05-20T17:59:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8c0606f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81abc60 con 0x5627cc462d80 2024-05-20T17:59:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8c0606f740 10 client.0 _ll_register_callbacks cb 0x7f8bf0098320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T17:59:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bfd760640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 0x5627cc857820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bfd760640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 0x5627cc857820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:48546/0 (socket says 172.21.15.129:48546) 2024-05-20T17:59:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bfd760640 1 -- 172.21.15.129:0/3842216813 learned_addr learned my addr 172.21.15.129:0/3842216813 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc462d80 0x5627cc859f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bf7fff640 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc8572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bfd760640 1 -- 172.21.15.129:0/3842216813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc462d80 msgr2=0x5627cc859f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bfd760640 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc462d80 0x5627cc859f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bfd760640 1 -- 172.21.15.129:0/3842216813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 msgr2=0x5627cc8572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bfd760640 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc8572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bfd760640 1 -- 172.21.15.129:0/3842216813 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627cc844460 con 0x5627cc4834d0 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc462d80 0x5627cc859f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bf7fff640 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc8572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bfd760640 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 0x5627cc857820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00cdb00 tx=0x7f8bf000efd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=378a3cd24b16089e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.804+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3842216813 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf009b9e0 con 0x5627cc4834d0 2024-05-20T17:59:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3842216813 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f8bf00a4a90 con 0x5627cc4834d0 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8c0606f740 1 -- 172.21.15.129:0/3842216813 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bf009c080 con 0x5627cc4834d0 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3842216813 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8bf00cd2c0 con 0x5627cc4834d0 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8c0606f740 10 client.25409 fetch_fsmap learned FSMap version 22 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8c0606f740 1 -- 172.21.15.129:0/3842216813 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5627cc85a920 con 0x5627cc4834d0 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3842216813 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf009c080 con 0x5627cc4834d0 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3842216813 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f8bf00cd660 con 0x5627cc4834d0 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8c0606f740 10 client.25409 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8c0606f740 10 client.25409 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8c0606f740 20 client.25409 populate_metadata read hostname 'smithi129' 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627cc85adb0 0x5627cc85d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.805+0000 7f8c0606f740 4 client.25409 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.806+0000 7f8c0606f740 1 -- 172.21.15.129:0/3842216813 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5627cc7afdc0 con 0x5627cc85adb0 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.806+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627cc85adb0 0x5627cc85d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T17:59:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.806+0000 7f8bd7fff640 10 client.25409 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.806+0000 7f8bfcf5f640 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627cc85adb0 0x5627cc85d1d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23bceb8d8126275c server_cookie=85c088c6afa7719e in_seq=0 out_seq=0 2024-05-20T17:59:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.810+0000 7f8bd7fff640 1 -- 172.21.15.129:0/3842216813 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5627cc7afdc0 con 0x5627cc85adb0 2024-05-20T17:59:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.810+0000 7f8bd7fff640 10 client.25409 handle_command_reply: tid=0 2024-05-20T17:59:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.810+0000 7f8c0606f740 1 client.25409 shutdown 2024-05-20T17:59:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8c0606f740 1 -- 172.21.15.129:0/3842216813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 msgr2=0x5627cc857820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 0x5627cc857820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00cdb00 tx=0x7f8bf000efd0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8c0606f740 1 -- 172.21.15.129:0/3842216813 shutdown_connections 2024-05-20T17:59:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627cc85adb0 0x5627cc85d1d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627cc4834d0 0x5627cc857820 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8bd7fff640 0 client.25409 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T17:59:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627cc47b2f0 0x5627cc8572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8c0606f740 1 --2- 172.21.15.129:0/3842216813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627cc462d80 0x5627cc859f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8c0606f740 1 -- 172.21.15.129:0/3842216813 >> 172.21.15.129:0/3842216813 conn(0x5627cc498300 msgr2=0x5627cc84cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8c0606f740 1 -- 172.21.15.129:0/3842216813 shutdown_connections 2024-05-20T17:59:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8c0606f740 1 -- 172.21.15.129:0/3842216813 wait complete. 2024-05-20T17:59:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8c0606f740 20 client.25409 trim_cache size 0 max 16384 2024-05-20T17:59:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T17:59:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T17:59:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T17:59:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T17:59:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8bff1c3640 1 -- 172.21.15.129:0/3706267664 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8be80a2e90 msgr2=0x7f8be80a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8bff1c3640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8be80a2e90 0x7f8be80a53a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x5627cc844430 tx=0x7f8bec005f90 comp rx=0 tx=0).stop 2024-05-20T17:59:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8bff1c3640 1 -- 172.21.15.129:0/3706267664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bf810d090 msgr2=0x7f8bf8199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T17:59:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.811+0000 7f8bff1c3640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bf810d090 0x7f8bf8199660 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000eb10 tx=0x7f8bf000c2d0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.812+0000 7f8bff1c3640 1 -- 172.21.15.129:0/3706267664 shutdown_connections 2024-05-20T17:59:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.812+0000 7f8bff1c3640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8be80a2e90 0x7f8be80a53a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.812+0000 7f8bff1c3640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8bf810d090 0x7f8bf8199660 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.812+0000 7f8bff1c3640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8bf810a6a0 0x7f8bf819fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.812+0000 7f8bff1c3640 1 --2- 172.21.15.129:0/3706267664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bf8103c40 0x7f8bf819f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T17:59:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.812+0000 7f8bff1c3640 1 -- 172.21.15.129:0/3706267664 >> 172.21.15.129:0/3706267664 conn(0x7f8bf8077160 msgr2=0x7f8bf8100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T17:59:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.812+0000 7f8bff1c3640 1 -- 172.21.15.129:0/3706267664 shutdown_connections 2024-05-20T17:59:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T17:59:57.812+0000 7f8bff1c3640 1 -- 172.21.15.129:0/3706267664 wait complete. 2024-05-20T17:59:59.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 17:59:58 smithi154 ceph-mon[36482]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 307 B/s wr, 3 op/s; 38 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T17:59:59.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 17:59:58 smithi129 ceph-mon[32029]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 307 B/s wr, 3 op/s; 38 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T17:59:59.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 17:59:58 smithi204 ceph-mon[37076]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 307 B/s wr, 3 op/s; 38 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T18:00:01.231 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:00 smithi204 ceph-mon[37076]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 525 B/s wr, 4 op/s; 33 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T18:00:01.231 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:00 smithi204 ceph-mon[37076]: overall HEALTH_OK 2024-05-20T18:00:01.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:00 smithi154 ceph-mon[36482]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 525 B/s wr, 4 op/s; 33 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T18:00:01.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:00 smithi154 ceph-mon[36482]: overall HEALTH_OK 2024-05-20T18:00:01.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:00 smithi129 ceph-mon[32029]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 525 B/s wr, 4 op/s; 33 KiB/s, 0 keys/s, 0 objects/s recovering 2024-05-20T18:00:01.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:00 smithi129 ceph-mon[32029]: overall HEALTH_OK 2024-05-20T18:00:03.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:02 smithi154 ceph-mon[36482]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-20T18:00:03.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:02 smithi129 ceph-mon[32029]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-20T18:00:03.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:02 smithi204 ceph-mon[37076]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-20T18:00:05.241 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:04 smithi154 ceph-mon[36482]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-20T18:00:05.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:04 smithi129 ceph-mon[32029]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-20T18:00:05.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:04 smithi204 ceph-mon[37076]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-20T18:00:07.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:06 smithi154 ceph-mon[36482]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-05-20T18:00:07.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:06 smithi129 ceph-mon[32029]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-05-20T18:00:07.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:06 smithi204 ceph-mon[37076]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-05-20T18:00:09.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:08 smithi154 ceph-mon[36482]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:00:09.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:08 smithi129 ceph-mon[32029]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:00:09.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:08 smithi204 ceph-mon[37076]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:00:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.957+0000 7fdf9512f640 1 -- 172.21.15.129:0/1724505393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf90107eb0 msgr2=0x7fdf90101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.957+0000 7fdf9512f640 1 --2- 172.21.15.129:0/1724505393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf90107eb0 0x7fdf90101b00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fdf84009900 tx=0x7fdf8402a410 comp rx=0 tx=0).stop 2024-05-20T18:00:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.957+0000 7fdf9512f640 1 -- 172.21.15.129:0/1724505393 shutdown_connections 2024-05-20T18:00:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.957+0000 7fdf9512f640 1 --2- 172.21.15.129:0/1724505393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf90102170 0x7fdf9010e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.957+0000 7fdf9512f640 1 --2- 172.21.15.129:0/1724505393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf90107eb0 0x7fdf90101b00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.957+0000 7fdf9512f640 1 --2- 172.21.15.129:0/1724505393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf90107a10 0x7fdf901015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.957+0000 7fdf9512f640 1 -- 172.21.15.129:0/1724505393 >> 172.21.15.129:0/1724505393 conn(0x7fdf90077160 msgr2=0x7fdf900775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.958+0000 7fdf9512f640 1 -- 172.21.15.129:0/1724505393 shutdown_connections 2024-05-20T18:00:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.958+0000 7fdf9512f640 1 -- 172.21.15.129:0/1724505393 wait complete. 2024-05-20T18:00:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.958+0000 7fdf9512f640 1 Processor -- start 2024-05-20T18:00:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf9512f640 1 -- start start 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf9512f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf90102170 0x7fdf901984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf9512f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf90107a10 0x7fdf90198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf9512f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf90107eb0 0x7fdf90198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf9512f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90111d60 con 0x7fdf90107a10 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf9512f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90111be0 con 0x7fdf90107eb0 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf9512f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdf90111ee0 con 0x7fdf90102170 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf8e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf90107a10 0x7fdf90198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf8e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf90107a10 0x7fdf90198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48642/0 (socket says 172.21.15.129:48642) 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf8e59c640 1 -- 172.21.15.129:0/1274803759 learned_addr learned my addr 172.21.15.129:0/1274803759 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf90107eb0 0x7fdf90198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.959+0000 7fdf8ed9d640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf90102170 0x7fdf901984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf8e59c640 1 -- 172.21.15.129:0/1274803759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf90102170 msgr2=0x7fdf901984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf8e59c640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf90102170 0x7fdf901984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf8e59c640 1 -- 172.21.15.129:0/1274803759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf90107eb0 msgr2=0x7fdf90198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf8e59c640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf90107eb0 0x7fdf90198f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf8e59c640 1 -- 172.21.15.129:0/1274803759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf901a7b70 con 0x7fdf90107a10 2024-05-20T18:00:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf90107eb0 0x7fdf90198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:00:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf8e59c640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf90107a10 0x7fdf90198a30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdf84005940 tx=0x7fdf840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf77fff640 1 -- 172.21.15.129:0/1274803759 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8403f030 con 0x7fdf90107a10 2024-05-20T18:00:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf9512f640 1 -- 172.21.15.129:0/1274803759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf901a7e00 con 0x7fdf90107a10 2024-05-20T18:00:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf77fff640 1 -- 172.21.15.129:0/1274803759 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf84003ce0 con 0x7fdf90107a10 2024-05-20T18:00:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf77fff640 1 -- 172.21.15.129:0/1274803759 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84035070 con 0x7fdf90107a10 2024-05-20T18:00:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.960+0000 7fdf9512f640 1 -- 172.21.15.129:0/1274803759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf901a8260 con 0x7fdf90107a10 2024-05-20T18:00:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.964+0000 7fdf77fff640 1 -- 172.21.15.129:0/1274803759 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 24) ==== 138067+0+0 (secure 0 0 0) 0x7fdf84042030 con 0x7fdf90107a10 2024-05-20T18:00:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.964+0000 7fdf9bfd5740 1 Processor -- start 2024-05-20T18:00:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.964+0000 7fdf9bfd5740 1 -- start start 2024-05-20T18:00:10.963 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5611190871b0 0x561119426080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:10.963 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x5611194265c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:10.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611190664d0 0x561119426b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:10.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf9bfd5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56111938b340 con 0x56111905e2f0 2024-05-20T18:00:10.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf9bfd5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561119333290 con 0x5611190664d0 2024-05-20T18:00:10.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf9bfd5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561119387300 con 0x5611190871b0 2024-05-20T18:00:10.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf77fff640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdf780a2e10 0x7fdf780a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:10.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf8ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x5611194265c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:10.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf8ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x5611194265c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48650/0 (socket says 172.21.15.129:48650) 2024-05-20T18:00:10.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf8ed9d640 1 -- 172.21.15.129:0/262726682 learned_addr learned my addr 172.21.15.129:0/262726682 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/262726682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5611190871b0 0x561119426080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf77fff640 1 -- 172.21.15.129:0/1274803759 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fdf840ef460 con 0x7fdf90107a10 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdf780a2e10 0x7fdf780a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf8ed9d640 1 -- 172.21.15.129:0/262726682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5611190871b0 msgr2=0x561119426080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf8ed9d640 1 --2- 172.21.15.129:0/262726682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5611190871b0 0x561119426080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf8ed9d640 1 -- 172.21.15.129:0/262726682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611190664d0 msgr2=0x561119426b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf8ed9d640 1 --2- 172.21.15.129:0/262726682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611190664d0 0x561119426b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.965+0000 7fdf8ed9d640 1 -- 172.21.15.129:0/262726682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611193578e0 con 0x56111905e2f0 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.966+0000 7fdf8ed9d640 1 --2- 172.21.15.129:0/262726682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x5611194265c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000b770 tx=0x7fdf8000bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5caf0371595b78fa server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.966+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdf780a2e10 0x7fdf780a5320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009fd0 tx=0x7fdf88009f00 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.966+0000 7fdf757fa640 1 -- 172.21.15.129:0/262726682 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80014070 con 0x56111905e2f0 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.966+0000 7fdf757fa640 1 -- 172.21.15.129:0/262726682 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf800162c0 con 0x56111905e2f0 2024-05-20T18:00:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.966+0000 7fdf757fa640 1 -- 172.21.15.129:0/262726682 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf800107e0 con 0x56111905e2f0 2024-05-20T18:00:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.967+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/262726682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 msgr2=0x5611194265c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.967+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/262726682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x5611194265c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000b770 tx=0x7fdf8000bc30 comp rx=0 tx=0).stop 2024-05-20T18:00:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.967+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/262726682 shutdown_connections 2024-05-20T18:00:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.967+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/262726682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611190664d0 0x561119426b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.967+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/262726682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x5611194265c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.967+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/262726682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5611190871b0 0x561119426080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.967+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/262726682 >> 172.21.15.129:0/262726682 conn(0x56111907b300 msgr2=0x561119422830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.967+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/262726682 shutdown_connections 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.967+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/262726682 wait complete. 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.968+0000 7fdf9bfd5740 1 -- start start 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.968+0000 7fdf9bfd5740 1 Processor -- start 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.968+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56111905e2f0 0x5611194322e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5611190664d0 0x561119432820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190871b0 0x561119430720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf9bfd5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5611194331b0 con 0x5611190871b0 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56111905e2f0 0x5611194322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56111905e2f0 0x5611194322e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:44048/0 (socket says 172.21.15.129:44048) 2024-05-20T18:00:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8ed9d640 1 -- 172.21.15.129:0/861987360 learned_addr learned my addr 172.21.15.129:0/861987360 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/861987360 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561119346790 con 0x56111905e2f0 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190871b0 0x561119430720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/861987360 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561119433330 con 0x5611190664d0 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8e59c640 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5611190664d0 0x561119432820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf9bfd5740 10 client.0 _ll_register_callbacks cb 0x561119381a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8f59e640 1 -- 172.21.15.129:0/861987360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5611190664d0 msgr2=0x561119432820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5611190664d0 0x561119432820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8f59e640 1 -- 172.21.15.129:0/861987360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56111905e2f0 msgr2=0x5611194322e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56111905e2f0 0x5611194322e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8f59e640 1 -- 172.21.15.129:0/861987360 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56111909f210 con 0x5611190871b0 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8e59c640 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5611190664d0 0x561119432820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8ed9d640 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56111905e2f0 0x5611194322e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:00:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190871b0 0x561119430720 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fdf880098b0 tx=0x7fdf88009dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b6198e7fbd2b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.969+0000 7fdf72ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf72ff5640 1 -- 172.21.15.129:0/861987360 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88037070 con 0x5611190871b0 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf72ff5640 1 -- 172.21.15.129:0/861987360 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fdf8802fd70 con 0x5611190871b0 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/861987360 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf800162c0 con 0x5611190871b0 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf72ff5640 1 -- 172.21.15.129:0/861987360 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdf8802b040 con 0x5611190871b0 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf9bfd5740 10 client.15738 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/861987360 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56111916b620 con 0x5611190871b0 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf72ff5640 1 -- 172.21.15.129:0/861987360 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf800162c0 con 0x5611190871b0 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf72ff5640 1 -- 172.21.15.129:0/861987360 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fdf8802e040 con 0x5611190871b0 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf9bfd5740 10 client.15738 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf9bfd5740 10 client.15738 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf9bfd5740 20 client.15738 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561119045d80 0x561119442870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf9bfd5740 4 client.15738 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:00:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.970+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/861987360 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561119398a70 con 0x561119045d80 2024-05-20T18:00:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.971+0000 7fdf8ed9d640 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561119045d80 0x561119442870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.971+0000 7fdf72ff5640 10 client.15738 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.971+0000 7fdf8ed9d640 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561119045d80 0x561119442870 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f016abff00ae605 server_cookie=6991ffb03a0cad68 in_seq=0 out_seq=0 2024-05-20T18:00:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf72ff5640 1 -- 172.21.15.129:0/861987360 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561119398a70 con 0x561119045d80 2024-05-20T18:00:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf72ff5640 10 client.15738 handle_command_reply: tid=0 2024-05-20T18:00:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 1 client.15738 shutdown 2024-05-20T18:00:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/861987360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190871b0 msgr2=0x561119430720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190871b0 0x561119430720 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fdf880098b0 tx=0x7fdf88009dd0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/861987360 shutdown_connections 2024-05-20T18:00:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561119045d80 0x561119442870 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf72ff5640 0 client.15738 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190871b0 0x561119430720 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5611190664d0 0x561119432820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/861987360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56111905e2f0 0x5611194322e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/861987360 >> 172.21.15.129:0/861987360 conn(0x56111907b300 msgr2=0x561119427da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/861987360 shutdown_connections 2024-05-20T18:00:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/861987360 wait complete. 2024-05-20T18:00:10.975 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:10.972+0000 7fdf9bfd5740 20 client.15738 trim_cache size 0 max 16384 2024-05-20T18:00:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.002+0000 7fdf9bfd5740 1 Processor -- start 2024-05-20T18:00:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.002+0000 7fdf9bfd5740 1 -- start start 2024-05-20T18:00:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561119045d80 0x5611194283d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x561119448070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611190664d0 0x5611194485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf9bfd5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901aa8d0 con 0x56111905e2f0 2024-05-20T18:00:11.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf9bfd5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901ad150 con 0x5611190664d0 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf9bfd5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901aaf40 con 0x561119045d80 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf8e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x561119448070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf8e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x561119448070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48662/0 (socket says 172.21.15.129:48662) 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf8e59c640 1 -- 172.21.15.129:0/3924029981 learned_addr learned my addr 172.21.15.129:0/3924029981 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf8ed9d640 1 --2- 172.21.15.129:0/3924029981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561119045d80 0x5611194283d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/3924029981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611190664d0 0x5611194485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf8e59c640 1 -- 172.21.15.129:0/3924029981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561119045d80 msgr2=0x5611194283d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf8e59c640 1 --2- 172.21.15.129:0/3924029981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561119045d80 0x5611194283d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf8e59c640 1 -- 172.21.15.129:0/3924029981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611190664d0 msgr2=0x5611194485b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf8e59c640 1 --2- 172.21.15.129:0/3924029981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611190664d0 0x5611194485b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf8e59c640 1 -- 172.21.15.129:0/3924029981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561119428910 con 0x56111905e2f0 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.003+0000 7fdf8e59c640 1 --2- 172.21.15.129:0/3924029981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x561119448070 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fdf84000c00 tx=0x7fdf84005f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1de8713c9761204 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:11.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.004+0000 7fdf74ff9640 1 -- 172.21.15.129:0/3924029981 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84049040 con 0x56111905e2f0 2024-05-20T18:00:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.004+0000 7fdf74ff9640 1 -- 172.21.15.129:0/3924029981 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf840bb8a0 con 0x56111905e2f0 2024-05-20T18:00:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.004+0000 7fdf74ff9640 1 -- 172.21.15.129:0/3924029981 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf840b7f10 con 0x56111905e2f0 2024-05-20T18:00:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.005+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3924029981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 msgr2=0x561119448070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.005+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/3924029981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x561119448070 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fdf84000c00 tx=0x7fdf84005f20 comp rx=0 tx=0).stop 2024-05-20T18:00:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.005+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3924029981 shutdown_connections 2024-05-20T18:00:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.005+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/3924029981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611190664d0 0x5611194485b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.005+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/3924029981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56111905e2f0 0x561119448070 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.005+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/3924029981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561119045d80 0x5611194283d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.005+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3924029981 >> 172.21.15.129:0/3924029981 conn(0x56111907b300 msgr2=0x561119427cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.005+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3924029981 shutdown_connections 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.005+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3924029981 wait complete. 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf9bfd5740 1 -- start start 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf9bfd5740 1 Processor -- start 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561119045d80 0x5611194383e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56111905e2f0 0x561119438920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf9bfd5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190664d0 0x5611194358e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf9bfd5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901ab610 con 0x5611190664d0 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf9bfd5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901aca80 con 0x561119045d80 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf9bfd5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdf901abce0 con 0x56111905e2f0 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf9bfd5740 10 client.0 _ll_register_callbacks cb 0x7fdf88010f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:11.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf8f59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190664d0 0x5611194358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf8f59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190664d0 0x5611194358e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48664/0 (socket says 172.21.15.129:48664) 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.006+0000 7fdf8f59e640 1 -- 172.21.15.129:0/3784715051 learned_addr learned my addr 172.21.15.129:0/3784715051 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf8ed9d640 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561119045d80 0x5611194383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf8e59c640 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56111905e2f0 0x561119438920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf8f59e640 1 -- 172.21.15.129:0/3784715051 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56111905e2f0 msgr2=0x561119438920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56111905e2f0 0x561119438920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf8f59e640 1 -- 172.21.15.129:0/3784715051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561119045d80 msgr2=0x5611194383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561119045d80 0x5611194383e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf8f59e640 1 -- 172.21.15.129:0/3784715051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561119435ef0 con 0x5611190664d0 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf8f59e640 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190664d0 0x5611194358e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdf88005a20 tx=0x7fdf8802ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344feb560487c1d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf72ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf72ff5640 1 -- 172.21.15.129:0/3784715051 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88047030 con 0x5611190664d0 2024-05-20T18:00:11.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf72ff5640 1 -- 172.21.15.129:0/3784715051 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fdf8803f020 con 0x5611190664d0 2024-05-20T18:00:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.007+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3784715051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf840bb8a0 con 0x5611190664d0 2024-05-20T18:00:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.008+0000 7fdf72ff5640 1 -- 172.21.15.129:0/3784715051 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdf88052050 con 0x5611190664d0 2024-05-20T18:00:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.008+0000 7fdf9bfd5740 10 client.15744 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:11.009 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.008+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3784715051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5611194aa550 con 0x5611190664d0 2024-05-20T18:00:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.008+0000 7fdf72ff5640 1 -- 172.21.15.129:0/3784715051 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf840bb8a0 con 0x5611190664d0 2024-05-20T18:00:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.008+0000 7fdf72ff5640 1 -- 172.21.15.129:0/3784715051 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fdf8802e040 con 0x5611190664d0 2024-05-20T18:00:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.009+0000 7fdf9bfd5740 10 client.15744 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.009+0000 7fdf9bfd5740 10 client.15744 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.009+0000 7fdf9bfd5740 20 client.15744 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.009+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5611190871b0 0x5611194acc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.009+0000 7fdf9bfd5740 4 client.15744 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:00:11.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.009+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3784715051 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561119392dc0 con 0x5611190871b0 2024-05-20T18:00:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.009+0000 7fdf8ed9d640 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5611190871b0 0x5611194acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.009+0000 7fdf72ff5640 10 client.15744 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.009+0000 7fdf8ed9d640 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5611190871b0 0x5611194acc40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=897a153236250f03 server_cookie=dcec2958b56ff1d4 in_seq=0 out_seq=0 2024-05-20T18:00:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.016+0000 7fdf72ff5640 1 -- 172.21.15.129:0/3784715051 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561119392dc0 con 0x5611190871b0 2024-05-20T18:00:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.016+0000 7fdf72ff5640 10 client.15744 handle_command_reply: tid=0 2024-05-20T18:00:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.016+0000 7fdf9bfd5740 1 client.15744 shutdown 2024-05-20T18:00:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.016+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3784715051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190664d0 msgr2=0x5611194358e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.016+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190664d0 0x5611194358e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdf88005a20 tx=0x7fdf8802ff70 comp rx=0 tx=0).stop 2024-05-20T18:00:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.016+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3784715051 shutdown_connections 2024-05-20T18:00:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.016+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5611190871b0 0x5611194acc40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5611190664d0 0x5611194358e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56111905e2f0 0x561119438920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf72ff5640 0 client.15744 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf9bfd5740 1 --2- 172.21.15.129:0/3784715051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561119045d80 0x5611194383e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3784715051 >> 172.21.15.129:0/3784715051 conn(0x56111907b300 msgr2=0x561119425e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3784715051 shutdown_connections 2024-05-20T18:00:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf9bfd5740 1 -- 172.21.15.129:0/3784715051 wait complete. 2024-05-20T18:00:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf9bfd5740 20 client.15744 trim_cache size 0 max 16384 2024-05-20T18:00:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:00:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:00:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:00:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:00:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf9512f640 1 -- 172.21.15.129:0/1274803759 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdf780a2e10 msgr2=0x7fdf780a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf9512f640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdf780a2e10 0x7fdf780a5320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009fd0 tx=0x7fdf88009f00 comp rx=0 tx=0).stop 2024-05-20T18:00:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf9512f640 1 -- 172.21.15.129:0/1274803759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf90107a10 msgr2=0x7fdf90198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.017+0000 7fdf9512f640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf90107a10 0x7fdf90198a30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdf84005940 tx=0x7fdf840042f0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.018+0000 7fdf9512f640 1 -- 172.21.15.129:0/1274803759 shutdown_connections 2024-05-20T18:00:11.018 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.018+0000 7fdf9512f640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdf780a2e10 0x7fdf780a5320 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.018 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.018+0000 7fdf9512f640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf90107eb0 0x7fdf90198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.018 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.018+0000 7fdf9512f640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf90107a10 0x7fdf90198a30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.018 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.018+0000 7fdf9512f640 1 --2- 172.21.15.129:0/1274803759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf90102170 0x7fdf901984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:11.018 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.018+0000 7fdf9512f640 1 -- 172.21.15.129:0/1274803759 >> 172.21.15.129:0/1274803759 conn(0x7fdf90077160 msgr2=0x7fdf901030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:11.018 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.018+0000 7fdf9512f640 1 -- 172.21.15.129:0/1274803759 shutdown_connections 2024-05-20T18:00:11.018 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:11.018+0000 7fdf9512f640 1 -- 172.21.15.129:0/1274803759 wait complete. 2024-05-20T18:00:11.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:10 smithi154 ceph-mon[36482]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-05-20T18:00:11.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:10 smithi129 ceph-mon[32029]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-05-20T18:00:11.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:10 smithi204 ceph-mon[37076]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-05-20T18:00:13.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:12 smithi154 ceph-mon[36482]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:00:13.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:12 smithi129 ceph-mon[32029]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:00:13.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:12 smithi204 ceph-mon[37076]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:00:15.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:14 smithi154 ceph-mon[36482]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:00:15.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:14 smithi129 ceph-mon[32029]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:00:15.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:14 smithi204 ceph-mon[37076]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:00:16.481 INFO:tasks.workunit.client.0.smithi129.stderr:Note: switching to 'd2b507c724ef6c8295ee0a35d185cdab167fd61d'. 2024-05-20T18:00:16.481 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-05-20T18:00:16.481 INFO:tasks.workunit.client.0.smithi129.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-20T18:00:16.481 INFO:tasks.workunit.client.0.smithi129.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr:state without impacting any branches by switching back to a branch. 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr: git switch -c 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr:Or undo this operation with: 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr: git switch - 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-20T18:00:16.482 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-05-20T18:00:16.483 INFO:tasks.workunit.client.0.smithi129.stderr:HEAD is now at d2b507c724e Merge PR #56984 into wip-pdonnell-testing-20240517.203521-squid-debug 2024-05-20T18:00:16.488 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-20T18:00:16.553 INFO:tasks.workunit.client.0.smithi129.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-20T18:00:16.556 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-20T18:00:16.556 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-20T18:00:17.237 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-20T18:00:17.293 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-20T18:00:17.342 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-20T18:00:17.344 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-20T18:00:17.344 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-20T18:00:17.383 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-20T18:00:17.386 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T18:00:17.387 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-20T18:00:17.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:16 smithi154 ceph-mon[36482]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-20T18:00:17.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:16 smithi154 ceph-mon[36482]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-05-20T18:00:17.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:16 smithi129 ceph-mon[32029]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-20T18:00:17.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:16 smithi129 ceph-mon[32029]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-05-20T18:00:17.427 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-05-20T18:00:17.429 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-05-20T18:00:17.429 DEBUG:teuthology.orchestra.run.smithi129:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2b507c724ef6c8295ee0a35d185cdab167fd61d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-05-20T18:00:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:16 smithi204 ceph-mon[37076]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-05-20T18:00:17.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:16 smithi204 ceph-mon[37076]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-05-20T18:00:17.490 INFO:tasks.workunit.client.0.smithi129.stdout:getting iogen 2024-05-20T18:00:17.491 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'getting iogen' 2024-05-20T18:00:17.491 INFO:tasks.workunit.client.0.smithi129.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-05-20T18:00:17.518 INFO:tasks.workunit.client.0.smithi129.stderr:--2024-05-20 18:00:17-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-05-20T18:00:17.518 INFO:tasks.workunit.client.0.smithi129.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-05-20T18:00:17.555 INFO:tasks.workunit.client.0.smithi129.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-05-20T18:00:17.590 INFO:tasks.workunit.client.0.smithi129.stderr:HTTP request sent, awaiting response... 200 OK 2024-05-20T18:00:17.590 INFO:tasks.workunit.client.0.smithi129.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-05-20T18:00:17.592 INFO:tasks.workunit.client.0.smithi129.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-05-20T18:00:17.596 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-05-20T18:00:17.596 INFO:tasks.workunit.client.0.smithi129.stderr: 0K ....... 100% 764M=0s 2024-05-20T18:00:17.596 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-05-20T18:00:17.596 INFO:tasks.workunit.client.0.smithi129.stderr:2024-05-20 18:00:17 (764 MB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-05-20T18:00:17.596 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-05-20T18:00:17.596 INFO:tasks.workunit.client.0.smithi129.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-05-20T18:00:17.610 INFO:tasks.workunit.client.0.smithi129.stdout:iogen_3.1p0 2024-05-20T18:00:17.611 INFO:tasks.workunit.client.0.smithi129.stdout:iogen_3.1p0/src 2024-05-20T18:00:17.612 INFO:tasks.workunit.client.0.smithi129.stdout:iogen_3.1p0/src/iogen.8 2024-05-20T18:00:17.613 INFO:tasks.workunit.client.0.smithi129.stdout:iogen_3.1p0/src/iogen.c 2024-05-20T18:00:17.614 INFO:tasks.workunit.client.0.smithi129.stdout:iogen_3.1p0/README 2024-05-20T18:00:17.615 INFO:tasks.workunit.client.0.smithi129.stdout:iogen_3.1p0/Makefile.bsd 2024-05-20T18:00:17.616 INFO:tasks.workunit.client.0.smithi129.stdout:iogen_3.1p0/Makefile 2024-05-20T18:00:17.618 INFO:tasks.workunit.client.0.smithi129.stderr:+ cd iogen_3.1p0 2024-05-20T18:00:17.618 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'making iogen' 2024-05-20T18:00:17.618 INFO:tasks.workunit.client.0.smithi129.stdout:making iogen 2024-05-20T18:00:17.618 INFO:tasks.workunit.client.0.smithi129.stderr:+ make 2024-05-20T18:00:17.629 INFO:tasks.workunit.client.0.smithi129.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-05-20T18:00:17.852 INFO:tasks.workunit.client.0.smithi129.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-05-20T18:00:17.956 INFO:tasks.workunit.client.0.smithi129.stdout:running iogen 2024-05-20T18:00:17.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'running iogen' 2024-05-20T18:00:17.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ ./iogen -n 5 -s 2g 2024-05-20T18:00:17.957 INFO:tasks.workunit.client.0.smithi129.stderr:child spawned 85732 2024-05-20T18:00:17.958 INFO:tasks.workunit.client.0.smithi129.stderr:child spawned 85733 2024-05-20T18:00:17.958 INFO:tasks.workunit.client.0.smithi129.stderr:child spawned 85734 2024-05-20T18:00:17.958 INFO:tasks.workunit.client.0.smithi129.stderr:child spawned 85735 2024-05-20T18:00:17.958 INFO:tasks.workunit.client.0.smithi129.stderr:child spawned 85736 2024-05-20T18:00:17.958 INFO:tasks.workunit.client.0.smithi129.stdout:sleep for 10 min 2024-05-20T18:00:17.958 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'sleep for 10 min' 2024-05-20T18:00:17.958 INFO:tasks.workunit.client.0.smithi129.stderr:+ sleep 600 2024-05-20T18:00:19.168 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:19 smithi129 ceph-mon[32029]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-05-20T18:00:19.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:19 smithi154 ceph-mon[36482]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-05-20T18:00:19.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:19 smithi204 ceph-mon[37076]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-05-20T18:00:21.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:21 smithi154 ceph-mon[36482]: pgmap v57: 97 pgs: 97 active+clean; 848 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 71 MiB/s wr, 62 op/s 2024-05-20T18:00:21.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:21 smithi129 ceph-mon[32029]: pgmap v57: 97 pgs: 97 active+clean; 848 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 71 MiB/s wr, 62 op/s 2024-05-20T18:00:21.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:21 smithi204 ceph-mon[37076]: pgmap v57: 97 pgs: 97 active+clean; 848 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 71 MiB/s wr, 62 op/s 2024-05-20T18:00:22.625 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:00:22.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:00:22.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.808+0000 7ff206e7a640 1 -- 172.21.15.129:0/2794362575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff20011c0d0 msgr2=0x7ff20011e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:22.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.808+0000 7ff206e7a640 1 --2- 172.21.15.129:0/2794362575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff20011c0d0 0x7ff20011e4f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009900 tx=0x7ff1f002a480 comp rx=0 tx=0).stop 2024-05-20T18:00:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.809+0000 7ff206e7a640 1 -- 172.21.15.129:0/2794362575 shutdown_connections 2024-05-20T18:00:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.809+0000 7ff206e7a640 1 --2- 172.21.15.129:0/2794362575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff20011c0d0 0x7ff20011e4f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.809+0000 7ff206e7a640 1 --2- 172.21.15.129:0/2794362575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff200119770 0x7ff20011bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.809+0000 7ff206e7a640 1 --2- 172.21.15.129:0/2794362575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff200109060 0x7ff200109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.809+0000 7ff206e7a640 1 -- 172.21.15.129:0/2794362575 >> 172.21.15.129:0/2794362575 conn(0x7ff20006d750 msgr2=0x7ff20006dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:22.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.809+0000 7ff206e7a640 1 -- 172.21.15.129:0/2794362575 shutdown_connections 2024-05-20T18:00:22.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.809+0000 7ff206e7a640 1 -- 172.21.15.129:0/2794362575 wait complete. 2024-05-20T18:00:22.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.811+0000 7ff206e7a640 1 Processor -- start 2024-05-20T18:00:22.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.814+0000 7ff206e7a640 1 -- start start 2024-05-20T18:00:22.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.815+0000 7ff206e7a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff200109060 0x7ff2001a85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.815+0000 7ff206e7a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff200119770 0x7ff2001a8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.815+0000 7ff206e7a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff20011c0d0 0x7ff2001acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.815+0000 7ff206e7a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff200121c10 con 0x7ff200109060 2024-05-20T18:00:22.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.815+0000 7ff206e7a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff200121a90 con 0x7ff20011c0d0 2024-05-20T18:00:22.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.815+0000 7ff206e7a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff200121d90 con 0x7ff200119770 2024-05-20T18:00:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.815+0000 7ff204c16640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff200109060 0x7ff2001a85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.815+0000 7ff204c16640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff200109060 0x7ff2001a85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58980/0 (socket says 172.21.15.129:58980) 2024-05-20T18:00:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.815+0000 7ff204c16640 1 -- 172.21.15.129:0/2650302335 learned_addr learned my addr 172.21.15.129:0/2650302335 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.815+0000 7ff1fffff640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff200119770 0x7ff2001a8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.816+0000 7ff1fffff640 1 -- 172.21.15.129:0/2650302335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff20011c0d0 msgr2=0x7ff2001acdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:00:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.816+0000 7ff1fffff640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff20011c0d0 0x7ff2001acdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.816+0000 7ff1fffff640 1 -- 172.21.15.129:0/2650302335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff200109060 msgr2=0x7ff2001a85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:22.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.816+0000 7ff1fffff640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff200109060 0x7ff2001a85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.816+0000 7ff1fffff640 1 -- 172.21.15.129:0/2650302335 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2001ad390 con 0x7ff200119770 2024-05-20T18:00:22.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.816+0000 7ff1fffff640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff200119770 0x7ff2001a8b10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800cd10 tx=0x7ff1f800a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:22.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.818+0000 7ff1fdffb640 1 -- 172.21.15.129:0/2650302335 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f800f450 con 0x7ff200119770 2024-05-20T18:00:22.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.818+0000 7ff1fdffb640 1 -- 172.21.15.129:0/2650302335 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1f800fad0 con 0x7ff200119770 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.820+0000 7ff1fdffb640 1 -- 172.21.15.129:0/2650302335 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8018b90 con 0x7ff200119770 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.822+0000 7ff206e7a640 1 -- 172.21.15.129:0/2650302335 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2001ad680 con 0x7ff200119770 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.822+0000 7ff206e7a640 1 -- 172.21.15.129:0/2650302335 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2001adb50 con 0x7ff200119770 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.833+0000 7ff20de04740 1 Processor -- start 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.834+0000 7ff1fdffb640 1 -- 172.21.15.129:0/2650302335 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ff1f800fc80 con 0x7ff200119770 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.834+0000 7ff20de04740 1 -- start start 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.835+0000 7ff20de04740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cd7fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.835+0000 7ff20de04740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 0x55938cd80270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.835+0000 7ff20de04740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd807b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.835+0000 7ff20de04740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55938cce5d00 con 0x55938c9e1390 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.835+0000 7ff20de04740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55938cc8d4c0 con 0x55938c9b84d0 2024-05-20T18:00:22.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.835+0000 7ff20de04740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55938cce1cc0 con 0x55938c9c06b0 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.836+0000 7ff1fffff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd807b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.836+0000 7ff204c16640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 0x55938cd80270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.836+0000 7ff1fffff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd807b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:55158/0 (socket says 172.21.15.129:55158) 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.836+0000 7ff1fffff640 1 -- 172.21.15.129:0/775133376 learned_addr learned my addr 172.21.15.129:0/775133376 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.836+0000 7ff205417640 1 --2- 172.21.15.129:0/775133376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cd7fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.837+0000 7ff204c16640 1 -- 172.21.15.129:0/775133376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 msgr2=0x55938cd807b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.837+0000 7ff204c16640 1 --2- 172.21.15.129:0/775133376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd807b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.837+0000 7ff204c16640 1 -- 172.21.15.129:0/775133376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 msgr2=0x55938cd7fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.837+0000 7ff204c16640 1 --2- 172.21.15.129:0/775133376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cd7fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.837+0000 7ff204c16640 1 -- 172.21.15.129:0/775133376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55938cac37d0 con 0x55938c9b84d0 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.838+0000 7ff1fdffb640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff1e80a2d60 0x7ff1e80a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.839+0000 7ff1fdffb640 1 -- 172.21.15.129:0/2650302335 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7ff1f80cf060 con 0x7ff200119770 2024-05-20T18:00:22.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.840+0000 7ff205417640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff1e80a2d60 0x7ff1e80a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.842+0000 7ff205417640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff1e80a2d60 0x7ff1e80a5270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55938cd7cea0 tx=0x7ff1f0005d50 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.842+0000 7ff204c16640 1 --2- 172.21.15.129:0/775133376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 0x55938cd80270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009a50 tx=0x7ff1f4009f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=390b64d6d1e5e8f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.854+0000 7ff1e6ffd640 1 -- 172.21.15.129:0/775133376 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4017070 con 0x55938c9b84d0 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.854+0000 7ff1e6ffd640 1 -- 172.21.15.129:0/775133376 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1f4013470 con 0x55938c9b84d0 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.854+0000 7ff1e6ffd640 1 -- 172.21.15.129:0/775133376 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4012630 con 0x55938c9b84d0 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.857+0000 7ff20de04740 1 -- 172.21.15.129:0/775133376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 msgr2=0x55938cd80270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.857+0000 7ff20de04740 1 --2- 172.21.15.129:0/775133376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 0x55938cd80270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009a50 tx=0x7ff1f4009f10 comp rx=0 tx=0).stop 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.861+0000 7ff20de04740 1 -- 172.21.15.129:0/775133376 shutdown_connections 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.861+0000 7ff20de04740 1 --2- 172.21.15.129:0/775133376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 0x55938cd80270 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.861+0000 7ff20de04740 1 --2- 172.21.15.129:0/775133376 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd807b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.861+0000 7ff20de04740 1 --2- 172.21.15.129:0/775133376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cd7fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.861+0000 7ff20de04740 1 -- 172.21.15.129:0/775133376 >> 172.21.15.129:0/775133376 conn(0x55938c9d54e0 msgr2=0x55938cd7c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.864+0000 7ff20de04740 1 -- 172.21.15.129:0/775133376 shutdown_connections 2024-05-20T18:00:22.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.864+0000 7ff20de04740 1 -- 172.21.15.129:0/775133376 wait complete. 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.866+0000 7ff20de04740 1 -- start start 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.866+0000 7ff20de04740 1 Processor -- start 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.868+0000 7ff20de04740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 0x55938cd8c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.869+0000 7ff20de04740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd8c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.869+0000 7ff20de04740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cd8a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.869+0000 7ff204c16640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 0x55938cd8c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff204c16640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 0x55938cd8c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52184/0 (socket says 172.21.15.129:52184) 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff204c16640 1 -- 172.21.15.129:0/1633270021 learned_addr learned my addr 172.21.15.129:0/1633270021 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff20de04740 1 -- 172.21.15.129:0/1633270021 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55938cd8cfc0 con 0x55938c9e1390 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff20de04740 1 -- 172.21.15.129:0/1633270021 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55938cca0b90 con 0x55938c9b84d0 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff20de04740 1 -- 172.21.15.129:0/1633270021 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55938cd8d140 con 0x55938c9c06b0 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff1fffff640 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd8c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff20de04740 10 client.0 _ll_register_callbacks cb 0x55938ccdcda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:22.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff204c16640 1 -- 172.21.15.129:0/1633270021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 msgr2=0x55938cd8c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:22.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff204c16640 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd8c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff204c16640 1 -- 172.21.15.129:0/1633270021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 msgr2=0x55938cd8a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:22.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff204c16640 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cd8a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.870+0000 7ff204c16640 1 -- 172.21.15.129:0/1633270021 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55938c9f93f0 con 0x55938c9b84d0 2024-05-20T18:00:22.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.875+0000 7ff204c16640 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 0x55938cd8c110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400e660 tx=0x7ff1f4013e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce5444d24396bed5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:22.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.887+0000 7ff1dffff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:00:22.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.887+0000 7ff1dffff640 1 -- 172.21.15.129:0/1633270021 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4017050 con 0x55938c9b84d0 2024-05-20T18:00:22.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.887+0000 7ff1dffff640 1 -- 172.21.15.129:0/1633270021 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7ff1f401c650 con 0x55938c9b84d0 2024-05-20T18:00:22.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.888+0000 7ff20de04740 1 -- 172.21.15.129:0/1633270021 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1f4013470 con 0x55938c9b84d0 2024-05-20T18:00:22.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.890+0000 7ff1dffff640 1 -- 172.21.15.129:0/1633270021 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff1f401c850 con 0x55938c9b84d0 2024-05-20T18:00:22.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.893+0000 7ff20de04740 10 client.25430 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:22.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.893+0000 7ff20de04740 1 -- 172.21.15.129:0/1633270021 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55938c9b90f0 con 0x55938c9b84d0 2024-05-20T18:00:22.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.898+0000 7ff1dffff640 1 -- 172.21.15.129:0/1633270021 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1f4013470 con 0x55938c9b84d0 2024-05-20T18:00:22.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.899+0000 7ff1dffff640 1 -- 172.21.15.129:0/1633270021 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7ff1f4030af0 con 0x55938c9b84d0 2024-05-20T18:00:22.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.900+0000 7ff20de04740 10 client.25430 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:22.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.900+0000 7ff20de04740 10 client.25430 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:22.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.900+0000 7ff20de04740 20 client.25430 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:22.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.900+0000 7ff20de04740 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55938c99ff60 0x55938cd9c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.900+0000 7ff20de04740 4 client.25430 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:00:22.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.900+0000 7ff20de04740 1 -- 172.21.15.129:0/1633270021 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55938ccf3430 con 0x55938c99ff60 2024-05-20T18:00:22.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.903+0000 7ff1fffff640 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55938c99ff60 0x55938cd9c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.916+0000 7ff1fffff640 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55938c99ff60 0x55938cd9c9a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad10330a50cfdf3f server_cookie=691a5ba95e87ef8d in_seq=0 out_seq=0 2024-05-20T18:00:22.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.917+0000 7ff1dffff640 10 client.25430 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:22.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.922+0000 7ff1dffff640 1 -- 172.21.15.129:0/1633270021 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55938ccf3430 con 0x55938c99ff60 2024-05-20T18:00:22.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.922+0000 7ff1dffff640 10 client.25430 handle_command_reply: tid=0 2024-05-20T18:00:22.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.923+0000 7ff20de04740 1 client.25430 shutdown 2024-05-20T18:00:22.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.924+0000 7ff20de04740 1 -- 172.21.15.129:0/1633270021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 msgr2=0x55938cd8c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:22.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.924+0000 7ff20de04740 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 0x55938cd8c110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400e660 tx=0x7ff1f4013e30 comp rx=0 tx=0).stop 2024-05-20T18:00:22.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.924+0000 7ff20de04740 1 -- 172.21.15.129:0/1633270021 shutdown_connections 2024-05-20T18:00:22.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.924+0000 7ff20de04740 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55938c99ff60 0x55938cd9c9a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.924+0000 7ff20de04740 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cd8a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.924+0000 7ff20de04740 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd8c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.924+0000 7ff20de04740 1 --2- 172.21.15.129:0/1633270021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9b84d0 0x55938cd8c110 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400e660 tx=0x7ff1f4013e30 comp rx=0 tx=0).stop 2024-05-20T18:00:22.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.924+0000 7ff20de04740 1 -- 172.21.15.129:0/1633270021 >> 172.21.15.129:0/1633270021 conn(0x55938c9d54e0 msgr2=0x55938cd81a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:22.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.924+0000 7ff1dffff640 0 client.25430 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:22.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.924+0000 7ff20de04740 1 -- 172.21.15.129:0/1633270021 shutdown_connections 2024-05-20T18:00:22.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.924+0000 7ff20de04740 1 -- 172.21.15.129:0/1633270021 wait complete. 2024-05-20T18:00:22.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.925+0000 7ff20de04740 20 client.25430 trim_cache size 0 max 16384 2024-05-20T18:00:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.962+0000 7ff20de04740 1 Processor -- start 2024-05-20T18:00:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.962+0000 7ff20de04740 1 -- start start 2024-05-20T18:00:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.962+0000 7ff20de04740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9c06b0 0x55938cd8c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.962+0000 7ff20de04740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9e1390 0x55938cd89470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff20de04740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9753a0 0x55938cd9f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff20de04740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff20010a380 con 0x55938c9753a0 2024-05-20T18:00:22.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff20de04740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2001c6760 con 0x55938c9c06b0 2024-05-20T18:00:22.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff20de04740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff200117c60 con 0x55938c9e1390 2024-05-20T18:00:22.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff205417640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9753a0 0x55938cd9f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff205417640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9753a0 0x55938cd9f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59004/0 (socket says 172.21.15.129:59004) 2024-05-20T18:00:22.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff205417640 1 -- 172.21.15.129:0/96157990 learned_addr learned my addr 172.21.15.129:0/96157990 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:22.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff1fffff640 1 --2- 172.21.15.129:0/96157990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9e1390 0x55938cd89470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff205417640 1 -- 172.21.15.129:0/96157990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9e1390 msgr2=0x55938cd89470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:22.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff205417640 1 --2- 172.21.15.129:0/96157990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9e1390 0x55938cd89470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff205417640 1 -- 172.21.15.129:0/96157990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9c06b0 msgr2=0x55938cd8c080 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:00:22.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff205417640 1 --2- 172.21.15.129:0/96157990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9c06b0 0x55938cd8c080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.963+0000 7ff205417640 1 -- 172.21.15.129:0/96157990 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55938cd8b3b0 con 0x55938c9753a0 2024-05-20T18:00:22.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.964+0000 7ff205417640 1 --2- 172.21.15.129:0/96157990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9753a0 0x55938cd9f9d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0006010 tx=0x7ff1f002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50cd8f138ba3af9e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:22.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.964+0000 7ff1e67fc640 1 -- 172.21.15.129:0/96157990 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0035040 con 0x55938c9753a0 2024-05-20T18:00:22.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.964+0000 7ff1e67fc640 1 -- 172.21.15.129:0/96157990 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1f004d070 con 0x55938c9753a0 2024-05-20T18:00:22.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.964+0000 7ff1e67fc640 1 -- 172.21.15.129:0/96157990 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0032de0 con 0x55938c9753a0 2024-05-20T18:00:22.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.965+0000 7ff20de04740 1 -- 172.21.15.129:0/96157990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9753a0 msgr2=0x55938cd9f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:22.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.965+0000 7ff20de04740 1 --2- 172.21.15.129:0/96157990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9753a0 0x55938cd9f9d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0006010 tx=0x7ff1f002a480 comp rx=0 tx=0).stop 2024-05-20T18:00:22.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.966+0000 7ff20de04740 1 -- 172.21.15.129:0/96157990 shutdown_connections 2024-05-20T18:00:22.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.966+0000 7ff20de04740 1 --2- 172.21.15.129:0/96157990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9753a0 0x55938cd9f9d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.966+0000 7ff20de04740 1 --2- 172.21.15.129:0/96157990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9e1390 0x55938cd89470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.966+0000 7ff20de04740 1 --2- 172.21.15.129:0/96157990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9c06b0 0x55938cd8c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.966+0000 7ff20de04740 1 -- 172.21.15.129:0/96157990 >> 172.21.15.129:0/96157990 conn(0x55938c9d54e0 msgr2=0x55938cd88d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:22.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.967+0000 7ff20de04740 1 -- 172.21.15.129:0/96157990 shutdown_connections 2024-05-20T18:00:22.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.967+0000 7ff20de04740 1 -- 172.21.15.129:0/96157990 wait complete. 2024-05-20T18:00:22.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.968+0000 7ff20de04740 1 -- start start 2024-05-20T18:00:22.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.968+0000 7ff20de04740 1 Processor -- start 2024-05-20T18:00:22.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.969+0000 7ff20de04740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9753a0 0x55938cd9ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.970+0000 7ff20de04740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd9f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.970+0000 7ff20de04740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cdfcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.970+0000 7ff205417640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cdfcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.970+0000 7ff20de04740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff200118330 con 0x55938c9e1390 2024-05-20T18:00:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.970+0000 7ff20de04740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2001c6090 con 0x55938c9753a0 2024-05-20T18:00:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.970+0000 7ff20de04740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff200118a00 con 0x55938c9c06b0 2024-05-20T18:00:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.970+0000 7ff20de04740 10 client.0 _ll_register_callbacks cb 0x7ff1f80d2960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:22.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.970+0000 7ff205417640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cdfcaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59006/0 (socket says 172.21.15.129:59006) 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.970+0000 7ff205417640 1 -- 172.21.15.129:0/1635164731 learned_addr learned my addr 172.21.15.129:0/1635164731 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.971+0000 7ff205417640 1 -- 172.21.15.129:0/1635164731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 msgr2=0x55938cd9f340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.971+0000 7ff205417640 1 --2- 172.21.15.129:0/1635164731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd9f340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.971+0000 7ff205417640 1 -- 172.21.15.129:0/1635164731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9753a0 msgr2=0x55938cd9ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.971+0000 7ff205417640 1 --2- 172.21.15.129:0/1635164731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9753a0 0x55938cd9ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.971+0000 7ff205417640 1 -- 172.21.15.129:0/1635164731 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55938cdfcfe0 con 0x55938c9e1390 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.971+0000 7ff205417640 1 --2- 172.21.15.129:0/1635164731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cdfcaa0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0032ce0 tx=0x7ff1f0032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1411eb07f8735e8d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.971+0000 7ff1dffff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.971+0000 7ff1dffff640 1 -- 172.21.15.129:0/1635164731 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0055070 con 0x55938c9e1390 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.971+0000 7ff1dffff640 1 -- 172.21.15.129:0/1635164731 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7ff1f0004ad0 con 0x55938c9e1390 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.971+0000 7ff20de04740 1 -- 172.21.15.129:0/1635164731 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1f004d070 con 0x55938c9e1390 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.972+0000 7ff1dffff640 1 -- 172.21.15.129:0/1635164731 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7ff1f0061050 con 0x55938c9e1390 2024-05-20T18:00:22.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.972+0000 7ff20de04740 10 client.15759 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:22.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.972+0000 7ff20de04740 1 -- 172.21.15.129:0/1635164731 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55938ce04320 con 0x55938c9e1390 2024-05-20T18:00:22.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.972+0000 7ff1dffff640 1 -- 172.21.15.129:0/1635164731 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1f004d070 con 0x55938c9e1390 2024-05-20T18:00:22.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.973+0000 7ff1dffff640 1 -- 172.21.15.129:0/1635164731 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7ff1f003a040 con 0x55938c9e1390 2024-05-20T18:00:22.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.973+0000 7ff20de04740 10 client.15759 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:22.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.973+0000 7ff20de04740 10 client.15759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:22.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.973+0000 7ff20de04740 20 client.15759 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:22.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.973+0000 7ff20de04740 1 --2- 172.21.15.129:0/1635164731 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55938ce048f0 0x55938ce06d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:22.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.973+0000 7ff20de04740 4 client.15759 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:00:22.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.973+0000 7ff20de04740 1 -- 172.21.15.129:0/1635164731 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55938cced780 con 0x55938ce048f0 2024-05-20T18:00:22.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.975+0000 7ff204c16640 1 --2- 172.21.15.129:0/1635164731 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55938ce048f0 0x55938ce06d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:22.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.977+0000 7ff1dffff640 10 client.15759 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:22.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.977+0000 7ff204c16640 1 --2- 172.21.15.129:0/1635164731 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55938ce048f0 0x55938ce06d10 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f54714f34648769e server_cookie=a44ddb2555a7fd6b in_seq=0 out_seq=0 2024-05-20T18:00:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.980+0000 7ff1dffff640 1 -- 172.21.15.129:0/1635164731 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55938cced780 con 0x55938ce048f0 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.980+0000 7ff1dffff640 10 client.15759 handle_command_reply: tid=0 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.980+0000 7ff20de04740 1 client.15759 shutdown 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff20de04740 1 -- 172.21.15.129:0/1635164731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 msgr2=0x55938cdfcaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff20de04740 1 --2- 172.21.15.129:0/1635164731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cdfcaa0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0032ce0 tx=0x7ff1f0032d10 comp rx=0 tx=0).stop 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff20de04740 1 -- 172.21.15.129:0/1635164731 shutdown_connections 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff20de04740 1 --2- 172.21.15.129:0/1635164731 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55938ce048f0 0x55938ce06d10 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff20de04740 1 --2- 172.21.15.129:0/1635164731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55938c9e1390 0x55938cdfcaa0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff20de04740 1 --2- 172.21.15.129:0/1635164731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55938c9c06b0 0x55938cd9f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff20de04740 1 --2- 172.21.15.129:0/1635164731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55938c9753a0 0x55938cd9ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff20de04740 1 -- 172.21.15.129:0/1635164731 >> 172.21.15.129:0/1635164731 conn(0x55938c9d54e0 msgr2=0x55938cd7fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff1dffff640 0 client.15759 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff20de04740 1 -- 172.21.15.129:0/1635164731 shutdown_connections 2024-05-20T18:00:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff20de04740 1 -- 172.21.15.129:0/1635164731 wait complete. 2024-05-20T18:00:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.981+0000 7ff20de04740 20 client.15759 trim_cache size 0 max 16384 2024-05-20T18:00:22.981 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:00:22.981 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:00:22.981 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:00:22.981 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:00:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.982+0000 7ff206e7a640 1 -- 172.21.15.129:0/2650302335 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff1e80a2d60 msgr2=0x7ff1e80a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.982+0000 7ff206e7a640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff1e80a2d60 0x7ff1e80a5270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55938cd7cea0 tx=0x7ff1f0005d50 comp rx=0 tx=0).stop 2024-05-20T18:00:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.982+0000 7ff206e7a640 1 -- 172.21.15.129:0/2650302335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff200119770 msgr2=0x7ff2001a8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.982+0000 7ff206e7a640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff200119770 0x7ff2001a8b10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800cd10 tx=0x7ff1f800a620 comp rx=0 tx=0).stop 2024-05-20T18:00:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.982+0000 7ff206e7a640 1 -- 172.21.15.129:0/2650302335 shutdown_connections 2024-05-20T18:00:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.982+0000 7ff206e7a640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff1e80a2d60 0x7ff1e80a5270 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.982+0000 7ff206e7a640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff20011c0d0 0x7ff2001acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.982+0000 7ff206e7a640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff200119770 0x7ff2001a8b10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.982+0000 7ff206e7a640 1 --2- 172.21.15.129:0/2650302335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff200109060 0x7ff2001a85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.982+0000 7ff206e7a640 1 -- 172.21.15.129:0/2650302335 >> 172.21.15.129:0/2650302335 conn(0x7ff20006d750 msgr2=0x7ff200119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.983+0000 7ff206e7a640 1 -- 172.21.15.129:0/2650302335 shutdown_connections 2024-05-20T18:00:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:22.983+0000 7ff206e7a640 1 -- 172.21.15.129:0/2650302335 wait complete. 2024-05-20T18:00:23.000 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:00:23.000 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:00:23.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.325+0000 7f6000e5d640 1 -- 172.21.15.129:0/2233214621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc10c330 msgr2=0x7f5ffc10e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:23.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.325+0000 7f6000e5d640 1 --2- 172.21.15.129:0/2233214621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc10c330 0x7f5ffc10e750 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4009ca0 tx=0x7f5ff402a8c0 comp rx=0 tx=0).stop 2024-05-20T18:00:23.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.325+0000 7f6000e5d640 1 -- 172.21.15.129:0/2233214621 shutdown_connections 2024-05-20T18:00:23.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.325+0000 7f6000e5d640 1 --2- 172.21.15.129:0/2233214621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc10c330 0x7f5ffc10e750 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:23.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.325+0000 7f6000e5d640 1 --2- 172.21.15.129:0/2233214621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ffc101aa0 0x7f5ffc101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:23.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.325+0000 7f6000e5d640 1 --2- 172.21.15.129:0/2233214621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ffc1079d0 0x7f5ffc101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:23.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.325+0000 7f6000e5d640 1 -- 172.21.15.129:0/2233214621 >> 172.21.15.129:0/2233214621 conn(0x7f5ffc0770c0 msgr2=0x7f5ffc077500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:23.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.326+0000 7f6000e5d640 1 -- 172.21.15.129:0/2233214621 shutdown_connections 2024-05-20T18:00:23.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.326+0000 7f6000e5d640 1 -- 172.21.15.129:0/2233214621 wait complete. 2024-05-20T18:00:23.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.328+0000 7f6000e5d640 1 Processor -- start 2024-05-20T18:00:23.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.329+0000 7f6000e5d640 1 -- start start 2024-05-20T18:00:23.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.329+0000 7f6000e5d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ffc101aa0 0x7f5ffc073690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:23.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.329+0000 7f6000e5d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc1079d0 0x7f5ffc06d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:23.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.329+0000 7f6000e5d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ffc10c330 0x7f5ffc06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:23.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.329+0000 7f6000e5d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc06bf80 con 0x7f5ffc1079d0 2024-05-20T18:00:23.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.329+0000 7f6000e5d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc06be00 con 0x7f5ffc101aa0 2024-05-20T18:00:23.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.329+0000 7f6000e5d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5ffc06c100 con 0x7f5ffc10c330 2024-05-20T18:00:23.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.330+0000 7f5ff9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc1079d0 0x7f5ffc06d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:23.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.330+0000 7f5ff9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc1079d0 0x7f5ffc06d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59014/0 (socket says 172.21.15.129:59014) 2024-05-20T18:00:23.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.330+0000 7f5ff9d9b640 1 -- 172.21.15.129:0/1926903117 learned_addr learned my addr 172.21.15.129:0/1926903117 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:23.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.330+0000 7f5ffad9d640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ffc10c330 0x7f5ffc06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:23.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.330+0000 7f5ff9d9b640 1 -- 172.21.15.129:0/1926903117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ffc10c330 msgr2=0x7f5ffc06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:23.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.330+0000 7f5ff9d9b640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ffc10c330 0x7f5ffc06dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:23.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.331+0000 7f5ff9d9b640 1 -- 172.21.15.129:0/1926903117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ffc101aa0 msgr2=0x7f5ffc073690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:23.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.331+0000 7f5ff9d9b640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ffc101aa0 0x7f5ffc073690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:23.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.331+0000 7f5ff9d9b640 1 -- 172.21.15.129:0/1926903117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ffc06e500 con 0x7f5ffc1079d0 2024-05-20T18:00:23.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.331+0000 7f5ff9d9b640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc1079d0 0x7f5ffc06d7a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5ff000e9f0 tx=0x7f5ff000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:23.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.331+0000 7f5feb7fe640 1 -- 172.21.15.129:0/1926903117 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff000ccc0 con 0x7f5ffc1079d0 2024-05-20T18:00:23.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.331+0000 7f5feb7fe640 1 -- 172.21.15.129:0/1926903117 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ff00035f0 con 0x7f5ffc1079d0 2024-05-20T18:00:23.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.331+0000 7f5feb7fe640 1 -- 172.21.15.129:0/1926903117 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff000ce60 con 0x7f5ffc1079d0 2024-05-20T18:00:23.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.332+0000 7f6000e5d640 1 -- 172.21.15.129:0/1926903117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ffc1ad080 con 0x7f5ffc1079d0 2024-05-20T18:00:23.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.332+0000 7f6000e5d640 1 -- 172.21.15.129:0/1926903117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ffc069080 con 0x7f5ffc1079d0 2024-05-20T18:00:23.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.334+0000 7f6000e5d640 1 -- 172.21.15.129:0/1926903117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ffc102c80 con 0x7f5ffc1079d0 2024-05-20T18:00:23.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.339+0000 7f5feb7fe640 1 -- 172.21.15.129:0/1926903117 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5ff00037a0 con 0x7f5ffc1079d0 2024-05-20T18:00:23.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.340+0000 7f5feb7fe640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5fe40a2dc0 0x7f5fe40a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:23.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.340+0000 7f5feb7fe640 1 -- 172.21.15.129:0/1926903117 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5ff00cfb60 con 0x7f5ffc1079d0 2024-05-20T18:00:23.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.341+0000 7f5ffa59c640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5fe40a2dc0 0x7f5fe40a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:23.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.342+0000 7f5ffa59c640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5fe40a2dc0 0x7f5fe40a52d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5fec005fd0 tx=0x7f5fec005f40 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:23.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.345+0000 7f5feb7fe640 1 -- 172.21.15.129:0/1926903117 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5ff009c800 con 0x7f5ffc1079d0 2024-05-20T18:00:23.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:23 smithi154 ceph-mon[36482]: pgmap v58: 97 pgs: 97 active+clean; 848 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 71 MiB/s wr, 61 op/s 2024-05-20T18:00:23.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:23 smithi129 ceph-mon[32029]: pgmap v58: 97 pgs: 97 active+clean; 848 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 71 MiB/s wr, 61 op/s 2024-05-20T18:00:23.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:23 smithi204 ceph-mon[37076]: pgmap v58: 97 pgs: 97 active+clean; 848 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 71 MiB/s wr, 61 op/s 2024-05-20T18:00:23.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.789+0000 7f6000e5d640 1 -- 172.21.15.129:0/1926903117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ffc105900 con 0x7f5ffc1079d0 2024-05-20T18:00:23.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.793+0000 7f5feb7fe640 1 -- 172.21.15.129:0/1926903117 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11829 (secure 0 0 0) 0x7f5ff009c100 con 0x7f5ffc1079d0 2024-05-20T18:00:23.793 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:00:23.793 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:59:17.074606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25229":{"gid":25229,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1860679091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1860679091},{"type":"v1","addr":"172.21.15.154:6837","nonce":1860679091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:00:23.794 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-05-20T18:00:23.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.800+0000 7f6000e5d640 1 -- 172.21.15.129:0/1926903117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5fe40a2dc0 msgr2=0x7f5fe40a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:23.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.800+0000 7f6000e5d640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5fe40a2dc0 0x7f5fe40a52d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5fec005fd0 tx=0x7f5fec005f40 comp rx=0 tx=0).stop 2024-05-20T18:00:23.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.800+0000 7f6000e5d640 1 -- 172.21.15.129:0/1926903117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc1079d0 msgr2=0x7f5ffc06d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:23.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.800+0000 7f6000e5d640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc1079d0 0x7f5ffc06d7a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5ff000e9f0 tx=0x7f5ff000eeb0 comp rx=0 tx=0).stop 2024-05-20T18:00:23.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.800+0000 7f6000e5d640 1 -- 172.21.15.129:0/1926903117 shutdown_connections 2024-05-20T18:00:23.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.801+0000 7f6000e5d640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5fe40a2dc0 0x7f5fe40a52d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:23.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.801+0000 7f6000e5d640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ffc10c330 0x7f5ffc06dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:23.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.801+0000 7f6000e5d640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ffc1079d0 0x7f5ffc06d7a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:23.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.801+0000 7f6000e5d640 1 --2- 172.21.15.129:0/1926903117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ffc101aa0 0x7f5ffc073690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:23.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.801+0000 7f6000e5d640 1 -- 172.21.15.129:0/1926903117 >> 172.21.15.129:0/1926903117 conn(0x7f5ffc0770c0 msgr2=0x7f5ffc103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:23.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.803+0000 7f6000e5d640 1 -- 172.21.15.129:0/1926903117 shutdown_connections 2024-05-20T18:00:23.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:23.803+0000 7f6000e5d640 1 -- 172.21.15.129:0/1926903117 wait complete. 2024-05-20T18:00:23.853 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:00:24.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.051+0000 7fa04fb79640 1 -- 172.21.15.129:0/557774493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa048106e10 msgr2=0x7fa0481072a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.051+0000 7fa04fb79640 1 --2- 172.21.15.129:0/557774493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa048106e10 0x7fa0481072a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa040009ca0 tx=0x7fa04002a8e0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.053+0000 7fa04fb79640 1 -- 172.21.15.129:0/557774493 shutdown_connections 2024-05-20T18:00:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.053+0000 7fa04fb79640 1 --2- 172.21.15.129:0/557774493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0481063e0 0x7fa0481067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.053+0000 7fa04fb79640 1 --2- 172.21.15.129:0/557774493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa048106e10 0x7fa0481072a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.053+0000 7fa04fb79640 1 --2- 172.21.15.129:0/557774493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa048101990 0x7fa048101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.053+0000 7fa04fb79640 1 -- 172.21.15.129:0/557774493 >> 172.21.15.129:0/557774493 conn(0x7fa0480770b0 msgr2=0x7fa0480774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.054+0000 7fa04fb79640 1 -- 172.21.15.129:0/557774493 shutdown_connections 2024-05-20T18:00:24.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.054+0000 7fa04fb79640 1 -- 172.21.15.129:0/557774493 wait complete. 2024-05-20T18:00:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.056+0000 7fa04fb79640 1 Processor -- start 2024-05-20T18:00:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.056+0000 7fa04fb79640 1 -- start start 2024-05-20T18:00:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.056+0000 7fa04fb79640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa048101990 0x7fa0481a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.056+0000 7fa04fb79640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0481063e0 0x7fa0481a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.056+0000 7fa04fb79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa048106e10 0x7fa04819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.056+0000 7fa04fb79640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa048119320 con 0x7fa048101990 2024-05-20T18:00:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.056+0000 7fa04fb79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481191a0 con 0x7fa048106e10 2024-05-20T18:00:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.056+0000 7fa04fb79640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481194a0 con 0x7fa0481063e0 2024-05-20T18:00:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.056+0000 7fa04d915640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa048101990 0x7fa0481a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.056+0000 7fa04d915640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa048101990 0x7fa0481a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59040/0 (socket says 172.21.15.129:59040) 2024-05-20T18:00:24.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.056+0000 7fa04d915640 1 -- 172.21.15.129:0/2017309821 learned_addr learned my addr 172.21.15.129:0/2017309821 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:24.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.057+0000 7fa04e116640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa048106e10 0x7fa04819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.057+0000 7fa04d114640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0481063e0 0x7fa0481a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.057+0000 7fa04d915640 1 -- 172.21.15.129:0/2017309821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0481063e0 msgr2=0x7fa0481a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.057+0000 7fa04d915640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0481063e0 0x7fa0481a5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.057+0000 7fa04d915640 1 -- 172.21.15.129:0/2017309821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa048106e10 msgr2=0x7fa04819fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.057+0000 7fa04d915640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa048106e10 0x7fa04819fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.057+0000 7fa04d915640 1 -- 172.21.15.129:0/2017309821 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0481a0140 con 0x7fa048101990 2024-05-20T18:00:24.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.057+0000 7fa04e116640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa048106e10 0x7fa04819fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:00:24.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.057+0000 7fa04d915640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa048101990 0x7fa0481a58d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa03800e9f0 tx=0x7fa03800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.058+0000 7fa03effd640 1 -- 172.21.15.129:0/2017309821 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03800cd80 con 0x7fa048101990 2024-05-20T18:00:24.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.058+0000 7fa03effd640 1 -- 172.21.15.129:0/2017309821 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa038010910 con 0x7fa048101990 2024-05-20T18:00:24.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.058+0000 7fa04fb79640 1 -- 172.21.15.129:0/2017309821 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0481a0430 con 0x7fa048101990 2024-05-20T18:00:24.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.058+0000 7fa03effd640 1 -- 172.21.15.129:0/2017309821 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038018990 con 0x7fa048101990 2024-05-20T18:00:24.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.058+0000 7fa04fb79640 1 -- 172.21.15.129:0/2017309821 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0481ac770 con 0x7fa048101990 2024-05-20T18:00:24.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.063+0000 7fa056a1a740 1 Processor -- start 2024-05-20T18:00:24.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.064+0000 7fa056a1a740 1 -- start start 2024-05-20T18:00:24.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.065+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab860390 0x558cabbfed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.065+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab8374d0 0x558cabbff260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.066+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab83f6b0 0x558cabbff7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.066+0000 7fa056a1a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558cabb64c40 con 0x558cab8374d0 2024-05-20T18:00:24.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.066+0000 7fa056a1a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558cabb0c580 con 0x558cab860390 2024-05-20T18:00:24.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.066+0000 7fa056a1a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558cabb60c00 con 0x558cab83f6b0 2024-05-20T18:00:24.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.066+0000 7fa04e116640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab860390 0x558cabbfed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.066+0000 7fa04e116640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab860390 0x558cabbfed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52224/0 (socket says 172.21.15.129:52224) 2024-05-20T18:00:24.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.067+0000 7fa04e116640 1 -- 172.21.15.129:0/3256870671 learned_addr learned my addr 172.21.15.129:0/3256870671 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:24.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.067+0000 7fa04d114640 1 --2- 172.21.15.129:0/3256870671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab83f6b0 0x558cabbff7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.067+0000 7fa04e116640 1 -- 172.21.15.129:0/3256870671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab83f6b0 msgr2=0x558cabbff7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.067+0000 7fa04e116640 1 --2- 172.21.15.129:0/3256870671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab83f6b0 0x558cabbff7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.067+0000 7fa04e116640 1 -- 172.21.15.129:0/3256870671 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab8374d0 msgr2=0x558cabbff260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:00:24.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.067+0000 7fa04e116640 1 --2- 172.21.15.129:0/3256870671 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab8374d0 0x558cabbff260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.067+0000 7fa04e116640 1 -- 172.21.15.129:0/3256870671 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cab8783f0 con 0x558cab860390 2024-05-20T18:00:24.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.069+0000 7fa03effd640 1 -- 172.21.15.129:0/2017309821 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa038010ac0 con 0x7fa048101990 2024-05-20T18:00:24.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.070+0000 7fa04e116640 1 --2- 172.21.15.129:0/3256870671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab860390 0x558cabbfed20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa040009a60 tx=0x7fa040031500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=910e129e219a9c01 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.071+0000 7fa03effd640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0340a2e40 0x7fa0340a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.071+0000 7fa03effd640 1 -- 172.21.15.129:0/2017309821 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa0380cf880 con 0x7fa048101990 2024-05-20T18:00:24.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.071+0000 7fa033fff640 1 -- 172.21.15.129:0/3256870671 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040031990 con 0x558cab860390 2024-05-20T18:00:24.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.071+0000 7fa033fff640 1 -- 172.21.15.129:0/3256870671 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa040036040 con 0x558cab860390 2024-05-20T18:00:24.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.072+0000 7fa04d915640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0340a2e40 0x7fa0340a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.073+0000 7fa056a1a740 1 -- 172.21.15.129:0/3256870671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab860390 msgr2=0x558cabbfed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.073+0000 7fa04d915640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0340a2e40 0x7fa0340a5350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x558cabbfc810 tx=0x7fa03800cf20 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.073+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3256870671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab860390 0x558cabbfed20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa040009a60 tx=0x7fa040031500 comp rx=0 tx=0).stop 2024-05-20T18:00:24.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.073+0000 7fa056a1a740 1 -- 172.21.15.129:0/3256870671 shutdown_connections 2024-05-20T18:00:24.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.073+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3256870671 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab83f6b0 0x558cabbff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.073+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3256870671 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab8374d0 0x558cabbff260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.073+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3256870671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab860390 0x558cabbfed20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.073+0000 7fa056a1a740 1 -- 172.21.15.129:0/3256870671 >> 172.21.15.129:0/3256870671 conn(0x558cab8544e0 msgr2=0x558cabbfb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.073+0000 7fa056a1a740 1 -- 172.21.15.129:0/3256870671 shutdown_connections 2024-05-20T18:00:24.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.073+0000 7fa056a1a740 1 -- 172.21.15.129:0/3256870671 wait complete. 2024-05-20T18:00:24.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.073+0000 7fa056a1a740 1 -- start start 2024-05-20T18:00:24.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.074+0000 7fa056a1a740 1 Processor -- start 2024-05-20T18:00:24.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.074+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc08d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.074+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab83f6b0 0x558cabc09270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.074+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 0x558cabc097b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa056a1a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558cabc0c000 con 0x558cab81ef60 2024-05-20T18:00:24.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa04e116640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 0x558cabc097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa04e116640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 0x558cabc097b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59046/0 (socket says 172.21.15.129:59046) 2024-05-20T18:00:24.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa04e116640 1 -- 172.21.15.129:0/2001139949 learned_addr learned my addr 172.21.15.129:0/2001139949 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:24.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa056a1a740 1 -- 172.21.15.129:0/2001139949 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558cabb1fc40 con 0x558cab83f6b0 2024-05-20T18:00:24.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa056a1a740 1 -- 172.21.15.129:0/2001139949 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558cabc0c180 con 0x558cab8374d0 2024-05-20T18:00:24.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa04e116640 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc08d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa056a1a740 10 client.0 _ll_register_callbacks cb 0x558cab877be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:24.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa04e116640 1 -- 172.21.15.129:0/2001139949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 msgr2=0x558cabc08d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa04e116640 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc08d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa04e116640 1 -- 172.21.15.129:0/2001139949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab83f6b0 msgr2=0x558cabc09270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa04e116640 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab83f6b0 0x558cabc09270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa04e116640 1 -- 172.21.15.129:0/2001139949 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cab9427d0 con 0x558cab81ef60 2024-05-20T18:00:24.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.075+0000 7fa04e116640 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 0x558cabc097b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa04002aed0 tx=0x7fa04003d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31114c96095918e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.076+0000 7fa0317fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:00:24.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.076+0000 7fa056a1a740 1 -- 172.21.15.129:0/2001139949 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558cabb72370 con 0x558cab81ef60 2024-05-20T18:00:24.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.076+0000 7fa0317fa640 1 -- 172.21.15.129:0/2001139949 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040055060 con 0x558cab81ef60 2024-05-20T18:00:24.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.076+0000 7fa0317fa640 1 -- 172.21.15.129:0/2001139949 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa040031990 con 0x558cab81ef60 2024-05-20T18:00:24.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.076+0000 7fa0317fa640 1 -- 172.21.15.129:0/2001139949 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa040031b90 con 0x558cab81ef60 2024-05-20T18:00:24.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.076+0000 7fa056a1a740 10 client.15774 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:24.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.076+0000 7fa056a1a740 1 -- 172.21.15.129:0/2001139949 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({fsmap=22}) -- 0x558cabc18060 con 0x558cab81ef60 2024-05-20T18:00:24.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.077+0000 7fa0317fa640 1 -- 172.21.15.129:0/2001139949 <== mon.0 v2:172.21.15.129:3300/0 4 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fa04004f040 con 0x558cab81ef60 2024-05-20T18:00:24.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.077+0000 7fa056a1a740 10 client.15774 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:24.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.077+0000 7fa056a1a740 10 client.15774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:24.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.077+0000 7fa056a1a740 20 client.15774 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:24.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.077+0000 7fa056a1a740 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558cab7f43a0 0x558cabc1b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.077+0000 7fa056a1a740 4 client.15774 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:00:24.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.077+0000 7fa056a1a740 1 -- 172.21.15.129:0/2001139949 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cabb72370 con 0x558cab7f43a0 2024-05-20T18:00:24.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.078+0000 7fa04d114640 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558cab7f43a0 0x558cabc1b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.081+0000 7fa04d114640 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558cab7f43a0 0x558cabc1b3c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=324aebb260452ffc server_cookie=dbb37b4859fb2e4e in_seq=0 out_seq=0 2024-05-20T18:00:24.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.081+0000 7fa0317fa640 10 client.15774 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:24.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.083+0000 7fa0317fa640 1 -- 172.21.15.129:0/2001139949 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558cabb72370 con 0x558cab7f43a0 2024-05-20T18:00:24.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.083+0000 7fa0317fa640 10 client.15774 handle_command_reply: tid=0 2024-05-20T18:00:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.083+0000 7fa056a1a740 1 client.15774 shutdown 2024-05-20T18:00:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa056a1a740 1 -- 172.21.15.129:0/2001139949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 msgr2=0x558cabc097b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa056a1a740 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 0x558cabc097b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa04002aed0 tx=0x7fa04003d040 comp rx=0 tx=0).stop 2024-05-20T18:00:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa056a1a740 1 -- 172.21.15.129:0/2001139949 shutdown_connections 2024-05-20T18:00:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa056a1a740 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558cab7f43a0 0x558cabc1b3c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa056a1a740 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab83f6b0 0x558cabc09270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa056a1a740 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 0x558cabc097b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa056a1a740 1 --2- 172.21.15.129:0/2001139949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc08d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa056a1a740 1 -- 172.21.15.129:0/2001139949 >> 172.21.15.129:0/2001139949 conn(0x558cab8544e0 msgr2=0x558cabbfecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa0317fa640 0 client.15774 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:24.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa056a1a740 1 -- 172.21.15.129:0/2001139949 shutdown_connections 2024-05-20T18:00:24.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa056a1a740 1 -- 172.21.15.129:0/2001139949 wait complete. 2024-05-20T18:00:24.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.084+0000 7fa056a1a740 20 client.15774 trim_cache size 0 max 16384 2024-05-20T18:00:24.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.127+0000 7fa056a1a740 1 Processor -- start 2024-05-20T18:00:24.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.127+0000 7fa056a1a740 1 -- start start 2024-05-20T18:00:24.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.128+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 0x558cabbff330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.129+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.129+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab83f6b0 0x558cabc1e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.129+0000 7fa056a1a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481b2fd0 con 0x558cab81ef60 2024-05-20T18:00:24.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.129+0000 7fa056a1a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481b5800 con 0x558cab83f6b0 2024-05-20T18:00:24.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.129+0000 7fa056a1a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481b35f0 con 0x558cab8374d0 2024-05-20T18:00:24.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.132+0000 7fa04e116640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab83f6b0 0x558cabc1e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.132+0000 7fa04e116640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 0x558cabbff330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.132+0000 7fa04e116640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 0x558cabbff330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59048/0 (socket says 172.21.15.129:59048) 2024-05-20T18:00:24.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.132+0000 7fa04e116640 1 -- 172.21.15.129:0/3408953195 learned_addr learned my addr 172.21.15.129:0/3408953195 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:24.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.133+0000 7fa04d114640 1 --2- 172.21.15.129:0/3408953195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.135+0000 7fa04d114640 1 -- 172.21.15.129:0/3408953195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab83f6b0 msgr2=0x558cabc1e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.135+0000 7fa04d114640 1 --2- 172.21.15.129:0/3408953195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab83f6b0 0x558cabc1e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.135+0000 7fa04d114640 1 -- 172.21.15.129:0/3408953195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 msgr2=0x558cabbff330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.135+0000 7fa04d114640 1 --2- 172.21.15.129:0/3408953195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 0x558cabbff330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.135+0000 7fa04d114640 1 -- 172.21.15.129:0/3408953195 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cab8605c0 con 0x558cab8374d0 2024-05-20T18:00:24.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.136+0000 7fa04d114640 1 --2- 172.21.15.129:0/3408953195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc095d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x558cabc0c1b0 tx=0x7fa044005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f54ffe9ed880f9d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.137+0000 7fa0337fe640 1 -- 172.21.15.129:0/3408953195 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa044003b30 con 0x558cab8374d0 2024-05-20T18:00:24.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.137+0000 7fa0337fe640 1 -- 172.21.15.129:0/3408953195 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa044003cd0 con 0x558cab8374d0 2024-05-20T18:00:24.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.139+0000 7fa056a1a740 1 -- 172.21.15.129:0/3408953195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 msgr2=0x558cabc095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.139+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3408953195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc095d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x558cabc0c1b0 tx=0x7fa044005850 comp rx=0 tx=0).stop 2024-05-20T18:00:24.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.144+0000 7fa056a1a740 1 -- 172.21.15.129:0/3408953195 shutdown_connections 2024-05-20T18:00:24.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.144+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3408953195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab83f6b0 0x558cabc1e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.144+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3408953195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc095d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.144+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3408953195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab81ef60 0x558cabbff330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.144+0000 7fa056a1a740 1 -- 172.21.15.129:0/3408953195 >> 172.21.15.129:0/3408953195 conn(0x558cab8544e0 msgr2=0x558cabc14e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.157+0000 7fa056a1a740 1 -- 172.21.15.129:0/3408953195 shutdown_connections 2024-05-20T18:00:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.159+0000 7fa056a1a740 1 -- 172.21.15.129:0/3408953195 wait complete. 2024-05-20T18:00:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.160+0000 7fa056a1a740 1 -- start start 2024-05-20T18:00:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.160+0000 7fa056a1a740 1 Processor -- start 2024-05-20T18:00:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.161+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab81ef60 0x558cabc1b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.161+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc1bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.161+0000 7fa056a1a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab83f6b0 0x558cabc7b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.161+0000 7fa056a1a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481b3cc0 con 0x558cab83f6b0 2024-05-20T18:00:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.161+0000 7fa056a1a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481b5130 con 0x558cab81ef60 2024-05-20T18:00:24.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.161+0000 7fa056a1a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa0481b4390 con 0x558cab8374d0 2024-05-20T18:00:24.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.161+0000 7fa056a1a740 10 client.0 _ll_register_callbacks cb 0x7fa04400c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:24.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.162+0000 7fa04d114640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc1bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:24 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1926903117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:00:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.164+0000 7fa04d114640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc1bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:55262/0 (socket says 172.21.15.129:55262) 2024-05-20T18:00:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.164+0000 7fa04e116640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab83f6b0 0x558cabc7b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.164+0000 7fa04d114640 1 -- 172.21.15.129:0/3601949469 learned_addr learned my addr 172.21.15.129:0/3601949469 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:24.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.168+0000 7fa04e116640 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab81ef60 0x558cabc1b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.169+0000 7fa04e116640 1 -- 172.21.15.129:0/3601949469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 msgr2=0x558cabc1bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.169+0000 7fa04e116640 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc1bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.169+0000 7fa04e116640 1 -- 172.21.15.129:0/3601949469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab81ef60 msgr2=0x558cabc1b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.169+0000 7fa04e116640 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab81ef60 0x558cabc1b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.169+0000 7fa04e116640 1 -- 172.21.15.129:0/3601949469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cabc7ba60 con 0x558cab83f6b0 2024-05-20T18:00:24.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.178+0000 7fa04e116640 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab83f6b0 0x558cabc7b520 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa040004330 tx=0x7fa040045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d914a3824fb2049 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.178+0000 7fa0317fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:00:24.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.179+0000 7fa0317fa640 1 -- 172.21.15.129:0/3601949469 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040003b80 con 0x558cab83f6b0 2024-05-20T18:00:24.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.179+0000 7fa0317fa640 1 -- 172.21.15.129:0/3601949469 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa040003d20 con 0x558cab83f6b0 2024-05-20T18:00:24.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.179+0000 7fa056a1a740 1 -- 172.21.15.129:0/3601949469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558cabb6c6c0 con 0x558cab83f6b0 2024-05-20T18:00:24.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.180+0000 7fa0317fa640 1 -- 172.21.15.129:0/3601949469 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa0400305e0 con 0x558cab83f6b0 2024-05-20T18:00:24.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.180+0000 7fa056a1a740 10 client.15780 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:24.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.180+0000 7fa056a1a740 1 -- 172.21.15.129:0/3601949469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558cabc93df0 con 0x558cab83f6b0 2024-05-20T18:00:24.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.181+0000 7fa0317fa640 1 -- 172.21.15.129:0/3601949469 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558cabb6c6c0 con 0x558cab83f6b0 2024-05-20T18:00:24.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.183+0000 7fa0317fa640 1 -- 172.21.15.129:0/3601949469 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fa04005a550 con 0x558cab83f6b0 2024-05-20T18:00:24.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.190+0000 7fa056a1a740 10 client.15780 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:24.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.190+0000 7fa056a1a740 10 client.15780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:24.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.190+0000 7fa056a1a740 20 client.15780 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:24.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.190+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558cabc94450 0x558cabc96870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.190+0000 7fa056a1a740 4 client.15780 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:00:24.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.190+0000 7fa056a1a740 1 -- 172.21.15.129:0/3601949469 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558cabae9820 con 0x558cabc94450 2024-05-20T18:00:24.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.196+0000 7fa04d114640 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558cabc94450 0x558cabc96870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.207+0000 7fa04d114640 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558cabc94450 0x558cabc96870 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd3dbfeacd118a3c server_cookie=2c63f6737c0308b8 in_seq=0 out_seq=0 2024-05-20T18:00:24.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.207+0000 7fa0317fa640 10 client.15780 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:24.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.213+0000 7fa0317fa640 1 -- 172.21.15.129:0/3601949469 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558cabae9820 con 0x558cabc94450 2024-05-20T18:00:24.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.214+0000 7fa0317fa640 10 client.15780 handle_command_reply: tid=0 2024-05-20T18:00:24.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.214+0000 7fa056a1a740 1 client.15780 shutdown 2024-05-20T18:00:24.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.214+0000 7fa056a1a740 1 -- 172.21.15.129:0/3601949469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab83f6b0 msgr2=0x558cabc7b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.214+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab83f6b0 0x558cabc7b520 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa040004330 tx=0x7fa040045000 comp rx=0 tx=0).stop 2024-05-20T18:00:24.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.215+0000 7fa056a1a740 1 -- 172.21.15.129:0/3601949469 shutdown_connections 2024-05-20T18:00:24.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.215+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558cabc94450 0x558cabc96870 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.215+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558cab83f6b0 0x558cabc7b520 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.215+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558cab8374d0 0x558cabc1bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.215+0000 7fa056a1a740 1 --2- 172.21.15.129:0/3601949469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cab81ef60 0x558cabc1b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.215+0000 7fa056a1a740 1 -- 172.21.15.129:0/3601949469 >> 172.21.15.129:0/3601949469 conn(0x558cab8544e0 msgr2=0x558cabc14e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.215+0000 7fa0317fa640 0 client.15780 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:24.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.220+0000 7fa056a1a740 1 -- 172.21.15.129:0/3601949469 shutdown_connections 2024-05-20T18:00:24.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.220+0000 7fa056a1a740 1 -- 172.21.15.129:0/3601949469 wait complete. 2024-05-20T18:00:24.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.221+0000 7fa056a1a740 20 client.15780 trim_cache size 0 max 16384 2024-05-20T18:00:24.221 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:00:24.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.225+0000 7fa04fb79640 1 -- 172.21.15.129:0/2017309821 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0340a2e40 msgr2=0x7fa0340a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.225+0000 7fa04fb79640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0340a2e40 0x7fa0340a5350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x558cabbfc810 tx=0x7fa03800cf20 comp rx=0 tx=0).stop 2024-05-20T18:00:24.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.225+0000 7fa04fb79640 1 -- 172.21.15.129:0/2017309821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa048101990 msgr2=0x7fa0481a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.225+0000 7fa04fb79640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa048101990 0x7fa0481a58d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa03800e9f0 tx=0x7fa03800eeb0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.226+0000 7fa04fb79640 1 -- 172.21.15.129:0/2017309821 shutdown_connections 2024-05-20T18:00:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.226+0000 7fa04fb79640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0340a2e40 0x7fa0340a5350 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.226+0000 7fa04fb79640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa048106e10 0x7fa04819fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.226+0000 7fa04fb79640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0481063e0 0x7fa0481a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.226+0000 7fa04fb79640 1 --2- 172.21.15.129:0/2017309821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa048101990 0x7fa0481a58d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.226+0000 7fa04fb79640 1 -- 172.21.15.129:0/2017309821 >> 172.21.15.129:0/2017309821 conn(0x7fa0480770b0 msgr2=0x7fa048103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.229+0000 7fa04fb79640 1 -- 172.21.15.129:0/2017309821 shutdown_connections 2024-05-20T18:00:24.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:24.229+0000 7fa04fb79640 1 -- 172.21.15.129:0/2017309821 wait complete. 2024-05-20T18:00:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.357+0000 7f789751a640 1 -- 172.21.15.129:0/1017778880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7890109080 msgr2=0x7f7890109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.357+0000 7f789751a640 1 --2- 172.21.15.129:0/1017778880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7890109080 0x7f7890109490 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7880009860 tx=0x7f788002a410 comp rx=0 tx=0).stop 2024-05-20T18:00:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.360+0000 7f789751a640 1 -- 172.21.15.129:0/1017778880 shutdown_connections 2024-05-20T18:00:24.358 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.360+0000 7f789751a640 1 --2- 172.21.15.129:0/1017778880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7890119770 0x7f789011bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.360+0000 7f789751a640 1 --2- 172.21.15.129:0/1017778880 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f789011c0d0 0x7f789011e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.360+0000 7f789751a640 1 --2- 172.21.15.129:0/1017778880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7890109080 0x7f7890109490 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.359 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.360+0000 7f789751a640 1 -- 172.21.15.129:0/1017778880 >> 172.21.15.129:0/1017778880 conn(0x7f789006d780 msgr2=0x7f789006dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.366+0000 7f789751a640 1 -- 172.21.15.129:0/1017778880 shutdown_connections 2024-05-20T18:00:24.367 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.366+0000 7f789751a640 1 -- 172.21.15.129:0/1017778880 wait complete. 2024-05-20T18:00:24.374 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.376+0000 7f789751a640 1 Processor -- start 2024-05-20T18:00:24.375 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.376+0000 7f789751a640 1 -- start start 2024-05-20T18:00:24.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.377+0000 7f789751a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7890109080 0x7f78901a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.378+0000 7f789751a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f789011c0d0 0x7f78901a8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.379+0000 7f789751a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78901acec0 0x7f78901ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.379+0000 7f789751a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7890121ce0 con 0x7f7890109080 2024-05-20T18:00:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.379+0000 7f789751a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7890121b60 con 0x7f789011c0d0 2024-05-20T18:00:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.379+0000 7f7894ab5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f789011c0d0 0x7f78901a8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.379+0000 7f7894ab5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f789011c0d0 0x7f78901a8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52266/0 (socket says 172.21.15.129:52266) 2024-05-20T18:00:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.379+0000 7f7894ab5640 1 -- 172.21.15.129:0/2077101063 learned_addr learned my addr 172.21.15.129:0/2077101063 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:24.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.379+0000 7f789751a640 1 -- 172.21.15.129:0/2077101063 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7890121e60 con 0x7f78901acec0 2024-05-20T18:00:24.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.380+0000 7f7895ab7640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78901acec0 0x7f78901ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.381+0000 7f78952b6640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7890109080 0x7f78901a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.387+0000 7f7895ab7640 1 -- 172.21.15.129:0/2077101063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f789011c0d0 msgr2=0x7f78901a8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.387+0000 7f7895ab7640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f789011c0d0 0x7f78901a8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.387+0000 7f7895ab7640 1 -- 172.21.15.129:0/2077101063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7890109080 msgr2=0x7f78901a86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.387+0000 7f7895ab7640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7890109080 0x7f78901a86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.385 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.387+0000 7f7895ab7640 1 -- 172.21.15.129:0/2077101063 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78901ad910 con 0x7f78901acec0 2024-05-20T18:00:24.386 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.388+0000 7f7895ab7640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78901acec0 0x7f78901ad2d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f788c00d680 tx=0x7f788c00db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.390+0000 7f78867fc640 1 -- 172.21.15.129:0/2077101063 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c017070 con 0x7f78901acec0 2024-05-20T18:00:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.390+0000 7f78867fc640 1 -- 172.21.15.129:0/2077101063 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f788c009820 con 0x7f78901acec0 2024-05-20T18:00:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.390+0000 7f789751a640 1 -- 172.21.15.129:0/2077101063 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78901adc00 con 0x7f78901acec0 2024-05-20T18:00:24.390 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.391+0000 7f789751a640 1 -- 172.21.15.129:0/2077101063 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78901b5350 con 0x7f78901acec0 2024-05-20T18:00:24.392 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.394+0000 7f78867fc640 1 -- 172.21.15.129:0/2077101063 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c004ce0 con 0x7f78901acec0 2024-05-20T18:00:24.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:24 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1926903117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:00:24.400 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.402+0000 7f789e3c2740 1 Processor -- start 2024-05-20T18:00:24.401 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.403+0000 7f789e3c2740 1 -- start start 2024-05-20T18:00:24.403 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.405+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d761b0 0x55ed64115080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.416+0000 7f7894ab5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d761b0 0x55ed64115080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.416+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 0x55ed641155c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.417+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed64115b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.417+0000 7f789e3c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ed6407a340 con 0x55ed63d761b0 2024-05-20T18:00:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.417+0000 7f789e3c2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ed64022290 con 0x55ed63d554d0 2024-05-20T18:00:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.417+0000 7f789e3c2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ed64076300 con 0x55ed63d4d2f0 2024-05-20T18:00:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.418+0000 7f78952b6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 0x55ed641155c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.418+0000 7f78952b6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 0x55ed641155c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:55274/0 (socket says 172.21.15.129:55274) 2024-05-20T18:00:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.418+0000 7f78952b6640 1 -- 172.21.15.129:0/1937029496 learned_addr learned my addr 172.21.15.129:0/1937029496 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.419+0000 7f7894ab5640 1 --2- 172.21.15.129:0/1937029496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed64115b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.421+0000 7f7894ab5640 1 -- 172.21.15.129:0/1937029496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 msgr2=0x55ed641155c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.421+0000 7f7894ab5640 1 --2- 172.21.15.129:0/1937029496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 0x55ed641155c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.420 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.421+0000 7f7894ab5640 1 -- 172.21.15.129:0/1937029496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 msgr2=0x55ed64115b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.420 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.421+0000 7f7894ab5640 1 --2- 172.21.15.129:0/1937029496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed64115b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.420 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.421+0000 7f7894ab5640 1 -- 172.21.15.129:0/1937029496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed640468e0 con 0x55ed63d761b0 2024-05-20T18:00:24.420 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.421+0000 7f78952b6640 1 --2- 172.21.15.129:0/1937029496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 0x55ed641155c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:00:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.422+0000 7f78867fc640 1 -- 172.21.15.129:0/2077101063 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f788c005220 con 0x7f78901acec0 2024-05-20T18:00:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.423+0000 7f78867fc640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f787c0a2d20 0x7f787c0a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.424+0000 7f7894ab5640 1 --2- 172.21.15.129:0/1937029496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d761b0 0x55ed64115080 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7888005c40 tx=0x7f78880078c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17681d1a8afd52d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.424+0000 7f78867fc640 1 -- 172.21.15.129:0/2077101063 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f788c0cecc0 con 0x7f78901acec0 2024-05-20T18:00:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.426+0000 7f78952b6640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f787c0a2d20 0x7f787c0a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.426+0000 7f787b7fe640 1 -- 172.21.15.129:0/1937029496 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78880142d0 con 0x55ed63d761b0 2024-05-20T18:00:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.427+0000 7f787b7fe640 1 -- 172.21.15.129:0/1937029496 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f788800fac0 con 0x55ed63d761b0 2024-05-20T18:00:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.429+0000 7f789e3c2740 1 -- 172.21.15.129:0/1937029496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d761b0 msgr2=0x55ed64115080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.429+0000 7f789e3c2740 1 --2- 172.21.15.129:0/1937029496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d761b0 0x55ed64115080 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7888005c40 tx=0x7f78880078c0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.429+0000 7f789e3c2740 1 -- 172.21.15.129:0/1937029496 shutdown_connections 2024-05-20T18:00:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.429+0000 7f789e3c2740 1 --2- 172.21.15.129:0/1937029496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed64115b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.429+0000 7f789e3c2740 1 --2- 172.21.15.129:0/1937029496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 0x55ed641155c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.429+0000 7f789e3c2740 1 --2- 172.21.15.129:0/1937029496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d761b0 0x55ed64115080 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.429+0000 7f789e3c2740 1 -- 172.21.15.129:0/1937029496 >> 172.21.15.129:0/1937029496 conn(0x55ed63d6a300 msgr2=0x55ed64111830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:24 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1926903117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:00:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.437+0000 7f789e3c2740 1 -- 172.21.15.129:0/1937029496 shutdown_connections 2024-05-20T18:00:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.437+0000 7f789e3c2740 1 -- 172.21.15.129:0/1937029496 wait complete. 2024-05-20T18:00:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.438+0000 7f789e3c2740 1 -- start start 2024-05-20T18:00:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.439+0000 7f789e3c2740 1 Processor -- start 2024-05-20T18:00:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.440+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d4d2f0 0x55ed64121130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.440+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d554d0 0x55ed64121670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.440+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d761b0 0x55ed6411f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.441+0000 7f789e3c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ed64122200 con 0x55ed63d554d0 2024-05-20T18:00:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.441+0000 7f789e3c2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ed64035790 con 0x55ed63d4d2f0 2024-05-20T18:00:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.441+0000 7f789e3c2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ed64122380 con 0x55ed63d761b0 2024-05-20T18:00:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.441+0000 7f7895ab7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d761b0 0x55ed6411f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.441+0000 7f7895ab7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d761b0 0x55ed6411f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:55276/0 (socket says 172.21.15.129:55276) 2024-05-20T18:00:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.441+0000 7f7895ab7640 1 -- 172.21.15.129:0/2519434407 learned_addr learned my addr 172.21.15.129:0/2519434407 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.441+0000 7f789e3c2740 10 client.0 _ll_register_callbacks cb 0x55ed64070a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.442+0000 7f78952b6640 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d4d2f0 0x55ed64121130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.445+0000 7f7895ab7640 1 -- 172.21.15.129:0/2519434407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d4d2f0 msgr2=0x55ed64121130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.445+0000 7f7895ab7640 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d4d2f0 0x55ed64121130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.445+0000 7f7895ab7640 1 -- 172.21.15.129:0/2519434407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d554d0 msgr2=0x55ed64121670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.445+0000 7f7895ab7640 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d554d0 0x55ed64121670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.446+0000 7f7895ab7640 1 -- 172.21.15.129:0/2519434407 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed63d8e210 con 0x55ed63d761b0 2024-05-20T18:00:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.447+0000 7f78952b6640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f787c0a2d20 0x7f787c0a5230 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7880000c00 tx=0x7f7880004200 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.450+0000 7f7895ab7640 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d761b0 0x55ed6411f5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f788c097b40 tx=0x7f788c09b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b662871523a337a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.451+0000 7f7878ff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:00:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.451+0000 7f7878ff9640 1 -- 172.21.15.129:0/2519434407 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c020040 con 0x55ed63d761b0 2024-05-20T18:00:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.451+0000 7f789e3c2740 1 -- 172.21.15.129:0/2519434407 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f788800fac0 con 0x55ed63d761b0 2024-05-20T18:00:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.451+0000 7f7878ff9640 1 -- 172.21.15.129:0/2519434407 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f788c0bb0b0 con 0x55ed63d761b0 2024-05-20T18:00:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.452+0000 7f7878ff9640 1 -- 172.21.15.129:0/2519434407 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f788c09f520 con 0x55ed63d761b0 2024-05-20T18:00:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.452+0000 7f789e3c2740 10 client.25480 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:24.451 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.452+0000 7f789e3c2740 1 -- 172.21.15.129:0/2519434407 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ed63e5a620 con 0x55ed63d761b0 2024-05-20T18:00:24.452 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.454+0000 7f7878ff9640 1 -- 172.21.15.129:0/2519434407 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f788c0a3de0 con 0x55ed63d761b0 2024-05-20T18:00:24.453 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.455+0000 7f7878ff9640 1 -- 172.21.15.129:0/2519434407 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f788c095ce0 con 0x55ed63d761b0 2024-05-20T18:00:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.455+0000 7f789e3c2740 10 client.25480 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.455+0000 7f789e3c2740 10 client.25480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.456+0000 7f789e3c2740 20 client.25480 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.456+0000 7f789e3c2740 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ed63d34d80 0x55ed64131a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.456+0000 7f789e3c2740 4 client.25480 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:00:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.456+0000 7f789e3c2740 1 -- 172.21.15.129:0/2519434407 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f788800fac0 con 0x55ed63d34d80 2024-05-20T18:00:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.458+0000 7f78952b6640 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ed63d34d80 0x55ed64131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.465+0000 7f78952b6640 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ed63d34d80 0x55ed64131a70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d85eb024ff8f8a1 server_cookie=9245a73becb8aea in_seq=0 out_seq=0 2024-05-20T18:00:24.465 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.465+0000 7f7878ff9640 10 client.25480 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.478+0000 7f7878ff9640 1 -- 172.21.15.129:0/2519434407 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f788800fac0 con 0x55ed63d34d80 2024-05-20T18:00:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.478+0000 7f7878ff9640 10 client.25480 handle_command_reply: tid=0 2024-05-20T18:00:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.481+0000 7f789e3c2740 1 client.25480 shutdown 2024-05-20T18:00:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.484+0000 7f789e3c2740 1 -- 172.21.15.129:0/2519434407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d761b0 msgr2=0x55ed6411f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.484+0000 7f789e3c2740 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d761b0 0x55ed6411f5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f788c097b40 tx=0x7f788c09b870 comp rx=0 tx=0).stop 2024-05-20T18:00:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.485+0000 7f789e3c2740 1 -- 172.21.15.129:0/2519434407 shutdown_connections 2024-05-20T18:00:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.485+0000 7f789e3c2740 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ed63d34d80 0x55ed64131a70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.485+0000 7f789e3c2740 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d761b0 0x55ed6411f5b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.485+0000 7f789e3c2740 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d554d0 0x55ed64121670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.485+0000 7f789e3c2740 1 --2- 172.21.15.129:0/2519434407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d4d2f0 0x55ed64121130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.485+0000 7f789e3c2740 1 -- 172.21.15.129:0/2519434407 >> 172.21.15.129:0/2519434407 conn(0x55ed63d6a300 msgr2=0x55ed64114eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.485+0000 7f7878ff9640 0 client.25480 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.485+0000 7f789e3c2740 1 -- 172.21.15.129:0/2519434407 shutdown_connections 2024-05-20T18:00:24.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.485+0000 7f789e3c2740 1 -- 172.21.15.129:0/2519434407 wait complete. 2024-05-20T18:00:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.486+0000 7f789e3c2740 20 client.25480 trim_cache size 0 max 16384 2024-05-20T18:00:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.528+0000 7f789e3c2740 1 Processor -- start 2024-05-20T18:00:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.529+0000 7f789e3c2740 1 -- start start 2024-05-20T18:00:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.529+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d4d2f0 0x55ed64137270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.529+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed641377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.529+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d761b0 0x55ed641112c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.529+0000 7f789e3c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78901bbd80 con 0x55ed63d4d2f0 2024-05-20T18:00:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.529+0000 7f789e3c2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78901be600 con 0x55ed63d554d0 2024-05-20T18:00:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.529+0000 7f789e3c2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78901bc3f0 con 0x55ed63d761b0 2024-05-20T18:00:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.531+0000 7f7894ab5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed641377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.531+0000 7f7894ab5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed641377b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52290/0 (socket says 172.21.15.129:52290) 2024-05-20T18:00:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.531+0000 7f7894ab5640 1 -- 172.21.15.129:0/2779776409 learned_addr learned my addr 172.21.15.129:0/2779776409 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.532+0000 7f7894ab5640 1 -- 172.21.15.129:0/2779776409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d761b0 msgr2=0x55ed641112c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:00:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.532+0000 7f7894ab5640 1 --2- 172.21.15.129:0/2779776409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d761b0 0x55ed641112c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.532+0000 7f7894ab5640 1 -- 172.21.15.129:0/2779776409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d4d2f0 msgr2=0x55ed64137270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:00:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.532+0000 7f7894ab5640 1 --2- 172.21.15.129:0/2779776409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d4d2f0 0x55ed64137270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.532+0000 7f7894ab5640 1 -- 172.21.15.129:0/2779776409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed63d34fb0 con 0x55ed63d554d0 2024-05-20T18:00:24.534 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.533+0000 7f7894ab5640 1 --2- 172.21.15.129:0/2779776409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed641377b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f788801b000 tx=0x7f7888007bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99e1c25481b5d350 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.534 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.536+0000 7f787affd640 1 -- 172.21.15.129:0/2779776409 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78880142d0 con 0x55ed63d554d0 2024-05-20T18:00:24.534 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.536+0000 7f787affd640 1 -- 172.21.15.129:0/2779776409 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7888005720 con 0x55ed63d554d0 2024-05-20T18:00:24.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.538+0000 7f789e3c2740 1 -- 172.21.15.129:0/2779776409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 msgr2=0x55ed641377b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.538+0000 7f789e3c2740 1 --2- 172.21.15.129:0/2779776409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed641377b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f788801b000 tx=0x7f7888007bd0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.538+0000 7f789e3c2740 1 -- 172.21.15.129:0/2779776409 shutdown_connections 2024-05-20T18:00:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.538+0000 7f789e3c2740 1 --2- 172.21.15.129:0/2779776409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed641377b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.538+0000 7f789e3c2740 1 --2- 172.21.15.129:0/2779776409 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d761b0 0x55ed641112c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.538+0000 7f789e3c2740 1 --2- 172.21.15.129:0/2779776409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d4d2f0 0x55ed64137270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.537 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.538+0000 7f789e3c2740 1 -- 172.21.15.129:0/2779776409 >> 172.21.15.129:0/2779776409 conn(0x55ed63d6a300 msgr2=0x55ed6411de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.541+0000 7f789e3c2740 1 -- 172.21.15.129:0/2779776409 shutdown_connections 2024-05-20T18:00:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.541+0000 7f789e3c2740 1 -- 172.21.15.129:0/2779776409 wait complete. 2024-05-20T18:00:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.542+0000 7f789e3c2740 1 -- start start 2024-05-20T18:00:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.542+0000 7f789e3c2740 1 Processor -- start 2024-05-20T18:00:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.543+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 0x55ed641278d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.543+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed64124c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.544+0000 7f789e3c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d761b0 0x55ed64125180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.544+0000 7f789e3c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78901bcac0 con 0x55ed63d761b0 2024-05-20T18:00:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.544+0000 7f78952b6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 0x55ed641278d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.544+0000 7f78952b6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 0x55ed641278d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:55286/0 (socket says 172.21.15.129:55286) 2024-05-20T18:00:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.544+0000 7f78952b6640 1 -- 172.21.15.129:0/1232919659 learned_addr learned my addr 172.21.15.129:0/1232919659 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.544+0000 7f789e3c2740 1 -- 172.21.15.129:0/1232919659 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78901bdf30 con 0x55ed63d554d0 2024-05-20T18:00:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.544+0000 7f7894ab5640 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed64124c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.544+0000 7f789e3c2740 1 -- 172.21.15.129:0/1232919659 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78901bd190 con 0x55ed63d4d2f0 2024-05-20T18:00:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.545+0000 7f789e3c2740 10 client.0 _ll_register_callbacks cb 0x7f788c097410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:24.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.545+0000 7f7894ab5640 1 -- 172.21.15.129:0/1232919659 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 msgr2=0x55ed641278d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.545+0000 7f7894ab5640 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 0x55ed641278d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.545+0000 7f7894ab5640 1 -- 172.21.15.129:0/1232919659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d761b0 msgr2=0x55ed64125180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:24.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.545+0000 7f7894ab5640 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d761b0 0x55ed64125180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.546+0000 7f7894ab5640 1 -- 172.21.15.129:0/1232919659 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed641256c0 con 0x55ed63d554d0 2024-05-20T18:00:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.547+0000 7f7894ab5640 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed64124c40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7888007620 tx=0x7f7888014820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7a064d747fa8867 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.547+0000 7f7878ff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:00:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.547+0000 7f7878ff9640 1 -- 172.21.15.129:0/1232919659 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7888005920 con 0x55ed63d554d0 2024-05-20T18:00:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.548+0000 7f7878ff9640 1 -- 172.21.15.129:0/1232919659 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f788800fac0 con 0x55ed63d554d0 2024-05-20T18:00:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.548+0000 7f789e3c2740 1 -- 172.21.15.129:0/1232919659 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ed64087a70 con 0x55ed63d554d0 2024-05-20T18:00:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.548+0000 7f7878ff9640 1 -- 172.21.15.129:0/1232919659 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f788800fcc0 con 0x55ed63d554d0 2024-05-20T18:00:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.548+0000 7f789e3c2740 10 client.25457 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.548+0000 7f789e3c2740 1 -- 172.21.15.129:0/1232919659 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ed64199a30 con 0x55ed63d554d0 2024-05-20T18:00:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.549+0000 7f7878ff9640 1 -- 172.21.15.129:0/1232919659 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ed64087a70 con 0x55ed63d554d0 2024-05-20T18:00:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.549+0000 7f7878ff9640 1 -- 172.21.15.129:0/1232919659 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f7888029070 con 0x55ed63d554d0 2024-05-20T18:00:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.549+0000 7f789e3c2740 10 client.25457 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.549+0000 7f789e3c2740 10 client.25457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.549+0000 7f789e3c2740 20 client.25457 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.550+0000 7f789e3c2740 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ed63d0a1c0 0x55ed6419be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.550+0000 7f789e3c2740 4 client.25457 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:00:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.550+0000 7f789e3c2740 1 -- 172.21.15.129:0/1232919659 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ed64081dc0 con 0x55ed63d0a1c0 2024-05-20T18:00:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.551+0000 7f78952b6640 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ed63d0a1c0 0x55ed6419be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.553+0000 7f7878ff9640 10 client.25457 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.553+0000 7f78952b6640 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ed63d0a1c0 0x55ed6419be70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9107dd3ef27f9185 server_cookie=dd8ac3b58c462859 in_seq=0 out_seq=0 2024-05-20T18:00:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.730+0000 7f7878ff9640 1 -- 172.21.15.129:0/1232919659 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ed64081dc0 con 0x55ed63d0a1c0 2024-05-20T18:00:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.730+0000 7f7878ff9640 10 client.25457 handle_command_reply: tid=0 2024-05-20T18:00:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.730+0000 7f789e3c2740 1 client.25457 shutdown 2024-05-20T18:00:24.729 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.730+0000 7f789e3c2740 1 -- 172.21.15.129:0/1232919659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 msgr2=0x55ed64124c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.730+0000 7f789e3c2740 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed64124c40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7888007620 tx=0x7f7888014820 comp rx=0 tx=0).stop 2024-05-20T18:00:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.731+0000 7f789e3c2740 1 -- 172.21.15.129:0/1232919659 shutdown_connections 2024-05-20T18:00:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.731+0000 7f789e3c2740 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ed63d0a1c0 0x55ed6419be70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.731+0000 7f789e3c2740 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ed63d554d0 0x55ed64124c40 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7888007620 tx=0x7f7888014820 comp rx=0 tx=0).stop 2024-05-20T18:00:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.731+0000 7f789e3c2740 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ed63d761b0 0x55ed64125180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.731+0000 7f789e3c2740 1 --2- 172.21.15.129:0/1232919659 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ed63d4d2f0 0x55ed641278d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.731+0000 7f789e3c2740 1 -- 172.21.15.129:0/1232919659 >> 172.21.15.129:0/1232919659 conn(0x55ed63d6a300 msgr2=0x55ed64118cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.730 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.731+0000 7f7878ff9640 0 client.25457 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.731+0000 7f789e3c2740 1 -- 172.21.15.129:0/1232919659 shutdown_connections 2024-05-20T18:00:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.731+0000 7f789e3c2740 1 -- 172.21.15.129:0/1232919659 wait complete. 2024-05-20T18:00:24.731 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.732+0000 7f789e3c2740 20 client.25457 trim_cache size 0 max 16384 2024-05-20T18:00:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:00:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:00:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:00:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:00:24.739 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.741+0000 7f789751a640 1 -- 172.21.15.129:0/2077101063 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f787c0a2d20 msgr2=0x7f787c0a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.739 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.741+0000 7f789751a640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f787c0a2d20 0x7f787c0a5230 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7880000c00 tx=0x7f7880004200 comp rx=0 tx=0).stop 2024-05-20T18:00:24.739 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.741+0000 7f789751a640 1 -- 172.21.15.129:0/2077101063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78901acec0 msgr2=0x7f78901ad2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.741+0000 7f789751a640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78901acec0 0x7f78901ad2d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f788c00d680 tx=0x7f788c00db40 comp rx=0 tx=0).stop 2024-05-20T18:00:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.741+0000 7f789751a640 1 -- 172.21.15.129:0/2077101063 shutdown_connections 2024-05-20T18:00:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.744+0000 7f789751a640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f787c0a2d20 0x7f787c0a5230 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.744+0000 7f789751a640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f789011c0d0 0x7f78901a8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.744+0000 7f789751a640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78901acec0 0x7f78901ad2d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.744+0000 7f789751a640 1 --2- 172.21.15.129:0/2077101063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7890109080 0x7f78901a86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:24.743 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.744+0000 7f789751a640 1 -- 172.21.15.129:0/2077101063 >> 172.21.15.129:0/2077101063 conn(0x7f789006d780 msgr2=0x7f7890119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.747+0000 7f789751a640 1 -- 172.21.15.129:0/2077101063 shutdown_connections 2024-05-20T18:00:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:24.747+0000 7f789751a640 1 -- 172.21.15.129:0/2077101063 wait complete. 2024-05-20T18:00:25.281 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:00:25.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:00:25.282 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:00:25.338 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:25 smithi129 ceph-mon[32029]: pgmap v59: 97 pgs: 97 active+clean; 848 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 71 MiB/s wr, 61 op/s 2024-05-20T18:00:25.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:25 smithi154 ceph-mon[36482]: pgmap v59: 97 pgs: 97 active+clean; 848 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 71 MiB/s wr, 61 op/s 2024-05-20T18:00:25.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:25 smithi204 ceph-mon[37076]: pgmap v59: 97 pgs: 97 active+clean; 848 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 71 MiB/s wr, 61 op/s 2024-05-20T18:00:25.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.666+0000 7f676cefc640 1 -- 172.21.15.129:0/4225520880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f676810f3a0 msgr2=0x7f6768108fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:25.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.666+0000 7f676cefc640 1 --2- 172.21.15.129:0/4225520880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f676810f3a0 0x7f6768108fa0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6760005fd0 tx=0x7f676002b430 comp rx=0 tx=0).stop 2024-05-20T18:00:25.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.666+0000 7f676cefc640 1 -- 172.21.15.129:0/4225520880 shutdown_connections 2024-05-20T18:00:25.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.666+0000 7f676cefc640 1 --2- 172.21.15.129:0/4225520880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67681094e0 0x7f67681098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.666+0000 7f676cefc640 1 --2- 172.21.15.129:0/4225520880 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6768113ca0 0x7f67681160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.666+0000 7f676cefc640 1 --2- 172.21.15.129:0/4225520880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f676810f3a0 0x7f6768108fa0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.666+0000 7f676cefc640 1 -- 172.21.15.129:0/4225520880 >> 172.21.15.129:0/4225520880 conn(0x7f676806d580 msgr2=0x7f676806d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:25.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.667+0000 7f676cefc640 1 -- 172.21.15.129:0/4225520880 shutdown_connections 2024-05-20T18:00:25.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.667+0000 7f676cefc640 1 -- 172.21.15.129:0/4225520880 wait complete. 2024-05-20T18:00:25.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.668+0000 7f676cefc640 1 Processor -- start 2024-05-20T18:00:25.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.668+0000 7f676cefc640 1 -- start start 2024-05-20T18:00:25.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.669+0000 7f676cefc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67681094e0 0x7f676819fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.669+0000 7f676cefc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f676810f3a0 0x7f67681a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.669+0000 7f676cefc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6768113ca0 0x7f67681a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.669+0000 7f676cefc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6768119350 con 0x7f6768113ca0 2024-05-20T18:00:25.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.669+0000 7f676cefc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67681191d0 con 0x7f676810f3a0 2024-05-20T18:00:25.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.669+0000 7f676cefc640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f67681194d0 con 0x7f67681094e0 2024-05-20T18:00:25.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.669+0000 7f6767fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6768113ca0 0x7f67681a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.669+0000 7f6767fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6768113ca0 0x7f67681a45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59098/0 (socket says 172.21.15.129:59098) 2024-05-20T18:00:25.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.669+0000 7f6767fff640 1 -- 172.21.15.129:0/1899567783 learned_addr learned my addr 172.21.15.129:0/1899567783 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:25.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.670+0000 7f6766ffd640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f676810f3a0 0x7f67681a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.670+0000 7f67677fe640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67681094e0 0x7f676819fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.671+0000 7f6767fff640 1 -- 172.21.15.129:0/1899567783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67681094e0 msgr2=0x7f676819fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:25.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.671+0000 7f6767fff640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67681094e0 0x7f676819fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.671+0000 7f6767fff640 1 -- 172.21.15.129:0/1899567783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f676810f3a0 msgr2=0x7f67681a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:25.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.671+0000 7f6767fff640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f676810f3a0 0x7f67681a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.671+0000 7f6767fff640 1 -- 172.21.15.129:0/1899567783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67681a4c30 con 0x7f6768113ca0 2024-05-20T18:00:25.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.671+0000 7f6766ffd640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f676810f3a0 0x7f67681a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:00:25.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.671+0000 7f67677fe640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67681094e0 0x7f676819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:00:25.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.671+0000 7f6767fff640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6768113ca0 0x7f67681a45f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f675c00d990 tx=0x7f675c00de50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:25.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.673+0000 7f6764ff9640 1 -- 172.21.15.129:0/1899567783 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c01b070 con 0x7f6768113ca0 2024-05-20T18:00:25.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.674+0000 7f6764ff9640 1 -- 172.21.15.129:0/1899567783 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f675c00ed60 con 0x7f6768113ca0 2024-05-20T18:00:25.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.674+0000 7f6764ff9640 1 -- 172.21.15.129:0/1899567783 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c014860 con 0x7f6768113ca0 2024-05-20T18:00:25.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.674+0000 7f676cefc640 1 -- 172.21.15.129:0/1899567783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67681a4f20 con 0x7f6768113ca0 2024-05-20T18:00:25.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.675+0000 7f676cefc640 1 -- 172.21.15.129:0/1899567783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67681a5420 con 0x7f6768113ca0 2024-05-20T18:00:25.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.680+0000 7f6775006740 1 Processor -- start 2024-05-20T18:00:25.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.682+0000 7f6764ff9640 1 -- 172.21.15.129:0/1899567783 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f675c01e070 con 0x7f6768113ca0 2024-05-20T18:00:25.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.682+0000 7f6775006740 1 -- start start 2024-05-20T18:00:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.682+0000 7f6775006740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab90410 0x56042af39740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.682+0000 7f6775006740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 0x56042af39c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.682+0000 7f6775006740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab6f730 0x56042af3d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.682+0000 7f6775006740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56042ae94990 con 0x56042ab6f730 2024-05-20T18:00:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.682+0000 7f6775006740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56042ae3c5b0 con 0x56042ab90410 2024-05-20T18:00:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.682+0000 7f6775006740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56042ae90950 con 0x56042ab67550 2024-05-20T18:00:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.683+0000 7f6766ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab6f730 0x56042af3d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.683+0000 7f6766ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab6f730 0x56042af3d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59106/0 (socket says 172.21.15.129:59106) 2024-05-20T18:00:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.683+0000 7f6766ffd640 1 -- 172.21.15.129:0/2195189531 learned_addr learned my addr 172.21.15.129:0/2195189531 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.683+0000 7f67677fe640 1 --2- 172.21.15.129:0/2195189531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 0x56042af39c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.684+0000 7f6764ff9640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f67500a2f00 0x7f67500a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.684+0000 7f6766ffd640 1 -- 172.21.15.129:0/2195189531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 msgr2=0x56042af39c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.684+0000 7f6766ffd640 1 --2- 172.21.15.129:0/2195189531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 0x56042af39c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.684+0000 7f6766ffd640 1 -- 172.21.15.129:0/2195189531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab90410 msgr2=0x56042af39740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.684+0000 7f6766ffd640 1 --2- 172.21.15.129:0/2195189531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab90410 0x56042af39740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.684+0000 7f6766ffd640 1 -- 172.21.15.129:0/2195189531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56042ac707e0 con 0x56042ab6f730 2024-05-20T18:00:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.684+0000 7f6766ffd640 1 --2- 172.21.15.129:0/2195189531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab6f730 0x56042af3d720 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6758006ca0 tx=0x7f67580105c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b414d26eb9564 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.684+0000 7f6764ff9640 1 -- 172.21.15.129:0/1899567783 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f675c0ce830 con 0x7f6768113ca0 2024-05-20T18:00:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.684+0000 7f6767fff640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f67500a2f00 0x7f67500a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.685+0000 7f674dffb640 1 -- 172.21.15.129:0/2195189531 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6758018020 con 0x56042ab6f730 2024-05-20T18:00:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.685+0000 7f674dffb640 1 -- 172.21.15.129:0/2195189531 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6758014b20 con 0x56042ab6f730 2024-05-20T18:00:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.685+0000 7f674dffb640 1 -- 172.21.15.129:0/2195189531 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6758013c10 con 0x56042ab6f730 2024-05-20T18:00:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.686+0000 7f6775006740 1 -- 172.21.15.129:0/2195189531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab6f730 msgr2=0x56042af3d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.686+0000 7f6775006740 1 --2- 172.21.15.129:0/2195189531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab6f730 0x56042af3d720 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6758006ca0 tx=0x7f67580105c0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.686+0000 7f6767fff640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f67500a2f00 0x7f67500a5410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f675c09aba0 tx=0x7f675c00ef10 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.687+0000 7f6775006740 1 -- 172.21.15.129:0/2195189531 shutdown_connections 2024-05-20T18:00:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.687+0000 7f6775006740 1 --2- 172.21.15.129:0/2195189531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab6f730 0x56042af3d720 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.687+0000 7f6775006740 1 --2- 172.21.15.129:0/2195189531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 0x56042af39c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.687+0000 7f6775006740 1 --2- 172.21.15.129:0/2195189531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab90410 0x56042af39740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.687+0000 7f6775006740 1 -- 172.21.15.129:0/2195189531 >> 172.21.15.129:0/2195189531 conn(0x56042ab84560 msgr2=0x56042af2b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.688+0000 7f6775006740 1 -- 172.21.15.129:0/2195189531 shutdown_connections 2024-05-20T18:00:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.688+0000 7f6775006740 1 -- 172.21.15.129:0/2195189531 wait complete. 2024-05-20T18:00:25.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.689+0000 7f6775006740 1 -- start start 2024-05-20T18:00:25.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.690+0000 7f6775006740 1 Processor -- start 2024-05-20T18:00:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.692+0000 7f6775006740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 0x56042af48aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.692+0000 7f6775006740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab6f730 0x56042af48fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.692+0000 7f6775006740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab90410 0x56042af42e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.693+0000 7f6775006740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56042af3e150 con 0x56042ab90410 2024-05-20T18:00:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.693+0000 7f6775006740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56042ae4fb40 con 0x56042ab6f730 2024-05-20T18:00:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.693+0000 7f6775006740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56042af3e2d0 con 0x56042ab67550 2024-05-20T18:00:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.693+0000 7f6775006740 10 client.0 _ll_register_callbacks cb 0x56042ae440f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:25.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.694+0000 7f6766ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab90410 0x56042af42e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.694+0000 7f6766ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 0x56042af48aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.694+0000 7f6766ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 0x56042af48aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:55314/0 (socket says 172.21.15.129:55314) 2024-05-20T18:00:25.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.694+0000 7f6766ffd640 1 -- 172.21.15.129:0/3400761371 learned_addr learned my addr 172.21.15.129:0/3400761371 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:25.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.696+0000 7f6766ffd640 1 -- 172.21.15.129:0/3400761371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab6f730 msgr2=0x56042af48fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:25.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.696+0000 7f6766ffd640 1 --2- 172.21.15.129:0/3400761371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab6f730 0x56042af48fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.696+0000 7f6766ffd640 1 -- 172.21.15.129:0/3400761371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab90410 msgr2=0x56042af42e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:25.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.696+0000 7f6766ffd640 1 --2- 172.21.15.129:0/3400761371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab90410 0x56042af42e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.696+0000 7f6766ffd640 1 -- 172.21.15.129:0/3400761371 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56042acd0f70 con 0x56042ab67550 2024-05-20T18:00:25.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.697+0000 7f6766ffd640 1 --2- 172.21.15.129:0/3400761371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 0x56042af48aa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x56042af2a520 tx=0x7f6758009960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c8354a45247fa22 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:25.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.700+0000 7f6746ffd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:00:25.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.700+0000 7f6746ffd640 1 -- 172.21.15.129:0/3400761371 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6758018030 con 0x56042ab67550 2024-05-20T18:00:25.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.710+0000 7f6746ffd640 1 -- 172.21.15.129:0/3400761371 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f6758014880 con 0x56042ab67550 2024-05-20T18:00:25.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.711+0000 7f6775006740 1 -- 172.21.15.129:0/3400761371 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6758014b20 con 0x56042ab67550 2024-05-20T18:00:25.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.712+0000 7f6746ffd640 1 -- 172.21.15.129:0/3400761371 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6758013eb0 con 0x56042ab67550 2024-05-20T18:00:25.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.723+0000 7f6775006740 10 client.25489 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:25.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.723+0000 7f6775006740 1 -- 172.21.15.129:0/3400761371 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56042ab68170 con 0x56042ab67550 2024-05-20T18:00:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.730+0000 7f6746ffd640 1 -- 172.21.15.129:0/3400761371 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6758014b20 con 0x56042ab67550 2024-05-20T18:00:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.730+0000 7f6746ffd640 1 -- 172.21.15.129:0/3400761371 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f675801d430 con 0x56042ab67550 2024-05-20T18:00:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.731+0000 7f6775006740 10 client.25489 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.731+0000 7f6775006740 10 client.25489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.731+0000 7f6775006740 20 client.25489 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.731+0000 7f6775006740 1 --2- 172.21.15.129:0/3400761371 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56042ab4f1f0 0x56042af549d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.731+0000 7f6775006740 4 client.25489 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:00:25.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.731+0000 7f6775006740 1 -- 172.21.15.129:0/3400761371 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56042aea20c0 con 0x56042ab4f1f0 2024-05-20T18:00:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.735+0000 7f67677fe640 1 --2- 172.21.15.129:0/3400761371 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56042ab4f1f0 0x56042af549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.743+0000 7f6746ffd640 10 client.25489 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:25.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.743+0000 7f67677fe640 1 --2- 172.21.15.129:0/3400761371 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56042ab4f1f0 0x56042af549d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=677178ebf7af6a92 server_cookie=f83487d6a0b619e2 in_seq=0 out_seq=0 2024-05-20T18:00:25.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.751+0000 7f6746ffd640 1 -- 172.21.15.129:0/3400761371 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56042aea20c0 con 0x56042ab4f1f0 2024-05-20T18:00:25.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.751+0000 7f6746ffd640 10 client.25489 handle_command_reply: tid=0 2024-05-20T18:00:25.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.751+0000 7f6775006740 1 client.25489 shutdown 2024-05-20T18:00:25.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.752+0000 7f6775006740 1 -- 172.21.15.129:0/3400761371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 msgr2=0x56042af48aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:25.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.752+0000 7f6775006740 1 --2- 172.21.15.129:0/3400761371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 0x56042af48aa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x56042af2a520 tx=0x7f6758009960 comp rx=0 tx=0).stop 2024-05-20T18:00:25.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.763+0000 7f6775006740 1 -- 172.21.15.129:0/3400761371 shutdown_connections 2024-05-20T18:00:25.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.763+0000 7f6775006740 1 --2- 172.21.15.129:0/3400761371 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56042ab4f1f0 0x56042af549d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.763+0000 7f6775006740 1 --2- 172.21.15.129:0/3400761371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab6f730 0x56042af48fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.763+0000 7f6775006740 1 --2- 172.21.15.129:0/3400761371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab90410 0x56042af42e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.763+0000 7f6775006740 1 --2- 172.21.15.129:0/3400761371 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab67550 0x56042af48aa0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.763+0000 7f6746ffd640 0 client.25489 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:25.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.763+0000 7f6775006740 1 -- 172.21.15.129:0/3400761371 >> 172.21.15.129:0/3400761371 conn(0x56042ab84560 msgr2=0x56042af2b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:25.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.765+0000 7f6775006740 1 -- 172.21.15.129:0/3400761371 shutdown_connections 2024-05-20T18:00:25.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.766+0000 7f6775006740 1 -- 172.21.15.129:0/3400761371 wait complete. 2024-05-20T18:00:25.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.766+0000 7f6775006740 20 client.25489 trim_cache size 0 max 16384 2024-05-20T18:00:25.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.855+0000 7f6775006740 1 Processor -- start 2024-05-20T18:00:25.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.857+0000 7f6775006740 1 -- start start 2024-05-20T18:00:25.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.858+0000 7f6775006740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab67550 0x56042af49000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.859+0000 7f6775006740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 0x56042af408a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.859+0000 7f6775006740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab90410 0x56042af38bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.859+0000 7f6775006740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f67681b30c0 con 0x56042ab67550 2024-05-20T18:00:25.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.859+0000 7f6775006740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67681b5940 con 0x56042ab90410 2024-05-20T18:00:25.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.859+0000 7f6775006740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f67681b3730 con 0x56042ab6f730 2024-05-20T18:00:25.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.862+0000 7f67677fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 0x56042af408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.862+0000 7f67677fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 0x56042af408a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:55322/0 (socket says 172.21.15.129:55322) 2024-05-20T18:00:25.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.862+0000 7f67677fe640 1 -- 172.21.15.129:0/1904441554 learned_addr learned my addr 172.21.15.129:0/1904441554 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:25.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.862+0000 7f67677fe640 1 -- 172.21.15.129:0/1904441554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab90410 msgr2=0x56042af38bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:00:25.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.862+0000 7f67677fe640 1 --2- 172.21.15.129:0/1904441554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab90410 0x56042af38bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.862+0000 7f67677fe640 1 -- 172.21.15.129:0/1904441554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab67550 msgr2=0x56042af49000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:25.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.862+0000 7f67677fe640 1 --2- 172.21.15.129:0/1904441554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab67550 0x56042af49000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.862+0000 7f67677fe640 1 -- 172.21.15.129:0/1904441554 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56042af54d40 con 0x56042ab6f730 2024-05-20T18:00:25.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.863+0000 7f67677fe640 1 --2- 172.21.15.129:0/1904441554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 0x56042af408a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f676000d090 tx=0x7f676000d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3deefe3933587f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:25.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.886+0000 7f674d7fa640 1 -- 172.21.15.129:0/1904441554 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6760037040 con 0x56042ab6f730 2024-05-20T18:00:25.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.886+0000 7f674d7fa640 1 -- 172.21.15.129:0/1904441554 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56042aea20c0 con 0x56042ab6f730 2024-05-20T18:00:25.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.886+0000 7f674d7fa640 1 -- 172.21.15.129:0/1904441554 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6760004040 con 0x56042ab6f730 2024-05-20T18:00:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.890+0000 7f6775006740 1 -- 172.21.15.129:0/1904441554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 msgr2=0x56042af408a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.890+0000 7f6775006740 1 --2- 172.21.15.129:0/1904441554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 0x56042af408a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f676000d090 tx=0x7f676000d000 comp rx=0 tx=0).stop 2024-05-20T18:00:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.890+0000 7f6775006740 1 -- 172.21.15.129:0/1904441554 shutdown_connections 2024-05-20T18:00:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.890+0000 7f6775006740 1 --2- 172.21.15.129:0/1904441554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab90410 0x56042af38bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.890+0000 7f6775006740 1 --2- 172.21.15.129:0/1904441554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 0x56042af408a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.890+0000 7f6775006740 1 --2- 172.21.15.129:0/1904441554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab67550 0x56042af49000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.891+0000 7f6775006740 1 -- 172.21.15.129:0/1904441554 >> 172.21.15.129:0/1904441554 conn(0x56042ab84560 msgr2=0x56042af384c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.891+0000 7f6775006740 1 -- 172.21.15.129:0/1904441554 shutdown_connections 2024-05-20T18:00:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.891+0000 7f6775006740 1 -- 172.21.15.129:0/1904441554 wait complete. 2024-05-20T18:00:25.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.894+0000 7f6775006740 1 -- start start 2024-05-20T18:00:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.896+0000 7f6775006740 1 Processor -- start 2024-05-20T18:00:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.897+0000 7f6775006740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab67550 0x56042af4a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.899+0000 7f6775006740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 0x56042af4ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.899+0000 7f6775006740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab90410 0x56042af4b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.900+0000 7f6775006740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f67681b3e00 con 0x56042ab90410 2024-05-20T18:00:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.900+0000 7f6775006740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67681b5270 con 0x56042ab67550 2024-05-20T18:00:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.900+0000 7f6775006740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f67681b44d0 con 0x56042ab6f730 2024-05-20T18:00:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.900+0000 7f6775006740 10 client.0 _ll_register_callbacks cb 0x7f6760035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.901+0000 7f6766ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab90410 0x56042af4b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.901+0000 7f6766ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab90410 0x56042af4b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59134/0 (socket says 172.21.15.129:59134) 2024-05-20T18:00:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.901+0000 7f6766ffd640 1 -- 172.21.15.129:0/1074964026 learned_addr learned my addr 172.21.15.129:0/1074964026 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.901+0000 7f6766ffd640 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab67550 0x56042af4a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.904+0000 7f67677fe640 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 0x56042af4ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.908+0000 7f67677fe640 1 -- 172.21.15.129:0/1074964026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab67550 msgr2=0x56042af4a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:25.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.908+0000 7f67677fe640 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab67550 0x56042af4a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.908+0000 7f67677fe640 1 -- 172.21.15.129:0/1074964026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab90410 msgr2=0x56042af4b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:25.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.908+0000 7f67677fe640 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab90410 0x56042af4b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.908+0000 7f67677fe640 1 -- 172.21.15.129:0/1074964026 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56042afbbb20 con 0x56042ab6f730 2024-05-20T18:00:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.916+0000 7f67677fe640 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 0x56042af4ace0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f67600327f0 tx=0x7f6760003970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66d24f42575b60ca server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.916+0000 7f6746ffd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:00:25.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.920+0000 7f6746ffd640 1 -- 172.21.15.129:0/1074964026 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6760037040 con 0x56042ab6f730 2024-05-20T18:00:25.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.920+0000 7f6746ffd640 1 -- 172.21.15.129:0/1074964026 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f6760007c50 con 0x56042ab6f730 2024-05-20T18:00:25.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.920+0000 7f6775006740 1 -- 172.21.15.129:0/1074964026 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56042aea20c0 con 0x56042ab6f730 2024-05-20T18:00:25.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.924+0000 7f6746ffd640 1 -- 172.21.15.129:0/1074964026 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6760032850 con 0x56042ab6f730 2024-05-20T18:00:25.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.925+0000 7f6775006740 10 client.25495 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:25.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.925+0000 7f6775006740 1 -- 172.21.15.129:0/1074964026 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56042afbcd70 con 0x56042ab6f730 2024-05-20T18:00:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.929+0000 7f6746ffd640 1 -- 172.21.15.129:0/1074964026 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67600329f0 con 0x56042ab6f730 2024-05-20T18:00:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.929+0000 7f6746ffd640 1 -- 172.21.15.129:0/1074964026 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f676004b070 con 0x56042ab6f730 2024-05-20T18:00:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.929+0000 7f6775006740 10 client.25495 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.929+0000 7f6775006740 10 client.25495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.929+0000 7f6775006740 20 client.25495 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.929+0000 7f6775006740 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56042ab24630 0x56042afbf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.930+0000 7f6775006740 4 client.25495 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:00:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.930+0000 7f6775006740 1 -- 172.21.15.129:0/1074964026 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56042aea20c0 con 0x56042ab24630 2024-05-20T18:00:25.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.933+0000 7f6766ffd640 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56042ab24630 0x56042afbf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:25.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.944+0000 7f6766ffd640 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56042ab24630 0x56042afbf4c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d6e4a9dd211c1c server_cookie=39fa6013618a7ee8 in_seq=0 out_seq=0 2024-05-20T18:00:25.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.946+0000 7f6746ffd640 10 client.25495 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:25.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.949+0000 7f6746ffd640 1 -- 172.21.15.129:0/1074964026 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56042aea20c0 con 0x56042ab24630 2024-05-20T18:00:25.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.949+0000 7f6746ffd640 10 client.25495 handle_command_reply: tid=0 2024-05-20T18:00:25.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.949+0000 7f6775006740 1 client.25495 shutdown 2024-05-20T18:00:25.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.955+0000 7f6775006740 1 -- 172.21.15.129:0/1074964026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 msgr2=0x56042af4ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:25.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.956+0000 7f6775006740 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 0x56042af4ace0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f67600327f0 tx=0x7f6760003970 comp rx=0 tx=0).stop 2024-05-20T18:00:25.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.957+0000 7f6775006740 1 -- 172.21.15.129:0/1074964026 shutdown_connections 2024-05-20T18:00:25.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.957+0000 7f6775006740 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56042ab24630 0x56042afbf4c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.958+0000 7f6746ffd640 0 client.25495 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:25.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.958+0000 7f6775006740 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56042ab90410 0x56042af4b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.958+0000 7f6775006740 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56042ab6f730 0x56042af4ace0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.958+0000 7f6775006740 1 --2- 172.21.15.129:0/1074964026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042ab67550 0x56042af4a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.958+0000 7f6775006740 1 -- 172.21.15.129:0/1074964026 >> 172.21.15.129:0/1074964026 conn(0x56042ab84560 msgr2=0x56042af48d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:25.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.964+0000 7f6775006740 1 -- 172.21.15.129:0/1074964026 shutdown_connections 2024-05-20T18:00:25.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.968+0000 7f6775006740 1 -- 172.21.15.129:0/1074964026 wait complete. 2024-05-20T18:00:25.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.968+0000 7f6775006740 20 client.25495 trim_cache size 0 max 16384 2024-05-20T18:00:25.975 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:00:25.975 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:00:25.975 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "53eef3bd-f4db-4574-8993-7bb6ac2cd79b", 2024-05-20T18:00:25.976 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:00:25.976 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:00:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.977+0000 7f676cefc640 1 -- 172.21.15.129:0/1899567783 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f67500a2f00 msgr2=0x7f67500a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.977+0000 7f676cefc640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f67500a2f00 0x7f67500a5410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f675c09aba0 tx=0x7f675c00ef10 comp rx=0 tx=0).stop 2024-05-20T18:00:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.977+0000 7f676cefc640 1 -- 172.21.15.129:0/1899567783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6768113ca0 msgr2=0x7f67681a45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.977+0000 7f676cefc640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6768113ca0 0x7f67681a45f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f675c00d990 tx=0x7f675c00de50 comp rx=0 tx=0).stop 2024-05-20T18:00:25.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.980+0000 7f676cefc640 1 -- 172.21.15.129:0/1899567783 shutdown_connections 2024-05-20T18:00:25.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.981+0000 7f676cefc640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f67500a2f00 0x7f67500a5410 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.981+0000 7f676cefc640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f676810f3a0 0x7f67681a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.981+0000 7f676cefc640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6768113ca0 0x7f67681a45f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.981+0000 7f676cefc640 1 --2- 172.21.15.129:0/1899567783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67681094e0 0x7f676819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:25.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.981+0000 7f676cefc640 1 -- 172.21.15.129:0/1899567783 >> 172.21.15.129:0/1899567783 conn(0x7f676806d580 msgr2=0x7f676810fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:25.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.984+0000 7f676cefc640 1 -- 172.21.15.129:0/1899567783 shutdown_connections 2024-05-20T18:00:25.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:25.985+0000 7f676cefc640 1 -- 172.21.15.129:0/1899567783 wait complete. 2024-05-20T18:00:26.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:00:26.035 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:00:26.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.487+0000 7f2011b9d640 1 -- 172.21.15.129:0/1968891898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200c106a90 msgr2=0x7f200c10e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:26.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.487+0000 7f2011b9d640 1 --2- 172.21.15.129:0/1968891898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200c106a90 0x7f200c10e230 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2004009ca0 tx=0x7f200402a8b0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.488+0000 7f2011b9d640 1 -- 172.21.15.129:0/1968891898 shutdown_connections 2024-05-20T18:00:26.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.488+0000 7f2011b9d640 1 --2- 172.21.15.129:0/1968891898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200c106a90 0x7f200c10e230 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.488+0000 7f2011b9d640 1 --2- 172.21.15.129:0/1968891898 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f200c105fd0 0x7f200c106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.488+0000 7f2011b9d640 1 --2- 172.21.15.129:0/1968891898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200c101a40 0x7f200c101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.488+0000 7f2011b9d640 1 -- 172.21.15.129:0/1968891898 >> 172.21.15.129:0/1968891898 conn(0x7f200c0fcec0 msgr2=0x7f200c0ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:26.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.488+0000 7f2011b9d640 1 -- 172.21.15.129:0/1968891898 shutdown_connections 2024-05-20T18:00:26.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.488+0000 7f2011b9d640 1 -- 172.21.15.129:0/1968891898 wait complete. 2024-05-20T18:00:26.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.489+0000 7f2011b9d640 1 Processor -- start 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.489+0000 7f2011b9d640 1 -- start start 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.489+0000 7f2011b9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200c101a40 0x7f200c10def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.489+0000 7f2011b9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f200c105fd0 0x7f200c10a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.489+0000 7f2011b9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200c106a90 0x7f200c10a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.489+0000 7f2011b9d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1109e0 con 0x7f200c106a90 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.489+0000 7f2011b9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f200c110860 con 0x7f200c101a40 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2010b9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200c101a40 0x7f200c10def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2011b9d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f200c110b60 con 0x7f200c105fd0 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f200bfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f200c105fd0 0x7f200c10a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2010b9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200c101a40 0x7f200c10def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52360/0 (socket says 172.21.15.129:52360) 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2010b9b640 1 -- 172.21.15.129:0/2957097840 learned_addr learned my addr 172.21.15.129:0/2957097840 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f201139c640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200c106a90 0x7f200c10a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2010b9b640 1 -- 172.21.15.129:0/2957097840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f200c105fd0 msgr2=0x7f200c10a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2010b9b640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f200c105fd0 0x7f200c10a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2010b9b640 1 -- 172.21.15.129:0/2957097840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200c106a90 msgr2=0x7f200c10a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2010b9b640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200c106a90 0x7f200c10a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2010b9b640 1 -- 172.21.15.129:0/2957097840 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200c10aba0 con 0x7f200c101a40 2024-05-20T18:00:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2010b9b640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200c101a40 0x7f200c10def0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00e9f0 tx=0x7f1ffc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2009ffb640 1 -- 172.21.15.129:0/2957097840 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc00cd80 con 0x7f200c101a40 2024-05-20T18:00:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2009ffb640 1 -- 172.21.15.129:0/2957097840 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ffc010910 con 0x7f200c101a40 2024-05-20T18:00:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2011b9d640 1 -- 172.21.15.129:0/2957097840 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200c1ac110 con 0x7f200c101a40 2024-05-20T18:00:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.490+0000 7f2009ffb640 1 -- 172.21.15.129:0/2957097840 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc018990 con 0x7f200c101a40 2024-05-20T18:00:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.491+0000 7f2011b9d640 1 -- 172.21.15.129:0/2957097840 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f200c1ac4c0 con 0x7f200c101a40 2024-05-20T18:00:26.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.494+0000 7f2009ffb640 1 -- 172.21.15.129:0/2957097840 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f1ffc018b30 con 0x7f200c101a40 2024-05-20T18:00:26.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f2009ffb640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ff40a2cf0 0x7f1ff40a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f2019d7d740 1 Processor -- start 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f2019d7d740 1 -- start start 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f2009ffb640 1 -- 172.21.15.129:0/2957097840 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f1ffc0cf790 con 0x7f200c101a40 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f200bfff640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ff40a2cf0 0x7f1ff40a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 0x5589074a9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589070e24d0 0x5589074aa470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074aa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f2019d7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55890740fd00 con 0x5589070ea6b0 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f2019d7d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5589073b74c0 con 0x5589070e24d0 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f2019d7d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55890740bcc0 con 0x55890710b390 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f201139c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074aa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f201139c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074aa9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59168/0 (socket says 172.21.15.129:59168) 2024-05-20T18:00:26.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f201139c640 1 -- 172.21.15.129:0/4265973090 learned_addr learned my addr 172.21.15.129:0/4265973090 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:26.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f200bfff640 1 --2- 172.21.15.129:0/4265973090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589070e24d0 0x5589074aa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.495+0000 7f2010b9b640 1 --2- 172.21.15.129:0/4265973090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 0x5589074a9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.496+0000 7f200bfff640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ff40a2cf0 0x7f1ff40a5200 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f200c100590 tx=0x7f200000b390 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:26.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.496+0000 7f2010b9b640 1 -- 172.21.15.129:0/4265973090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589070e24d0 msgr2=0x5589074aa470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:26.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.496+0000 7f2010b9b640 1 --2- 172.21.15.129:0/4265973090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589070e24d0 0x5589074aa470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.496+0000 7f2010b9b640 1 -- 172.21.15.129:0/4265973090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 msgr2=0x5589074aa9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:26.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.496+0000 7f2010b9b640 1 --2- 172.21.15.129:0/4265973090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074aa9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.496+0000 7f2010b9b640 1 -- 172.21.15.129:0/4265973090 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589071ed7d0 con 0x55890710b390 2024-05-20T18:00:26.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.496+0000 7f2010b9b640 1 --2- 172.21.15.129:0/4265973090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 0x5589074a9f30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0cced0 tx=0x7f1ffc09be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c078cb5672fb1ec6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:26.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.496+0000 7f1ff2ffd640 1 -- 172.21.15.129:0/4265973090 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc010ac0 con 0x55890710b390 2024-05-20T18:00:26.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.496+0000 7f1ff2ffd640 1 -- 172.21.15.129:0/4265973090 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ffc010430 con 0x55890710b390 2024-05-20T18:00:26.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.496+0000 7f1ff2ffd640 1 -- 172.21.15.129:0/4265973090 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc09b400 con 0x55890710b390 2024-05-20T18:00:26.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.498+0000 7f2019d7d740 1 -- 172.21.15.129:0/4265973090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 msgr2=0x5589074a9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:26.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.498+0000 7f2019d7d740 1 --2- 172.21.15.129:0/4265973090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 0x5589074a9f30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0cced0 tx=0x7f1ffc09be10 comp rx=0 tx=0).stop 2024-05-20T18:00:26.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.498+0000 7f2019d7d740 1 -- 172.21.15.129:0/4265973090 shutdown_connections 2024-05-20T18:00:26.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.499+0000 7f2019d7d740 1 --2- 172.21.15.129:0/4265973090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589070e24d0 0x5589074aa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.499+0000 7f2019d7d740 1 --2- 172.21.15.129:0/4265973090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074aa9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.499+0000 7f2019d7d740 1 --2- 172.21.15.129:0/4265973090 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 0x5589074a9f30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.499+0000 7f2019d7d740 1 -- 172.21.15.129:0/4265973090 >> 172.21.15.129:0/4265973090 conn(0x5589070ff4e0 msgr2=0x5589074a66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:26.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.499+0000 7f2019d7d740 1 -- 172.21.15.129:0/4265973090 shutdown_connections 2024-05-20T18:00:26.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.500+0000 7f2019d7d740 1 -- 172.21.15.129:0/4265973090 wait complete. 2024-05-20T18:00:26.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.501+0000 7f2019d7d740 1 -- start start 2024-05-20T18:00:26.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.501+0000 7f2019d7d740 1 Processor -- start 2024-05-20T18:00:26.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.501+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5589070e24d0 0x5589074b6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.501+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074b6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.501+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55890710b390 0x5589074b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.501+0000 7f2019d7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5589074b7280 con 0x5589070ea6b0 2024-05-20T18:00:26.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.501+0000 7f2019d7d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5589073cab90 con 0x55890710b390 2024-05-20T18:00:26.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.501+0000 7f2019d7d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5589074b7400 con 0x5589070e24d0 2024-05-20T18:00:26.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.501+0000 7f2019d7d740 10 client.0 _ll_register_callbacks cb 0x558907406da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:26.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f200bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f2010b9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5589070e24d0 0x5589074b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f200bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074b6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59170/0 (socket says 172.21.15.129:59170) 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f200bfff640 1 -- 172.21.15.129:0/654090182 learned_addr learned my addr 172.21.15.129:0/654090182 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f201139c640 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55890710b390 0x5589074b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f2010b9b640 1 -- 172.21.15.129:0/654090182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55890710b390 msgr2=0x5589074b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f2010b9b640 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55890710b390 0x5589074b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f2010b9b640 1 -- 172.21.15.129:0/654090182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 msgr2=0x5589074b6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f2010b9b640 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074b6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f2010b9b640 1 -- 172.21.15.129:0/654090182 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589071233f0 con 0x5589070e24d0 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f2010b9b640 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5589070e24d0 0x5589074b6020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc09cfd0 tx=0x7f1ffc018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dd118351dd5c6cc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f1febfff640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.502+0000 7f1febfff640 1 -- 172.21.15.129:0/654090182 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc0b9020 con 0x5589070e24d0 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f1febfff640 1 -- 172.21.15.129:0/654090182 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f1ffc09b1a0 con 0x5589070e24d0 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f2019d7d740 1 -- 172.21.15.129:0/654090182 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ffc010430 con 0x5589070e24d0 2024-05-20T18:00:26.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f1febfff640 1 -- 172.21.15.129:0/654090182 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1ffc09b3e0 con 0x5589070e24d0 2024-05-20T18:00:26.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f2019d7d740 10 client.25507 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:26.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f2019d7d740 1 -- 172.21.15.129:0/654090182 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5589070e30f0 con 0x5589070e24d0 2024-05-20T18:00:26.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f1febfff640 1 -- 172.21.15.129:0/654090182 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ffc010430 con 0x5589070e24d0 2024-05-20T18:00:26.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f1febfff640 1 -- 172.21.15.129:0/654090182 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f1ffc0a1300 con 0x5589070e24d0 2024-05-20T18:00:26.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f2019d7d740 10 client.25507 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:26.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f2019d7d740 10 client.25507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:26.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f2019d7d740 20 client.25507 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:26.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f2019d7d740 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5589070c9f60 0x5589074c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f2019d7d740 4 client.25507 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:00:26.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.503+0000 7f2019d7d740 1 -- 172.21.15.129:0/654090182 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55890741d430 con 0x5589070c9f60 2024-05-20T18:00:26.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.504+0000 7f200bfff640 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5589070c9f60 0x5589074c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.504+0000 7f1febfff640 10 client.25507 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:26.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.504+0000 7f200bfff640 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5589070c9f60 0x5589074c6bf0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58d60d1bb6967fd1 server_cookie=e83fb1a93a569637 in_seq=0 out_seq=0 2024-05-20T18:00:26.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.505+0000 7f1febfff640 1 -- 172.21.15.129:0/654090182 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55890741d430 con 0x5589070c9f60 2024-05-20T18:00:26.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.505+0000 7f1febfff640 10 client.25507 handle_command_reply: tid=0 2024-05-20T18:00:26.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.505+0000 7f2019d7d740 1 client.25507 shutdown 2024-05-20T18:00:26.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.506+0000 7f2019d7d740 1 -- 172.21.15.129:0/654090182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5589070e24d0 msgr2=0x5589074b6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:26.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.506+0000 7f2019d7d740 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5589070e24d0 0x5589074b6020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc09cfd0 tx=0x7f1ffc018420 comp rx=0 tx=0).stop 2024-05-20T18:00:26.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.507+0000 7f2019d7d740 1 -- 172.21.15.129:0/654090182 shutdown_connections 2024-05-20T18:00:26.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.507+0000 7f2019d7d740 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5589070c9f60 0x5589074c6bf0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.507+0000 7f2019d7d740 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55890710b390 0x5589074b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.507+0000 7f2019d7d740 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074b6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.507+0000 7f2019d7d740 1 --2- 172.21.15.129:0/654090182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5589070e24d0 0x5589074b6020 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc09cfd0 tx=0x7f1ffc018420 comp rx=0 tx=0).stop 2024-05-20T18:00:26.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.507+0000 7f2019d7d740 1 -- 172.21.15.129:0/654090182 >> 172.21.15.129:0/654090182 conn(0x5589070ff4e0 msgr2=0x5589074abc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:26.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.507+0000 7f1febfff640 0 client.25507 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.510+0000 7f2019d7d740 1 -- 172.21.15.129:0/654090182 shutdown_connections 2024-05-20T18:00:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.511+0000 7f2019d7d740 1 -- 172.21.15.129:0/654090182 wait complete. 2024-05-20T18:00:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.511+0000 7f2019d7d740 20 client.25507 trim_cache size 0 max 16384 2024-05-20T18:00:26.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.551+0000 7f2019d7d740 1 Processor -- start 2024-05-20T18:00:26.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.551+0000 7f2019d7d740 1 -- start start 2024-05-20T18:00:26.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.551+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.551+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55890710b390 0x5589074a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.551+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890709f3a0 0x5589074adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.551+0000 7f2019d7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1b2fb0 con 0x5589070ea6b0 2024-05-20T18:00:26.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.551+0000 7f2019d7d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1b5830 con 0x55890710b390 2024-05-20T18:00:26.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.551+0000 7f2019d7d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1b3620 con 0x55890709f3a0 2024-05-20T18:00:26.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.553+0000 7f201139c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890709f3a0 0x5589074adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.554+0000 7f201139c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890709f3a0 0x5589074adbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:55382/0 (socket says 172.21.15.129:55382) 2024-05-20T18:00:26.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.554+0000 7f201139c640 1 -- 172.21.15.129:0/2488213513 learned_addr learned my addr 172.21.15.129:0/2488213513 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:26.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.558+0000 7f201139c640 1 -- 172.21.15.129:0/2488213513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55890710b390 msgr2=0x5589074a67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:26.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.558+0000 7f201139c640 1 --2- 172.21.15.129:0/2488213513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55890710b390 0x5589074a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.558+0000 7f201139c640 1 -- 172.21.15.129:0/2488213513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 msgr2=0x5589074b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:26.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.558+0000 7f201139c640 1 --2- 172.21.15.129:0/2488213513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.558+0000 7f201139c640 1 -- 172.21.15.129:0/2488213513 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589074a1ea0 con 0x55890709f3a0 2024-05-20T18:00:26.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.562+0000 7f201139c640 1 --2- 172.21.15.129:0/2488213513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890709f3a0 0x5589074adbe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2004030510 tx=0x7f2004005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aca889562715f71 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:26.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.573+0000 7f1ff27fc640 1 -- 172.21.15.129:0/2488213513 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2004030930 con 0x55890709f3a0 2024-05-20T18:00:26.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.573+0000 7f1ff27fc640 1 -- 172.21.15.129:0/2488213513 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f200404b070 con 0x55890709f3a0 2024-05-20T18:00:26.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.576+0000 7f2019d7d740 1 -- 172.21.15.129:0/2488213513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890709f3a0 msgr2=0x5589074adbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:26.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.576+0000 7f2019d7d740 1 --2- 172.21.15.129:0/2488213513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890709f3a0 0x5589074adbe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2004030510 tx=0x7f2004005fd0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.577+0000 7f2019d7d740 1 -- 172.21.15.129:0/2488213513 shutdown_connections 2024-05-20T18:00:26.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.577+0000 7f2019d7d740 1 --2- 172.21.15.129:0/2488213513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55890710b390 0x5589074a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.577+0000 7f2019d7d740 1 --2- 172.21.15.129:0/2488213513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890709f3a0 0x5589074adbe0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.577+0000 7f2019d7d740 1 --2- 172.21.15.129:0/2488213513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5589070ea6b0 0x5589074b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.577+0000 7f2019d7d740 1 -- 172.21.15.129:0/2488213513 >> 172.21.15.129:0/2488213513 conn(0x5589070ff4e0 msgr2=0x5589074a60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:26.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.593+0000 7f2019d7d740 1 -- 172.21.15.129:0/2488213513 shutdown_connections 2024-05-20T18:00:26.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.601+0000 7f2019d7d740 1 -- 172.21.15.129:0/2488213513 wait complete. 2024-05-20T18:00:26.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.602+0000 7f2019d7d740 1 -- start start 2024-05-20T18:00:26.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.603+0000 7f2019d7d740 1 Processor -- start 2024-05-20T18:00:26.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.608+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55890709f3a0 0x5589074c6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.608+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589070ea6b0 0x5589074c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.608+0000 7f2019d7d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 0x5589074c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.608+0000 7f2019d7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1b3cf0 con 0x55890709f3a0 2024-05-20T18:00:26.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.608+0000 7f2019d7d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1b5160 con 0x5589070ea6b0 2024-05-20T18:00:26.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.608+0000 7f2019d7d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1b43c0 con 0x55890710b390 2024-05-20T18:00:26.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.609+0000 7f2019d7d740 10 client.0 _ll_register_callbacks cb 0x7f2004009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:26.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.614+0000 7f2010b9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55890709f3a0 0x5589074c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.614+0000 7f2010b9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55890709f3a0 0x5589074c6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59188/0 (socket says 172.21.15.129:59188) 2024-05-20T18:00:26.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.614+0000 7f2010b9b640 1 -- 172.21.15.129:0/904369894 learned_addr learned my addr 172.21.15.129:0/904369894 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:26.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.615+0000 7f201139c640 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 0x5589074c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.617+0000 7f200bfff640 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589070ea6b0 0x5589074c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.617+0000 7f201139c640 1 -- 172.21.15.129:0/904369894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589070ea6b0 msgr2=0x5589074c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.618+0000 7f201139c640 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589070ea6b0 0x5589074c6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.618+0000 7f201139c640 1 -- 172.21.15.129:0/904369894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55890709f3a0 msgr2=0x5589074c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.618+0000 7f201139c640 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55890709f3a0 0x5589074c6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.618+0000 7f201139c640 1 -- 172.21.15.129:0/904369894 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558907528cf0 con 0x55890710b390 2024-05-20T18:00:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.618+0000 7f201139c640 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 0x5589074c70f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2004030540 tx=0x7f2004030d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e8e2ffe17db7d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.619+0000 7f1febfff640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:00:26.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.620+0000 7f1febfff640 1 -- 172.21.15.129:0/904369894 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2004003d80 con 0x55890710b390 2024-05-20T18:00:26.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.620+0000 7f2019d7d740 1 -- 172.21.15.129:0/904369894 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f200404b070 con 0x55890710b390 2024-05-20T18:00:26.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.620+0000 7f1febfff640 1 -- 172.21.15.129:0/904369894 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f2004003f20 con 0x55890710b390 2024-05-20T18:00:26.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.622+0000 7f1febfff640 1 -- 172.21.15.129:0/904369894 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2004031440 con 0x55890710b390 2024-05-20T18:00:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.623+0000 7f2019d7d740 10 client.25513 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.623+0000 7f2019d7d740 1 -- 172.21.15.129:0/904369894 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558907529f60 con 0x55890710b390 2024-05-20T18:00:26.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.628+0000 7f1febfff640 1 -- 172.21.15.129:0/904369894 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f200404b070 con 0x55890710b390 2024-05-20T18:00:26.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.628+0000 7f1febfff640 1 -- 172.21.15.129:0/904369894 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f2004058740 con 0x55890710b390 2024-05-20T18:00:26.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.629+0000 7f2019d7d740 10 client.25513 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:26.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.629+0000 7f2019d7d740 10 client.25513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:26.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.629+0000 7f2019d7d740 20 client.25513 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:26.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.629+0000 7f2019d7d740 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55890752a660 0x558907541d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:26.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.629+0000 7f2019d7d740 4 client.25513 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:00:26.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.629+0000 7f2019d7d740 1 -- 172.21.15.129:0/904369894 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558907417780 con 0x55890752a660 2024-05-20T18:00:26.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.634+0000 7f2010b9b640 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55890752a660 0x558907541d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:26.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.647+0000 7f2010b9b640 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55890752a660 0x558907541d30 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f18fe14e73feb46 server_cookie=8c9964d9a2f4cf22 in_seq=0 out_seq=0 2024-05-20T18:00:26.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.647+0000 7f1febfff640 10 client.25513 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:26.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.654+0000 7f1febfff640 1 -- 172.21.15.129:0/904369894 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558907417780 con 0x55890752a660 2024-05-20T18:00:26.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.654+0000 7f1febfff640 10 client.25513 handle_command_reply: tid=0 2024-05-20T18:00:26.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.654+0000 7f2019d7d740 1 client.25513 shutdown 2024-05-20T18:00:26.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.656+0000 7f2019d7d740 1 -- 172.21.15.129:0/904369894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 msgr2=0x5589074c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.656+0000 7f2019d7d740 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 0x5589074c70f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2004030540 tx=0x7f2004030d30 comp rx=0 tx=0).stop 2024-05-20T18:00:26.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.657+0000 7f2019d7d740 1 -- 172.21.15.129:0/904369894 shutdown_connections 2024-05-20T18:00:26.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.657+0000 7f2019d7d740 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55890752a660 0x558907541d30 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.657+0000 7f2019d7d740 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589070ea6b0 0x5589074c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.657+0000 7f2019d7d740 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55890710b390 0x5589074c70f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.657+0000 7f2019d7d740 1 --2- 172.21.15.129:0/904369894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55890709f3a0 0x5589074c6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.657+0000 7f2019d7d740 1 -- 172.21.15.129:0/904369894 >> 172.21.15.129:0/904369894 conn(0x5589070ff4e0 msgr2=0x5589074b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:26.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.658+0000 7f1febfff640 0 client.25513 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:26.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.658+0000 7f2019d7d740 1 -- 172.21.15.129:0/904369894 shutdown_connections 2024-05-20T18:00:26.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.658+0000 7f2019d7d740 1 -- 172.21.15.129:0/904369894 wait complete. 2024-05-20T18:00:26.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.658+0000 7f2019d7d740 20 client.25513 trim_cache size 0 max 16384 2024-05-20T18:00:26.658 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:00:26.658 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:00:26.658 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:00:26.658 INFO:teuthology.orchestra.run.smithi129.stdout: "53eef3bd-f4db-4574-8993-7bb6ac2cd79b": { 2024-05-20T18:00:26.658 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:00:26.658 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "53eef3bd-f4db-4574-8993-7bb6ac2cd79b", 2024-05-20T18:00:26.658 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:00:26.658 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:00:26.658 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:00:26.658 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:00:26.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.661+0000 7f2011b9d640 1 -- 172.21.15.129:0/2957097840 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ff40a2cf0 msgr2=0x7f1ff40a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:26.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.661+0000 7f2011b9d640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ff40a2cf0 0x7f1ff40a5200 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f200c100590 tx=0x7f200000b390 comp rx=0 tx=0).stop 2024-05-20T18:00:26.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.661+0000 7f2011b9d640 1 -- 172.21.15.129:0/2957097840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200c101a40 msgr2=0x7f200c10def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:26.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.661+0000 7f2011b9d640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200c101a40 0x7f200c10def0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00e9f0 tx=0x7f1ffc00eeb0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.661+0000 7f2011b9d640 1 -- 172.21.15.129:0/2957097840 shutdown_connections 2024-05-20T18:00:26.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.661+0000 7f2011b9d640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ff40a2cf0 0x7f1ff40a5200 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.661+0000 7f2011b9d640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200c106a90 0x7f200c10a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.661+0000 7f2011b9d640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f200c105fd0 0x7f200c10a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.662+0000 7f2011b9d640 1 --2- 172.21.15.129:0/2957097840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200c101a40 0x7f200c10def0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:26.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.662+0000 7f2011b9d640 1 -- 172.21.15.129:0/2957097840 >> 172.21.15.129:0/2957097840 conn(0x7f200c0fcec0 msgr2=0x7f200c104130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:26.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.662+0000 7f2011b9d640 1 -- 172.21.15.129:0/2957097840 shutdown_connections 2024-05-20T18:00:26.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:26.662+0000 7f2011b9d640 1 -- 172.21.15.129:0/2957097840 wait complete. 2024-05-20T18:00:26.704 INFO:tasks.cephfs.filesystem:scrub status for tag:53eef3bd-f4db-4574-8993-7bb6ac2cd79b - {'path': '/', 'tag': '53eef3bd-f4db-4574-8993-7bb6ac2cd79b', 'options': 'recursive,force'} 2024-05-20T18:00:27.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:27 smithi154 ceph-mon[36482]: pgmap v60: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 430 MiB/s wr, 341 op/s 2024-05-20T18:00:27.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:27 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:00:27.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:27 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:00:27.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:27 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:00:27.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:27 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:00:27.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:27 smithi129 ceph-mon[32029]: pgmap v60: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 430 MiB/s wr, 341 op/s 2024-05-20T18:00:27.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:27 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:00:27.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:27 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:00:27.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:27 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:00:27.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:27 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:00:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:27 smithi204 ceph-mon[37076]: pgmap v60: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 430 MiB/s wr, 341 op/s 2024-05-20T18:00:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:27 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:00:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:27 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:00:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:27 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:00:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:27 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:00:29.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:29 smithi154 ceph-mon[36482]: pgmap v61: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 430 MiB/s wr, 339 op/s 2024-05-20T18:00:29.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:29 smithi129 ceph-mon[32029]: pgmap v61: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 430 MiB/s wr, 339 op/s 2024-05-20T18:00:29.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:29 smithi204 ceph-mon[37076]: pgmap v61: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 430 MiB/s wr, 339 op/s 2024-05-20T18:00:31.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:31 smithi154 ceph-mon[36482]: pgmap v62: 97 pgs: 97 active+clean; 7.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 648 MiB/s wr, 461 op/s 2024-05-20T18:00:31.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:31 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:00:31.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:31 smithi129 ceph-mon[32029]: pgmap v62: 97 pgs: 97 active+clean; 7.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 648 MiB/s wr, 461 op/s 2024-05-20T18:00:31.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:31 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:00:31.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:31 smithi204 ceph-mon[37076]: pgmap v62: 97 pgs: 97 active+clean; 7.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 648 MiB/s wr, 461 op/s 2024-05-20T18:00:31.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:31 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:00:33.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:33 smithi154 ceph-mon[36482]: pgmap v63: 97 pgs: 97 active+clean; 9.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 MiB/s wr, 500 op/s 2024-05-20T18:00:33.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:33 smithi129 ceph-mon[32029]: pgmap v63: 97 pgs: 97 active+clean; 9.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 MiB/s wr, 500 op/s 2024-05-20T18:00:33.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:33 smithi204 ceph-mon[37076]: pgmap v63: 97 pgs: 97 active+clean; 9.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 MiB/s wr, 500 op/s 2024-05-20T18:00:35.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:35 smithi154 ceph-mon[36482]: pgmap v64: 97 pgs: 97 active+clean; 9.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 MiB/s wr, 500 op/s 2024-05-20T18:00:35.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:35 smithi129 ceph-mon[32029]: pgmap v64: 97 pgs: 97 active+clean; 9.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 MiB/s wr, 500 op/s 2024-05-20T18:00:35.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:35 smithi204 ceph-mon[37076]: pgmap v64: 97 pgs: 97 active+clean; 9.2 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 MiB/s wr, 500 op/s 2024-05-20T18:00:37.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:37 smithi154 ceph-mon[36482]: pgmap v65: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 GiB/s wr, 1.80k op/s 2024-05-20T18:00:37.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:37 smithi129 ceph-mon[32029]: pgmap v65: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 GiB/s wr, 1.80k op/s 2024-05-20T18:00:37.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:37 smithi204 ceph-mon[37076]: pgmap v65: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 GiB/s wr, 1.80k op/s 2024-05-20T18:00:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.214+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1668355634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a4109200 msgr2=0x7f07a4109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.214+0000 7f07a8dc2640 1 --2- 172.21.15.129:0/1668355634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a4109200 0x7f07a4109610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0798009860 tx=0x7f079802a380 comp rx=0 tx=0).stop 2024-05-20T18:00:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.219+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1668355634 shutdown_connections 2024-05-20T18:00:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.219+0000 7f07a8dc2640 1 --2- 172.21.15.129:0/1668355634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410e250 0x7f07a41159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.219+0000 7f07a8dc2640 1 --2- 172.21.15.129:0/1668355634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a410d790 0x7f07a410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.219+0000 7f07a8dc2640 1 --2- 172.21.15.129:0/1668355634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a4109200 0x7f07a4109610 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.219+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1668355634 >> 172.21.15.129:0/1668355634 conn(0x7f07a4075d50 msgr2=0x7f07a4076190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.222+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1668355634 shutdown_connections 2024-05-20T18:00:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.222+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1668355634 wait complete. 2024-05-20T18:00:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.224+0000 7f07a8dc2640 1 Processor -- start 2024-05-20T18:00:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.224+0000 7f07a8dc2640 1 -- start start 2024-05-20T18:00:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.224+0000 7f07a8dc2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a4109200 0x7f07a4083540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.225+0000 7f07a8dc2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410d790 0x7f07a407b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.225+0000 7f07a8dc2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a410e250 0x7f07a407bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.225+0000 7f07a1d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410d790 0x7f07a407b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.225+0000 7f07a1d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410d790 0x7f07a407b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:58766/0 (socket says 172.21.15.129:58766) 2024-05-20T18:00:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.225+0000 7f07a8dc2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07a4119370 con 0x7f07a4109200 2024-05-20T18:00:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.225+0000 7f07a8dc2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07a41191f0 con 0x7f07a410d790 2024-05-20T18:00:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.225+0000 7f07a8dc2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f07a41194f0 con 0x7f07a410e250 2024-05-20T18:00:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.226+0000 7f07a1d9b640 1 -- 172.21.15.129:0/1259375295 learned_addr learned my addr 172.21.15.129:0/1259375295 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.228+0000 7f07a2d9d640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a410e250 0x7f07a407bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.228+0000 7f07a1d9b640 1 -- 172.21.15.129:0/1259375295 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a410e250 msgr2=0x7f07a407bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.228+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a410e250 0x7f07a407bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.228+0000 7f07a1d9b640 1 -- 172.21.15.129:0/1259375295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a4109200 msgr2=0x7f07a4083540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.228+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a4109200 0x7f07a4083540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.228+0000 7f07a1d9b640 1 -- 172.21.15.129:0/1259375295 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a407c210 con 0x7f07a410d790 2024-05-20T18:00:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.229+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410d790 0x7f07a407b5a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f079400b9c0 tx=0x7f079400be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.234+0000 7f07937fe640 1 -- 172.21.15.129:0/1259375295 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794019070 con 0x7f07a410d790 2024-05-20T18:00:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.234+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1259375295 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a419fa60 con 0x7f07a410d790 2024-05-20T18:00:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.234+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1259375295 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a419ffb0 con 0x7f07a410d790 2024-05-20T18:00:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.236+0000 7f07937fe640 1 -- 172.21.15.129:0/1259375295 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f079400cba0 con 0x7f07a410d790 2024-05-20T18:00:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.236+0000 7f07937fe640 1 -- 172.21.15.129:0/1259375295 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940126b0 con 0x7f07a410d790 2024-05-20T18:00:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.245+0000 7f07afc5f740 1 Processor -- start 2024-05-20T18:00:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.245+0000 7f07afc5f740 1 -- start start 2024-05-20T18:00:38.246 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.248+0000 7f07afc5f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c561b0 0x55db36ff5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.249+0000 7f07a2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c561b0 0x55db36ff5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.249+0000 7f07a2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c561b0 0x55db36ff5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36762/0 (socket says 172.21.15.129:36762) 2024-05-20T18:00:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.249+0000 7f07a2d9d640 1 -- 172.21.15.129:0/2563110932 learned_addr learned my addr 172.21.15.129:0/2563110932 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.250+0000 7f07afc5f740 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c2d2f0 0x55db36ff5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.250+0000 7f07afc5f740 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 0x55db36ff5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.251+0000 7f07afc5f740 1 -- 172.21.15.129:0/2563110932 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55db36f5a340 con 0x55db36c561b0 2024-05-20T18:00:38.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.251+0000 7f07afc5f740 1 -- 172.21.15.129:0/2563110932 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55db36f02290 con 0x55db36c354d0 2024-05-20T18:00:38.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.251+0000 7f07afc5f740 1 -- 172.21.15.129:0/2563110932 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55db36f56300 con 0x55db36c2d2f0 2024-05-20T18:00:38.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.251+0000 7f07a259c640 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c2d2f0 0x55db36ff5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.251 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.252+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 0x55db36ff5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.251 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.253+0000 7f07937fe640 1 -- 172.21.15.129:0/1259375295 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f079401c070 con 0x7f07a410d790 2024-05-20T18:00:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.254+0000 7f07a1d9b640 1 -- 172.21.15.129:0/2563110932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c2d2f0 msgr2=0x55db36ff5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.254+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c2d2f0 0x55db36ff5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.254+0000 7f07a1d9b640 1 -- 172.21.15.129:0/2563110932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c561b0 msgr2=0x55db36ff5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.254+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c561b0 0x55db36ff5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.254+0000 7f07a1d9b640 1 -- 172.21.15.129:0/2563110932 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db36f268e0 con 0x55db36c354d0 2024-05-20T18:00:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.255+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 0x55db36ff5b80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f07940cbfb0 tx=0x7f07940d4a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67e8d1dfacb3bbef server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:38.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.258+0000 7f07937fe640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f078c0a2f00 0x7f078c0a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.259+0000 7f07937fe640 1 -- 172.21.15.129:0/1259375295 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f07940ce7c0 con 0x7f07a410d790 2024-05-20T18:00:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.262+0000 7f0790ff9640 1 -- 172.21.15.129:0/2563110932 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940dd070 con 0x55db36c354d0 2024-05-20T18:00:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.263+0000 7f07a2d9d640 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c561b0 0x55db36ff5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:00:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.263+0000 7f07a259c640 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c2d2f0 0x55db36ff5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:00:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.263+0000 7f0790ff9640 1 -- 172.21.15.129:0/2563110932 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07940d6e10 con 0x55db36c354d0 2024-05-20T18:00:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.263+0000 7f07a2d9d640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f078c0a2f00 0x7f078c0a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.263 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.265+0000 7f07afc5f740 1 -- 172.21.15.129:0/2563110932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 msgr2=0x55db36ff5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:38.263 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.265+0000 7f07afc5f740 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 0x55db36ff5b80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f07940cbfb0 tx=0x7f07940d4a60 comp rx=0 tx=0).stop 2024-05-20T18:00:38.264 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.266+0000 7f07afc5f740 1 -- 172.21.15.129:0/2563110932 shutdown_connections 2024-05-20T18:00:38.264 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.266+0000 7f07afc5f740 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 0x55db36ff5b80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.264 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.266+0000 7f07afc5f740 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c2d2f0 0x55db36ff5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.264 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.266+0000 7f07afc5f740 1 --2- 172.21.15.129:0/2563110932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c561b0 0x55db36ff5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.264 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.266+0000 7f07afc5f740 1 -- 172.21.15.129:0/2563110932 >> 172.21.15.129:0/2563110932 conn(0x55db36c4a300 msgr2=0x55db36ff1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:38.266 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.268+0000 7f07a2d9d640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f078c0a2f00 0x7f078c0a5410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55db36ff2410 tx=0x7f079c009260 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:38.267 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.269+0000 7f07afc5f740 1 -- 172.21.15.129:0/2563110932 shutdown_connections 2024-05-20T18:00:38.268 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.269+0000 7f07afc5f740 1 -- 172.21.15.129:0/2563110932 wait complete. 2024-05-20T18:00:38.268 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.270+0000 7f07afc5f740 1 -- start start 2024-05-20T18:00:38.268 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.270+0000 7f07afc5f740 1 Processor -- start 2024-05-20T18:00:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.271+0000 7f07afc5f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 0x55db37010930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.271+0000 7f07afc5f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 0x55db37010e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.271+0000 7f07afc5f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c561b0 0x55db370169e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.271+0000 7f07afc5f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55db37013410 con 0x55db36c2d2f0 2024-05-20T18:00:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.271+0000 7f07afc5f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55db36f15790 con 0x55db36c354d0 2024-05-20T18:00:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.271+0000 7f07afc5f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55db37013590 con 0x55db36c561b0 2024-05-20T18:00:38.270 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.271+0000 7f07afc5f740 10 client.0 _ll_register_callbacks cb 0x55db36f50a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:38.270 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.272+0000 7f07a2d9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c561b0 0x55db370169e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.274+0000 7f07a1d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 0x55db37010e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.274+0000 7f07a1d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 0x55db37010e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:58790/0 (socket says 172.21.15.129:58790) 2024-05-20T18:00:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.274+0000 7f07a1d9b640 1 -- 172.21.15.129:0/657633907 learned_addr learned my addr 172.21.15.129:0/657633907 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.274+0000 7f07a259c640 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 0x55db37010930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.278+0000 7f07a259c640 1 -- 172.21.15.129:0/657633907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c561b0 msgr2=0x55db370169e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.278+0000 7f07a259c640 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c561b0 0x55db370169e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.278+0000 7f07a259c640 1 -- 172.21.15.129:0/657633907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 msgr2=0x55db37010e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.278+0000 7f07a259c640 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 0x55db37010e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.278+0000 7f07a259c640 1 -- 172.21.15.129:0/657633907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db36c6e210 con 0x55db36c2d2f0 2024-05-20T18:00:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.279+0000 7f07a2d9d640 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c561b0 0x55db370169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:00:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.279+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 0x55db37010e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:00:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.295+0000 7f07a259c640 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 0x55db37010930 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x55db36ff2d60 tx=0x7f0798004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e57d8ad3946d206 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.295+0000 7f0789ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:00:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.296+0000 7f0789ffb640 1 -- 172.21.15.129:0/657633907 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798040020 con 0x55db36c2d2f0 2024-05-20T18:00:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.296+0000 7f0789ffb640 1 -- 172.21.15.129:0/657633907 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f07980307d0 con 0x55db36c2d2f0 2024-05-20T18:00:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.296+0000 7f07afc5f740 1 -- 172.21.15.129:0/657633907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07940d6e10 con 0x55db36c2d2f0 2024-05-20T18:00:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.297+0000 7f0789ffb640 1 -- 172.21.15.129:0/657633907 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0798037070 con 0x55db36c2d2f0 2024-05-20T18:00:38.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.305+0000 7f07afc5f740 10 client.15822 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:38.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.305+0000 7f07afc5f740 1 -- 172.21.15.129:0/657633907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55db36d3a620 con 0x55db36c2d2f0 2024-05-20T18:00:38.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.308+0000 7f0789ffb640 1 -- 172.21.15.129:0/657633907 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07940d6e10 con 0x55db36c2d2f0 2024-05-20T18:00:38.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.308+0000 7f0789ffb640 1 -- 172.21.15.129:0/657633907 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f0798004a60 con 0x55db36c2d2f0 2024-05-20T18:00:38.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.315+0000 7f07afc5f740 10 client.15822 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:38.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.315+0000 7f07afc5f740 10 client.15822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:38.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.315+0000 7f07afc5f740 20 client.15822 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:38.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.315+0000 7f07afc5f740 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55db36c14d80 0x55db37022ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.315+0000 7f07afc5f740 4 client.15822 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:00:38.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.315+0000 7f07afc5f740 1 -- 172.21.15.129:0/657633907 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db36f67a70 con 0x55db36c14d80 2024-05-20T18:00:38.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.317+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55db36c14d80 0x55db37022ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.333+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55db36c14d80 0x55db37022ae0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d86f79229b3919f8 server_cookie=6202c82341396fe5 in_seq=0 out_seq=0 2024-05-20T18:00:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.334+0000 7f0789ffb640 10 client.15822 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.337+0000 7f0789ffb640 1 -- 172.21.15.129:0/657633907 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55db36f67a70 con 0x55db36c14d80 2024-05-20T18:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.337+0000 7f0789ffb640 10 client.15822 handle_command_reply: tid=0 2024-05-20T18:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.337+0000 7f07afc5f740 1 client.15822 shutdown 2024-05-20T18:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.337+0000 7f07afc5f740 1 -- 172.21.15.129:0/657633907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 msgr2=0x55db37010930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.337+0000 7f07afc5f740 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 0x55db37010930 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x55db36ff2d60 tx=0x7f0798004270 comp rx=0 tx=0).stop 2024-05-20T18:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.337+0000 7f07afc5f740 1 -- 172.21.15.129:0/657633907 shutdown_connections 2024-05-20T18:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.337+0000 7f07afc5f740 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55db36c14d80 0x55db37022ae0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.338+0000 7f07afc5f740 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c354d0 0x55db37010e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.338+0000 7f07afc5f740 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c561b0 0x55db370169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.338+0000 7f07afc5f740 1 --2- 172.21.15.129:0/657633907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 0x55db37010930 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.338+0000 7f07afc5f740 1 -- 172.21.15.129:0/657633907 >> 172.21.15.129:0/657633907 conn(0x55db36c4a300 msgr2=0x55db36ff6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.341+0000 7f0789ffb640 0 client.15822 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:38.345 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.346+0000 7f07afc5f740 1 -- 172.21.15.129:0/657633907 shutdown_connections 2024-05-20T18:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.348+0000 7f07afc5f740 1 -- 172.21.15.129:0/657633907 wait complete. 2024-05-20T18:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.349+0000 7f07afc5f740 20 client.15822 trim_cache size 0 max 16384 2024-05-20T18:00:38.386 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.388+0000 7f07afc5f740 1 Processor -- start 2024-05-20T18:00:38.387 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.388+0000 7f07afc5f740 1 -- start start 2024-05-20T18:00:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.389+0000 7f07afc5f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c2d2f0 0x55db3700fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.390+0000 7f07afc5f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c354d0 0x55db36ff5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.390+0000 7f07afc5f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c561b0 0x55db36ff5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.390+0000 7f07a1d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c354d0 0x55db36ff5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.390+0000 7f07a1d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c354d0 0x55db36ff5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36772/0 (socket says 172.21.15.129:36772) 2024-05-20T18:00:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.390+0000 7f07a1d9b640 1 -- 172.21.15.129:0/227254107 learned_addr learned my addr 172.21.15.129:0/227254107 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.390+0000 7f07afc5f740 1 -- 172.21.15.129:0/227254107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f078400c8d0 con 0x55db36c354d0 2024-05-20T18:00:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.390+0000 7f07afc5f740 1 -- 172.21.15.129:0/227254107 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f078400f150 con 0x55db36c561b0 2024-05-20T18:00:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.390+0000 7f07afc5f740 1 -- 172.21.15.129:0/227254107 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f078400cf40 con 0x55db36c2d2f0 2024-05-20T18:00:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.391+0000 7f07a1d9b640 1 -- 172.21.15.129:0/227254107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c2d2f0 msgr2=0x55db3700fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.391+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/227254107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c2d2f0 0x55db3700fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.391+0000 7f07a1d9b640 1 -- 172.21.15.129:0/227254107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c561b0 msgr2=0x55db36ff5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.391+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/227254107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c561b0 0x55db36ff5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.391+0000 7f07a1d9b640 1 -- 172.21.15.129:0/227254107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db36ff2610 con 0x55db36c354d0 2024-05-20T18:00:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.396+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/227254107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c354d0 0x55db36ff5710 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f07940976e0 tx=0x7f07940d08c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2220466be15e03e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.399+0000 7f078bfff640 1 -- 172.21.15.129:0/227254107 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940dd040 con 0x55db36c354d0 2024-05-20T18:00:38.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.399+0000 7f078bfff640 1 -- 172.21.15.129:0/227254107 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f079409b980 con 0x55db36c354d0 2024-05-20T18:00:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.400+0000 7f078bfff640 1 -- 172.21.15.129:0/227254107 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940d4380 con 0x55db36c354d0 2024-05-20T18:00:38.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.401+0000 7f07afc5f740 1 -- 172.21.15.129:0/227254107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c354d0 msgr2=0x55db36ff5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.401+0000 7f07afc5f740 1 --2- 172.21.15.129:0/227254107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c354d0 0x55db36ff5710 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f07940976e0 tx=0x7f07940d08c0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.401+0000 7f07afc5f740 1 -- 172.21.15.129:0/227254107 shutdown_connections 2024-05-20T18:00:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.401+0000 7f07afc5f740 1 --2- 172.21.15.129:0/227254107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c561b0 0x55db36ff5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.401+0000 7f07afc5f740 1 --2- 172.21.15.129:0/227254107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c354d0 0x55db36ff5710 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.401+0000 7f07afc5f740 1 --2- 172.21.15.129:0/227254107 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c2d2f0 0x55db3700fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.400 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.401+0000 7f07afc5f740 1 -- 172.21.15.129:0/227254107 >> 172.21.15.129:0/227254107 conn(0x55db36c4a300 msgr2=0x55db36ff5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.402+0000 7f07afc5f740 1 -- 172.21.15.129:0/227254107 shutdown_connections 2024-05-20T18:00:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.402+0000 7f07afc5f740 1 -- 172.21.15.129:0/227254107 wait complete. 2024-05-20T18:00:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.403+0000 7f07afc5f740 1 -- start start 2024-05-20T18:00:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.403+0000 7f07afc5f740 1 Processor -- start 2024-05-20T18:00:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.404+0000 7f07afc5f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 0x55db3701ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.404+0000 7f07afc5f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c354d0 0x55db3701d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.404+0000 7f07afc5f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c561b0 0x55db37015ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.404+0000 7f07afc5f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f078400d610 con 0x55db36c2d2f0 2024-05-20T18:00:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.404+0000 7f07afc5f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f078400ea80 con 0x55db36c561b0 2024-05-20T18:00:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.404+0000 7f07afc5f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f078400dce0 con 0x55db36c354d0 2024-05-20T18:00:38.404 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.404+0000 7f07afc5f740 10 client.0 _ll_register_callbacks cb 0x7f0798035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:38.405 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.405+0000 7f07a259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 0x55db3701ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.405 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.405+0000 7f07a259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 0x55db3701ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36782/0 (socket says 172.21.15.129:36782) 2024-05-20T18:00:38.405 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.405+0000 7f07a259c640 1 -- 172.21.15.129:0/4116748419 learned_addr learned my addr 172.21.15.129:0/4116748419 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:38.405 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.405+0000 7f07a2d9d640 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c561b0 0x55db37015ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.405 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.405+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c354d0 0x55db3701d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.406+0000 7f07a259c640 1 -- 172.21.15.129:0/4116748419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c354d0 msgr2=0x55db3701d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:38.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.406+0000 7f07a259c640 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c354d0 0x55db3701d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.406+0000 7f07a259c640 1 -- 172.21.15.129:0/4116748419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c561b0 msgr2=0x55db37015ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:38.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.406+0000 7f07a259c640 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c561b0 0x55db37015ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:38.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.406+0000 7f07a259c640 1 -- 172.21.15.129:0/4116748419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db37016660 con 0x55db36c2d2f0 2024-05-20T18:00:38.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.407+0000 7f07a259c640 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 0x55db3701ce10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0798031000 tx=0x7f0798005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd07fb6fba36455 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:38.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.407+0000 7f0789ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:00:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.413+0000 7f0789ffb640 1 -- 172.21.15.129:0/4116748419 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798052070 con 0x55db36c2d2f0 2024-05-20T18:00:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.413+0000 7f0789ffb640 1 -- 172.21.15.129:0/4116748419 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f07980305d0 con 0x55db36c2d2f0 2024-05-20T18:00:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.414+0000 7f07afc5f740 1 -- 172.21.15.129:0/4116748419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f079409b980 con 0x55db36c2d2f0 2024-05-20T18:00:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.416+0000 7f0789ffb640 1 -- 172.21.15.129:0/4116748419 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f079805b050 con 0x55db36c2d2f0 2024-05-20T18:00:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.416+0000 7f07afc5f740 10 client.15828 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.416+0000 7f07afc5f740 1 -- 172.21.15.129:0/4116748419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55db3701b380 con 0x55db36c2d2f0 2024-05-20T18:00:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.417+0000 7f0789ffb640 1 -- 172.21.15.129:0/4116748419 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f079409b980 con 0x55db36c2d2f0 2024-05-20T18:00:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.417+0000 7f0789ffb640 1 -- 172.21.15.129:0/4116748419 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f0798004fd0 con 0x55db36c2d2f0 2024-05-20T18:00:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.417+0000 7f07afc5f740 10 client.15828 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.417+0000 7f07afc5f740 10 client.15828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.417+0000 7f07afc5f740 20 client.15828 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.417+0000 7f07afc5f740 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55db36bea1c0 0x55db3708d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.417+0000 7f07afc5f740 4 client.15828 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:00:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.417+0000 7f07afc5f740 1 -- 172.21.15.129:0/4116748419 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55db36f61dc0 con 0x55db36bea1c0 2024-05-20T18:00:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.418+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55db36bea1c0 0x55db3708d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.422+0000 7f0789ffb640 10 client.15828 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:38.422+0000 7f07a1d9b640 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55db36bea1c0 0x55db3708d060 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be0544fe1fd6fd2c server_cookie=9fb2143549d3a751 in_seq=0 out_seq=0 2024-05-20T18:00:39.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:39 smithi204 ceph-mon[37076]: pgmap v66: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 MiB/s wr, 1.52k op/s 2024-05-20T18:00:39.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:39 smithi154 ceph-mon[36482]: pgmap v66: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 MiB/s wr, 1.52k op/s 2024-05-20T18:00:39.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:39 smithi129 ceph-mon[32029]: pgmap v66: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 MiB/s wr, 1.52k op/s 2024-05-20T18:00:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.248+0000 7f0789ffb640 1 -- 172.21.15.129:0/4116748419 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55db36f61dc0 con 0x55db36bea1c0 2024-05-20T18:00:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.248+0000 7f0789ffb640 10 client.15828 handle_command_reply: tid=0 2024-05-20T18:00:41.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.248+0000 7f07afc5f740 1 client.15828 shutdown 2024-05-20T18:00:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.249+0000 7f07afc5f740 1 -- 172.21.15.129:0/4116748419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 msgr2=0x55db3701ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.249+0000 7f07afc5f740 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 0x55db3701ce10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0798031000 tx=0x7f0798005850 comp rx=0 tx=0).stop 2024-05-20T18:00:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.250+0000 7f07afc5f740 1 -- 172.21.15.129:0/4116748419 shutdown_connections 2024-05-20T18:00:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.250+0000 7f07afc5f740 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55db36bea1c0 0x55db3708d060 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.250+0000 7f07afc5f740 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db36c561b0 0x55db37015ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.250+0000 7f07afc5f740 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55db36c354d0 0x55db3701d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.250+0000 7f07afc5f740 1 --2- 172.21.15.129:0/4116748419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55db36c2d2f0 0x55db3701ce10 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.250+0000 7f07afc5f740 1 -- 172.21.15.129:0/4116748419 >> 172.21.15.129:0/4116748419 conn(0x55db36c4a300 msgr2=0x55db36ff5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.250+0000 7f0789ffb640 0 client.15828 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.250+0000 7f07afc5f740 1 -- 172.21.15.129:0/4116748419 shutdown_connections 2024-05-20T18:00:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.251+0000 7f07afc5f740 1 -- 172.21.15.129:0/4116748419 wait complete. 2024-05-20T18:00:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.252+0000 7f07afc5f740 20 client.15828 trim_cache size 0 max 16384 2024-05-20T18:00:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:00:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:00:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:00:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:00:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.253+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1259375295 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f078c0a2f00 msgr2=0x7f078c0a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.253+0000 7f07a8dc2640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f078c0a2f00 0x7f078c0a5410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55db36ff2410 tx=0x7f079c009260 comp rx=0 tx=0).stop 2024-05-20T18:00:41.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.253+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1259375295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410d790 msgr2=0x7f07a407b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:41.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.253+0000 7f07a8dc2640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410d790 0x7f07a407b5a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f079400b9c0 tx=0x7f079400be80 comp rx=0 tx=0).stop 2024-05-20T18:00:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.256+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1259375295 shutdown_connections 2024-05-20T18:00:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.256+0000 7f07a8dc2640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f078c0a2f00 0x7f078c0a5410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.256+0000 7f07a8dc2640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410d790 0x7f07a407b5a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.256+0000 7f07a8dc2640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a410e250 0x7f07a407bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.256+0000 7f07a8dc2640 1 --2- 172.21.15.129:0/1259375295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a4109200 0x7f07a4083540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.257+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1259375295 >> 172.21.15.129:0/1259375295 conn(0x7f07a4075d50 msgr2=0x7f07a4079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.258+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1259375295 shutdown_connections 2024-05-20T18:00:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:41.260+0000 7f07a8dc2640 1 -- 172.21.15.129:0/1259375295 wait complete. 2024-05-20T18:00:41.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:41 smithi154 ceph-mon[36482]: pgmap v67: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 923 MiB/s wr, 2.39k op/s 2024-05-20T18:00:41.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:41 smithi129 ceph-mon[32029]: pgmap v67: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 923 MiB/s wr, 2.39k op/s 2024-05-20T18:00:41.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:41 smithi204 ceph-mon[37076]: pgmap v67: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 923 MiB/s wr, 2.39k op/s 2024-05-20T18:00:43.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:43 smithi154 ceph-mon[36482]: pgmap v68: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 844 MiB/s wr, 2.83k op/s 2024-05-20T18:00:43.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:43 smithi129 ceph-mon[32029]: pgmap v68: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 844 MiB/s wr, 2.83k op/s 2024-05-20T18:00:43.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:43 smithi204 ceph-mon[37076]: pgmap v68: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 844 MiB/s wr, 2.83k op/s 2024-05-20T18:00:45.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:45 smithi154 ceph-mon[36482]: pgmap v69: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 699 MiB/s wr, 2.73k op/s 2024-05-20T18:00:45.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:45 smithi129 ceph-mon[32029]: pgmap v69: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 699 MiB/s wr, 2.73k op/s 2024-05-20T18:00:45.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:45 smithi204 ceph-mon[37076]: pgmap v69: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 699 MiB/s wr, 2.73k op/s 2024-05-20T18:00:47.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:47 smithi154 ceph-mon[36482]: pgmap v70: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.0 GiB/s wr, 3.46k op/s 2024-05-20T18:00:47.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:47 smithi129 ceph-mon[32029]: pgmap v70: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.0 GiB/s wr, 3.46k op/s 2024-05-20T18:00:47.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:47 smithi204 ceph-mon[37076]: pgmap v70: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.0 GiB/s wr, 3.46k op/s 2024-05-20T18:00:49.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:49 smithi154 ceph-mon[36482]: pgmap v71: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 709 MiB/s wr, 2.15k op/s 2024-05-20T18:00:49.667 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:49 smithi129 ceph-mon[32029]: pgmap v71: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 709 MiB/s wr, 2.15k op/s 2024-05-20T18:00:49.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:49 smithi204 ceph-mon[37076]: pgmap v71: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 709 MiB/s wr, 2.15k op/s 2024-05-20T18:00:51.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:51 smithi154 ceph-mon[36482]: pgmap v72: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 935 MiB/s wr, 2.69k op/s 2024-05-20T18:00:51.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:51 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:00:51.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:51 smithi129 ceph-mon[32029]: pgmap v72: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 935 MiB/s wr, 2.69k op/s 2024-05-20T18:00:51.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:51 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:00:51.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:51 smithi204 ceph-mon[37076]: pgmap v72: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 935 MiB/s wr, 2.69k op/s 2024-05-20T18:00:51.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:51 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:00:52.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:52 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:52.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:52 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:52.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:52 smithi154 ceph-mon[36482]: pgmap v73: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 871 MiB/s wr, 2.15k op/s 2024-05-20T18:00:52.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:52 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:52.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:52 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:52.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:52 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:52.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:52 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:52.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:52 smithi129 ceph-mon[32029]: pgmap v73: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 871 MiB/s wr, 2.15k op/s 2024-05-20T18:00:52.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:52 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:52.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:52 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:52.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:52 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:52.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:52 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:52.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:52 smithi204 ceph-mon[37076]: pgmap v73: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 871 MiB/s wr, 2.15k op/s 2024-05-20T18:00:52.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:52 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:52.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:52 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:53.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:53 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:00:53.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:53 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:00:53.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:53 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:53.921 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:53 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:00:53.921 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:53 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:00:53.921 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:53 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:53.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:53 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:00:53.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:53 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:00:53.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:53 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:00:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.612+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1438253071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d941095e0 msgr2=0x7f2d941099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.612+0000 7f2d9a2de640 1 --2- 172.21.15.129:0/1438253071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d941095e0 0x7f2d941099f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c009ca0 tx=0x7f2d8c02a8f0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.613+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1438253071 shutdown_connections 2024-05-20T18:00:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.613+0000 7f2d9a2de640 1 --2- 172.21.15.129:0/1438253071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d9410e630 0x7f2d94115dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.613+0000 7f2d9a2de640 1 --2- 172.21.15.129:0/1438253071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9410db70 0x7f2d9410dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.613+0000 7f2d9a2de640 1 --2- 172.21.15.129:0/1438253071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d941095e0 0x7f2d941099f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.613+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1438253071 >> 172.21.15.129:0/1438253071 conn(0x7f2d9406ea40 msgr2=0x7f2d9406ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.622+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1438253071 shutdown_connections 2024-05-20T18:00:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.623+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1438253071 wait complete. 2024-05-20T18:00:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.624+0000 7f2d9a2de640 1 Processor -- start 2024-05-20T18:00:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.633+0000 7f2d9a2de640 1 -- start start 2024-05-20T18:00:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.637+0000 7f2d9a2de640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d941095e0 0x7f2d941a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.637+0000 7f2d9a2de640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9410db70 0x7f2d941a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.637+0000 7f2d9a2de640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d9410e630 0x7f2d9419fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.637+0000 7f2d9a2de640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2d94119300 con 0x7f2d9410db70 2024-05-20T18:00:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.637+0000 7f2d9a2de640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d94119180 con 0x7f2d941095e0 2024-05-20T18:00:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.637+0000 7f2d9a2de640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2d94119480 con 0x7f2d9410e630 2024-05-20T18:00:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.638+0000 7f2d93fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d941095e0 0x7f2d941a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.638+0000 7f2d93fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d941095e0 0x7f2d941a58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38860/0 (socket says 172.21.15.129:38860) 2024-05-20T18:00:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.638+0000 7f2d93fff640 1 -- 172.21.15.129:0/1021770283 learned_addr learned my addr 172.21.15.129:0/1021770283 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.639+0000 7f2d93fff640 1 -- 172.21.15.129:0/1021770283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d9410e630 msgr2=0x7f2d9419fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.639+0000 7f2d93fff640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d9410e630 0x7f2d9419fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.639+0000 7f2d93fff640 1 -- 172.21.15.129:0/1021770283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9410db70 msgr2=0x7f2d941a5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.639+0000 7f2d93fff640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9410db70 0x7f2d941a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.639+0000 7f2d93fff640 1 -- 172.21.15.129:0/1021770283 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d941a0160 con 0x7f2d941095e0 2024-05-20T18:00:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.640+0000 7f2d93fff640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d941095e0 0x7f2d941a58b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c02aee0 tx=0x7f2d8c031f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.643+0000 7f2d917fa640 1 -- 172.21.15.129:0/1021770283 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c02fa50 con 0x7f2d941095e0 2024-05-20T18:00:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.644+0000 7f2d917fa640 1 -- 172.21.15.129:0/1021770283 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8c02d070 con 0x7f2d941095e0 2024-05-20T18:00:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.644+0000 7f2d917fa640 1 -- 172.21.15.129:0/1021770283 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c0466e0 con 0x7f2d941095e0 2024-05-20T18:00:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.647+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1021770283 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d941a03f0 con 0x7f2d941095e0 2024-05-20T18:00:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.648+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1021770283 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d941a08f0 con 0x7f2d941095e0 2024-05-20T18:00:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.656+0000 7f2d917fa640 1 -- 172.21.15.129:0/1021770283 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f2d8c02fbf0 con 0x7f2d941095e0 2024-05-20T18:00:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.657+0000 7f2d917fa640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2d7c0a7310 0x7f2d7c0a9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.658+0000 7f2d917fa640 1 -- 172.21.15.129:0/1021770283 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f2d8c0f0940 con 0x7f2d941095e0 2024-05-20T18:00:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.659+0000 7f2d937fe640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2d7c0a7310 0x7f2d7c0a9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.668+0000 7f2d937fe640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2d7c0a7310 0x7f2d7c0a9820 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2d94115140 tx=0x7f2d84006c90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.671+0000 7f2da117e740 1 Processor -- start 2024-05-20T18:00:54.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.671+0000 7f2da117e740 1 -- start start 2024-05-20T18:00:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.672+0000 7f2da117e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3c71b0 0x55b03e766270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.672+0000 7f2da117e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 0x55b03e7667b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.672+0000 7f2da117e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e3a64d0 0x55b03e766cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.672+0000 7f2da117e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b03e6cb340 con 0x55b03e3c71b0 2024-05-20T18:00:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.672+0000 7f2da117e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b03e673290 con 0x55b03e3a64d0 2024-05-20T18:00:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.672+0000 7f2da117e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b03e6c7300 con 0x55b03e39e2f0 2024-05-20T18:00:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.672+0000 7f2d93fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3c71b0 0x55b03e766270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.672+0000 7f2d93fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3c71b0 0x55b03e766270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41890/0 (socket says 172.21.15.129:41890) 2024-05-20T18:00:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.672+0000 7f2d93fff640 1 -- 172.21.15.129:0/1902053804 learned_addr learned my addr 172.21.15.129:0/1902053804 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:54.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.673+0000 7f2d937fe640 1 --2- 172.21.15.129:0/1902053804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 0x55b03e7667b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.673+0000 7f2d937fe640 1 -- 172.21.15.129:0/1902053804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e3a64d0 msgr2=0x55b03e766cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:00:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.673+0000 7f2d937fe640 1 --2- 172.21.15.129:0/1902053804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e3a64d0 0x55b03e766cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.674+0000 7f2d937fe640 1 -- 172.21.15.129:0/1902053804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3c71b0 msgr2=0x55b03e766270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.674+0000 7f2d937fe640 1 --2- 172.21.15.129:0/1902053804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3c71b0 0x55b03e766270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.674+0000 7f2d937fe640 1 -- 172.21.15.129:0/1902053804 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b03e6978e0 con 0x55b03e39e2f0 2024-05-20T18:00:54.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.674+0000 7f2d937fe640 1 --2- 172.21.15.129:0/1902053804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 0x55b03e7667b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2d9406b2e0 tx=0x7f2d8400e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=729ec171dee2fec5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.679+0000 7f2d7a7fc640 1 -- 172.21.15.129:0/1902053804 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d84023070 con 0x55b03e39e2f0 2024-05-20T18:00:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.679+0000 7f2d7a7fc640 1 -- 172.21.15.129:0/1902053804 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8400be10 con 0x55b03e39e2f0 2024-05-20T18:00:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.683+0000 7f2da117e740 1 -- 172.21.15.129:0/1902053804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 msgr2=0x55b03e7667b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.683+0000 7f2da117e740 1 --2- 172.21.15.129:0/1902053804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 0x55b03e7667b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2d9406b2e0 tx=0x7f2d8400e000 comp rx=0 tx=0).stop 2024-05-20T18:00:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.684+0000 7f2da117e740 1 -- 172.21.15.129:0/1902053804 shutdown_connections 2024-05-20T18:00:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.684+0000 7f2da117e740 1 --2- 172.21.15.129:0/1902053804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e3a64d0 0x55b03e766cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.684+0000 7f2da117e740 1 --2- 172.21.15.129:0/1902053804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 0x55b03e7667b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.684+0000 7f2da117e740 1 --2- 172.21.15.129:0/1902053804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3c71b0 0x55b03e766270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.684+0000 7f2da117e740 1 -- 172.21.15.129:0/1902053804 >> 172.21.15.129:0/1902053804 conn(0x55b03e3bb300 msgr2=0x55b03e762a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:54.684 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.686+0000 7f2da117e740 1 -- 172.21.15.129:0/1902053804 shutdown_connections 2024-05-20T18:00:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.686+0000 7f2da117e740 1 -- 172.21.15.129:0/1902053804 wait complete. 2024-05-20T18:00:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.687+0000 7f2da117e740 1 -- start start 2024-05-20T18:00:54.686 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.688+0000 7f2da117e740 1 Processor -- start 2024-05-20T18:00:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.689+0000 7f2da117e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e39e2f0 0x55b03e77f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.687 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.689+0000 7f2da117e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3a64d0 0x55b03e77f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.689+0000 7f2da117e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3c71b0 0x55b03e779650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.689+0000 7f2da117e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b03e7733f0 con 0x55b03e3a64d0 2024-05-20T18:00:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.689+0000 7f2da117e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b03e686790 con 0x55b03e39e2f0 2024-05-20T18:00:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.689+0000 7f2da117e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b03e773570 con 0x55b03e3c71b0 2024-05-20T18:00:54.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.689+0000 7f2da117e740 10 client.0 _ll_register_callbacks cb 0x55b03e6c1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:54.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.694+0000 7f2d9887b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3c71b0 0x55b03e779650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.694+0000 7f2d937fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3a64d0 0x55b03e77f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.694+0000 7f2d937fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3a64d0 0x55b03e77f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41894/0 (socket says 172.21.15.129:41894) 2024-05-20T18:00:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.694+0000 7f2d937fe640 1 -- 172.21.15.129:0/2660120658 learned_addr learned my addr 172.21.15.129:0/2660120658 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:54.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.694+0000 7f2d93fff640 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e39e2f0 0x55b03e77f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.702+0000 7f2d9887b640 1 -- 172.21.15.129:0/2660120658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e39e2f0 msgr2=0x55b03e77f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.702+0000 7f2d9887b640 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e39e2f0 0x55b03e77f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.702+0000 7f2d9887b640 1 -- 172.21.15.129:0/2660120658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3a64d0 msgr2=0x55b03e77f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.702+0000 7f2d9887b640 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3a64d0 0x55b03e77f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.702+0000 7f2d9887b640 1 -- 172.21.15.129:0/2660120658 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b03e3df210 con 0x55b03e3c71b0 2024-05-20T18:00:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.705+0000 7f2d9887b640 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3c71b0 0x55b03e779650 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2d8800bb30 tx=0x7f2d8800d2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c25976d9f5cd763 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.707+0000 7f2d737fe640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:00:54.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.722+0000 7f2d737fe640 1 -- 172.21.15.129:0/2660120658 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d88016070 con 0x55b03e3c71b0 2024-05-20T18:00:54.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.722+0000 7f2d737fe640 1 -- 172.21.15.129:0/2660120658 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f2d88004020 con 0x55b03e3c71b0 2024-05-20T18:00:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.726+0000 7f2da117e740 1 -- 172.21.15.129:0/2660120658 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d8400be10 con 0x55b03e3c71b0 2024-05-20T18:00:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.727+0000 7f2d737fe640 1 -- 172.21.15.129:0/2660120658 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2d8801f050 con 0x55b03e3c71b0 2024-05-20T18:00:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.727+0000 7f2da117e740 10 client.25534 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.727+0000 7f2da117e740 1 -- 172.21.15.129:0/2660120658 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b03e4ab620 con 0x55b03e3c71b0 2024-05-20T18:00:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.734+0000 7f2d737fe640 1 -- 172.21.15.129:0/2660120658 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8400be10 con 0x55b03e3c71b0 2024-05-20T18:00:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.734+0000 7f2d737fe640 1 -- 172.21.15.129:0/2660120658 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f2d88010480 con 0x55b03e3c71b0 2024-05-20T18:00:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.735+0000 7f2da117e740 10 client.25534 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.735+0000 7f2da117e740 10 client.25534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.735+0000 7f2da117e740 20 client.25534 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.735+0000 7f2da117e740 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b03e385d80 0x55b03e78b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.735+0000 7f2da117e740 4 client.25534 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:00:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.735+0000 7f2da117e740 1 -- 172.21.15.129:0/2660120658 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b03e6d8a70 con 0x55b03e385d80 2024-05-20T18:00:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.737+0000 7f2d93fff640 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b03e385d80 0x55b03e78b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.743+0000 7f2d93fff640 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b03e385d80 0x55b03e78b840 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d30eb14f8711942 server_cookie=3ce0a5cd8dc9194b in_seq=0 out_seq=0 2024-05-20T18:00:54.741 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.743+0000 7f2d737fe640 10 client.25534 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.751+0000 7f2d737fe640 1 -- 172.21.15.129:0/2660120658 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b03e6d8a70 con 0x55b03e385d80 2024-05-20T18:00:54.749 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.751+0000 7f2d737fe640 10 client.25534 handle_command_reply: tid=0 2024-05-20T18:00:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.751+0000 7f2da117e740 1 client.25534 shutdown 2024-05-20T18:00:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.752+0000 7f2da117e740 1 -- 172.21.15.129:0/2660120658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3c71b0 msgr2=0x55b03e779650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.752+0000 7f2da117e740 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3c71b0 0x55b03e779650 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2d8800bb30 tx=0x7f2d8800d2e0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.752+0000 7f2da117e740 1 -- 172.21.15.129:0/2660120658 shutdown_connections 2024-05-20T18:00:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.753+0000 7f2da117e740 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b03e385d80 0x55b03e78b840 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.753+0000 7f2da117e740 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3c71b0 0x55b03e779650 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.753+0000 7f2da117e740 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3a64d0 0x55b03e77f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.753+0000 7f2da117e740 1 --2- 172.21.15.129:0/2660120658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e39e2f0 0x55b03e77f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.753+0000 7f2da117e740 1 -- 172.21.15.129:0/2660120658 >> 172.21.15.129:0/2660120658 conn(0x55b03e3bb300 msgr2=0x55b03e7660a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.754+0000 7f2d737fe640 0 client.25534 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.755+0000 7f2da117e740 1 -- 172.21.15.129:0/2660120658 shutdown_connections 2024-05-20T18:00:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.755+0000 7f2da117e740 1 -- 172.21.15.129:0/2660120658 wait complete. 2024-05-20T18:00:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.755+0000 7f2da117e740 20 client.25534 trim_cache size 0 max 16384 2024-05-20T18:00:54.819 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.821+0000 7f2da117e740 1 Processor -- start 2024-05-20T18:00:54.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.821+0000 7f2da117e740 1 -- start start 2024-05-20T18:00:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.822+0000 7f2da117e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 0x55b03e773f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.822+0000 7f2da117e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3a64d0 0x55b03e772870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.822+0000 7f2da117e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e3c71b0 0x55b03e791040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.822+0000 7f2da117e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2d941b2f90 con 0x55b03e3a64d0 2024-05-20T18:00:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.822+0000 7f2da117e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d941b5810 con 0x55b03e3c71b0 2024-05-20T18:00:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.822+0000 7f2da117e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2d941b3600 con 0x55b03e39e2f0 2024-05-20T18:00:54.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.823+0000 7f2d93fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 0x55b03e773f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.823+0000 7f2d93fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 0x55b03e773f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:49368/0 (socket says 172.21.15.129:49368) 2024-05-20T18:00:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.823+0000 7f2d93fff640 1 -- 172.21.15.129:0/713226277 learned_addr learned my addr 172.21.15.129:0/713226277 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.823+0000 7f2d9887b640 1 --2- 172.21.15.129:0/713226277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e3c71b0 0x55b03e791040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.825+0000 7f2d937fe640 1 --2- 172.21.15.129:0/713226277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3a64d0 0x55b03e772870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.825+0000 7f2d9887b640 1 -- 172.21.15.129:0/713226277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 msgr2=0x55b03e773f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:54.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.825+0000 7f2d9887b640 1 --2- 172.21.15.129:0/713226277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 0x55b03e773f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.825+0000 7f2d9887b640 1 -- 172.21.15.129:0/713226277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3a64d0 msgr2=0x55b03e772870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:54.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.825+0000 7f2d9887b640 1 --2- 172.21.15.129:0/713226277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3a64d0 0x55b03e772870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.825+0000 7f2d9887b640 1 -- 172.21.15.129:0/713226277 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b03e385fb0 con 0x55b03e3c71b0 2024-05-20T18:00:54.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.828+0000 7f2d9887b640 1 --2- 172.21.15.129:0/713226277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e3c71b0 0x55b03e791040 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2d880108d0 tx=0x7f2d8800ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3625fbf896ebba2c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:54.829 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.831+0000 7f2d79ffb640 1 -- 172.21.15.129:0/713226277 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d88016040 con 0x55b03e3c71b0 2024-05-20T18:00:54.830 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.831+0000 7f2d79ffb640 1 -- 172.21.15.129:0/713226277 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d88004570 con 0x55b03e3c71b0 2024-05-20T18:00:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.834+0000 7f2da117e740 1 -- 172.21.15.129:0/713226277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e3c71b0 msgr2=0x55b03e791040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.834+0000 7f2da117e740 1 --2- 172.21.15.129:0/713226277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e3c71b0 0x55b03e791040 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2d880108d0 tx=0x7f2d8800ded0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.834+0000 7f2da117e740 1 -- 172.21.15.129:0/713226277 shutdown_connections 2024-05-20T18:00:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.834+0000 7f2da117e740 1 --2- 172.21.15.129:0/713226277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e3c71b0 0x55b03e791040 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.835+0000 7f2da117e740 1 --2- 172.21.15.129:0/713226277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e3a64d0 0x55b03e772870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.835+0000 7f2da117e740 1 --2- 172.21.15.129:0/713226277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e39e2f0 0x55b03e773f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.833 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.835+0000 7f2da117e740 1 -- 172.21.15.129:0/713226277 >> 172.21.15.129:0/713226277 conn(0x55b03e3bb300 msgr2=0x55b03e773810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:54.836 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.838+0000 7f2da117e740 1 -- 172.21.15.129:0/713226277 shutdown_connections 2024-05-20T18:00:54.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.841+0000 7f2da117e740 1 -- 172.21.15.129:0/713226277 wait complete. 2024-05-20T18:00:54.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.841+0000 7f2da117e740 1 -- start start 2024-05-20T18:00:54.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.842+0000 7f2da117e740 1 Processor -- start 2024-05-20T18:00:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.844+0000 7f2da117e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e39e2f0 0x55b03e77ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.844+0000 7f2da117e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3a64d0 0x55b03e7804f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.845+0000 7f2da117e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e35b1c0 0x55b03e780a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.845+0000 7f2da117e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2d941b3cd0 con 0x55b03e39e2f0 2024-05-20T18:00:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.845+0000 7f2da117e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d941b5140 con 0x55b03e35b1c0 2024-05-20T18:00:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.845+0000 7f2da117e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2d941b43a0 con 0x55b03e3a64d0 2024-05-20T18:00:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.845+0000 7f2da117e740 10 client.0 _ll_register_callbacks cb 0x7f2d8800b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.846+0000 7f2d937fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3a64d0 0x55b03e7804f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.846+0000 7f2d937fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3a64d0 0x55b03e7804f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:49380/0 (socket says 172.21.15.129:49380) 2024-05-20T18:00:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.846+0000 7f2d937fe640 1 -- 172.21.15.129:0/3213643110 learned_addr learned my addr 172.21.15.129:0/3213643110 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.847+0000 7f2d9887b640 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e35b1c0 0x55b03e780a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.847+0000 7f2d93fff640 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e39e2f0 0x55b03e77ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.848+0000 7f2d9887b640 1 -- 172.21.15.129:0/3213643110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3a64d0 msgr2=0x55b03e7804f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:54.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.848+0000 7f2d9887b640 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3a64d0 0x55b03e7804f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.848+0000 7f2d9887b640 1 -- 172.21.15.129:0/3213643110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e39e2f0 msgr2=0x55b03e77ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:54.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.848+0000 7f2d9887b640 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e39e2f0 0x55b03e77ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:54.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.848+0000 7f2d9887b640 1 -- 172.21.15.129:0/3213643110 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b03e7f1be0 con 0x55b03e35b1c0 2024-05-20T18:00:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.850+0000 7f2d9887b640 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e35b1c0 0x55b03e780a30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2d88010850 tx=0x7f2d88010cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23f3875e82ab24fe server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.851+0000 7f2d737fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:00:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.851+0000 7f2d737fe640 1 -- 172.21.15.129:0/3213643110 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d88034070 con 0x55b03e35b1c0 2024-05-20T18:00:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.852+0000 7f2da117e740 1 -- 172.21.15.129:0/3213643110 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b03e6d2dc0 con 0x55b03e35b1c0 2024-05-20T18:00:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.852+0000 7f2d737fe640 1 -- 172.21.15.129:0/3213643110 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f2d88003a00 con 0x55b03e35b1c0 2024-05-20T18:00:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.852+0000 7f2d737fe640 1 -- 172.21.15.129:0/3213643110 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2d8803d050 con 0x55b03e35b1c0 2024-05-20T18:00:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.853+0000 7f2da117e740 10 client.25505 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.853+0000 7f2da117e740 1 -- 172.21.15.129:0/3213643110 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b03e7f2d10 con 0x55b03e35b1c0 2024-05-20T18:00:54.854 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.856+0000 7f2d737fe640 1 -- 172.21.15.129:0/3213643110 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b03e6d2dc0 con 0x55b03e35b1c0 2024-05-20T18:00:54.854 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.856+0000 7f2d737fe640 1 -- 172.21.15.129:0/3213643110 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f2d88037020 con 0x55b03e35b1c0 2024-05-20T18:00:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.857+0000 7f2da117e740 10 client.25505 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.857+0000 7f2da117e740 10 client.25505 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.857+0000 7f2da117e740 20 client.25505 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.857+0000 7f2da117e740 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b03e7f33c0 0x55b03e7f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.857+0000 7f2da117e740 4 client.25505 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:00:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.857+0000 7f2da117e740 1 -- 172.21.15.129:0/3213643110 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b03e6508e0 con 0x55b03e7f33c0 2024-05-20T18:00:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.862+0000 7f2d93fff640 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b03e7f33c0 0x55b03e7f57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:54.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.873+0000 7f2d93fff640 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b03e7f33c0 0x55b03e7f57e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=515af0f68d9924cd server_cookie=1ecc7833889dbc47 in_seq=0 out_seq=0 2024-05-20T18:00:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:54.877+0000 7f2d737fe640 10 client.25505 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:54.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:54 smithi129 ceph-mon[32029]: pgmap v74: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 731 MiB/s wr, 1.59k op/s 2024-05-20T18:00:54.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:54 smithi204 ceph-mon[37076]: pgmap v74: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 731 MiB/s wr, 1.59k op/s 2024-05-20T18:00:55.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:54 smithi154 ceph-mon[36482]: pgmap v74: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 731 MiB/s wr, 1.59k op/s 2024-05-20T18:00:56.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:00:56.706 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:00:57.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:56 smithi154 ceph-mon[36482]: pgmap v75: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 GiB/s wr, 2.56k op/s 2024-05-20T18:00:57.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:56 smithi129 ceph-mon[32029]: pgmap v75: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 GiB/s wr, 2.56k op/s 2024-05-20T18:00:57.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:56 smithi204 ceph-mon[37076]: pgmap v75: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 GiB/s wr, 2.56k op/s 2024-05-20T18:00:57.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.234+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3353535812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6410f270 msgr2=0x7f5d64108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:57.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.234+0000 7f5d6b4d6640 1 --2- 172.21.15.129:0/3353535812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6410f270 0x7f5d64108b50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c009ca0 tx=0x7f5d5c02a8f0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.243+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3353535812 shutdown_connections 2024-05-20T18:00:57.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.243+0000 7f5d6b4d6640 1 --2- 172.21.15.129:0/3353535812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d64109090 0x7f5d64109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.243+0000 7f5d6b4d6640 1 --2- 172.21.15.129:0/3353535812 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6410f270 0x7f5d64108b50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.243+0000 7f5d6b4d6640 1 --2- 172.21.15.129:0/3353535812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d6410e890 0x7f5d6410eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.243+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3353535812 >> 172.21.15.129:0/3353535812 conn(0x7f5d6406d580 msgr2=0x7f5d6406d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.246+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3353535812 shutdown_connections 2024-05-20T18:00:57.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.250+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3353535812 wait complete. 2024-05-20T18:00:57.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.252+0000 7f5d6b4d6640 1 Processor -- start 2024-05-20T18:00:57.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.259+0000 7f5d6b4d6640 1 -- start start 2024-05-20T18:00:57.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.259+0000 7f5d6b4d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d64109090 0x7f5d6411f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:57.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.259+0000 7f5d6b4d6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d6410e890 0x7f5d641182a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:57.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.259+0000 7f5d6b4d6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6410f270 0x7f5d641188d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:57.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.259+0000 7f5d6b4d6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d64121a90 con 0x7f5d6410e890 2024-05-20T18:00:57.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.259+0000 7f5d6b4d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d64121910 con 0x7f5d64109090 2024-05-20T18:00:57.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.259+0000 7f5d6b4d6640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d64121c10 con 0x7f5d6410f270 2024-05-20T18:00:57.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.263+0000 7f5d6a4d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d64109090 0x7f5d6411f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:57.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.266+0000 7f5d6a4d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d64109090 0x7f5d6411f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38918/0 (socket says 172.21.15.129:38918) 2024-05-20T18:00:57.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.266+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/3070053628 learned_addr learned my addr 172.21.15.129:0/3070053628 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:57.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.266+0000 7f5d6acd5640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6410f270 0x7f5d641188d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:57.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.266+0000 7f5d69cd3640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d6410e890 0x7f5d641182a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.271+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/3070053628 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6410f270 msgr2=0x7f5d641188d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.271+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6410f270 0x7f5d641188d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.271+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/3070053628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d6410e890 msgr2=0x7f5d641182a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.271+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d6410e890 0x7f5d641182a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.271+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/3070053628 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d6411cca0 con 0x7f5d64109090 2024-05-20T18:00:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.272+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d64109090 0x7f5d6411f180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000d950 tx=0x7f5d6000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.273+0000 7f5d5b7fe640 1 -- 172.21.15.129:0/3070053628 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60017070 con 0x7f5d64109090 2024-05-20T18:00:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.273+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3070053628 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d6411cf90 con 0x7f5d64109090 2024-05-20T18:00:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.273+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3070053628 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d6411d530 con 0x7f5d64109090 2024-05-20T18:00:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.273+0000 7f5d5b7fe640 1 -- 172.21.15.129:0/3070053628 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d60009c30 con 0x7f5d64109090 2024-05-20T18:00:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.273+0000 7f5d5b7fe640 1 -- 172.21.15.129:0/3070053628 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6000f690 con 0x7f5d64109090 2024-05-20T18:00:57.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.279+0000 7f5d735e5740 1 Processor -- start 2024-05-20T18:00:57.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.290+0000 7f5d5b7fe640 1 -- 172.21.15.129:0/3070053628 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5d6000f830 con 0x7f5d64109090 2024-05-20T18:00:57.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.291+0000 7f5d5b7fe640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d500a2e40 0x7f5d500a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:57.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.292+0000 7f5d5b7fe640 1 -- 172.21.15.129:0/3070053628 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5d600d0060 con 0x7f5d64109090 2024-05-20T18:00:57.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.293+0000 7f5d735e5740 1 -- start start 2024-05-20T18:00:57.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.294+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af512390 0x55b2af8b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:57.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.295+0000 7f5d6acd5640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d500a2e40 0x7f5d500a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:57.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.296+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af4e94d0 0x55b2af8b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:57.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.296+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4f16b0 0x55b2af8b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:57.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.296+0000 7f5d735e5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b2af816d00 con 0x55b2af4e94d0 2024-05-20T18:00:57.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.296+0000 7f5d735e5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b2af7be4c0 con 0x55b2af512390 2024-05-20T18:00:57.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.296+0000 7f5d735e5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b2af812cc0 con 0x55b2af4f16b0 2024-05-20T18:00:57.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.297+0000 7f5d6a4d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af512390 0x55b2af8b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:57.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.301+0000 7f5d6a4d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af512390 0x55b2af8b0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38934/0 (socket says 172.21.15.129:38934) 2024-05-20T18:00:57.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.301+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/1519260986 learned_addr learned my addr 172.21.15.129:0/1519260986 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:57.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.301+0000 7f5d6acd5640 1 --2- 172.21.15.129:0/1519260986 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4f16b0 0x55b2af8b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:57.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.302+0000 7f5d6acd5640 1 -- 172.21.15.129:0/1519260986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af512390 msgr2=0x55b2af8b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:57.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.302+0000 7f5d6acd5640 1 --2- 172.21.15.129:0/1519260986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af512390 0x55b2af8b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.302+0000 7f5d6acd5640 1 -- 172.21.15.129:0/1519260986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af4e94d0 msgr2=0x55b2af8b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:57.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.302+0000 7f5d6acd5640 1 --2- 172.21.15.129:0/1519260986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af4e94d0 0x55b2af8b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.302+0000 7f5d6acd5640 1 -- 172.21.15.129:0/1519260986 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2af5f47d0 con 0x55b2af4f16b0 2024-05-20T18:00:57.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.303+0000 7f5d6acd5640 1 --2- 172.21.15.129:0/1519260986 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4f16b0 0x55b2af8b19b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5d5400f610 tx=0x7f5d5401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=607a8ec19a7bf3a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:57.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.308+0000 7f5d58ff9640 1 -- 172.21.15.129:0/1519260986 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d54020650 con 0x55b2af4f16b0 2024-05-20T18:00:57.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.308+0000 7f5d6acd5640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d500a2e40 0x7f5d500a5350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009f10 tx=0x7f5d540137d0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:57.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.308+0000 7f5d58ff9640 1 -- 172.21.15.129:0/1519260986 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d54025070 con 0x55b2af4f16b0 2024-05-20T18:00:57.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.311+0000 7f5d735e5740 1 -- 172.21.15.129:0/1519260986 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4f16b0 msgr2=0x55b2af8b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:57.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.311+0000 7f5d735e5740 1 --2- 172.21.15.129:0/1519260986 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4f16b0 0x55b2af8b19b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5d5400f610 tx=0x7f5d5401b000 comp rx=0 tx=0).stop 2024-05-20T18:00:57.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.311+0000 7f5d735e5740 1 -- 172.21.15.129:0/1519260986 shutdown_connections 2024-05-20T18:00:57.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.311+0000 7f5d735e5740 1 --2- 172.21.15.129:0/1519260986 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4f16b0 0x55b2af8b19b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.311+0000 7f5d735e5740 1 --2- 172.21.15.129:0/1519260986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af4e94d0 0x55b2af8b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.311+0000 7f5d735e5740 1 --2- 172.21.15.129:0/1519260986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af512390 0x55b2af8b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.311+0000 7f5d735e5740 1 -- 172.21.15.129:0/1519260986 >> 172.21.15.129:0/1519260986 conn(0x55b2af5064e0 msgr2=0x55b2af8ad6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:57.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.312+0000 7f5d735e5740 1 -- 172.21.15.129:0/1519260986 shutdown_connections 2024-05-20T18:00:57.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.312+0000 7f5d735e5740 1 -- 172.21.15.129:0/1519260986 wait complete. 2024-05-20T18:00:57.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.312+0000 7f5d735e5740 1 -- start start 2024-05-20T18:00:57.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.313+0000 7f5d735e5740 1 Processor -- start 2024-05-20T18:00:57.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.313+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8baf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:57.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.313+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 0x55b2af8bb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:57.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.313+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 0x55b2af8bb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:57.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.313+0000 7f5d735e5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b2af8be270 con 0x55b2af512390 2024-05-20T18:00:57.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.313+0000 7f5d735e5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b2af7d1b90 con 0x55b2af4e94d0 2024-05-20T18:00:57.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.313+0000 7f5d735e5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b2af8be3f0 con 0x55b2af4d0f60 2024-05-20T18:00:57.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.313+0000 7f5d735e5740 10 client.0 _ll_register_callbacks cb 0x55b2af80dda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:57.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.314+0000 7f5d6a4d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8baf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:57.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.314+0000 7f5d6a4d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8baf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38948/0 (socket says 172.21.15.129:38948) 2024-05-20T18:00:57.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.314+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/100603718 learned_addr learned my addr 172.21.15.129:0/100603718 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:57.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.314+0000 7f5d69cd3640 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 0x55b2af8bb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:57.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.315+0000 7f5d6acd5640 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 0x55b2af8bb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:57.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.315+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/100603718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 msgr2=0x55b2af8bb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:57.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.315+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 0x55b2af8bb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.315+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/100603718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 msgr2=0x55b2af8bb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:57.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.315+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 0x55b2af8bb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:57.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.315+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/100603718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2af52a3f0 con 0x55b2af4e94d0 2024-05-20T18:00:57.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.315+0000 7f5d6acd5640 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 0x55b2af8bb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:00:57.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.316+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8baf70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5d60010f20 tx=0x7f5d6009bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81ca9261ad78855c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:57.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.316+0000 7f5d4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:00:57.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.317+0000 7f5d4dffb640 1 -- 172.21.15.129:0/100603718 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60020050 con 0x55b2af4e94d0 2024-05-20T18:00:57.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.317+0000 7f5d735e5740 1 -- 172.21.15.129:0/100603718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b2af824430 con 0x55b2af4e94d0 2024-05-20T18:00:57.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.317+0000 7f5d4dffb640 1 -- 172.21.15.129:0/100603718 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5d600b52c0 con 0x55b2af4e94d0 2024-05-20T18:00:57.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.319+0000 7f5d4dffb640 1 -- 172.21.15.129:0/100603718 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5d600cdbc0 con 0x55b2af4e94d0 2024-05-20T18:00:57.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.321+0000 7f5d735e5740 10 client.25517 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:57.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.321+0000 7f5d735e5740 1 -- 172.21.15.129:0/100603718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b2af4ea0f0 con 0x55b2af4e94d0 2024-05-20T18:00:57.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.325+0000 7f5d4dffb640 1 -- 172.21.15.129:0/100603718 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b2af824430 con 0x55b2af4e94d0 2024-05-20T18:00:57.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.325+0000 7f5d4dffb640 1 -- 172.21.15.129:0/100603718 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f5d6009b960 con 0x55b2af4e94d0 2024-05-20T18:00:57.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.325+0000 7f5d735e5740 10 client.25517 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:57.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.325+0000 7f5d735e5740 10 client.25517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:57.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.325+0000 7f5d735e5740 20 client.25517 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:57.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.325+0000 7f5d735e5740 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2af4a63a0 0x55b2af8cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:57.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.325+0000 7f5d735e5740 4 client.25517 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:00:57.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.325+0000 7f5d735e5740 1 -- 172.21.15.129:0/100603718 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2af81e780 con 0x55b2af4a63a0 2024-05-20T18:00:57.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.328+0000 7f5d69cd3640 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2af4a63a0 0x55b2af8cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:57.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.349+0000 7f5d69cd3640 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2af4a63a0 0x55b2af8cdea0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a17ea7947e756a2 server_cookie=b0e2368d6350f008 in_seq=0 out_seq=0 2024-05-20T18:00:57.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:57.349+0000 7f5d4dffb640 10 client.25517 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:59.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:00:58 smithi154 ceph-mon[36482]: pgmap v76: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 721 MiB/s wr, 1.83k op/s 2024-05-20T18:00:59.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:00:58 smithi129 ceph-mon[32029]: pgmap v76: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 721 MiB/s wr, 1.83k op/s 2024-05-20T18:00:59.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:00:58 smithi204 ceph-mon[37076]: pgmap v76: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 721 MiB/s wr, 1.83k op/s 2024-05-20T18:00:59.216 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.218+0000 7f2d737fe640 1 -- 172.21.15.129:0/3213643110 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b03e6508e0 con 0x55b03e7f33c0 2024-05-20T18:00:59.216 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.218+0000 7f2d737fe640 10 client.25505 handle_command_reply: tid=0 2024-05-20T18:00:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.218+0000 7f2da117e740 1 client.25505 shutdown 2024-05-20T18:00:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.218+0000 7f2da117e740 1 -- 172.21.15.129:0/3213643110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e35b1c0 msgr2=0x55b03e780a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.218+0000 7f2da117e740 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e35b1c0 0x55b03e780a30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2d88010850 tx=0x7f2d88010cf0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.224+0000 7f2da117e740 1 -- 172.21.15.129:0/3213643110 shutdown_connections 2024-05-20T18:00:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.224+0000 7f2da117e740 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b03e7f33c0 0x55b03e7f57e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.224+0000 7f2da117e740 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b03e35b1c0 0x55b03e780a30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.224+0000 7f2da117e740 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b03e3a64d0 0x55b03e7804f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.224+0000 7f2da117e740 1 --2- 172.21.15.129:0/3213643110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b03e39e2f0 0x55b03e77ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.224+0000 7f2da117e740 1 -- 172.21.15.129:0/3213643110 >> 172.21.15.129:0/3213643110 conn(0x55b03e3bb300 msgr2=0x55b03e77a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.224+0000 7f2d737fe640 0 client.25505 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.224+0000 7f2da117e740 1 -- 172.21.15.129:0/3213643110 shutdown_connections 2024-05-20T18:00:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.224+0000 7f2da117e740 1 -- 172.21.15.129:0/3213643110 wait complete. 2024-05-20T18:00:59.222 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.225+0000 7f2da117e740 20 client.25505 trim_cache size 0 max 16384 2024-05-20T18:00:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.225+0000 7f5d4dffb640 1 -- 172.21.15.129:0/100603718 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b2af81e780 con 0x55b2af4a63a0 2024-05-20T18:00:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.225+0000 7f5d4dffb640 10 client.25517 handle_command_reply: tid=0 2024-05-20T18:00:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.226+0000 7f5d735e5740 1 client.25517 shutdown 2024-05-20T18:00:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.226+0000 7f5d735e5740 1 -- 172.21.15.129:0/100603718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 msgr2=0x55b2af8baf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.226+0000 7f5d735e5740 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8baf70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5d60010f20 tx=0x7f5d6009bfa0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.226+0000 7f5d735e5740 1 -- 172.21.15.129:0/100603718 shutdown_connections 2024-05-20T18:00:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.226+0000 7f5d735e5740 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2af4a63a0 0x55b2af8cdea0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.226+0000 7f5d735e5740 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 0x55b2af8bb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.226+0000 7f5d735e5740 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 0x55b2af8bb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.226+0000 7f5d735e5740 1 --2- 172.21.15.129:0/100603718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8baf70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.226+0000 7f5d735e5740 1 -- 172.21.15.129:0/100603718 >> 172.21.15.129:0/100603718 conn(0x55b2af5064e0 msgr2=0x55b2af8b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.226+0000 7f5d4dffb640 0 client.25517 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.227+0000 7f5d735e5740 1 -- 172.21.15.129:0/100603718 shutdown_connections 2024-05-20T18:00:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.227+0000 7f5d735e5740 1 -- 172.21.15.129:0/100603718 wait complete. 2024-05-20T18:00:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.227+0000 7f5d735e5740 20 client.25517 trim_cache size 0 max 16384 2024-05-20T18:00:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:00:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:00:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:00:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:00:59.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.234+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1021770283 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2d7c0a7310 msgr2=0x7f2d7c0a9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:59.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.234+0000 7f2d9a2de640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2d7c0a7310 0x7f2d7c0a9820 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2d94115140 tx=0x7f2d84006c90 comp rx=0 tx=0).stop 2024-05-20T18:00:59.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.234+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1021770283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d941095e0 msgr2=0x7f2d941a58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:59.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.234+0000 7f2d9a2de640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d941095e0 0x7f2d941a58b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c02aee0 tx=0x7f2d8c031f70 comp rx=0 tx=0).stop 2024-05-20T18:00:59.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.235+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1021770283 shutdown_connections 2024-05-20T18:00:59.233 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.235+0000 7f2d9a2de640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2d7c0a7310 0x7f2d7c0a9820 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.233 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.235+0000 7f2d9a2de640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d9410e630 0x7f2d9419fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.233 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.235+0000 7f2d9a2de640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d9410db70 0x7f2d941a5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.233 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.235+0000 7f2d9a2de640 1 --2- 172.21.15.129:0/1021770283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d941095e0 0x7f2d941a58b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.235+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1021770283 >> 172.21.15.129:0/1021770283 conn(0x7f2d9406ea40 msgr2=0x7f2d9410cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.236+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1021770283 shutdown_connections 2024-05-20T18:00:59.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:00:59.236+0000 7f2d9a2de640 1 -- 172.21.15.129:0/1021770283 wait complete. 2024-05-20T18:00:59.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.288+0000 7f5d735e5740 1 Processor -- start 2024-05-20T18:00:59.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.288+0000 7f5d735e5740 1 -- start start 2024-05-20T18:00:59.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.289+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 0x55b2af8b9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:59.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.290+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8bb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:59.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.290+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 0x55b2af8bcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:59.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.290+0000 7f5d735e5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4800c920 con 0x55b2af512390 2024-05-20T18:00:59.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.290+0000 7f5d6a4d4640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 0x55b2af8b9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:59.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.290+0000 7f5d6a4d4640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 0x55b2af8b9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:49426/0 (socket says 172.21.15.129:49426) 2024-05-20T18:00:59.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.290+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/471966089 learned_addr learned my addr 172.21.15.129:0/471966089 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:59.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.290+0000 7f5d735e5740 1 -- 172.21.15.129:0/471966089 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4800f1a0 con 0x55b2af4e94d0 2024-05-20T18:00:59.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.290+0000 7f5d735e5740 1 -- 172.21.15.129:0/471966089 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4800cf90 con 0x55b2af4d0f60 2024-05-20T18:00:59.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.291+0000 7f5d6acd5640 1 --2- 172.21.15.129:0/471966089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 0x55b2af8bcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:59.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.291+0000 7f5d69cd3640 1 --2- 172.21.15.129:0/471966089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8bb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:59.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.292+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/471966089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 msgr2=0x55b2af8bb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:59.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.292+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/471966089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8bb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.292+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/471966089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 msgr2=0x55b2af8bcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:59.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.292+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/471966089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 0x55b2af8bcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.292+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/471966089 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2af8b94c0 con 0x55b2af4d0f60 2024-05-20T18:00:59.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.293+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/471966089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 0x55b2af8b9eb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5d6009c210 tx=0x7f5d6009be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7db5a4414990c8a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:59.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.295+0000 7f5d4ffff640 1 -- 172.21.15.129:0/471966089 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d600a08c0 con 0x55b2af4d0f60 2024-05-20T18:00:59.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.295+0000 7f5d4ffff640 1 -- 172.21.15.129:0/471966089 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d600a5180 con 0x55b2af4d0f60 2024-05-20T18:00:59.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.297+0000 7f5d735e5740 1 -- 172.21.15.129:0/471966089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 msgr2=0x55b2af8b9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:59.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.297+0000 7f5d735e5740 1 --2- 172.21.15.129:0/471966089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 0x55b2af8b9eb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5d6009c210 tx=0x7f5d6009be70 comp rx=0 tx=0).stop 2024-05-20T18:00:59.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.297+0000 7f5d735e5740 1 -- 172.21.15.129:0/471966089 shutdown_connections 2024-05-20T18:00:59.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.297+0000 7f5d735e5740 1 --2- 172.21.15.129:0/471966089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8bb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.297+0000 7f5d735e5740 1 --2- 172.21.15.129:0/471966089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 0x55b2af8bcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.297+0000 7f5d735e5740 1 --2- 172.21.15.129:0/471966089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af4d0f60 0x55b2af8b9eb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.297+0000 7f5d735e5740 1 -- 172.21.15.129:0/471966089 >> 172.21.15.129:0/471966089 conn(0x55b2af5064e0 msgr2=0x55b2af8bc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:59.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.298+0000 7f5d735e5740 1 -- 172.21.15.129:0/471966089 shutdown_connections 2024-05-20T18:00:59.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.298+0000 7f5d735e5740 1 -- 172.21.15.129:0/471966089 wait complete. 2024-05-20T18:00:59.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.298+0000 7f5d735e5740 1 -- start start 2024-05-20T18:00:59.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.299+0000 7f5d735e5740 1 Processor -- start 2024-05-20T18:00:59.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.300+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8be690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:59.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.300+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 0x55b2af8bebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:59.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.301+0000 7f5d735e5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af8bf110 0x55b2af934240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:59.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.301+0000 7f5d735e5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4800d660 con 0x55b2af512390 2024-05-20T18:00:59.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.301+0000 7f5d735e5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4800ead0 con 0x55b2af4e94d0 2024-05-20T18:00:59.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.301+0000 7f5d735e5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4800dd30 con 0x55b2af8bf110 2024-05-20T18:00:59.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.301+0000 7f5d735e5740 10 client.0 _ll_register_callbacks cb 0x7f5d600cd6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:00:59.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.301+0000 7f5d6a4d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8be690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:59.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.301+0000 7f5d6a4d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8be690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38960/0 (socket says 172.21.15.129:38960) 2024-05-20T18:00:59.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.301+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/1978059092 learned_addr learned my addr 172.21.15.129:0/1978059092 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:59.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.304+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/1978059092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af8bf110 msgr2=0x55b2af934240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:59.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.304+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/1978059092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af8bf110 0x55b2af934240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.304+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/1978059092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 msgr2=0x55b2af8bebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:00:59.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.304+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/1978059092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 0x55b2af8bebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.304+0000 7f5d6a4d4640 1 -- 172.21.15.129:0/1978059092 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2af934780 con 0x55b2af4e94d0 2024-05-20T18:00:59.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.321+0000 7f5d6a4d4640 1 --2- 172.21.15.129:0/1978059092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8be690 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5d6009a7c0 tx=0x7f5d6000ff90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8e8b46a2799c336 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:59.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.321+0000 7f5d4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:00:59.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.328+0000 7f5d4dffb640 1 -- 172.21.15.129:0/1978059092 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d600cdbd0 con 0x55b2af4e94d0 2024-05-20T18:00:59.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.328+0000 7f5d4dffb640 1 -- 172.21.15.129:0/1978059092 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5d60098460 con 0x55b2af4e94d0 2024-05-20T18:00:59.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.328+0000 7f5d735e5740 1 -- 172.21.15.129:0/1978059092 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b2af79b6f0 con 0x55b2af4e94d0 2024-05-20T18:00:59.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.330+0000 7f5d4dffb640 1 -- 172.21.15.129:0/1978059092 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5d600a08c0 con 0x55b2af4e94d0 2024-05-20T18:00:59.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.330+0000 7f5d735e5740 10 client.25523 fetch_fsmap learned FSMap version 22 2024-05-20T18:00:59.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.330+0000 7f5d735e5740 1 -- 172.21.15.129:0/1978059092 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b2af935a00 con 0x55b2af4e94d0 2024-05-20T18:00:59.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.337+0000 7f5d4dffb640 1 -- 172.21.15.129:0/1978059092 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b2af79b6f0 con 0x55b2af4e94d0 2024-05-20T18:00:59.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.337+0000 7f5d4dffb640 1 -- 172.21.15.129:0/1978059092 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f5d600a1a50 con 0x55b2af4e94d0 2024-05-20T18:00:59.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.340+0000 7f5d735e5740 10 client.25523 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:00:59.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.340+0000 7f5d735e5740 10 client.25523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:00:59.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.340+0000 7f5d735e5740 20 client.25523 populate_metadata read hostname 'smithi129' 2024-05-20T18:00:59.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.340+0000 7f5d735e5740 1 --2- 172.21.15.129:0/1978059092 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2af935e30 0x55b2af938250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:59.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.340+0000 7f5d735e5740 4 client.25523 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:00:59.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.340+0000 7f5d735e5740 1 -- 172.21.15.129:0/1978059092 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b2af7e8040 con 0x55b2af935e30 2024-05-20T18:00:59.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.342+0000 7f5d69cd3640 1 --2- 172.21.15.129:0/1978059092 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2af935e30 0x55b2af938250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:59.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.357+0000 7f5d69cd3640 1 --2- 172.21.15.129:0/1978059092 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2af935e30 0x55b2af938250 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=359bc564b50a787c server_cookie=4413fe876907f473 in_seq=0 out_seq=0 2024-05-20T18:00:59.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.357+0000 7f5d4dffb640 10 client.25523 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:59.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.359+0000 7f5d4dffb640 1 -- 172.21.15.129:0/1978059092 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b2af7e8040 con 0x55b2af935e30 2024-05-20T18:00:59.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.359+0000 7f5d4dffb640 10 client.25523 handle_command_reply: tid=0 2024-05-20T18:00:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.359+0000 7f5d735e5740 1 client.25523 shutdown 2024-05-20T18:00:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.360+0000 7f5d735e5740 1 -- 172.21.15.129:0/1978059092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 msgr2=0x55b2af8be690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:59.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.360+0000 7f5d735e5740 1 --2- 172.21.15.129:0/1978059092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8be690 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5d6009a7c0 tx=0x7f5d6000ff90 comp rx=0 tx=0).stop 2024-05-20T18:00:59.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.360+0000 7f5d735e5740 1 -- 172.21.15.129:0/1978059092 shutdown_connections 2024-05-20T18:00:59.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.360+0000 7f5d735e5740 1 --2- 172.21.15.129:0/1978059092 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2af935e30 0x55b2af938250 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.360+0000 7f5d735e5740 1 --2- 172.21.15.129:0/1978059092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2af8bf110 0x55b2af934240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.360+0000 7f5d735e5740 1 --2- 172.21.15.129:0/1978059092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2af512390 0x55b2af8bebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.360+0000 7f5d735e5740 1 --2- 172.21.15.129:0/1978059092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2af4e94d0 0x55b2af8be690 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5d6009a7c0 tx=0x7f5d6000ff90 comp rx=0 tx=0).stop 2024-05-20T18:00:59.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.360+0000 7f5d735e5740 1 -- 172.21.15.129:0/1978059092 >> 172.21.15.129:0/1978059092 conn(0x55b2af5064e0 msgr2=0x55b2af8b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:59.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.360+0000 7f5d4dffb640 0 client.25523 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:00:59.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.360+0000 7f5d735e5740 1 -- 172.21.15.129:0/1978059092 shutdown_connections 2024-05-20T18:00:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.361+0000 7f5d735e5740 1 -- 172.21.15.129:0/1978059092 wait complete. 2024-05-20T18:00:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.361+0000 7f5d735e5740 20 client.25523 trim_cache size 0 max 16384 2024-05-20T18:00:59.360 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:00:59.360 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:00:59.360 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:00:59.360 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:00:59.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.374+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3070053628 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d500a2e40 msgr2=0x7f5d500a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:59.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.374+0000 7f5d6b4d6640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d500a2e40 0x7f5d500a5350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009f10 tx=0x7f5d540137d0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.374+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3070053628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d64109090 msgr2=0x7f5d6411f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:59.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.374+0000 7f5d6b4d6640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d64109090 0x7f5d6411f180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000d950 tx=0x7f5d6000de10 comp rx=0 tx=0).stop 2024-05-20T18:00:59.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.375+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3070053628 shutdown_connections 2024-05-20T18:00:59.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.375+0000 7f5d6b4d6640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d500a2e40 0x7f5d500a5350 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.375+0000 7f5d6b4d6640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d6410f270 0x7f5d641188d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.375+0000 7f5d6b4d6640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d6410e890 0x7f5d641182a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.375+0000 7f5d6b4d6640 1 --2- 172.21.15.129:0/3070053628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d64109090 0x7f5d6411f180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.375+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3070053628 >> 172.21.15.129:0/3070053628 conn(0x7f5d6406d580 msgr2=0x7f5d640728c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:59.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.376+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3070053628 shutdown_connections 2024-05-20T18:00:59.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.376+0000 7f5d6b4d6640 1 -- 172.21.15.129:0/3070053628 wait complete. 2024-05-20T18:00:59.452 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:00:59.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:00:59.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.837+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3533059557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe7010f3b0 msgr2=0x7fbe70108ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:59.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.837+0000 7fbe6f59e640 1 --2- 172.21.15.129:0/3533059557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe7010f3b0 0x7fbe70108ff0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009860 tx=0x7fbe6402a380 comp rx=0 tx=0).stop 2024-05-20T18:00:59.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.840+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3533059557 shutdown_connections 2024-05-20T18:00:59.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.840+0000 7fbe6f59e640 1 --2- 172.21.15.129:0/3533059557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe70109660 0x7fbe70115a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.840+0000 7fbe6f59e640 1 --2- 172.21.15.129:0/3533059557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe7010f3b0 0x7fbe70108ff0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.840+0000 7fbe6f59e640 1 --2- 172.21.15.129:0/3533059557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe7010ef10 0x7fbe70108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.840+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3533059557 >> 172.21.15.129:0/3533059557 conn(0x7fbe70075b10 msgr2=0x7fbe70075f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:00:59.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.846+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3533059557 shutdown_connections 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.846+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3533059557 wait complete. 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.847+0000 7fbe6f59e640 1 Processor -- start 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.847+0000 7fbe6f59e640 1 -- start start 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.848+0000 7fbe6f59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe70109660 0x7fbe7006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.848+0000 7fbe6f59e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe7010ef10 0x7fbe7006df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.848+0000 7fbe6f59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe7010f3b0 0x7fbe7006e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.848+0000 7fbe6e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe70109660 0x7fbe7006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.848+0000 7fbe6e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe70109660 0x7fbe7006da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45182/0 (socket says 172.21.15.129:45182) 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.848+0000 7fbe6e59c640 1 -- 172.21.15.129:0/3509869598 learned_addr learned my addr 172.21.15.129:0/3509869598 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.848+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7006bf20 con 0x7fbe70109660 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.848+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7006bda0 con 0x7fbe7010f3b0 2024-05-20T18:00:59.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.848+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7006c0a0 con 0x7fbe7010ef10 2024-05-20T18:00:59.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.849+0000 7fbe6ed9d640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe7010f3b0 0x7fbe7006e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:59.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.849+0000 7fbe6e59c640 1 -- 172.21.15.129:0/3509869598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe7010ef10 msgr2=0x7fbe7006df90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:00:59.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.849+0000 7fbe6e59c640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe7010ef10 0x7fbe7006df90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.849+0000 7fbe6e59c640 1 -- 172.21.15.129:0/3509869598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe7010f3b0 msgr2=0x7fbe7006e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:00:59.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.849+0000 7fbe6e59c640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe7010f3b0 0x7fbe7006e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:00:59.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.849+0000 7fbe6e59c640 1 -- 172.21.15.129:0/3509869598 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe7011e980 con 0x7fbe70109660 2024-05-20T18:00:59.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.850+0000 7fbe6e59c640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe70109660 0x7fbe7006da50 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800b9e0 tx=0x7fbe6800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:59.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.855+0000 7fbe5f7fe640 1 -- 172.21.15.129:0/3509869598 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe68019070 con 0x7fbe70109660 2024-05-20T18:00:59.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.855+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe7011ec10 con 0x7fbe70109660 2024-05-20T18:00:59.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.856+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe7011f160 con 0x7fbe70109660 2024-05-20T18:00:59.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.856+0000 7fbe5f7fe640 1 -- 172.21.15.129:0/3509869598 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe6800cba0 con 0x7fbe70109660 2024-05-20T18:00:59.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.856+0000 7fbe5f7fe640 1 -- 172.21.15.129:0/3509869598 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe68012670 con 0x7fbe70109660 2024-05-20T18:00:59.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.862+0000 7fbe5f7fe640 1 -- 172.21.15.129:0/3509869598 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fbe6801c070 con 0x7fbe70109660 2024-05-20T18:00:59.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.863+0000 7fbe5f7fe640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbe580a2e80 0x7fbe580a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:00:59.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.863+0000 7fbe6dd9b640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbe580a2e80 0x7fbe580a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:00:59.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.863+0000 7fbe5f7fe640 1 -- 172.21.15.129:0/3509869598 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fbe680ce840 con 0x7fbe70109660 2024-05-20T18:00:59.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.864+0000 7fbe6dd9b640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbe580a2e80 0x7fbe580a5390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbe64035040 tx=0x7fbe64000bc0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:00:59.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.866+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe50005290 con 0x7fbe70109660 2024-05-20T18:00:59.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:00:59.878+0000 7fbe5f7fe640 1 -- 172.21.15.129:0/3509869598 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbe6809b560 con 0x7fbe70109660 2024-05-20T18:01:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.702+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe50005580 con 0x7fbe70109660 2024-05-20T18:01:00.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.711+0000 7fbe5f7fe640 1 -- 172.21.15.129:0/3509869598 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11829 (secure 0 0 0) 0x7fbe6809ae60 con 0x7fbe70109660 2024-05-20T18:01:00.711 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:01:00.711 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:59:17.074606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25229":{"gid":25229,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1860679091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1860679091},{"type":"v1","addr":"172.21.15.154:6837","nonce":1860679091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:01:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-05-20T18:01:00.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.725+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbe580a2e80 msgr2=0x7fbe580a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:00.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.725+0000 7fbe6f59e640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbe580a2e80 0x7fbe580a5390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbe64035040 tx=0x7fbe64000bc0 comp rx=0 tx=0).stop 2024-05-20T18:01:00.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.725+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe70109660 msgr2=0x7fbe7006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:00.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.725+0000 7fbe6f59e640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe70109660 0x7fbe7006da50 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800b9e0 tx=0x7fbe6800bea0 comp rx=0 tx=0).stop 2024-05-20T18:01:00.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.725+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 shutdown_connections 2024-05-20T18:01:00.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.725+0000 7fbe6f59e640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbe580a2e80 0x7fbe580a5390 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:00.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.725+0000 7fbe6f59e640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe7010f3b0 0x7fbe7006e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:00.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.725+0000 7fbe6f59e640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbe7010ef10 0x7fbe7006df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:00.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.725+0000 7fbe6f59e640 1 --2- 172.21.15.129:0/3509869598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe70109660 0x7fbe7006da50 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:00.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.725+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 >> 172.21.15.129:0/3509869598 conn(0x7fbe70075b10 msgr2=0x7fbe70106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:00.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.726+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 shutdown_connections 2024-05-20T18:01:00.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:00.726+0000 7fbe6f59e640 1 -- 172.21.15.129:0/3509869598 wait complete. 2024-05-20T18:01:00.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:01:01.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:00 smithi154 ceph-mon[36482]: pgmap v77: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 940 MiB/s wr, 2.59k op/s 2024-05-20T18:01:01.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:00 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3509869598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:01:01.167 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:00 smithi129 ceph-mon[32029]: pgmap v77: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 940 MiB/s wr, 2.59k op/s 2024-05-20T18:01:01.167 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:00 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3509869598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:01:01.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:00 smithi204 ceph-mon[37076]: pgmap v77: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 940 MiB/s wr, 2.59k op/s 2024-05-20T18:01:01.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:00 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3509869598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:01:01.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.368+0000 7f711e75e640 1 -- 172.21.15.129:0/3714328617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7118119770 msgr2=0x7f711811bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:01.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.368+0000 7f711e75e640 1 --2- 172.21.15.129:0/3714328617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7118119770 0x7f711811bb90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f711001acb0 tx=0x7f711003b990 comp rx=0 tx=0).stop 2024-05-20T18:01:01.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.371+0000 7f711e75e640 1 -- 172.21.15.129:0/3714328617 shutdown_connections 2024-05-20T18:01:01.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.371+0000 7f711e75e640 1 --2- 172.21.15.129:0/3714328617 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7118119770 0x7f711811bb90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.371+0000 7f711e75e640 1 --2- 172.21.15.129:0/3714328617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f711811c0d0 0x7f711811e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.371+0000 7f711e75e640 1 --2- 172.21.15.129:0/3714328617 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7118108f70 0x7f7118109380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.371+0000 7f711e75e640 1 -- 172.21.15.129:0/3714328617 >> 172.21.15.129:0/3714328617 conn(0x7f711806d5b0 msgr2=0x7f711806da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:01.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.373+0000 7f711e75e640 1 -- 172.21.15.129:0/3714328617 shutdown_connections 2024-05-20T18:01:01.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.373+0000 7f711e75e640 1 -- 172.21.15.129:0/3714328617 wait complete. 2024-05-20T18:01:01.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.374+0000 7f711e75e640 1 Processor -- start 2024-05-20T18:01:01.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.375+0000 7f711e75e640 1 -- start start 2024-05-20T18:01:01.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.376+0000 7f711e75e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7118108f70 0x7f7118115510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.376+0000 7f711e75e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7118119770 0x7f7118115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.376+0000 7f711e75e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f711811c0d0 0x7f7118115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.376+0000 7f711e75e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71181139e0 con 0x7f711811c0d0 2024-05-20T18:01:01.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.376+0000 7f711e75e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7118113860 con 0x7f7118119770 2024-05-20T18:01:01.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.376+0000 7f711e75e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7118113b60 con 0x7f7118108f70 2024-05-20T18:01:01.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.377+0000 7f71177fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7118119770 0x7f7118115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.377+0000 7f71177fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7118119770 0x7f7118115a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:58930/0 (socket says 172.21.15.129:58930) 2024-05-20T18:01:01.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.377+0000 7f71177fe640 1 -- 172.21.15.129:0/1735258405 learned_addr learned my addr 172.21.15.129:0/1735258405 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:01.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.378+0000 7f7117fff640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7118108f70 0x7f7118115510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.379+0000 7f71177fe640 1 -- 172.21.15.129:0/1735258405 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7118108f70 msgr2=0x7f7118115510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:01.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.379+0000 7f71177fe640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7118108f70 0x7f7118115510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.379+0000 7f71177fe640 1 -- 172.21.15.129:0/1735258405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f711811c0d0 msgr2=0x7f7118115f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:01.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.379+0000 7f71177fe640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f711811c0d0 0x7f7118115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.380+0000 7f71177fe640 1 -- 172.21.15.129:0/1735258405 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71181abea0 con 0x7f7118119770 2024-05-20T18:01:01.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.383+0000 7f71177fe640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7118119770 0x7f7118115a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7110006010 tx=0x7f7110003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:01.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.383+0000 7f71157fa640 1 -- 172.21.15.129:0/1735258405 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110045030 con 0x7f7118119770 2024-05-20T18:01:01.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.384+0000 7f711e75e640 1 -- 172.21.15.129:0/1735258405 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71181ac130 con 0x7f7118119770 2024-05-20T18:01:01.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.384+0000 7f711e75e640 1 -- 172.21.15.129:0/1735258405 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71181ac620 con 0x7f7118119770 2024-05-20T18:01:01.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.384+0000 7f71157fa640 1 -- 172.21.15.129:0/1735258405 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f711004f070 con 0x7f7118119770 2024-05-20T18:01:01.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.384+0000 7f71157fa640 1 -- 172.21.15.129:0/1735258405 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110004860 con 0x7f7118119770 2024-05-20T18:01:01.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.407+0000 7f7125608740 1 Processor -- start 2024-05-20T18:01:01.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.413+0000 7f7125608740 1 -- start start 2024-05-20T18:01:01.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.413+0000 7f7125608740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6146f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.413+0000 7f7125608740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d7f4d0 0x5579b61474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.413+0000 7f7125608740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d876b0 0x5579b6147a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.413+0000 7f7125608740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5579b60acc40 con 0x5579b5d7f4d0 2024-05-20T18:01:01.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.413+0000 7f7125608740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5579b6054580 con 0x5579b5da8390 2024-05-20T18:01:01.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.413+0000 7f7125608740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5579b60a8c00 con 0x5579b5d876b0 2024-05-20T18:01:01.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.414+0000 7f711ccfb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6146f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.414+0000 7f711ccfb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6146f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:58946/0 (socket says 172.21.15.129:58946) 2024-05-20T18:01:01.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.414+0000 7f711ccfb640 1 -- 172.21.15.129:0/84743439 learned_addr learned my addr 172.21.15.129:0/84743439 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:01.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.414+0000 7f71177fe640 1 --2- 172.21.15.129:0/84743439 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d876b0 0x5579b6147a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.415+0000 7f711ccfb640 1 -- 172.21.15.129:0/84743439 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d876b0 msgr2=0x5579b6147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:01.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.415+0000 7f711ccfb640 1 --2- 172.21.15.129:0/84743439 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d876b0 0x5579b6147a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.415+0000 7f711ccfb640 1 -- 172.21.15.129:0/84743439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d7f4d0 msgr2=0x5579b61474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:01.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.415+0000 7f711ccfb640 1 --2- 172.21.15.129:0/84743439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d7f4d0 0x5579b61474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.415+0000 7f711ccfb640 1 -- 172.21.15.129:0/84743439 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579b5dc03f0 con 0x5579b5da8390 2024-05-20T18:01:01.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.416+0000 7f711ccfb640 1 --2- 172.21.15.129:0/84743439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6146f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f710c009770 tx=0x7f710c009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fff024e9de73570 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:01.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.422+0000 7f71157fa640 1 -- 172.21.15.129:0/1735258405 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7110048020 con 0x7f7118119770 2024-05-20T18:01:01.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.422+0000 7f70fe7fc640 1 -- 172.21.15.129:0/84743439 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c015020 con 0x5579b5da8390 2024-05-20T18:01:01.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.423+0000 7f70fe7fc640 1 -- 172.21.15.129:0/84743439 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f710c011420 con 0x5579b5da8390 2024-05-20T18:01:01.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.423+0000 7f71157fa640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f71000a2f00 0x7f71000a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.423+0000 7f71157fa640 1 -- 172.21.15.129:0/1735258405 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f7110101650 con 0x7f7118119770 2024-05-20T18:01:01.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.424+0000 7f7125608740 1 -- 172.21.15.129:0/84743439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 msgr2=0x5579b6146f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:01.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.424+0000 7f7125608740 1 --2- 172.21.15.129:0/84743439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6146f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f710c009770 tx=0x7f710c009c30 comp rx=0 tx=0).stop 2024-05-20T18:01:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.426+0000 7f7117fff640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f71000a2f00 0x7f71000a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.431+0000 7f7125608740 1 -- 172.21.15.129:0/84743439 shutdown_connections 2024-05-20T18:01:01.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.431+0000 7f7125608740 1 --2- 172.21.15.129:0/84743439 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d876b0 0x5579b6147a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.431+0000 7f7125608740 1 --2- 172.21.15.129:0/84743439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d7f4d0 0x5579b61474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.431+0000 7f7125608740 1 --2- 172.21.15.129:0/84743439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6146f90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.431+0000 7f7125608740 1 -- 172.21.15.129:0/84743439 >> 172.21.15.129:0/84743439 conn(0x5579b5d9c4e0 msgr2=0x5579b6143740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:01.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.431+0000 7f7125608740 1 -- 172.21.15.129:0/84743439 shutdown_connections 2024-05-20T18:01:01.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.431+0000 7f7125608740 1 -- 172.21.15.129:0/84743439 wait complete. 2024-05-20T18:01:01.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.433+0000 7f7125608740 1 -- start start 2024-05-20T18:01:01.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.434+0000 7f7125608740 1 Processor -- start 2024-05-20T18:01:01.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.434+0000 7f7117fff640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f71000a2f00 0x7f71000a5410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x5579b6148550 tx=0x7f7108008040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.435+0000 7f7125608740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d7f4d0 0x5579b6151410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.435+0000 7f7125608740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d876b0 0x5579b6151950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.435+0000 7f7125608740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b615d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.435+0000 7f7125608740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5579b6154330 con 0x5579b5d876b0 2024-05-20T18:01:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.435+0000 7f7125608740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5579b6067c40 con 0x5579b5da8390 2024-05-20T18:01:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.435+0000 7f7125608740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5579b61544b0 con 0x5579b5d7f4d0 2024-05-20T18:01:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.435+0000 7f7125608740 10 client.0 _ll_register_callbacks cb 0x5579b5dbfbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.436+0000 7f711ccfb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b615d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.436+0000 7f711ccfb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b615d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:58962/0 (socket says 172.21.15.129:58962) 2024-05-20T18:01:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.436+0000 7f711ccfb640 1 -- 172.21.15.129:0/3713132911 learned_addr learned my addr 172.21.15.129:0/3713132911 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.436+0000 7f7117fff640 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d7f4d0 0x5579b6151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.437+0000 7f71177fe640 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d876b0 0x5579b6151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.439+0000 7f7117fff640 1 -- 172.21.15.129:0/3713132911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 msgr2=0x5579b615d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.439+0000 7f7117fff640 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b615d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.439+0000 7f7117fff640 1 -- 172.21.15.129:0/3713132911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d876b0 msgr2=0x5579b6151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.439+0000 7f7117fff640 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d876b0 0x5579b6151950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.439+0000 7f7117fff640 1 -- 172.21.15.129:0/3713132911 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579b5e8a7d0 con 0x5579b5d7f4d0 2024-05-20T18:01:01.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.441+0000 7f7117fff640 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d7f4d0 0x5579b6151410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7108006e10 tx=0x7f71080046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9ca3bd89118d807 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:01.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.442+0000 7f70f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:01:01.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.442+0000 7f70f77fe640 1 -- 172.21.15.129:0/3713132911 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7108024060 con 0x5579b5d7f4d0 2024-05-20T18:01:01.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.442+0000 7f70f77fe640 1 -- 172.21.15.129:0/3713132911 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f710801f400 con 0x5579b5d7f4d0 2024-05-20T18:01:01.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.443+0000 7f7125608740 1 -- 172.21.15.129:0/3713132911 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f710c011420 con 0x5579b5d7f4d0 2024-05-20T18:01:01.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.445+0000 7f70f77fe640 1 -- 172.21.15.129:0/3713132911 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7108014e10 con 0x5579b5d7f4d0 2024-05-20T18:01:01.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.445+0000 7f7125608740 10 client.25564 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:01.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.445+0000 7f7125608740 1 -- 172.21.15.129:0/3713132911 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5579b61607d0 con 0x5579b5d7f4d0 2024-05-20T18:01:01.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.447+0000 7f70f77fe640 1 -- 172.21.15.129:0/3713132911 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f710c011420 con 0x5579b5d7f4d0 2024-05-20T18:01:01.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.447+0000 7f70f77fe640 1 -- 172.21.15.129:0/3713132911 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f7108013070 con 0x5579b5d7f4d0 2024-05-20T18:01:01.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.448+0000 7f7125608740 10 client.25564 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:01.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.448+0000 7f7125608740 10 client.25564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:01.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.448+0000 7f7125608740 20 client.25564 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:01.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.448+0000 7f7125608740 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5579b5d66f60 0x5579b6163cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.448+0000 7f7125608740 4 client.25564 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:01:01.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.448+0000 7f7125608740 1 -- 172.21.15.129:0/3713132911 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579b60ba370 con 0x5579b5d66f60 2024-05-20T18:01:01.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.451+0000 7f71177fe640 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5579b5d66f60 0x5579b6163cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.456+0000 7f70f77fe640 10 client.25564 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:01.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.456+0000 7f71177fe640 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5579b5d66f60 0x5579b6163cc0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d5a6bd2e151dc68 server_cookie=8d408af2dbc4dfad in_seq=0 out_seq=0 2024-05-20T18:01:01.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.461+0000 7f70f77fe640 1 -- 172.21.15.129:0/3713132911 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5579b60ba370 con 0x5579b5d66f60 2024-05-20T18:01:01.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.461+0000 7f70f77fe640 10 client.25564 handle_command_reply: tid=0 2024-05-20T18:01:01.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.461+0000 7f7125608740 1 client.25564 shutdown 2024-05-20T18:01:01.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.465+0000 7f7125608740 1 -- 172.21.15.129:0/3713132911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d7f4d0 msgr2=0x5579b6151410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:01.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.465+0000 7f7125608740 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d7f4d0 0x5579b6151410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7108006e10 tx=0x7f71080046c0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.468+0000 7f7125608740 1 -- 172.21.15.129:0/3713132911 shutdown_connections 2024-05-20T18:01:01.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.468+0000 7f7125608740 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5579b5d66f60 0x5579b6163cc0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.468+0000 7f7125608740 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b615d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.468+0000 7f7125608740 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d876b0 0x5579b6151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.468+0000 7f7125608740 1 --2- 172.21.15.129:0/3713132911 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d7f4d0 0x5579b6151410 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.468+0000 7f7125608740 1 -- 172.21.15.129:0/3713132911 >> 172.21.15.129:0/3713132911 conn(0x5579b5d9c4e0 msgr2=0x5579b6148cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:01.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.468+0000 7f70f77fe640 0 client.25564 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:01.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.469+0000 7f7125608740 1 -- 172.21.15.129:0/3713132911 shutdown_connections 2024-05-20T18:01:01.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.469+0000 7f7125608740 1 -- 172.21.15.129:0/3713132911 wait complete. 2024-05-20T18:01:01.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.469+0000 7f7125608740 20 client.25564 trim_cache size 0 max 16384 2024-05-20T18:01:01.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.561+0000 7f7125608740 1 Processor -- start 2024-05-20T18:01:01.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.565+0000 7f7125608740 1 -- start start 2024-05-20T18:01:01.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.566+0000 7f7125608740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d7f4d0 0x5579b615d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.566+0000 7f7125608740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d876b0 0x5579b61694c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.566+0000 7f7125608740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6169a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.567+0000 7f7125608740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f710400c8d0 con 0x5579b5d876b0 2024-05-20T18:01:01.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.567+0000 7f7125608740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f710400f150 con 0x5579b5da8390 2024-05-20T18:01:01.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.567+0000 7f7125608740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f710400cf40 con 0x5579b5d7f4d0 2024-05-20T18:01:01.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.568+0000 7f711ccfb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6169a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.568+0000 7f711ccfb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6169a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:58978/0 (socket says 172.21.15.129:58978) 2024-05-20T18:01:01.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.568+0000 7f711ccfb640 1 -- 172.21.15.129:0/1029614223 learned_addr learned my addr 172.21.15.129:0/1029614223 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:01.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.570+0000 7f711ccfb640 1 -- 172.21.15.129:0/1029614223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d7f4d0 msgr2=0x5579b615d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:01.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.570+0000 7f711ccfb640 1 --2- 172.21.15.129:0/1029614223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d7f4d0 0x5579b615d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.570+0000 7f711ccfb640 1 -- 172.21.15.129:0/1029614223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d876b0 msgr2=0x5579b61694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:01.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.570+0000 7f711ccfb640 1 --2- 172.21.15.129:0/1029614223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d876b0 0x5579b61694c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.570+0000 7f711ccfb640 1 -- 172.21.15.129:0/1029614223 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579b61487a0 con 0x5579b5da8390 2024-05-20T18:01:01.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.572+0000 7f711ccfb640 1 --2- 172.21.15.129:0/1029614223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6169a00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f710c009770 tx=0x7f710c011e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c50f1b0bb1380e72 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:01.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.577+0000 7f70fdffb640 1 -- 172.21.15.129:0/1029614223 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c015040 con 0x5579b5da8390 2024-05-20T18:01:01.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.578+0000 7f70fdffb640 1 -- 172.21.15.129:0/1029614223 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f710c027070 con 0x5579b5da8390 2024-05-20T18:01:01.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.582+0000 7f7125608740 1 -- 172.21.15.129:0/1029614223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 msgr2=0x5579b6169a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:01.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.582+0000 7f7125608740 1 --2- 172.21.15.129:0/1029614223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6169a00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f710c009770 tx=0x7f710c011e80 comp rx=0 tx=0).stop 2024-05-20T18:01:01.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.584+0000 7f7125608740 1 -- 172.21.15.129:0/1029614223 shutdown_connections 2024-05-20T18:01:01.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.584+0000 7f7125608740 1 --2- 172.21.15.129:0/1029614223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5da8390 0x5579b6169a00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.584+0000 7f7125608740 1 --2- 172.21.15.129:0/1029614223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5d876b0 0x5579b61694c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.584+0000 7f7125608740 1 --2- 172.21.15.129:0/1029614223 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d7f4d0 0x5579b615d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.584+0000 7f7125608740 1 -- 172.21.15.129:0/1029614223 >> 172.21.15.129:0/1029614223 conn(0x5579b5d9c4e0 msgr2=0x5579b6147750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:01.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.585+0000 7f7125608740 1 -- 172.21.15.129:0/1029614223 shutdown_connections 2024-05-20T18:01:01.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.586+0000 7f7125608740 1 -- 172.21.15.129:0/1029614223 wait complete. 2024-05-20T18:01:01.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.587+0000 7f7125608740 1 -- start start 2024-05-20T18:01:01.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.588+0000 7f7125608740 1 Processor -- start 2024-05-20T18:01:01.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.589+0000 7f7125608740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5d7f4d0 0x5579b6159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.611+0000 7f7125608740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d876b0 0x5579b6154de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.611+0000 7f7125608740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5da8390 0x5579b6155320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.611+0000 7f7125608740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f710400d610 con 0x5579b5da8390 2024-05-20T18:01:01.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.611+0000 7f7125608740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f710400ea80 con 0x5579b5d7f4d0 2024-05-20T18:01:01.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.611+0000 7f7125608740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f710400dce0 con 0x5579b5d876b0 2024-05-20T18:01:01.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.611+0000 7f7125608740 10 client.0 _ll_register_callbacks cb 0x7f710800d9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:01.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.611+0000 7f7117fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5d7f4d0 0x5579b6159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.612+0000 7f7117fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5d7f4d0 0x5579b6159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:58992/0 (socket says 172.21.15.129:58992) 2024-05-20T18:01:01.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.612+0000 7f7117fff640 1 -- 172.21.15.129:0/3043556719 learned_addr learned my addr 172.21.15.129:0/3043556719 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:01.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.613+0000 7f71177fe640 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d876b0 0x5579b6154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.613+0000 7f7117fff640 1 -- 172.21.15.129:0/3043556719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d876b0 msgr2=0x5579b6154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:01.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.613+0000 7f7117fff640 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d876b0 0x5579b6154de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.613+0000 7f7117fff640 1 -- 172.21.15.129:0/3043556719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5da8390 msgr2=0x5579b6155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:01.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.613+0000 7f7117fff640 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5da8390 0x5579b6155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.613+0000 7f7117fff640 1 -- 172.21.15.129:0/3043556719 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579b6155860 con 0x5579b5d7f4d0 2024-05-20T18:01:01.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.617+0000 7f7117fff640 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5d7f4d0 0x5579b6159a40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f710801e7b0 tx=0x7f710801ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=930e6a88e2b148fd server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:01.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.622+0000 7f70f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:01:01.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.622+0000 7f70f77fe640 1 -- 172.21.15.129:0/3043556719 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710801e3a0 con 0x5579b5d7f4d0 2024-05-20T18:01:01.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.622+0000 7f7125608740 1 -- 172.21.15.129:0/3043556719 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f710c027070 con 0x5579b5d7f4d0 2024-05-20T18:01:01.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.622+0000 7f70f77fe640 1 -- 172.21.15.129:0/3043556719 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f7108005bd0 con 0x5579b5d7f4d0 2024-05-20T18:01:01.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.629+0000 7f70f77fe640 1 -- 172.21.15.129:0/3043556719 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7108024020 con 0x5579b5d7f4d0 2024-05-20T18:01:01.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.629+0000 7f7125608740 10 client.25544 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:01.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.629+0000 7f7125608740 1 -- 172.21.15.129:0/3043556719 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5579b61cbbf0 con 0x5579b5d7f4d0 2024-05-20T18:01:01.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.635+0000 7f70f77fe640 1 -- 172.21.15.129:0/3043556719 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f710c027070 con 0x5579b5d7f4d0 2024-05-20T18:01:01.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.638+0000 7f70f77fe640 1 -- 172.21.15.129:0/3043556719 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f7108031430 con 0x5579b5d7f4d0 2024-05-20T18:01:01.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.638+0000 7f7125608740 10 client.25544 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:01.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.638+0000 7f7125608740 10 client.25544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:01.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.638+0000 7f7125608740 20 client.25544 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:01.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.638+0000 7f7125608740 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5579b5d3c3a0 0x5579b61ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:01.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.638+0000 7f7125608740 4 client.25544 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:01:01.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.638+0000 7f7125608740 1 -- 172.21.15.129:0/3043556719 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5579b60b46c0 con 0x5579b5d3c3a0 2024-05-20T18:01:01.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.642+0000 7f71177fe640 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5579b5d3c3a0 0x5579b61ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:01.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.659+0000 7f71177fe640 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5579b5d3c3a0 0x5579b61ce3e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd6be5bc97c34130 server_cookie=7c8c6db3dcbc9a08 in_seq=0 out_seq=0 2024-05-20T18:01:01.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.660+0000 7f70f77fe640 10 client.25544 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:01.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.665+0000 7f70f77fe640 1 -- 172.21.15.129:0/3043556719 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5579b60b46c0 con 0x5579b5d3c3a0 2024-05-20T18:01:01.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.665+0000 7f70f77fe640 10 client.25544 handle_command_reply: tid=0 2024-05-20T18:01:01.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.665+0000 7f7125608740 1 client.25544 shutdown 2024-05-20T18:01:01.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.667+0000 7f7125608740 1 -- 172.21.15.129:0/3043556719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5d7f4d0 msgr2=0x5579b6159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:01.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.667+0000 7f7125608740 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5d7f4d0 0x5579b6159a40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f710801e7b0 tx=0x7f710801ee90 comp rx=0 tx=0).stop 2024-05-20T18:01:01.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.667+0000 7f7125608740 1 -- 172.21.15.129:0/3043556719 shutdown_connections 2024-05-20T18:01:01.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.667+0000 7f7125608740 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5579b5d3c3a0 0x5579b61ce3e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.667+0000 7f7125608740 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5579b5da8390 0x5579b6155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.667+0000 7f7125608740 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5579b5d876b0 0x5579b6154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.667+0000 7f7125608740 1 --2- 172.21.15.129:0/3043556719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5579b5d7f4d0 0x5579b6159a40 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f710801e7b0 tx=0x7f710801ee90 comp rx=0 tx=0).stop 2024-05-20T18:01:01.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.667+0000 7f7125608740 1 -- 172.21.15.129:0/3043556719 >> 172.21.15.129:0/3043556719 conn(0x5579b5d9c4e0 msgr2=0x5579b61528b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:01.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.667+0000 7f70f77fe640 0 client.25544 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:01.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.667+0000 7f7125608740 1 -- 172.21.15.129:0/3043556719 shutdown_connections 2024-05-20T18:01:01.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.667+0000 7f7125608740 1 -- 172.21.15.129:0/3043556719 wait complete. 2024-05-20T18:01:01.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.668+0000 7f7125608740 20 client.25544 trim_cache size 0 max 16384 2024-05-20T18:01:01.666 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:01:01.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.676+0000 7f711e75e640 1 -- 172.21.15.129:0/1735258405 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f71000a2f00 msgr2=0x7f71000a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:01.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.676+0000 7f711e75e640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f71000a2f00 0x7f71000a5410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x5579b6148550 tx=0x7f7108008040 comp rx=0 tx=0).stop 2024-05-20T18:01:01.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.676+0000 7f711e75e640 1 -- 172.21.15.129:0/1735258405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7118119770 msgr2=0x7f7118115a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:01.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.676+0000 7f711e75e640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7118119770 0x7f7118115a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7110006010 tx=0x7f7110003970 comp rx=0 tx=0).stop 2024-05-20T18:01:01.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.677+0000 7f711e75e640 1 -- 172.21.15.129:0/1735258405 shutdown_connections 2024-05-20T18:01:01.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.677+0000 7f711e75e640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f71000a2f00 0x7f71000a5410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.677+0000 7f711e75e640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7118119770 0x7f7118115a50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.677+0000 7f711e75e640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f711811c0d0 0x7f7118115f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.677+0000 7f711e75e640 1 --2- 172.21.15.129:0/1735258405 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7118108f70 0x7f7118115510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:01.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.677+0000 7f711e75e640 1 -- 172.21.15.129:0/1735258405 >> 172.21.15.129:0/1735258405 conn(0x7f711806d5b0 msgr2=0x7f7118119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:01.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.678+0000 7f711e75e640 1 -- 172.21.15.129:0/1735258405 shutdown_connections 2024-05-20T18:01:01.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:01.679+0000 7f711e75e640 1 -- 172.21.15.129:0/1735258405 wait complete. 2024-05-20T18:01:02.754 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:01:02.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:01:02.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:01:03.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.102+0000 7fb56833e640 1 -- 172.21.15.129:0/3566319831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb560101ac0 msgr2=0x7fb560101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.102+0000 7fb56833e640 1 --2- 172.21.15.129:0/3566319831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb560101ac0 0x7fb560101ed0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb55000d910 tx=0x7fb55002c380 comp rx=0 tx=0).stop 2024-05-20T18:01:03.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.102+0000 7fb56833e640 1 -- 172.21.15.129:0/3566319831 shutdown_connections 2024-05-20T18:01:03.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.102+0000 7fb56833e640 1 --2- 172.21.15.129:0/3566319831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb56010c350 0x7fb56010e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.102+0000 7fb56833e640 1 --2- 172.21.15.129:0/3566319831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb560101ac0 0x7fb560101ed0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.102+0000 7fb56833e640 1 --2- 172.21.15.129:0/3566319831 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5601079f0 0x7fb560101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.102+0000 7fb56833e640 1 -- 172.21.15.129:0/3566319831 >> 172.21.15.129:0/3566319831 conn(0x7fb560077130 msgr2=0x7fb560077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.102+0000 7fb56833e640 1 -- 172.21.15.129:0/3566319831 shutdown_connections 2024-05-20T18:01:03.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.102+0000 7fb56833e640 1 -- 172.21.15.129:0/3566319831 wait complete. 2024-05-20T18:01:03.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.103+0000 7fb56833e640 1 Processor -- start 2024-05-20T18:01:03.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.103+0000 7fb56833e640 1 -- start start 2024-05-20T18:01:03.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.103+0000 7fb56833e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb560101ac0 0x7fb5601a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.103+0000 7fb56833e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5601079f0 0x7fb5601a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.103+0000 7fb56833e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb56010c350 0x7fb56019b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.103+0000 7fb56833e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb560110c40 con 0x7fb56010c350 2024-05-20T18:01:03.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.103+0000 7fb56833e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb560110ac0 con 0x7fb560101ac0 2024-05-20T18:01:03.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.103+0000 7fb56833e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb560110dc0 con 0x7fb5601079f0 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5658d9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5601079f0 0x7fb5601a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5660da640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb560101ac0 0x7fb5601a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5658d9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5601079f0 0x7fb5601a1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:43690/0 (socket says 172.21.15.129:43690) 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5658d9640 1 -- 172.21.15.129:0/4066261985 learned_addr learned my addr 172.21.15.129:0/4066261985 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5668db640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb56010c350 0x7fb56019b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5658d9640 1 -- 172.21.15.129:0/4066261985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb560101ac0 msgr2=0x7fb5601a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5658d9640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb560101ac0 0x7fb5601a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5658d9640 1 -- 172.21.15.129:0/4066261985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb56010c350 msgr2=0x7fb56019b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5658d9640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb56010c350 0x7fb56019b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5658d9640 1 -- 172.21.15.129:0/4066261985 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56019bed0 con 0x7fb5601079f0 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5658d9640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5601079f0 0x7fb5601a1b00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb550006010 tx=0x7fb550034740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5577fe640 1 -- 172.21.15.129:0/4066261985 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb550034b40 con 0x7fb5601079f0 2024-05-20T18:01:03.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb56833e640 1 -- 172.21.15.129:0/4066261985 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56019c160 con 0x7fb5601079f0 2024-05-20T18:01:03.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5577fe640 1 -- 172.21.15.129:0/4066261985 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb550034ce0 con 0x7fb5601079f0 2024-05-20T18:01:03.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb56833e640 1 -- 172.21.15.129:0/4066261985 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5601a8410 con 0x7fb5601079f0 2024-05-20T18:01:03.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.104+0000 7fb5577fe640 1 -- 172.21.15.129:0/4066261985 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55003c420 con 0x7fb5601079f0 2024-05-20T18:01:03.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.107+0000 7fb5577fe640 1 -- 172.21.15.129:0/4066261985 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fb55000c050 con 0x7fb5601079f0 2024-05-20T18:01:03.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.107+0000 7fb56f1df740 1 Processor -- start 2024-05-20T18:01:03.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb56f1df740 1 -- start start 2024-05-20T18:01:03.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223fc8410 0x556224367620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223f9f550 0x556224367b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fa7730 0x5562243680a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb56f1df740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5562242cc990 con 0x556223f9f550 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb56f1df740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5562242745b0 con 0x556223fc8410 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb56f1df740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5562242c8950 con 0x556223fa7730 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb5668db640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223fc8410 0x556224367620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb5577fe640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb54c0a2d00 0x7fb54c0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb5668db640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223fc8410 0x556224367620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59022/0 (socket says 172.21.15.129:59022) 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb5668db640 1 -- 172.21.15.129:0/1545402921 learned_addr learned my addr 172.21.15.129:0/1545402921 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb5660da640 1 --2- 172.21.15.129:0/1545402921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223f9f550 0x556224367b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb5658d9640 1 --2- 172.21.15.129:0/1545402921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fa7730 0x5562243680a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.108+0000 7fb5668db640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb54c0a2d00 0x7fb54c0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.109+0000 7fb5668db640 1 -- 172.21.15.129:0/1545402921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fa7730 msgr2=0x5562243680a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.109+0000 7fb5668db640 1 --2- 172.21.15.129:0/1545402921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fa7730 0x5562243680a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.109+0000 7fb5668db640 1 -- 172.21.15.129:0/1545402921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223f9f550 msgr2=0x556224367b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.109+0000 7fb5668db640 1 --2- 172.21.15.129:0/1545402921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223f9f550 0x556224367b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.109+0000 7fb5668db640 1 -- 172.21.15.129:0/1545402921 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562240a87e0 con 0x556223fc8410 2024-05-20T18:01:03.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.109+0000 7fb5577fe640 1 -- 172.21.15.129:0/4066261985 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fb5500eec20 con 0x7fb5601079f0 2024-05-20T18:01:03.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.109+0000 7fb5668db640 1 --2- 172.21.15.129:0/1545402921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223fc8410 0x556224367620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb558005c40 tx=0x7fb55800f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e089dc7773ae57da server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.109+0000 7fb5668db640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb54c0a2d00 0x7fb54c0a5210 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb55800f8c0 tx=0x7fb55801f000 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.109+0000 7fb554bfe640 1 -- 172.21.15.129:0/1545402921 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55800fd30 con 0x556223fc8410 2024-05-20T18:01:03.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.109+0000 7fb554bfe640 1 -- 172.21.15.129:0/1545402921 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb558022030 con 0x556223fc8410 2024-05-20T18:01:03.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.109+0000 7fb554bfe640 1 -- 172.21.15.129:0/1545402921 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5580108c0 con 0x556223fc8410 2024-05-20T18:01:03.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.110+0000 7fb56f1df740 1 -- 172.21.15.129:0/1545402921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223fc8410 msgr2=0x556224367620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.110+0000 7fb56f1df740 1 --2- 172.21.15.129:0/1545402921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223fc8410 0x556224367620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb558005c40 tx=0x7fb55800f3c0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.110+0000 7fb56f1df740 1 -- 172.21.15.129:0/1545402921 shutdown_connections 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.110+0000 7fb56f1df740 1 --2- 172.21.15.129:0/1545402921 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fa7730 0x5562243680a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.110+0000 7fb56f1df740 1 --2- 172.21.15.129:0/1545402921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223f9f550 0x556224367b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.110+0000 7fb56f1df740 1 --2- 172.21.15.129:0/1545402921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223fc8410 0x556224367620 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.110+0000 7fb56f1df740 1 -- 172.21.15.129:0/1545402921 >> 172.21.15.129:0/1545402921 conn(0x556223fbc560 msgr2=0x556224363dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.110+0000 7fb56f1df740 1 -- 172.21.15.129:0/1545402921 shutdown_connections 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.110+0000 7fb56f1df740 1 -- 172.21.15.129:0/1545402921 wait complete. 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb56f1df740 1 -- start start 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb56f1df740 1 Processor -- start 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f9f550 0x556224373670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223fa7730 0x556224373bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fc8410 0x556224371b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb56f1df740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556224374910 con 0x556223fa7730 2024-05-20T18:01:03.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb56f1df740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556224287b40 con 0x556223f9f550 2024-05-20T18:01:03.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb56f1df740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x556224374a90 con 0x556223fc8410 2024-05-20T18:01:03.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb5660da640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f9f550 0x556224373670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb5658d9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223fa7730 0x556224373bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb5660da640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f9f550 0x556224373670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59026/0 (socket says 172.21.15.129:59026) 2024-05-20T18:01:03.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb5660da640 1 -- 172.21.15.129:0/473479979 learned_addr learned my addr 172.21.15.129:0/473479979 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:03.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb56f1df740 10 client.0 _ll_register_callbacks cb 0x55622427c0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:03.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb5668db640 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fc8410 0x556224371b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb5660da640 1 -- 172.21.15.129:0/473479979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fc8410 msgr2=0x556224371b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb5660da640 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fc8410 0x556224371b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb5660da640 1 -- 172.21.15.129:0/473479979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223fa7730 msgr2=0x556224373bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb5660da640 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223fa7730 0x556224373bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.111+0000 7fb5660da640 1 -- 172.21.15.129:0/473479979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556224108f70 con 0x556223f9f550 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.112+0000 7fb5660da640 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f9f550 0x556224373670 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0126d0 tx=0x7fb55c012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db28a79f278a83e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.112+0000 7fb549ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.112+0000 7fb549ffb640 1 -- 172.21.15.129:0/473479979 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c004030 con 0x556223f9f550 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.112+0000 7fb549ffb640 1 -- 172.21.15.129:0/473479979 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fb55c0041d0 con 0x556223f9f550 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.112+0000 7fb56f1df740 1 -- 172.21.15.129:0/473479979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb558022030 con 0x556223f9f550 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.112+0000 7fb549ffb640 1 -- 172.21.15.129:0/473479979 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb55c0043d0 con 0x556223f9f550 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.112+0000 7fb56f1df740 10 client.25556 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.112+0000 7fb56f1df740 1 -- 172.21.15.129:0/473479979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x556223fa0170 con 0x556223f9f550 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.113+0000 7fb549ffb640 1 -- 172.21.15.129:0/473479979 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb558022030 con 0x556223f9f550 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.114+0000 7fb549ffb640 1 -- 172.21.15.129:0/473479979 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fb55c003bc0 con 0x556223f9f550 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.114+0000 7fb56f1df740 10 client.25556 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:03.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.114+0000 7fb56f1df740 10 client.25556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:03.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.114+0000 7fb56f1df740 20 client.25556 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:03.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.114+0000 7fb56f1df740 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556223f871f0 0x5562243841a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.114+0000 7fb56f1df740 4 client.25556 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:01:03.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.114+0000 7fb56f1df740 1 -- 172.21.15.129:0/473479979 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562242da0c0 con 0x556223f871f0 2024-05-20T18:01:03.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.114+0000 7fb5658d9640 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556223f871f0 0x5562243841a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.114+0000 7fb549ffb640 10 client.25556 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:03.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.114+0000 7fb5658d9640 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556223f871f0 0x5562243841a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd7f696439c6583e server_cookie=de06a6c095446b06 in_seq=0 out_seq=0 2024-05-20T18:01:03.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb549ffb640 1 -- 172.21.15.129:0/473479979 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5562242da0c0 con 0x556223f871f0 2024-05-20T18:01:03.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb549ffb640 10 client.25556 handle_command_reply: tid=0 2024-05-20T18:01:03.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb56f1df740 1 client.25556 shutdown 2024-05-20T18:01:03.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb56f1df740 1 -- 172.21.15.129:0/473479979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f9f550 msgr2=0x556224373670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb56f1df740 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f9f550 0x556224373670 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0126d0 tx=0x7fb55c012b90 comp rx=0 tx=0).stop 2024-05-20T18:01:03.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb56f1df740 1 -- 172.21.15.129:0/473479979 shutdown_connections 2024-05-20T18:01:03.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb56f1df740 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556223f871f0 0x5562243841a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb56f1df740 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fc8410 0x556224371b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb549ffb640 0 client.25556 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:03.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb56f1df740 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223fa7730 0x556224373bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb56f1df740 1 --2- 172.21.15.129:0/473479979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f9f550 0x556224373670 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb56f1df740 1 -- 172.21.15.129:0/473479979 >> 172.21.15.129:0/473479979 conn(0x556223fbc560 msgr2=0x556224367450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb56f1df740 1 -- 172.21.15.129:0/473479979 shutdown_connections 2024-05-20T18:01:03.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.117+0000 7fb56f1df740 1 -- 172.21.15.129:0/473479979 wait complete. 2024-05-20T18:01:03.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.118+0000 7fb56f1df740 20 client.25556 trim_cache size 0 max 16384 2024-05-20T18:01:03.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:02 smithi154 ceph-mon[36482]: pgmap v78: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 852 MiB/s wr, 2.52k op/s 2024-05-20T18:01:03.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.150+0000 7fb56f1df740 1 Processor -- start 2024-05-20T18:01:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.155+0000 7fb56f1df740 1 -- start start 2024-05-20T18:01:03.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.156+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 0x556224363ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.157+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223f9f550 0x556224379d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.157+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223fa7730 0x55622437a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.157+0000 7fb5660da640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 0x556224363ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.157+0000 7fb5660da640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 0x556224363ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59040/0 (socket says 172.21.15.129:59040) 2024-05-20T18:01:03.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.157+0000 7fb5660da640 1 -- 172.21.15.129:0/611484521 learned_addr learned my addr 172.21.15.129:0/611484521 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:03.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.157+0000 7fb5658d9640 1 --2- 172.21.15.129:0/611484521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223f9f550 0x556224379d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.158+0000 7fb56f1df740 1 -- 172.21.15.129:0/611484521 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601aec70 con 0x556223fa7730 2024-05-20T18:01:03.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.158+0000 7fb56f1df740 1 -- 172.21.15.129:0/611484521 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601b14f0 con 0x556223f871f0 2024-05-20T18:01:03.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.158+0000 7fb56f1df740 1 -- 172.21.15.129:0/611484521 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601af2e0 con 0x556223f9f550 2024-05-20T18:01:03.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.159+0000 7fb5668db640 1 --2- 172.21.15.129:0/611484521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223fa7730 0x55622437a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.160+0000 7fb5668db640 1 -- 172.21.15.129:0/611484521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223f9f550 msgr2=0x556224379d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.160+0000 7fb5668db640 1 --2- 172.21.15.129:0/611484521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223f9f550 0x556224379d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.160+0000 7fb5668db640 1 -- 172.21.15.129:0/611484521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 msgr2=0x556224363ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.160+0000 7fb5668db640 1 --2- 172.21.15.129:0/611484521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 0x556224363ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.160+0000 7fb5668db640 1 -- 172.21.15.129:0/611484521 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55622436d440 con 0x556223fa7730 2024-05-20T18:01:03.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.161+0000 7fb5668db640 1 --2- 172.21.15.129:0/611484521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223fa7730 0x55622437a260 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb55800a010 tx=0x7fb558009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94448da9f0529d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.161+0000 7fb54bfff640 1 -- 172.21.15.129:0/611484521 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb558013120 con 0x556223fa7730 2024-05-20T18:01:03.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.161+0000 7fb54bfff640 1 -- 172.21.15.129:0/611484521 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5580132c0 con 0x556223fa7730 2024-05-20T18:01:03.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.162+0000 7fb54bfff640 1 -- 172.21.15.129:0/611484521 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb558010710 con 0x556223fa7730 2024-05-20T18:01:03.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:02 smithi129 ceph-mon[32029]: pgmap v78: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 852 MiB/s wr, 2.52k op/s 2024-05-20T18:01:03.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.163+0000 7fb56f1df740 1 -- 172.21.15.129:0/611484521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223fa7730 msgr2=0x55622437a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.163+0000 7fb56f1df740 1 --2- 172.21.15.129:0/611484521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223fa7730 0x55622437a260 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb55800a010 tx=0x7fb558009b30 comp rx=0 tx=0).stop 2024-05-20T18:01:03.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.164+0000 7fb56f1df740 1 -- 172.21.15.129:0/611484521 shutdown_connections 2024-05-20T18:01:03.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.164+0000 7fb56f1df740 1 --2- 172.21.15.129:0/611484521 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223fa7730 0x55622437a260 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.164+0000 7fb56f1df740 1 --2- 172.21.15.129:0/611484521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223f9f550 0x556224379d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.164+0000 7fb56f1df740 1 --2- 172.21.15.129:0/611484521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 0x556224363ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.164+0000 7fb56f1df740 1 -- 172.21.15.129:0/611484521 >> 172.21.15.129:0/611484521 conn(0x556223fbc560 msgr2=0x556224363790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.164+0000 7fb56f1df740 1 -- 172.21.15.129:0/611484521 shutdown_connections 2024-05-20T18:01:03.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.164+0000 7fb56f1df740 1 -- 172.21.15.129:0/611484521 wait complete. 2024-05-20T18:01:03.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.165+0000 7fb56f1df740 1 -- start start 2024-05-20T18:01:03.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.165+0000 7fb56f1df740 1 Processor -- start 2024-05-20T18:01:03.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.166+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 0x55622437a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.166+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223f9f550 0x556224377600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.166+0000 7fb56f1df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fa7730 0x556224377b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.166+0000 7fb56f1df740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601af9b0 con 0x556223f9f550 2024-05-20T18:01:03.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.166+0000 7fb56f1df740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601b0e20 con 0x556223f871f0 2024-05-20T18:01:03.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.166+0000 7fb56f1df740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601b0080 con 0x556223fa7730 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.166+0000 7fb5660da640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 0x55622437a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.166+0000 7fb5660da640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 0x55622437a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59052/0 (socket says 172.21.15.129:59052) 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.166+0000 7fb5660da640 1 -- 172.21.15.129:0/2091324010 learned_addr learned my addr 172.21.15.129:0/2091324010 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.167+0000 7fb5668db640 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fa7730 0x556224377b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.167+0000 7fb56f1df740 10 client.0 _ll_register_callbacks cb 0x7fb55c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.167+0000 7fb5660da640 1 -- 172.21.15.129:0/2091324010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fa7730 msgr2=0x556224377b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.167+0000 7fb5660da640 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fa7730 0x556224377b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.167+0000 7fb5660da640 1 -- 172.21.15.129:0/2091324010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223f9f550 msgr2=0x556224377600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.167+0000 7fb5660da640 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223f9f550 0x556224377600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.167+0000 7fb5660da640 1 -- 172.21.15.129:0/2091324010 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556224378280 con 0x556223f871f0 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.168+0000 7fb5660da640 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 0x55622437a290 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb55c000c00 tx=0x7fb55c003670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=725caeac7c2e80ea server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.168+0000 7fb549ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:01:03.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.170+0000 7fb549ffb640 1 -- 172.21.15.129:0/2091324010 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c028040 con 0x556223f871f0 2024-05-20T18:01:03.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.170+0000 7fb549ffb640 1 -- 172.21.15.129:0/2091324010 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fb55c003940 con 0x556223f871f0 2024-05-20T18:01:03.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.170+0000 7fb56f1df740 1 -- 172.21.15.129:0/2091324010 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5562242d4410 con 0x556223f871f0 2024-05-20T18:01:03.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.171+0000 7fb549ffb640 1 -- 172.21.15.129:0/2091324010 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb55c003b40 con 0x556223f871f0 2024-05-20T18:01:03.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.171+0000 7fb56f1df740 10 client.25562 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:03.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.171+0000 7fb56f1df740 1 -- 172.21.15.129:0/2091324010 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5562243f4bd0 con 0x556223f871f0 2024-05-20T18:01:03.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.174+0000 7fb549ffb640 1 -- 172.21.15.129:0/2091324010 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb55c003ce0 con 0x556223f871f0 2024-05-20T18:01:03.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.174+0000 7fb549ffb640 1 -- 172.21.15.129:0/2091324010 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fb55c025b70 con 0x556223f871f0 2024-05-20T18:01:03.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.176+0000 7fb56f1df740 10 client.25562 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:03.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.176+0000 7fb56f1df740 10 client.25562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:03.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.176+0000 7fb56f1df740 20 client.25562 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:03.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.176+0000 7fb56f1df740 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556223fc8410 0x5562243f72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.176+0000 7fb56f1df740 4 client.25562 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:01:03.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.176+0000 7fb56f1df740 1 -- 172.21.15.129:0/2091324010 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5562242d4410 con 0x556223fc8410 2024-05-20T18:01:03.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.177+0000 7fb5658d9640 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556223fc8410 0x5562243f72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:02 smithi204 ceph-mon[37076]: pgmap v78: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 852 MiB/s wr, 2.52k op/s 2024-05-20T18:01:03.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.184+0000 7fb5658d9640 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556223fc8410 0x5562243f72e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8412a621b893084 server_cookie=cafdfc39257f742f in_seq=0 out_seq=0 2024-05-20T18:01:03.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.185+0000 7fb549ffb640 10 client.25562 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:03.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.190+0000 7fb549ffb640 1 -- 172.21.15.129:0/2091324010 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5562242d4410 con 0x556223fc8410 2024-05-20T18:01:03.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.190+0000 7fb549ffb640 10 client.25562 handle_command_reply: tid=0 2024-05-20T18:01:03.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.190+0000 7fb56f1df740 1 client.25562 shutdown 2024-05-20T18:01:03.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.192+0000 7fb56f1df740 1 -- 172.21.15.129:0/2091324010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 msgr2=0x55622437a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.192+0000 7fb56f1df740 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 0x55622437a290 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb55c000c00 tx=0x7fb55c003670 comp rx=0 tx=0).stop 2024-05-20T18:01:03.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.194+0000 7fb56f1df740 1 -- 172.21.15.129:0/2091324010 shutdown_connections 2024-05-20T18:01:03.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.194+0000 7fb56f1df740 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556223fc8410 0x5562243f72e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.194+0000 7fb56f1df740 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556223fa7730 0x556224377b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.194+0000 7fb56f1df740 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556223f9f550 0x556224377600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.194+0000 7fb56f1df740 1 --2- 172.21.15.129:0/2091324010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556223f871f0 0x55622437a290 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.194+0000 7fb56f1df740 1 -- 172.21.15.129:0/2091324010 >> 172.21.15.129:0/2091324010 conn(0x556223fbc560 msgr2=0x55622436b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.195+0000 7fb549ffb640 0 client.25562 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:03.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.196+0000 7fb56f1df740 1 -- 172.21.15.129:0/2091324010 shutdown_connections 2024-05-20T18:01:03.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.196+0000 7fb56f1df740 1 -- 172.21.15.129:0/2091324010 wait complete. 2024-05-20T18:01:03.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.199+0000 7fb56f1df740 20 client.25562 trim_cache size 0 max 16384 2024-05-20T18:01:03.199 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:01:03.199 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:01:03.200 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "f3ad9695-5da6-4055-bab9-da281f7a355c", 2024-05-20T18:01:03.200 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:01:03.200 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:01:03.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.204+0000 7fb56833e640 1 -- 172.21.15.129:0/4066261985 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb54c0a2d00 msgr2=0x7fb54c0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.204+0000 7fb56833e640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb54c0a2d00 0x7fb54c0a5210 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb55800f8c0 tx=0x7fb55801f000 comp rx=0 tx=0).stop 2024-05-20T18:01:03.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.204+0000 7fb56833e640 1 -- 172.21.15.129:0/4066261985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5601079f0 msgr2=0x7fb5601a1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.204+0000 7fb56833e640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5601079f0 0x7fb5601a1b00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb550006010 tx=0x7fb550034740 comp rx=0 tx=0).stop 2024-05-20T18:01:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.211+0000 7fb56833e640 1 -- 172.21.15.129:0/4066261985 shutdown_connections 2024-05-20T18:01:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.211+0000 7fb56833e640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb54c0a2d00 0x7fb54c0a5210 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.211+0000 7fb56833e640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb56010c350 0x7fb56019b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.211+0000 7fb56833e640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb5601079f0 0x7fb5601a1b00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.211+0000 7fb56833e640 1 --2- 172.21.15.129:0/4066261985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb560101ac0 0x7fb5601a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.211+0000 7fb56833e640 1 -- 172.21.15.129:0/4066261985 >> 172.21.15.129:0/4066261985 conn(0x7fb560077130 msgr2=0x7fb560103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.212+0000 7fb56833e640 1 -- 172.21.15.129:0/4066261985 shutdown_connections 2024-05-20T18:01:03.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.213+0000 7fb56833e640 1 -- 172.21.15.129:0/4066261985 wait complete. 2024-05-20T18:01:03.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:01:03.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:01:03.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.524+0000 7fc05cd96640 1 -- 172.21.15.129:0/3311700348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058108f80 msgr2=0x7fc058102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.524+0000 7fc05cd96640 1 --2- 172.21.15.129:0/3311700348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058108f80 0x7fc058102c10 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc04c009900 tx=0x7fc04c02a440 comp rx=0 tx=0).stop 2024-05-20T18:01:03.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.524+0000 7fc05cd96640 1 -- 172.21.15.129:0/3311700348 shutdown_connections 2024-05-20T18:01:03.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.524+0000 7fc05cd96640 1 --2- 172.21.15.129:0/3311700348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc058103280 0x7fc05810f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.524+0000 7fc05cd96640 1 --2- 172.21.15.129:0/3311700348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058108f80 0x7fc058102c10 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.524+0000 7fc05cd96640 1 --2- 172.21.15.129:0/3311700348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc058108ae0 0x7fc0581026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.524+0000 7fc05cd96640 1 -- 172.21.15.129:0/3311700348 >> 172.21.15.129:0/3311700348 conn(0x7fc0580780b0 msgr2=0x7fc0580784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.526+0000 7fc05cd96640 1 -- 172.21.15.129:0/3311700348 shutdown_connections 2024-05-20T18:01:03.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.526+0000 7fc05cd96640 1 -- 172.21.15.129:0/3311700348 wait complete. 2024-05-20T18:01:03.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.527+0000 7fc05cd96640 1 Processor -- start 2024-05-20T18:01:03.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.527+0000 7fc05cd96640 1 -- start start 2024-05-20T18:01:03.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.527+0000 7fc05cd96640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc058103280 0x7fc058112990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.527+0000 7fc05cd96640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058108ae0 0x7fc058112ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.527+0000 7fc05cd96640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc058113410 0x7fc0580715a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.527+0000 7fc05cd96640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc058110d60 con 0x7fc058108ae0 2024-05-20T18:01:03.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.527+0000 7fc05cd96640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc058110be0 con 0x7fc058113410 2024-05-20T18:01:03.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.527+0000 7fc05cd96640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc058110ee0 con 0x7fc058103280 2024-05-20T18:01:03.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.528+0000 7fc055d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058108ae0 0x7fc058112ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.528+0000 7fc055d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058108ae0 0x7fc058112ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45318/0 (socket says 172.21.15.129:45318) 2024-05-20T18:01:03.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.528+0000 7fc055d9b640 1 -- 172.21.15.129:0/2426364103 learned_addr learned my addr 172.21.15.129:0/2426364103 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:03.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.528+0000 7fc055d9b640 1 -- 172.21.15.129:0/2426364103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc058103280 msgr2=0x7fc058112990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:01:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.528+0000 7fc055d9b640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc058103280 0x7fc058112990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.528+0000 7fc055d9b640 1 -- 172.21.15.129:0/2426364103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc058113410 msgr2=0x7fc0580715a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.528+0000 7fc055d9b640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc058113410 0x7fc0580715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.528+0000 7fc055d9b640 1 -- 172.21.15.129:0/2426364103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc058071ae0 con 0x7fc058108ae0 2024-05-20T18:01:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.528+0000 7fc055d9b640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058108ae0 0x7fc058112ed0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc04c006010 tx=0x7fc04c003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.529+0000 7fc0477fe640 1 -- 172.21.15.129:0/2426364103 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c03f020 con 0x7fc058108ae0 2024-05-20T18:01:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.529+0000 7fc05cd96640 1 -- 172.21.15.129:0/2426364103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc058071d70 con 0x7fc058108ae0 2024-05-20T18:01:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.529+0000 7fc0477fe640 1 -- 172.21.15.129:0/2426364103 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc04c03b460 con 0x7fc058108ae0 2024-05-20T18:01:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.529+0000 7fc0477fe640 1 -- 172.21.15.129:0/2426364103 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c035070 con 0x7fc058108ae0 2024-05-20T18:01:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.529+0000 7fc05cd96640 1 -- 172.21.15.129:0/2426364103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc058072230 con 0x7fc058108ae0 2024-05-20T18:01:03.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.533+0000 7fc063c44740 1 Processor -- start 2024-05-20T18:01:03.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.534+0000 7fc063c44740 1 -- start start 2024-05-20T18:01:03.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.535+0000 7fc063c44740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe9e390 0x55bfec23cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.535+0000 7fc063c44740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe754d0 0x55bfec23d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.535+0000 7fc063c44740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec23d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.535+0000 7fc063c44740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bfec1a2d00 con 0x55bfebe7d6b0 2024-05-20T18:01:03.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.535+0000 7fc063c44740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bfec14a4c0 con 0x55bfebe754d0 2024-05-20T18:01:03.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.535+0000 7fc063c44740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55bfec19ecc0 con 0x55bfebe9e390 2024-05-20T18:01:03.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.535+0000 7fc0477fe640 1 -- 172.21.15.129:0/2426364103 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc04c042020 con 0x7fc058108ae0 2024-05-20T18:01:03.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.536+0000 7fc055d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec23d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.536+0000 7fc055d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec23d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45320/0 (socket says 172.21.15.129:45320) 2024-05-20T18:01:03.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.536+0000 7fc055d9b640 1 -- 172.21.15.129:0/3224810839 learned_addr learned my addr 172.21.15.129:0/3224810839 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:03.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.536+0000 7fc056d9d640 1 --2- 172.21.15.129:0/3224810839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe9e390 0x55bfec23cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.536+0000 7fc05659c640 1 --2- 172.21.15.129:0/3224810839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe754d0 0x55bfec23d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.536+0000 7fc055d9b640 1 -- 172.21.15.129:0/3224810839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe9e390 msgr2=0x55bfec23cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.536+0000 7fc055d9b640 1 --2- 172.21.15.129:0/3224810839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe9e390 0x55bfec23cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.536+0000 7fc055d9b640 1 -- 172.21.15.129:0/3224810839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe754d0 msgr2=0x55bfec23d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.536+0000 7fc055d9b640 1 --2- 172.21.15.129:0/3224810839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe754d0 0x55bfec23d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.537+0000 7fc055d9b640 1 -- 172.21.15.129:0/3224810839 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfebf807d0 con 0x55bfebe7d6b0 2024-05-20T18:01:03.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.537+0000 7fc05659c640 1 --2- 172.21.15.129:0/3224810839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe754d0 0x55bfec23d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:01:03.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.537+0000 7fc055d9b640 1 --2- 172.21.15.129:0/3224810839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec23d7b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0ed1c0 tx=0x7fc04c0f3bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d44cedc6bed08d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.539+0000 7fc0477fe640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0400a2d60 0x7fc0400a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.539+0000 7fc044ff9640 1 -- 172.21.15.129:0/3224810839 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c0f3e10 con 0x55bfebe7d6b0 2024-05-20T18:01:03.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.539+0000 7fc044ff9640 1 -- 172.21.15.129:0/3224810839 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc04c0fb3f0 con 0x55bfebe7d6b0 2024-05-20T18:01:03.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.539+0000 7fc044ff9640 1 -- 172.21.15.129:0/3224810839 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c0fa5b0 con 0x55bfebe7d6b0 2024-05-20T18:01:03.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.540+0000 7fc056d9d640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0400a2d60 0x7fc0400a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.540+0000 7fc063c44740 1 -- 172.21.15.129:0/3224810839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 msgr2=0x55bfec23d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.540+0000 7fc063c44740 1 --2- 172.21.15.129:0/3224810839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec23d7b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0ed1c0 tx=0x7fc04c0f3bc0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.540+0000 7fc0477fe640 1 -- 172.21.15.129:0/2426364103 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fc04c0efba0 con 0x7fc058108ae0 2024-05-20T18:01:03.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.541+0000 7fc056d9d640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0400a2d60 0x7fc0400a5270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55bfec239ea0 tx=0x7fc050009450 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.542+0000 7fc063c44740 1 -- 172.21.15.129:0/3224810839 shutdown_connections 2024-05-20T18:01:03.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.542+0000 7fc063c44740 1 --2- 172.21.15.129:0/3224810839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe754d0 0x55bfec23d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.542+0000 7fc063c44740 1 --2- 172.21.15.129:0/3224810839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec23d7b0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.542+0000 7fc063c44740 1 --2- 172.21.15.129:0/3224810839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe9e390 0x55bfec23cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.542+0000 7fc063c44740 1 -- 172.21.15.129:0/3224810839 >> 172.21.15.129:0/3224810839 conn(0x55bfebe924e0 msgr2=0x55bfec2394e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.542+0000 7fc063c44740 1 -- 172.21.15.129:0/3224810839 shutdown_connections 2024-05-20T18:01:03.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.542+0000 7fc063c44740 1 -- 172.21.15.129:0/3224810839 wait complete. 2024-05-20T18:01:03.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.543+0000 7fc063c44740 1 -- start start 2024-05-20T18:01:03.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.543+0000 7fc063c44740 1 Processor -- start 2024-05-20T18:01:03.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.545+0000 7fc063c44740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe754d0 0x55bfec248ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.545+0000 7fc063c44740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe7d6b0 0x55bfec249420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.545+0000 7fc05659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe754d0 0x55bfec248ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.545+0000 7fc063c44740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe9e390 0x55bfec2472d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.545+0000 7fc05659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe754d0 0x55bfec248ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45322/0 (socket says 172.21.15.129:45322) 2024-05-20T18:01:03.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.545+0000 7fc063c44740 1 -- 172.21.15.129:0/1522505526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bfec1a2d00 con 0x55bfebe754d0 2024-05-20T18:01:03.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.545+0000 7fc05659c640 1 -- 172.21.15.129:0/1522505526 learned_addr learned my addr 172.21.15.129:0/1522505526 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:03.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.545+0000 7fc063c44740 1 -- 172.21.15.129:0/1522505526 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bfec15db90 con 0x55bfebe7d6b0 2024-05-20T18:01:03.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.545+0000 7fc063c44740 1 -- 172.21.15.129:0/1522505526 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55bfec249e40 con 0x55bfebe9e390 2024-05-20T18:01:03.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.545+0000 7fc063c44740 10 client.0 _ll_register_callbacks cb 0x55bfec199da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:03.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.546+0000 7fc05659c640 1 -- 172.21.15.129:0/1522505526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe9e390 msgr2=0x55bfec2472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.546+0000 7fc05659c640 1 --2- 172.21.15.129:0/1522505526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe9e390 0x55bfec2472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.546+0000 7fc05659c640 1 -- 172.21.15.129:0/1522505526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe7d6b0 msgr2=0x55bfec249420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.546+0000 7fc05659c640 1 --2- 172.21.15.129:0/1522505526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe7d6b0 0x55bfec249420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.546+0000 7fc05659c640 1 -- 172.21.15.129:0/1522505526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfebeb63f0 con 0x55bfebe754d0 2024-05-20T18:01:03.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.546+0000 7fc05659c640 1 --2- 172.21.15.129:0/1522505526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe754d0 0x55bfec248ec0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc04800af20 tx=0x7fc04800c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772af576ee089be1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.546+0000 7fc03dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:01:03.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.547+0000 7fc03dffb640 1 -- 172.21.15.129:0/1522505526 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc048015070 con 0x55bfebe754d0 2024-05-20T18:01:03.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.547+0000 7fc03dffb640 1 -- 172.21.15.129:0/1522505526 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fc0480115f0 con 0x55bfebe754d0 2024-05-20T18:01:03.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.547+0000 7fc063c44740 1 -- 172.21.15.129:0/1522505526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc04c0fb3f0 con 0x55bfebe754d0 2024-05-20T18:01:03.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.548+0000 7fc03dffb640 1 -- 172.21.15.129:0/1522505526 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc048021050 con 0x55bfebe754d0 2024-05-20T18:01:03.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.548+0000 7fc063c44740 10 client.15888 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:03.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.548+0000 7fc063c44740 1 -- 172.21.15.129:0/1522505526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55bfebe760f0 con 0x55bfebe754d0 2024-05-20T18:01:03.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.548+0000 7fc03dffb640 1 -- 172.21.15.129:0/1522505526 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc04c0fb3f0 con 0x55bfebe754d0 2024-05-20T18:01:03.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.549+0000 7fc03dffb640 1 -- 172.21.15.129:0/1522505526 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fc048019a60 con 0x55bfebe754d0 2024-05-20T18:01:03.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.549+0000 7fc063c44740 10 client.15888 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:03.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.549+0000 7fc063c44740 10 client.15888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:03.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.549+0000 7fc063c44740 20 client.15888 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:03.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.549+0000 7fc063c44740 1 --2- 172.21.15.129:0/1522505526 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bfebe5cf60 0x55bfec259a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.549+0000 7fc063c44740 4 client.15888 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:01:03.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.549+0000 7fc063c44740 1 -- 172.21.15.129:0/1522505526 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfec1b0430 con 0x55bfebe5cf60 2024-05-20T18:01:03.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.550+0000 7fc055d9b640 1 --2- 172.21.15.129:0/1522505526 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bfebe5cf60 0x55bfec259a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.552+0000 7fc03dffb640 10 client.15888 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:03.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.552+0000 7fc055d9b640 1 --2- 172.21.15.129:0/1522505526 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bfebe5cf60 0x55bfec259a50 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c28cd4954d810a1e server_cookie=4e86b4d2a179537e in_seq=0 out_seq=0 2024-05-20T18:01:03.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc03dffb640 1 -- 172.21.15.129:0/1522505526 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bfec1b0430 con 0x55bfebe5cf60 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc03dffb640 10 client.15888 handle_command_reply: tid=0 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 1 client.15888 shutdown 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 1 -- 172.21.15.129:0/1522505526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe754d0 msgr2=0x55bfec248ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 1 --2- 172.21.15.129:0/1522505526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe754d0 0x55bfec248ec0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc04800af20 tx=0x7fc04800c5e0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 1 -- 172.21.15.129:0/1522505526 shutdown_connections 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 1 --2- 172.21.15.129:0/1522505526 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bfebe5cf60 0x55bfec259a50 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 1 --2- 172.21.15.129:0/1522505526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe7d6b0 0x55bfec249420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc03dffb640 0 client.15888 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 1 --2- 172.21.15.129:0/1522505526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe9e390 0x55bfec2472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 1 --2- 172.21.15.129:0/1522505526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe754d0 0x55bfec248ec0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 1 -- 172.21.15.129:0/1522505526 >> 172.21.15.129:0/1522505526 conn(0x55bfebe924e0 msgr2=0x55bfec23ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 1 -- 172.21.15.129:0/1522505526 shutdown_connections 2024-05-20T18:01:03.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 1 -- 172.21.15.129:0/1522505526 wait complete. 2024-05-20T18:01:03.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.554+0000 7fc063c44740 20 client.15888 trim_cache size 0 max 16384 2024-05-20T18:01:03.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.593+0000 7fc063c44740 1 Processor -- start 2024-05-20T18:01:03.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.593+0000 7fc063c44740 1 -- start start 2024-05-20T18:01:03.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.594+0000 7fc063c44740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe5cf60 0x55bfec246210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.594+0000 7fc063c44740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe754d0 0x55bfec246f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.594+0000 7fc063c44740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec25c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.594+0000 7fc063c44740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581c47f0 con 0x55bfebe7d6b0 2024-05-20T18:01:03.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.594+0000 7fc063c44740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581c7070 con 0x55bfebe754d0 2024-05-20T18:01:03.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.594+0000 7fc05659c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe5cf60 0x55bfec246210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.594+0000 7fc05659c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe5cf60 0x55bfec246210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:43782/0 (socket says 172.21.15.129:43782) 2024-05-20T18:01:03.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.594+0000 7fc05659c640 1 -- 172.21.15.129:0/1470292229 learned_addr learned my addr 172.21.15.129:0/1470292229 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:03.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.595+0000 7fc056d9d640 1 --2- 172.21.15.129:0/1470292229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec25c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.595+0000 7fc063c44740 1 -- 172.21.15.129:0/1470292229 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581c4e60 con 0x55bfebe5cf60 2024-05-20T18:01:03.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.596+0000 7fc056d9d640 1 -- 172.21.15.129:0/1470292229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe5cf60 msgr2=0x55bfec246210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.596+0000 7fc056d9d640 1 --2- 172.21.15.129:0/1470292229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe5cf60 0x55bfec246210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.596+0000 7fc056d9d640 1 -- 172.21.15.129:0/1470292229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe754d0 msgr2=0x55bfec246f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:01:03.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.596+0000 7fc056d9d640 1 --2- 172.21.15.129:0/1470292229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe754d0 0x55bfec246f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.596+0000 7fc056d9d640 1 -- 172.21.15.129:0/1470292229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfec23a0b0 con 0x55bfebe7d6b0 2024-05-20T18:01:03.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.596+0000 7fc056d9d640 1 --2- 172.21.15.129:0/1470292229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec25c990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc050017730 tx=0x7fc050004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8610bd6f532358 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.597+0000 7fc03ffff640 1 -- 172.21.15.129:0/1470292229 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050027070 con 0x55bfebe7d6b0 2024-05-20T18:01:03.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.597+0000 7fc03ffff640 1 -- 172.21.15.129:0/1470292229 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc050017d60 con 0x55bfebe7d6b0 2024-05-20T18:01:03.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.599+0000 7fc03ffff640 1 -- 172.21.15.129:0/1470292229 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050016600 con 0x55bfebe7d6b0 2024-05-20T18:01:03.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.599+0000 7fc063c44740 1 -- 172.21.15.129:0/1470292229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 msgr2=0x55bfec25c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.599+0000 7fc063c44740 1 --2- 172.21.15.129:0/1470292229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec25c990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc050017730 tx=0x7fc050004ad0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.599+0000 7fc063c44740 1 -- 172.21.15.129:0/1470292229 shutdown_connections 2024-05-20T18:01:03.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.599+0000 7fc063c44740 1 --2- 172.21.15.129:0/1470292229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe754d0 0x55bfec246f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.599+0000 7fc063c44740 1 --2- 172.21.15.129:0/1470292229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec25c990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.599+0000 7fc063c44740 1 --2- 172.21.15.129:0/1470292229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe5cf60 0x55bfec246210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.599+0000 7fc063c44740 1 -- 172.21.15.129:0/1470292229 >> 172.21.15.129:0/1470292229 conn(0x55bfebe924e0 msgr2=0x55bfec246810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.600+0000 7fc063c44740 1 -- 172.21.15.129:0/1470292229 shutdown_connections 2024-05-20T18:01:03.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.600+0000 7fc063c44740 1 -- 172.21.15.129:0/1470292229 wait complete. 2024-05-20T18:01:03.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.601+0000 7fc063c44740 1 -- start start 2024-05-20T18:01:03.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.601+0000 7fc063c44740 1 Processor -- start 2024-05-20T18:01:03.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.601+0000 7fc063c44740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe5cf60 0x55bfec2ba630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.601+0000 7fc063c44740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec2bab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.601+0000 7fc063c44740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe9e390 0x55bfec25c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.601+0000 7fc063c44740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581c5530 con 0x55bfebe7d6b0 2024-05-20T18:01:03.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.601+0000 7fc063c44740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581c69a0 con 0x55bfebe9e390 2024-05-20T18:01:03.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.601+0000 7fc063c44740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581c5c00 con 0x55bfebe5cf60 2024-05-20T18:01:03.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.601+0000 7fc063c44740 10 client.0 _ll_register_callbacks cb 0x7fc048006ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:03.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.602+0000 7fc055d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec2bab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.602+0000 7fc055d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec2bab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45340/0 (socket says 172.21.15.129:45340) 2024-05-20T18:01:03.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.602+0000 7fc055d9b640 1 -- 172.21.15.129:0/2099856166 learned_addr learned my addr 172.21.15.129:0/2099856166 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:03.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.602+0000 7fc05659c640 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe5cf60 0x55bfec2ba630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.602+0000 7fc056d9d640 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe9e390 0x55bfec25c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.603+0000 7fc056d9d640 1 -- 172.21.15.129:0/2099856166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe5cf60 msgr2=0x55bfec2ba630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.603+0000 7fc056d9d640 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe5cf60 0x55bfec2ba630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.603+0000 7fc056d9d640 1 -- 172.21.15.129:0/2099856166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 msgr2=0x55bfec2bab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:03.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.603+0000 7fc056d9d640 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec2bab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.603+0000 7fc056d9d640 1 -- 172.21.15.129:0/2099856166 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfec25c9a0 con 0x55bfebe9e390 2024-05-20T18:01:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.604+0000 7fc055d9b640 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec2bab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:01:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.604+0000 7fc056d9d640 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe9e390 0x55bfec25c390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc050025040 tx=0x7fc050017f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e92e1079f9c2f296 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.604+0000 7fc03dffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:01:03.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.605+0000 7fc03dffb640 1 -- 172.21.15.129:0/2099856166 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050031020 con 0x55bfebe9e390 2024-05-20T18:01:03.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.605+0000 7fc03dffb640 1 -- 172.21.15.129:0/2099856166 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fc0500228b0 con 0x55bfebe9e390 2024-05-20T18:01:03.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.605+0000 7fc063c44740 1 -- 172.21.15.129:0/2099856166 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc050017d60 con 0x55bfebe9e390 2024-05-20T18:01:03.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.606+0000 7fc03dffb640 1 -- 172.21.15.129:0/2099856166 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fc050027040 con 0x55bfebe9e390 2024-05-20T18:01:03.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.606+0000 7fc063c44740 10 client.25565 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:03.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.606+0000 7fc063c44740 1 -- 172.21.15.129:0/2099856166 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55bfec2c1ce0 con 0x55bfebe9e390 2024-05-20T18:01:03.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.607+0000 7fc03dffb640 1 -- 172.21.15.129:0/2099856166 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc050017d60 con 0x55bfebe9e390 2024-05-20T18:01:03.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.608+0000 7fc03dffb640 1 -- 172.21.15.129:0/2099856166 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fc05002a030 con 0x55bfebe9e390 2024-05-20T18:01:03.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.608+0000 7fc063c44740 10 client.25565 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:03.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.608+0000 7fc063c44740 10 client.25565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:03.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.608+0000 7fc063c44740 20 client.25565 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:03.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.608+0000 7fc063c44740 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bfebe323a0 0x55bfec2c4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:03.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.608+0000 7fc063c44740 4 client.25565 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:01:03.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.608+0000 7fc063c44740 1 -- 172.21.15.129:0/2099856166 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bfec1aa780 con 0x55bfebe323a0 2024-05-20T18:01:03.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.609+0000 7fc05659c640 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bfebe323a0 0x55bfec2c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:03.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.611+0000 7fc05659c640 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bfebe323a0 0x55bfec2c4310 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cae65f89097e1937 server_cookie=c0eb445aa7ff6809 in_seq=0 out_seq=0 2024-05-20T18:01:03.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.615+0000 7fc03dffb640 10 client.25565 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:03.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.615+0000 7fc03dffb640 1 -- 172.21.15.129:0/2099856166 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bfec1aa780 con 0x55bfebe323a0 2024-05-20T18:01:03.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.615+0000 7fc03dffb640 10 client.25565 handle_command_reply: tid=0 2024-05-20T18:01:03.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.615+0000 7fc063c44740 1 client.25565 shutdown 2024-05-20T18:01:03.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.621+0000 7fc063c44740 1 -- 172.21.15.129:0/2099856166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe9e390 msgr2=0x55bfec25c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.621+0000 7fc063c44740 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe9e390 0x55bfec25c390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc050025040 tx=0x7fc050017f10 comp rx=0 tx=0).stop 2024-05-20T18:01:03.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.627+0000 7fc063c44740 1 -- 172.21.15.129:0/2099856166 shutdown_connections 2024-05-20T18:01:03.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.627+0000 7fc063c44740 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bfebe323a0 0x55bfec2c4310 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.627+0000 7fc063c44740 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfebe9e390 0x55bfec25c390 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.627+0000 7fc063c44740 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bfebe7d6b0 0x55bfec2bab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.627+0000 7fc063c44740 1 --2- 172.21.15.129:0/2099856166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bfebe5cf60 0x55bfec2ba630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.627+0000 7fc063c44740 1 -- 172.21.15.129:0/2099856166 >> 172.21.15.129:0/2099856166 conn(0x55bfebe924e0 msgr2=0x55bfec23ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.628+0000 7fc03dffb640 0 client.25565 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:03.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.633+0000 7fc063c44740 1 -- 172.21.15.129:0/2099856166 shutdown_connections 2024-05-20T18:01:03.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.633+0000 7fc063c44740 1 -- 172.21.15.129:0/2099856166 wait complete. 2024-05-20T18:01:03.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.634+0000 7fc063c44740 20 client.25565 trim_cache size 0 max 16384 2024-05-20T18:01:03.634 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:01:03.634 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:01:03.635 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:01:03.635 INFO:teuthology.orchestra.run.smithi129.stdout: "f3ad9695-5da6-4055-bab9-da281f7a355c": { 2024-05-20T18:01:03.635 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:01:03.635 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "f3ad9695-5da6-4055-bab9-da281f7a355c", 2024-05-20T18:01:03.635 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:01:03.635 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:01:03.635 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:01:03.635 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:01:03.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.636+0000 7fc05cd96640 1 -- 172.21.15.129:0/2426364103 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0400a2d60 msgr2=0x7fc0400a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.636+0000 7fc05cd96640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0400a2d60 0x7fc0400a5270 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55bfec239ea0 tx=0x7fc050009450 comp rx=0 tx=0).stop 2024-05-20T18:01:03.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.636+0000 7fc05cd96640 1 -- 172.21.15.129:0/2426364103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058108ae0 msgr2=0x7fc058112ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:03.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.636+0000 7fc05cd96640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058108ae0 0x7fc058112ed0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc04c006010 tx=0x7fc04c003cd0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.637+0000 7fc05cd96640 1 -- 172.21.15.129:0/2426364103 shutdown_connections 2024-05-20T18:01:03.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.637+0000 7fc05cd96640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0400a2d60 0x7fc0400a5270 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.637+0000 7fc05cd96640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc058113410 0x7fc0580715a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.637+0000 7fc05cd96640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc058108ae0 0x7fc058112ed0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.637+0000 7fc05cd96640 1 --2- 172.21.15.129:0/2426364103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc058103280 0x7fc058112990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:03.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.637+0000 7fc05cd96640 1 -- 172.21.15.129:0/2426364103 >> 172.21.15.129:0/2426364103 conn(0x7fc0580780b0 msgr2=0x7fc0581012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:03.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.637+0000 7fc05cd96640 1 -- 172.21.15.129:0/2426364103 shutdown_connections 2024-05-20T18:01:03.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:03.638+0000 7fc05cd96640 1 -- 172.21.15.129:0/2426364103 wait complete. 2024-05-20T18:01:03.662 INFO:tasks.cephfs.filesystem:scrub status for tag:f3ad9695-5da6-4055-bab9-da281f7a355c - {'path': '/', 'tag': 'f3ad9695-5da6-4055-bab9-da281f7a355c', 'options': 'recursive,force'} 2024-05-20T18:01:03.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:03 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:01:03.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:03 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:03.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:03 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:01:04.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:03 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:01:04.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:03 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:04.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:03 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:01:04.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:03 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:01:04.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:03 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:04.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:03 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:01:05.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:04 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:05.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:04 smithi154 ceph-mon[36482]: pgmap v79: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 711 MiB/s wr, 2.20k op/s 2024-05-20T18:01:05.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:04 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:05.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:04 smithi129 ceph-mon[32029]: pgmap v79: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 711 MiB/s wr, 2.20k op/s 2024-05-20T18:01:05.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:04 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:05.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:04 smithi204 ceph-mon[37076]: pgmap v79: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 711 MiB/s wr, 2.20k op/s 2024-05-20T18:01:06.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:05 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:01:06.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:05 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:01:06.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:05 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:01:07.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:06 smithi154 ceph-mon[36482]: pgmap v80: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 GiB/s wr, 2.96k op/s 2024-05-20T18:01:07.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:06 smithi129 ceph-mon[32029]: pgmap v80: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 GiB/s wr, 2.96k op/s 2024-05-20T18:01:07.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:06 smithi204 ceph-mon[37076]: pgmap v80: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.0 GiB/s wr, 2.96k op/s 2024-05-20T18:01:09.145 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:08 smithi154 ceph-mon[36482]: pgmap v81: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 710 MiB/s wr, 2.00k op/s 2024-05-20T18:01:09.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:08 smithi129 ceph-mon[32029]: pgmap v81: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 710 MiB/s wr, 2.00k op/s 2024-05-20T18:01:09.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:08 smithi204 ceph-mon[37076]: pgmap v81: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 710 MiB/s wr, 2.00k op/s 2024-05-20T18:01:11.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:10 smithi154 ceph-mon[36482]: pgmap v82: 97 pgs: 97 active+clean; 11 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 919 MiB/s wr, 3.46k op/s 2024-05-20T18:01:11.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:10 smithi129 ceph-mon[32029]: pgmap v82: 97 pgs: 97 active+clean; 11 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 919 MiB/s wr, 3.46k op/s 2024-05-20T18:01:11.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:10 smithi204 ceph-mon[37076]: pgmap v82: 97 pgs: 97 active+clean; 11 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 919 MiB/s wr, 3.46k op/s 2024-05-20T18:01:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.615+0000 7fe4feea6640 1 -- 172.21.15.129:0/4004624493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f810dd00 msgr2=0x7fe4f810e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.615+0000 7fe4feea6640 1 --2- 172.21.15.129:0/4004624493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f810dd00 0x7fe4f810e130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0009ca0 tx=0x7fe4f002a8c0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.616+0000 7fe4feea6640 1 -- 172.21.15.129:0/4004624493 shutdown_connections 2024-05-20T18:01:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.616+0000 7fe4feea6640 1 --2- 172.21.15.129:0/4004624493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f810dd00 0x7fe4f810e130 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0009ca0 tx=0x7fe4f002a8c0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.616+0000 7fe4feea6640 1 --2- 172.21.15.129:0/4004624493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4f810e7c0 0x7fe4f8115f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.616+0000 7fe4feea6640 1 --2- 172.21.15.129:0/4004624493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4f8109770 0x7fe4f8109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.616+0000 7fe4feea6640 1 -- 172.21.15.129:0/4004624493 >> 172.21.15.129:0/4004624493 conn(0x7fe4f806d780 msgr2=0x7fe4f806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.617+0000 7fe4feea6640 1 -- 172.21.15.129:0/4004624493 shutdown_connections 2024-05-20T18:01:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.617+0000 7fe4feea6640 1 -- 172.21.15.129:0/4004624493 wait complete. 2024-05-20T18:01:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.618+0000 7fe4feea6640 1 Processor -- start 2024-05-20T18:01:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.618+0000 7fe4feea6640 1 -- start start 2024-05-20T18:01:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.619+0000 7fe4feea6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4f8109770 0x7fe4f811ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.619+0000 7fe4feea6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4f810e7c0 0x7fe4f8120540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.619+0000 7fe4feea6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f8125c30 0x7fe4f811b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.619+0000 7fe4feea6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f8119150 con 0x7fe4f810e7c0 2024-05-20T18:01:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.619+0000 7fe4feea6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f8118fd0 con 0x7fe4f8125c30 2024-05-20T18:01:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.619+0000 7fe4feea6640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f81192d0 con 0x7fe4f8109770 2024-05-20T18:01:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.620+0000 7fe4f7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4f810e7c0 0x7fe4f8120540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.620+0000 7fe4f7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4f810e7c0 0x7fe4f8120540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54928/0 (socket says 172.21.15.129:54928) 2024-05-20T18:01:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.620+0000 7fe4f7fff640 1 -- 172.21.15.129:0/4277203400 learned_addr learned my addr 172.21.15.129:0/4277203400 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.620+0000 7fe4fd443640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f8125c30 0x7fe4f811b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.620+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4f8109770 0x7fe4f811ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.620+0000 7fe4f7fff640 1 -- 172.21.15.129:0/4277203400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4f8109770 msgr2=0x7fe4f811ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.621+0000 7fe4f7fff640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4f8109770 0x7fe4f811ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.621+0000 7fe4f7fff640 1 -- 172.21.15.129:0/4277203400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f8125c30 msgr2=0x7fe4f811b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.621+0000 7fe4f7fff640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f8125c30 0x7fe4f811b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.621+0000 7fe4f7fff640 1 -- 172.21.15.129:0/4277203400 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f811b660 con 0x7fe4f810e7c0 2024-05-20T18:01:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.621+0000 7fe4f7fff640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4f810e7c0 0x7fe4f8120540 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe4f002ae30 tx=0x7fe4f0030ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.622+0000 7fe4f5ffb640 1 -- 172.21.15.129:0/4277203400 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0003b70 con 0x7fe4f810e7c0 2024-05-20T18:01:12.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.622+0000 7fe4feea6640 1 -- 172.21.15.129:0/4277203400 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4f811b8f0 con 0x7fe4f810e7c0 2024-05-20T18:01:12.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.623+0000 7fe4feea6640 1 -- 172.21.15.129:0/4277203400 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4f811be10 con 0x7fe4f810e7c0 2024-05-20T18:01:12.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.623+0000 7fe4f5ffb640 1 -- 172.21.15.129:0/4277203400 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4f0030e90 con 0x7fe4f810e7c0 2024-05-20T18:01:12.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.625+0000 7fe4f5ffb640 1 -- 172.21.15.129:0/4277203400 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f00046c0 con 0x7fe4f810e7c0 2024-05-20T18:01:12.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.629+0000 7fe505e20740 1 Processor -- start 2024-05-20T18:01:12.629 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.631+0000 7fe4f5ffb640 1 -- 172.21.15.129:0/4277203400 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fe4f0031450 con 0x7fe4f810e7c0 2024-05-20T18:01:12.629 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.631+0000 7fe505e20740 1 -- start start 2024-05-20T18:01:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.632+0000 7fe505e20740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f259b1b0 0x55b2f293a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.632+0000 7fe505e20740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f25722f0 0x55b2f293a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.632+0000 7fe505e20740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f257a4d0 0x55b2f293ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.631 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.632+0000 7fe505e20740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b2f289f340 con 0x55b2f257a4d0 2024-05-20T18:01:12.631 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.632+0000 7fe505e20740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b2f2847290 con 0x55b2f259b1b0 2024-05-20T18:01:12.631 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.632+0000 7fe505e20740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b2f289b300 con 0x55b2f25722f0 2024-05-20T18:01:12.631 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.632+0000 7fe4fd443640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f259b1b0 0x55b2f293a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.631 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.632+0000 7fe4fd443640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f259b1b0 0x55b2f293a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52520/0 (socket says 172.21.15.129:52520) 2024-05-20T18:01:12.631 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.632+0000 7fe4fd443640 1 -- 172.21.15.129:0/3003989410 learned_addr learned my addr 172.21.15.129:0/3003989410 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:12.631 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.632+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/3003989410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f25722f0 0x55b2f293a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.633+0000 7fe4fcc42640 1 -- 172.21.15.129:0/3003989410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f259b1b0 msgr2=0x55b2f293a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:12.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.633+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/3003989410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f259b1b0 0x55b2f293a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.633+0000 7fe4fcc42640 1 -- 172.21.15.129:0/3003989410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f257a4d0 msgr2=0x55b2f293ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:12.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.633+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/3003989410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f257a4d0 0x55b2f293ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.633+0000 7fe4fcc42640 1 -- 172.21.15.129:0/3003989410 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2f286b8e0 con 0x55b2f25722f0 2024-05-20T18:01:12.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.633+0000 7fe4f5ffb640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4ec0a2c60 0x7fe4ec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.633+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/3003989410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f25722f0 0x55b2f293a5c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00099a0 tx=0x7fe4e0009e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=314688c4d6610dca server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:12.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.633+0000 7fe4f5ffb640 1 -- 172.21.15.129:0/4277203400 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fe4f00f06a0 con 0x7fe4f810e7c0 2024-05-20T18:01:12.633 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.634+0000 7fe4e6ffd640 1 -- 172.21.15.129:0/3003989410 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0017070 con 0x55b2f25722f0 2024-05-20T18:01:12.633 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.634+0000 7fe4e6ffd640 1 -- 172.21.15.129:0/3003989410 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4e0013470 con 0x55b2f25722f0 2024-05-20T18:01:12.633 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.634+0000 7fe4e6ffd640 1 -- 172.21.15.129:0/3003989410 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0012630 con 0x55b2f25722f0 2024-05-20T18:01:12.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.636+0000 7fe505e20740 1 -- 172.21.15.129:0/3003989410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f25722f0 msgr2=0x55b2f293a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:12.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.636+0000 7fe505e20740 1 --2- 172.21.15.129:0/3003989410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f25722f0 0x55b2f293a5c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00099a0 tx=0x7fe4e0009e60 comp rx=0 tx=0).stop 2024-05-20T18:01:12.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.639+0000 7fe4fd443640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4ec0a2c60 0x7fe4ec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.646 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.648+0000 7fe505e20740 1 -- 172.21.15.129:0/3003989410 shutdown_connections 2024-05-20T18:01:12.646 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.648+0000 7fe505e20740 1 --2- 172.21.15.129:0/3003989410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f257a4d0 0x55b2f293ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.646 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.648+0000 7fe505e20740 1 --2- 172.21.15.129:0/3003989410 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f25722f0 0x55b2f293a5c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.646 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.648+0000 7fe505e20740 1 --2- 172.21.15.129:0/3003989410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f259b1b0 0x55b2f293a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.648+0000 7fe505e20740 1 -- 172.21.15.129:0/3003989410 >> 172.21.15.129:0/3003989410 conn(0x55b2f258f300 msgr2=0x55b2f2936830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:12.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.648+0000 7fe505e20740 1 -- 172.21.15.129:0/3003989410 shutdown_connections 2024-05-20T18:01:12.648 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.650+0000 7fe4fd443640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4ec0a2c60 0x7fe4ec0a5170 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55b2f29371f0 tx=0x7fe4e800c040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:12.649 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.651+0000 7fe505e20740 1 -- 172.21.15.129:0/3003989410 wait complete. 2024-05-20T18:01:12.650 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.652+0000 7fe505e20740 1 -- start start 2024-05-20T18:01:12.650 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.652+0000 7fe505e20740 1 Processor -- start 2024-05-20T18:01:12.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.658+0000 7fe505e20740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f29461b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.658+0000 7fe505e20740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 0x55b2f29466f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.658+0000 7fe505e20740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 0x55b2f29446c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.658+0000 7fe505e20740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b2f29471b0 con 0x55b2f259b1b0 2024-05-20T18:01:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.658+0000 7fe505e20740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b2f285a790 con 0x55b2f25722f0 2024-05-20T18:01:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.658+0000 7fe505e20740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b2f2947330 con 0x55b2f257a4d0 2024-05-20T18:01:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.658+0000 7fe505e20740 10 client.0 _ll_register_callbacks cb 0x55b2f2895a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:12.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.664+0000 7fe4fcc42640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f29461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.664+0000 7fe4fcc42640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f29461b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52532/0 (socket says 172.21.15.129:52532) 2024-05-20T18:01:12.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.664+0000 7fe4fcc42640 1 -- 172.21.15.129:0/3082464642 learned_addr learned my addr 172.21.15.129:0/3082464642 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:12.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.664+0000 7fe4f7fff640 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 0x55b2f29466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.665+0000 7fe4fd443640 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 0x55b2f29446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.668 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.666+0000 7fe4fcc42640 1 -- 172.21.15.129:0/3082464642 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 msgr2=0x55b2f29466f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:12.668 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.666+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 0x55b2f29466f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.668 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.666+0000 7fe4fcc42640 1 -- 172.21.15.129:0/3082464642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 msgr2=0x55b2f29446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:12.668 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.666+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 0x55b2f29446c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.668 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.666+0000 7fe4fcc42640 1 -- 172.21.15.129:0/3082464642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2f25b3210 con 0x55b2f25722f0 2024-05-20T18:01:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.667+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f29461b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009970 tx=0x7fe4e000ea80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7701881f2f1b39ec server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.670+0000 7fe4dffff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:01:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.670+0000 7fe4dffff640 1 -- 172.21.15.129:0/3082464642 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0017050 con 0x55b2f25722f0 2024-05-20T18:01:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.670+0000 7fe505e20740 1 -- 172.21.15.129:0/3082464642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4e0013470 con 0x55b2f25722f0 2024-05-20T18:01:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.670+0000 7fe4dffff640 1 -- 172.21.15.129:0/3082464642 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fe4e001c650 con 0x55b2f25722f0 2024-05-20T18:01:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.671+0000 7fe4dffff640 1 -- 172.21.15.129:0/3082464642 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe4e001c890 con 0x55b2f25722f0 2024-05-20T18:01:12.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.672+0000 7fe505e20740 10 client.25577 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:12.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.672+0000 7fe505e20740 1 -- 172.21.15.129:0/3082464642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b2f267f620 con 0x55b2f25722f0 2024-05-20T18:01:12.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.672+0000 7fe4dffff640 1 -- 172.21.15.129:0/3082464642 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4e0013470 con 0x55b2f25722f0 2024-05-20T18:01:12.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.674+0000 7fe4dffff640 1 -- 172.21.15.129:0/3082464642 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fe4e0026b90 con 0x55b2f25722f0 2024-05-20T18:01:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.674+0000 7fe505e20740 10 client.25577 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.674+0000 7fe505e20740 10 client.25577 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.674+0000 7fe505e20740 20 client.25577 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.675+0000 7fe505e20740 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2f2559d80 0x55b2f2956cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.675+0000 7fe505e20740 4 client.25577 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:01:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.675+0000 7fe505e20740 1 -- 172.21.15.129:0/3082464642 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2f28aca70 con 0x55b2f2559d80 2024-05-20T18:01:12.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.675+0000 7fe4f7fff640 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2f2559d80 0x55b2f2956cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.676+0000 7fe4f7fff640 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2f2559d80 0x55b2f2956cb0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcbefb17c9b5b254 server_cookie=9b66e7161401baf4 in_seq=0 out_seq=0 2024-05-20T18:01:12.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.676+0000 7fe4dffff640 10 client.25577 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.677+0000 7fe4dffff640 1 -- 172.21.15.129:0/3082464642 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b2f28aca70 con 0x55b2f2559d80 2024-05-20T18:01:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.677+0000 7fe4dffff640 10 client.25577 handle_command_reply: tid=0 2024-05-20T18:01:12.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.677+0000 7fe505e20740 1 client.25577 shutdown 2024-05-20T18:01:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.684+0000 7fe505e20740 1 -- 172.21.15.129:0/3082464642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 msgr2=0x55b2f29461b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.684+0000 7fe505e20740 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f29461b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009970 tx=0x7fe4e000ea80 comp rx=0 tx=0).stop 2024-05-20T18:01:12.686 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.688+0000 7fe505e20740 1 -- 172.21.15.129:0/3082464642 shutdown_connections 2024-05-20T18:01:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.688+0000 7fe505e20740 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2f2559d80 0x55b2f2956cb0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.688+0000 7fe505e20740 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 0x55b2f29446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.688+0000 7fe505e20740 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 0x55b2f29466f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.688+0000 7fe505e20740 1 --2- 172.21.15.129:0/3082464642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f29461b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.688+0000 7fe505e20740 1 -- 172.21.15.129:0/3082464642 >> 172.21.15.129:0/3082464642 conn(0x55b2f258f300 msgr2=0x55b2f293bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.688+0000 7fe4dffff640 0 client.25577 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:12.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.694+0000 7fe505e20740 1 -- 172.21.15.129:0/3082464642 shutdown_connections 2024-05-20T18:01:12.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.694+0000 7fe505e20740 1 -- 172.21.15.129:0/3082464642 wait complete. 2024-05-20T18:01:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.701+0000 7fe505e20740 20 client.25577 trim_cache size 0 max 16384 2024-05-20T18:01:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.758+0000 7fe505e20740 1 Processor -- start 2024-05-20T18:01:12.757 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.758+0000 7fe505e20740 1 -- start start 2024-05-20T18:01:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.760+0000 7fe505e20740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f2944660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.760+0000 7fe505e20740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 0x55b2f295c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.760+0000 7fe505e20740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 0x55b2f295c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.761+0000 7fe505e20740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d800c790 con 0x55b2f259b1b0 2024-05-20T18:01:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.761+0000 7fe505e20740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d800ef60 con 0x55b2f25722f0 2024-05-20T18:01:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.761+0000 7fe505e20740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d800cd50 con 0x55b2f257a4d0 2024-05-20T18:01:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.761+0000 7fe4fcc42640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f2944660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.761+0000 7fe4fcc42640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f2944660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52536/0 (socket says 172.21.15.129:52536) 2024-05-20T18:01:12.760 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.761+0000 7fe4fcc42640 1 -- 172.21.15.129:0/3568149764 learned_addr learned my addr 172.21.15.129:0/3568149764 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.762+0000 7fe4fcc42640 1 -- 172.21.15.129:0/3568149764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 msgr2=0x55b2f295c4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:01:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.762+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/3568149764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 0x55b2f295c4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.762+0000 7fe4fcc42640 1 -- 172.21.15.129:0/3568149764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 msgr2=0x55b2f295c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.762+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/3568149764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 0x55b2f295c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.762+0000 7fe4fcc42640 1 -- 172.21.15.129:0/3568149764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2f2937420 con 0x55b2f25722f0 2024-05-20T18:01:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.763+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/3568149764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f2944660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009ea0 tx=0x7fe4e00093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89fa03ce5239dc9e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.766+0000 7fe4e67fc640 1 -- 172.21.15.129:0/3568149764 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0033030 con 0x55b2f25722f0 2024-05-20T18:01:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.766+0000 7fe4e67fc640 1 -- 172.21.15.129:0/3568149764 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4e0013b50 con 0x55b2f25722f0 2024-05-20T18:01:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.767+0000 7fe505e20740 1 -- 172.21.15.129:0/3568149764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 msgr2=0x55b2f2944660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.767+0000 7fe505e20740 1 --2- 172.21.15.129:0/3568149764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f2944660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009ea0 tx=0x7fe4e00093e0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.771+0000 7fe505e20740 1 -- 172.21.15.129:0/3568149764 shutdown_connections 2024-05-20T18:01:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.771+0000 7fe505e20740 1 --2- 172.21.15.129:0/3568149764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 0x55b2f295c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.771+0000 7fe505e20740 1 --2- 172.21.15.129:0/3568149764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 0x55b2f295c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.771+0000 7fe505e20740 1 --2- 172.21.15.129:0/3568149764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f2944660 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.771 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.771+0000 7fe505e20740 1 -- 172.21.15.129:0/3568149764 >> 172.21.15.129:0/3568149764 conn(0x55b2f258f300 msgr2=0x55b2f2943f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.778+0000 7fe505e20740 1 -- 172.21.15.129:0/3568149764 shutdown_connections 2024-05-20T18:01:12.776 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.778+0000 7fe505e20740 1 -- 172.21.15.129:0/3568149764 wait complete. 2024-05-20T18:01:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.779+0000 7fe505e20740 1 -- start start 2024-05-20T18:01:12.777 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.779+0000 7fe505e20740 1 Processor -- start 2024-05-20T18:01:12.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.780+0000 7fe505e20740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f294c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.781+0000 7fe505e20740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 0x55b2f2947610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.781+0000 7fe505e20740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 0x55b2f2947b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.781+0000 7fe505e20740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d800d420 con 0x55b2f259b1b0 2024-05-20T18:01:12.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.781+0000 7fe505e20740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d800e890 con 0x55b2f25722f0 2024-05-20T18:01:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.781+0000 7fe4fcc42640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f294c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.781+0000 7fe4fcc42640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f294c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52538/0 (socket says 172.21.15.129:52538) 2024-05-20T18:01:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.781+0000 7fe4fcc42640 1 -- 172.21.15.129:0/1049855933 learned_addr learned my addr 172.21.15.129:0/1049855933 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.781+0000 7fe505e20740 1 -- 172.21.15.129:0/1049855933 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d800daf0 con 0x55b2f257a4d0 2024-05-20T18:01:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.781+0000 7fe505e20740 10 client.0 _ll_register_callbacks cb 0x7fe4e800a370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.782+0000 7fe4f7fff640 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 0x55b2f2947610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.783+0000 7fe4fcc42640 1 -- 172.21.15.129:0/1049855933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 msgr2=0x55b2f2947610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.783+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 0x55b2f2947610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.783+0000 7fe4fcc42640 1 -- 172.21.15.129:0/1049855933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 msgr2=0x55b2f2947b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:12.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.783+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 0x55b2f2947b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:12.784 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.783+0000 7fe4fcc42640 1 -- 172.21.15.129:0/1049855933 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2f2948090 con 0x55b2f25722f0 2024-05-20T18:01:12.784 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.783+0000 7fe4fcc42640 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f294c870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0010010 tx=0x7fe4e00092c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=994485f9de601106 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:12.788 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.790+0000 7fe4dffff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:01:12.788 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.790+0000 7fe4dffff640 1 -- 172.21.15.129:0/1049855933 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e003c070 con 0x55b2f25722f0 2024-05-20T18:01:12.788 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.790+0000 7fe4dffff640 1 -- 172.21.15.129:0/1049855933 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fe4e002ee00 con 0x55b2f25722f0 2024-05-20T18:01:12.788 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.790+0000 7fe505e20740 1 -- 172.21.15.129:0/1049855933 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4e0013b50 con 0x55b2f25722f0 2024-05-20T18:01:12.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.790+0000 7fe4dffff640 1 -- 172.21.15.129:0/1049855933 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fe4e0047050 con 0x55b2f25722f0 2024-05-20T18:01:12.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.791+0000 7fe505e20740 10 client.25583 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:12.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.791+0000 7fe505e20740 1 -- 172.21.15.129:0/1049855933 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55b2f29bdbc0 con 0x55b2f25722f0 2024-05-20T18:01:12.789 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.791+0000 7fe4dffff640 1 -- 172.21.15.129:0/1049855933 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4e0013b50 con 0x55b2f25722f0 2024-05-20T18:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.791+0000 7fe4dffff640 1 -- 172.21.15.129:0/1049855933 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fe4e00384f0 con 0x55b2f25722f0 2024-05-20T18:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.791+0000 7fe505e20740 10 client.25583 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.791+0000 7fe505e20740 10 client.25583 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.791+0000 7fe505e20740 20 client.25583 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.792+0000 7fe505e20740 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2f252f1c0 0x55b2f29c0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.792+0000 7fe505e20740 4 client.25583 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.792+0000 7fe505e20740 1 -- 172.21.15.129:0/1049855933 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b2f28a6dc0 con 0x55b2f252f1c0 2024-05-20T18:01:12.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.792+0000 7fe4f7fff640 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2f252f1c0 0x55b2f29c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:12.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.795+0000 7fe4dffff640 10 client.25583 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:12.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:12.795+0000 7fe4f7fff640 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2f252f1c0 0x55b2f29c0230 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b139ded7d7cdea3 server_cookie=68189562ab8fd2f in_seq=0 out_seq=0 2024-05-20T18:01:13.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:12 smithi129 ceph-mon[32029]: pgmap v83: 97 pgs: 97 active+clean; 12 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 821 MiB/s wr, 3.64k op/s 2024-05-20T18:01:13.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:12 smithi204 ceph-mon[37076]: pgmap v83: 97 pgs: 97 active+clean; 12 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 821 MiB/s wr, 3.64k op/s 2024-05-20T18:01:13.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:12 smithi154 ceph-mon[36482]: pgmap v83: 97 pgs: 97 active+clean; 12 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 821 MiB/s wr, 3.64k op/s 2024-05-20T18:01:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe4dffff640 1 -- 172.21.15.129:0/1049855933 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b2f28a6dc0 con 0x55b2f252f1c0 2024-05-20T18:01:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe4dffff640 10 client.25583 handle_command_reply: tid=0 2024-05-20T18:01:14.722 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe505e20740 1 client.25583 shutdown 2024-05-20T18:01:14.722 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe505e20740 1 -- 172.21.15.129:0/1049855933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 msgr2=0x55b2f294c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:14.722 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe505e20740 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f294c870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0010010 tx=0x7fe4e00092c0 comp rx=0 tx=0).stop 2024-05-20T18:01:14.722 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe505e20740 1 -- 172.21.15.129:0/1049855933 shutdown_connections 2024-05-20T18:01:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe505e20740 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b2f252f1c0 0x55b2f29c0230 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe505e20740 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b2f259b1b0 0x55b2f2947b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe505e20740 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b2f257a4d0 0x55b2f2947610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe505e20740 1 --2- 172.21.15.129:0/1049855933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f25722f0 0x55b2f294c870 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe505e20740 1 -- 172.21.15.129:0/1049855933 >> 172.21.15.129:0/1049855933 conn(0x55b2f258f300 msgr2=0x55b2f2945c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.723+0000 7fe4dffff640 0 client.25583 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.724+0000 7fe505e20740 1 -- 172.21.15.129:0/1049855933 shutdown_connections 2024-05-20T18:01:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.724+0000 7fe505e20740 1 -- 172.21.15.129:0/1049855933 wait complete. 2024-05-20T18:01:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.724+0000 7fe505e20740 20 client.25583 trim_cache size 0 max 16384 2024-05-20T18:01:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:01:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:01:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:01:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:01:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.728+0000 7fe4feea6640 1 -- 172.21.15.129:0/4277203400 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4ec0a2c60 msgr2=0x7fe4ec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.729+0000 7fe4feea6640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4ec0a2c60 0x7fe4ec0a5170 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55b2f29371f0 tx=0x7fe4e800c040 comp rx=0 tx=0).stop 2024-05-20T18:01:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.729+0000 7fe4feea6640 1 -- 172.21.15.129:0/4277203400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4f810e7c0 msgr2=0x7fe4f8120540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.729+0000 7fe4feea6640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4f810e7c0 0x7fe4f8120540 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe4f002ae30 tx=0x7fe4f0030ca0 comp rx=0 tx=0).stop 2024-05-20T18:01:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.729+0000 7fe4feea6640 1 -- 172.21.15.129:0/4277203400 shutdown_connections 2024-05-20T18:01:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.729+0000 7fe4feea6640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4ec0a2c60 0x7fe4ec0a5170 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.729+0000 7fe4feea6640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4f8125c30 0x7fe4f811b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.729+0000 7fe4feea6640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4f810e7c0 0x7fe4f8120540 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.729+0000 7fe4feea6640 1 --2- 172.21.15.129:0/4277203400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4f8109770 0x7fe4f811ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.729+0000 7fe4feea6640 1 -- 172.21.15.129:0/4277203400 >> 172.21.15.129:0/4277203400 conn(0x7fe4f806d780 msgr2=0x7fe4f810cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.730+0000 7fe4feea6640 1 -- 172.21.15.129:0/4277203400 shutdown_connections 2024-05-20T18:01:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:14.730+0000 7fe4feea6640 1 -- 172.21.15.129:0/4277203400 wait complete. 2024-05-20T18:01:15.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:14 smithi129 ceph-mon[32029]: pgmap v84: 97 pgs: 97 active+clean; 12 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 684 MiB/s wr, 3.17k op/s 2024-05-20T18:01:15.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:14 smithi204 ceph-mon[37076]: pgmap v84: 97 pgs: 97 active+clean; 12 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 684 MiB/s wr, 3.17k op/s 2024-05-20T18:01:15.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:14 smithi154 ceph-mon[36482]: pgmap v84: 97 pgs: 97 active+clean; 12 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 684 MiB/s wr, 3.17k op/s 2024-05-20T18:01:17.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:16 smithi204 ceph-mon[37076]: pgmap v85: 97 pgs: 97 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 910 MiB/s wr, 5.22k op/s 2024-05-20T18:01:17.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:16 smithi154 ceph-mon[36482]: pgmap v85: 97 pgs: 97 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 910 MiB/s wr, 5.22k op/s 2024-05-20T18:01:17.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:16 smithi129 ceph-mon[32029]: pgmap v85: 97 pgs: 97 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 910 MiB/s wr, 5.22k op/s 2024-05-20T18:01:19.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:18 smithi129 ceph-mon[32029]: pgmap v86: 97 pgs: 97 active+clean; 13 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 655 MiB/s wr, 5.41k op/s 2024-05-20T18:01:19.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:18 smithi154 ceph-mon[36482]: pgmap v86: 97 pgs: 97 active+clean; 13 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 655 MiB/s wr, 5.41k op/s 2024-05-20T18:01:19.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:18 smithi204 ceph-mon[37076]: pgmap v86: 97 pgs: 97 active+clean; 13 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 655 MiB/s wr, 5.41k op/s 2024-05-20T18:01:21.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:20 smithi154 ceph-mon[36482]: pgmap v87: 97 pgs: 97 active+clean; 14 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 837 MiB/s wr, 6.69k op/s 2024-05-20T18:01:21.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:20 smithi129 ceph-mon[32029]: pgmap v87: 97 pgs: 97 active+clean; 14 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 837 MiB/s wr, 6.69k op/s 2024-05-20T18:01:21.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:20 smithi204 ceph-mon[37076]: pgmap v87: 97 pgs: 97 active+clean; 14 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 837 MiB/s wr, 6.69k op/s 2024-05-20T18:01:23.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:22 smithi154 ceph-mon[36482]: pgmap v88: 97 pgs: 97 active+clean; 15 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 747 MiB/s wr, 6k op/s 2024-05-20T18:01:23.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:22 smithi129 ceph-mon[32029]: pgmap v88: 97 pgs: 97 active+clean; 15 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 747 MiB/s wr, 6k op/s 2024-05-20T18:01:23.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:22 smithi204 ceph-mon[37076]: pgmap v88: 97 pgs: 97 active+clean; 15 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 747 MiB/s wr, 6k op/s 2024-05-20T18:01:25.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:24 smithi154 ceph-mon[36482]: pgmap v89: 97 pgs: 97 active+clean; 15 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 625 MiB/s wr, 5.06k op/s 2024-05-20T18:01:25.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:24 smithi129 ceph-mon[32029]: pgmap v89: 97 pgs: 97 active+clean; 15 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 625 MiB/s wr, 5.06k op/s 2024-05-20T18:01:25.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:24 smithi204 ceph-mon[37076]: pgmap v89: 97 pgs: 97 active+clean; 15 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 625 MiB/s wr, 5.06k op/s 2024-05-20T18:01:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:27 smithi154 ceph-mon[36482]: pgmap v90: 97 pgs: 97 active+clean; 17 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 831 MiB/s wr, 5.80k op/s 2024-05-20T18:01:27.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:27 smithi129 ceph-mon[32029]: pgmap v90: 97 pgs: 97 active+clean; 17 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 831 MiB/s wr, 5.80k op/s 2024-05-20T18:01:27.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:27 smithi204 ceph-mon[37076]: pgmap v90: 97 pgs: 97 active+clean; 17 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 831 MiB/s wr, 5.80k op/s 2024-05-20T18:01:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.106+0000 7f078fa44640 1 -- 172.21.15.129:0/3421686200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078810e370 msgr2=0x7f0788079e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:28.106 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.106+0000 7f078fa44640 1 --2- 172.21.15.129:0/3421686200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078810e370 0x7f0788079e10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0780009ca0 tx=0x7f078002a8d0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.117+0000 7f078fa44640 1 -- 172.21.15.129:0/3421686200 shutdown_connections 2024-05-20T18:01:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.117+0000 7f078fa44640 1 --2- 172.21.15.129:0/3421686200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f078810e370 0x7f0788079e10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.117+0000 7f078fa44640 1 --2- 172.21.15.129:0/3421686200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f078810d8b0 0x7f078810dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.117+0000 7f078fa44640 1 --2- 172.21.15.129:0/3421686200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0788109320 0x7f0788109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.117+0000 7f078fa44640 1 -- 172.21.15.129:0/3421686200 >> 172.21.15.129:0/3421686200 conn(0x7f078806d780 msgr2=0x7f078806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:28.123 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.125+0000 7f078fa44640 1 -- 172.21.15.129:0/3421686200 shutdown_connections 2024-05-20T18:01:28.134 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.130+0000 7f078fa44640 1 -- 172.21.15.129:0/3421686200 wait complete. 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.132+0000 7f078fa44640 1 Processor -- start 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.132+0000 7f078fa44640 1 -- start start 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.132+0000 7f078fa44640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0788109320 0x7f078807f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.132+0000 7f078fa44640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f078810d8b0 0x7f078807f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.132+0000 7f078fa44640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f078810e370 0x7f0788131b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.132+0000 7f078fa44640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f078807d6b0 con 0x7f078810e370 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.132+0000 7f078fa44640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f078807d530 con 0x7f078810d8b0 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.132+0000 7f078fa44640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f078807d830 con 0x7f0788109320 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.135+0000 7f078d7e0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0788109320 0x7f078807f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.136+0000 7f078d7e0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0788109320 0x7f078807f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:56410/0 (socket says 172.21.15.129:56410) 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.136+0000 7f078d7e0640 1 -- 172.21.15.129:0/2817630941 learned_addr learned my addr 172.21.15.129:0/2817630941 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:28.135 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.136+0000 7f078dfe1640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f078810e370 0x7f0788131b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.136+0000 7f078cfdf640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f078810d8b0 0x7f078807f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.138+0000 7f078cfdf640 1 -- 172.21.15.129:0/2817630941 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0788109320 msgr2=0x7f078807f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.138+0000 7f078cfdf640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0788109320 0x7f078807f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.138+0000 7f078cfdf640 1 -- 172.21.15.129:0/2817630941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f078810e370 msgr2=0x7f0788131b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.138+0000 7f078cfdf640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f078810e370 0x7f0788131b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.138+0000 7f078cfdf640 1 -- 172.21.15.129:0/2817630941 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0788132110 con 0x7f078810d8b0 2024-05-20T18:01:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.139+0000 7f078cfdf640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f078810d8b0 0x7f078807f8a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f077800d6a0 tx=0x7f077800db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.143+0000 7f077e7fc640 1 -- 172.21.15.129:0/2817630941 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778017070 con 0x7f078810d8b0 2024-05-20T18:01:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.143+0000 7f078fa44640 1 -- 172.21.15.129:0/2817630941 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0788132400 con 0x7f078810d8b0 2024-05-20T18:01:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.143+0000 7f078fa44640 1 -- 172.21.15.129:0/2817630941 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0788138bb0 con 0x7f078810d8b0 2024-05-20T18:01:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.143+0000 7f077e7fc640 1 -- 172.21.15.129:0/2817630941 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0778009820 con 0x7f078810d8b0 2024-05-20T18:01:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.143+0000 7f077e7fc640 1 -- 172.21.15.129:0/2817630941 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778004d00 con 0x7f078810d8b0 2024-05-20T18:01:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.160+0000 7f077e7fc640 1 -- 172.21.15.129:0/2817630941 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f0778005220 con 0x7f078810d8b0 2024-05-20T18:01:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.161+0000 7f07969c6740 1 Processor -- start 2024-05-20T18:01:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.163+0000 7f077e7fc640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07740a2f00 0x7f07740a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.163+0000 7f077e7fc640 1 -- 172.21.15.129:0/2817630941 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f07780cecc0 con 0x7f078810d8b0 2024-05-20T18:01:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.168+0000 7f07969c6740 1 -- start start 2024-05-20T18:01:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.170+0000 7f07969c6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88a4e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.170+0000 7f07969c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 0x55cd88a4e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.170+0000 7f07969c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a4eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.170+0000 7f07969c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cd889b3340 con 0x55cd886862f0 2024-05-20T18:01:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.170+0000 7f07969c6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cd8895b290 con 0x55cd8868e4d0 2024-05-20T18:01:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.170+0000 7f07969c6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55cd889af300 con 0x55cd886af1b0 2024-05-20T18:01:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.172+0000 7f078dfe1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a4eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.172+0000 7f078dfe1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a4eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:39540/0 (socket says 172.21.15.129:39540) 2024-05-20T18:01:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.172+0000 7f078dfe1640 1 -- 172.21.15.129:0/4173882892 learned_addr learned my addr 172.21.15.129:0/4173882892 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.173+0000 7f078d7e0640 1 --2- 172.21.15.129:0/4173882892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88a4e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.174+0000 7f078dfe1640 1 -- 172.21.15.129:0/4173882892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 msgr2=0x55cd88a4e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.174+0000 7f078dfe1640 1 --2- 172.21.15.129:0/4173882892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88a4e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.174+0000 7f078dfe1640 1 -- 172.21.15.129:0/4173882892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 msgr2=0x55cd88a4e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.174+0000 7f078dfe1640 1 --2- 172.21.15.129:0/4173882892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 0x55cd88a4e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.174+0000 7f078dfe1640 1 -- 172.21.15.129:0/4173882892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd8897f8e0 con 0x55cd8868e4d0 2024-05-20T18:01:28.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.178+0000 7f078dfe1640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07740a2f00 0x7f07740a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.179+0000 7f078dfe1640 1 --2- 172.21.15.129:0/4173882892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a4eb00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0780009880 tx=0x7f07800098b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d154b3fcdabed2c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.181+0000 7f07737fe640 1 -- 172.21.15.129:0/4173882892 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078004d070 con 0x55cd8868e4d0 2024-05-20T18:01:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.181+0000 7f07737fe640 1 -- 172.21.15.129:0/4173882892 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0780036040 con 0x55cd8868e4d0 2024-05-20T18:01:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.181+0000 7f07737fe640 1 -- 172.21.15.129:0/4173882892 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07800305d0 con 0x55cd8868e4d0 2024-05-20T18:01:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.183+0000 7f07969c6740 1 -- 172.21.15.129:0/4173882892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 msgr2=0x55cd88a4eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.183+0000 7f07969c6740 1 --2- 172.21.15.129:0/4173882892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a4eb00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0780009880 tx=0x7f07800098b0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.194+0000 7f07969c6740 1 -- 172.21.15.129:0/4173882892 shutdown_connections 2024-05-20T18:01:28.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.194+0000 7f07969c6740 1 --2- 172.21.15.129:0/4173882892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a4eb00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.194+0000 7f07969c6740 1 --2- 172.21.15.129:0/4173882892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 0x55cd88a4e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.194+0000 7f07969c6740 1 --2- 172.21.15.129:0/4173882892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88a4e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.194+0000 7f07969c6740 1 -- 172.21.15.129:0/4173882892 >> 172.21.15.129:0/4173882892 conn(0x55cd886a3300 msgr2=0x55cd88a4a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:28.193 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.195+0000 7f07969c6740 1 -- 172.21.15.129:0/4173882892 shutdown_connections 2024-05-20T18:01:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.195+0000 7f07969c6740 1 -- 172.21.15.129:0/4173882892 wait complete. 2024-05-20T18:01:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.196+0000 7f07969c6740 1 -- start start 2024-05-20T18:01:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.196+0000 7f07969c6740 1 Processor -- start 2024-05-20T18:01:28.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.198+0000 7f07969c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 0x55cd88a5a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.198+0000 7f07969c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a5a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.198+0000 7f07969c6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88a585b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.198+0000 7f07969c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cd88a5b1b0 con 0x55cd886862f0 2024-05-20T18:01:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.198+0000 7f07969c6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cd8896e790 con 0x55cd8868e4d0 2024-05-20T18:01:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.198+0000 7f07969c6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55cd88a5b330 con 0x55cd886af1b0 2024-05-20T18:01:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.198+0000 7f07969c6740 10 client.0 _ll_register_callbacks cb 0x55cd889a9a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.200+0000 7f078dfe1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88a585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.200+0000 7f078dfe1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88a585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:56424/0 (socket says 172.21.15.129:56424) 2024-05-20T18:01:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.200+0000 7f078dfe1640 1 -- 172.21.15.129:0/2704860053 learned_addr learned my addr 172.21.15.129:0/2704860053 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.200+0000 7f078dfe1640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07740a2f00 0x7f07740a5410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0780030040 tx=0x7f07800098b0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.200+0000 7f078d7e0640 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 0x55cd88a5a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.201+0000 7f078cfdf640 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a5a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.201+0000 7f078dfe1640 1 -- 172.21.15.129:0/2704860053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 msgr2=0x55cd88a5a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.201+0000 7f078dfe1640 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a5a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.201+0000 7f078dfe1640 1 -- 172.21.15.129:0/2704860053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 msgr2=0x55cd88a5a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.201+0000 7f078dfe1640 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 0x55cd88a5a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.201+0000 7f078dfe1640 1 -- 172.21.15.129:0/2704860053 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd886c7210 con 0x55cd886af1b0 2024-05-20T18:01:28.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.202+0000 7f078dfe1640 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88a585b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0780009bf0 tx=0x7f07800095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2b459a952072087 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.203+0000 7f0770ff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:01:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.203+0000 7f0770ff9640 1 -- 172.21.15.129:0/2704860053 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078005a030 con 0x55cd886af1b0 2024-05-20T18:01:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.203+0000 7f07969c6740 1 -- 172.21.15.129:0/2704860053 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0780036040 con 0x55cd886af1b0 2024-05-20T18:01:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.204+0000 7f0770ff9640 1 -- 172.21.15.129:0/2704860053 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f0780031830 con 0x55cd886af1b0 2024-05-20T18:01:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.204+0000 7f0770ff9640 1 -- 172.21.15.129:0/2704860053 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f078004d040 con 0x55cd886af1b0 2024-05-20T18:01:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.205+0000 7f07969c6740 10 client.25621 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.205+0000 7f07969c6740 1 -- 172.21.15.129:0/2704860053 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55cd88793620 con 0x55cd886af1b0 2024-05-20T18:01:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.206+0000 7f0770ff9640 1 -- 172.21.15.129:0/2704860053 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0780036040 con 0x55cd886af1b0 2024-05-20T18:01:28.205 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.206+0000 7f0770ff9640 1 -- 172.21.15.129:0/2704860053 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f0780048840 con 0x55cd886af1b0 2024-05-20T18:01:28.208 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.209+0000 7f07969c6740 10 client.25621 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:28.208 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.209+0000 7f07969c6740 10 client.25621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:28.208 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.209+0000 7f07969c6740 20 client.25621 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:28.208 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.209+0000 7f07969c6740 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cd8866dd80 0x55cd88a6a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.208 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.209+0000 7f07969c6740 4 client.25621 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:01:28.208 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.209+0000 7f07969c6740 1 -- 172.21.15.129:0/2704860053 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd889c0a70 con 0x55cd8866dd80 2024-05-20T18:01:28.208 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.210+0000 7f078d7e0640 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cd8866dd80 0x55cd88a6a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.216+0000 7f0770ff9640 10 client.25621 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.216+0000 7f078d7e0640 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cd8866dd80 0x55cd88a6a990 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89102596e1e97ac9 server_cookie=edd385477c5be341 in_seq=0 out_seq=0 2024-05-20T18:01:28.223 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.225+0000 7f0770ff9640 1 -- 172.21.15.129:0/2704860053 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cd889c0a70 con 0x55cd8866dd80 2024-05-20T18:01:28.223 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.225+0000 7f0770ff9640 10 client.25621 handle_command_reply: tid=0 2024-05-20T18:01:28.223 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.225+0000 7f07969c6740 1 client.25621 shutdown 2024-05-20T18:01:28.226 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.227+0000 7f07969c6740 1 -- 172.21.15.129:0/2704860053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 msgr2=0x55cd88a585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:28.226 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.227+0000 7f07969c6740 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88a585b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0780009bf0 tx=0x7f07800095c0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.231+0000 7f07969c6740 1 -- 172.21.15.129:0/2704860053 shutdown_connections 2024-05-20T18:01:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.231+0000 7f07969c6740 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cd8866dd80 0x55cd88a6a990 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.231+0000 7f07969c6740 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a5a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.231+0000 7f07969c6740 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88a585b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.231+0000 7f07969c6740 1 --2- 172.21.15.129:0/2704860053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 0x55cd88a5a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.231+0000 7f07969c6740 1 -- 172.21.15.129:0/2704860053 >> 172.21.15.129:0/2704860053 conn(0x55cd886a3300 msgr2=0x55cd88a4e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.231+0000 7f0770ff9640 0 client.25621 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.232+0000 7f07969c6740 1 -- 172.21.15.129:0/2704860053 shutdown_connections 2024-05-20T18:01:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.232+0000 7f07969c6740 1 -- 172.21.15.129:0/2704860053 wait complete. 2024-05-20T18:01:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.232+0000 7f07969c6740 20 client.25621 trim_cache size 0 max 16384 2024-05-20T18:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.279+0000 7f07969c6740 1 Processor -- start 2024-05-20T18:01:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.279+0000 7f07969c6740 1 -- start start 2024-05-20T18:01:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.280+0000 7f07969c6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886862f0 0x55cd88a523e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.280+0000 7f07969c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a605f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.280+0000 7f078d7e0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886862f0 0x55cd88a523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.281+0000 7f078d7e0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886862f0 0x55cd88a523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:56436/0 (socket says 172.21.15.129:56436) 2024-05-20T18:01:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.281+0000 7f078d7e0640 1 -- 172.21.15.129:0/1229914753 learned_addr learned my addr 172.21.15.129:0/1229914753 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.281+0000 7f07969c6740 1 --2- 172.21.15.129:0/1229914753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886af1b0 0x55cd88a60b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.281+0000 7f07969c6740 1 -- 172.21.15.129:0/1229914753 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f076c00c8c0 con 0x55cd886af1b0 2024-05-20T18:01:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.281+0000 7f078cfdf640 1 --2- 172.21.15.129:0/1229914753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.281+0000 7f07969c6740 1 -- 172.21.15.129:0/1229914753 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f076c00f140 con 0x55cd8868e4d0 2024-05-20T18:01:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.281+0000 7f07969c6740 1 -- 172.21.15.129:0/1229914753 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f076c00cf30 con 0x55cd886862f0 2024-05-20T18:01:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.282+0000 7f078dfe1640 1 --2- 172.21.15.129:0/1229914753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886af1b0 0x55cd88a60b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.283+0000 7f078cfdf640 1 -- 172.21.15.129:0/1229914753 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886862f0 msgr2=0x55cd88a523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:28.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.283+0000 7f078cfdf640 1 --2- 172.21.15.129:0/1229914753 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886862f0 0x55cd88a523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.283+0000 7f078cfdf640 1 -- 172.21.15.129:0/1229914753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886af1b0 msgr2=0x55cd88a60b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:28.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.283+0000 7f078cfdf640 1 --2- 172.21.15.129:0/1229914753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886af1b0 0x55cd88a60b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.283+0000 7f078cfdf640 1 -- 172.21.15.129:0/1229914753 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd88a53dd0 con 0x55cd8868e4d0 2024-05-20T18:01:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.287+0000 7f078cfdf640 1 --2- 172.21.15.129:0/1229914753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a605f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f07780cc3a0 tx=0x7f077809aad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89688f1839774c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.288+0000 7f0772ffd640 1 -- 172.21.15.129:0/1229914753 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780b4070 con 0x55cd8868e4d0 2024-05-20T18:01:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.288+0000 7f0772ffd640 1 -- 172.21.15.129:0/1229914753 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f077809f520 con 0x55cd8868e4d0 2024-05-20T18:01:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.288+0000 7f0772ffd640 1 -- 172.21.15.129:0/1229914753 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778020040 con 0x55cd8868e4d0 2024-05-20T18:01:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.290+0000 7f07969c6740 1 -- 172.21.15.129:0/1229914753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 msgr2=0x55cd88a605f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.290+0000 7f07969c6740 1 --2- 172.21.15.129:0/1229914753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a605f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f07780cc3a0 tx=0x7f077809aad0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.296+0000 7f07969c6740 1 -- 172.21.15.129:0/1229914753 shutdown_connections 2024-05-20T18:01:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.296+0000 7f07969c6740 1 --2- 172.21.15.129:0/1229914753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a605f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.296+0000 7f07969c6740 1 --2- 172.21.15.129:0/1229914753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886af1b0 0x55cd88a60b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.296+0000 7f07969c6740 1 --2- 172.21.15.129:0/1229914753 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886862f0 0x55cd88a523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.296+0000 7f07969c6740 1 -- 172.21.15.129:0/1229914753 >> 172.21.15.129:0/1229914753 conn(0x55cd886a3300 msgr2=0x55cd88a51cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.296+0000 7f07969c6740 1 -- 172.21.15.129:0/1229914753 shutdown_connections 2024-05-20T18:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.296+0000 7f07969c6740 1 -- 172.21.15.129:0/1229914753 wait complete. 2024-05-20T18:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.297+0000 7f07969c6740 1 -- start start 2024-05-20T18:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.297+0000 7f07969c6740 1 Processor -- start 2024-05-20T18:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.299+0000 7f07969c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 0x55cd88a5fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.299+0000 7f07969c6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a60310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.300+0000 7f07969c6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88ac9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.300+0000 7f07969c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f076c00d600 con 0x55cd886862f0 2024-05-20T18:01:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.300+0000 7f07969c6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f076c00ea70 con 0x55cd8868e4d0 2024-05-20T18:01:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.300+0000 7f07969c6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f076c00dcd0 con 0x55cd886af1b0 2024-05-20T18:01:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.300+0000 7f078cfdf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a60310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.300+0000 7f078cfdf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a60310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:39566/0 (socket says 172.21.15.129:39566) 2024-05-20T18:01:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.300+0000 7f078cfdf640 1 -- 172.21.15.129:0/3606393377 learned_addr learned my addr 172.21.15.129:0/3606393377 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.301+0000 7f078dfe1640 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88ac9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.301+0000 7f07969c6740 10 client.0 _ll_register_callbacks cb 0x7f078004b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.301+0000 7f078cfdf640 1 -- 172.21.15.129:0/3606393377 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 msgr2=0x55cd88ac9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.301+0000 7f078cfdf640 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88ac9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.301+0000 7f078cfdf640 1 -- 172.21.15.129:0/3606393377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 msgr2=0x55cd88a5fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.301+0000 7f078cfdf640 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 0x55cd88a5fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.301+0000 7f078cfdf640 1 -- 172.21.15.129:0/3606393377 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd88a5a4a0 con 0x55cd8868e4d0 2024-05-20T18:01:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.303+0000 7f07969c6740 1 -- 172.21.15.129:0/3606393377 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f077809f520 con 0x55cd8868e4d0 2024-05-20T18:01:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.304+0000 7f078cfdf640 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a60310 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f077809ae70 tx=0x7f077809b230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=627ae4185a3fe432 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.304+0000 7f0770ff9640 10 client.25595 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:01:28.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.305+0000 7f0770ff9640 1 -- 172.21.15.129:0/3606393377 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778020070 con 0x55cd8868e4d0 2024-05-20T18:01:28.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.306+0000 7f0770ff9640 1 -- 172.21.15.129:0/3606393377 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f07780bb0b0 con 0x55cd8868e4d0 2024-05-20T18:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.306+0000 7f0770ff9640 1 -- 172.21.15.129:0/3606393377 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f07780b4040 con 0x55cd8868e4d0 2024-05-20T18:01:28.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.307+0000 7f07969c6740 10 client.25595 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.307+0000 7f07969c6740 1 -- 172.21.15.129:0/3606393377 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({fsmap=22}) -- 0x55cd88ad25c0 con 0x55cd8868e4d0 2024-05-20T18:01:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.312+0000 7f0770ff9640 1 -- 172.21.15.129:0/3606393377 <== mon.2 v2:172.21.15.154:3300/0 4 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f0778095ce0 con 0x55cd8868e4d0 2024-05-20T18:01:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.312+0000 7f07969c6740 10 client.25595 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.313+0000 7f07969c6740 10 client.25595 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.313+0000 7f07969c6740 20 client.25595 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.313+0000 7f07969c6740 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cd886431c0 0x55cd88ad4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.313+0000 7f07969c6740 4 client.25595 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:01:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.313+0000 7f07969c6740 1 -- 172.21.15.129:0/3606393377 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f077809f520 con 0x55cd886431c0 2024-05-20T18:01:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.323+0000 7f078d7e0640 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cd886431c0 0x55cd88ad4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.360+0000 7f078d7e0640 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cd886431c0 0x55cd88ad4b70 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa44d0ee0ed201a5 server_cookie=d39aa3c94c86de4a in_seq=0 out_seq=0 2024-05-20T18:01:28.359 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:28.361+0000 7f0770ff9640 10 client.25595 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:29.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:29 smithi154 ceph-mon[36482]: pgmap v91: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 704 MiB/s wr, 4.08k op/s 2024-05-20T18:01:29.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:29 smithi129 ceph-mon[32029]: pgmap v91: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 704 MiB/s wr, 4.08k op/s 2024-05-20T18:01:29.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:29 smithi204 ceph-mon[37076]: pgmap v91: 97 pgs: 97 active+clean; 17 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 704 MiB/s wr, 4.08k op/s 2024-05-20T18:01:31.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:31 smithi154 ceph-mon[36482]: pgmap v92: 97 pgs: 97 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 814 MiB/s wr, 3.62k op/s 2024-05-20T18:01:31.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:31 smithi129 ceph-mon[32029]: pgmap v92: 97 pgs: 97 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 814 MiB/s wr, 3.62k op/s 2024-05-20T18:01:31.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:31 smithi204 ceph-mon[37076]: pgmap v92: 97 pgs: 97 active+clean; 18 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 814 MiB/s wr, 3.62k op/s 2024-05-20T18:01:33.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:33 smithi154 ceph-mon[36482]: pgmap v93: 97 pgs: 97 active+clean; 19 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 761 MiB/s wr, 2.48k op/s 2024-05-20T18:01:33.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:33 smithi129 ceph-mon[32029]: pgmap v93: 97 pgs: 97 active+clean; 19 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 761 MiB/s wr, 2.48k op/s 2024-05-20T18:01:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.426+0000 7f0770ff9640 1 -- 172.21.15.129:0/3606393377 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f077809f520 con 0x55cd886431c0 2024-05-20T18:01:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.426+0000 7f0770ff9640 10 client.25595 handle_command_reply: tid=0 2024-05-20T18:01:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.426+0000 7f07969c6740 1 client.25595 shutdown 2024-05-20T18:01:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.427+0000 7f07969c6740 1 -- 172.21.15.129:0/3606393377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 msgr2=0x55cd88a60310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.427+0000 7f07969c6740 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a60310 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f077809ae70 tx=0x7f077809b230 comp rx=0 tx=0).stop 2024-05-20T18:01:33.426 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.427+0000 7f07969c6740 1 -- 172.21.15.129:0/3606393377 shutdown_connections 2024-05-20T18:01:33.426 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.427+0000 7f07969c6740 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55cd886431c0 0x55cd88ad4b70 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:33.426 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.427+0000 7f07969c6740 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cd8868e4d0 0x55cd88a60310 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:33.426 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.427+0000 7f07969c6740 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd886af1b0 0x55cd88ac9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:33.426 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.427+0000 7f07969c6740 1 --2- 172.21.15.129:0/3606393377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd886862f0 0x55cd88a5fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:33.426 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.427+0000 7f07969c6740 1 -- 172.21.15.129:0/3606393377 >> 172.21.15.129:0/3606393377 conn(0x55cd886a3300 msgr2=0x55cd88a4de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:33.426 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.427+0000 7f0770ff9640 0 client.25595 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:33.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:33 smithi204 ceph-mon[37076]: pgmap v93: 97 pgs: 97 active+clean; 19 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 761 MiB/s wr, 2.48k op/s 2024-05-20T18:01:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.432+0000 7f07969c6740 1 -- 172.21.15.129:0/3606393377 shutdown_connections 2024-05-20T18:01:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.432+0000 7f07969c6740 1 -- 172.21.15.129:0/3606393377 wait complete. 2024-05-20T18:01:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.433+0000 7f07969c6740 20 client.25595 trim_cache size 0 max 16384 2024-05-20T18:01:33.432 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:01:33.432 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:01:33.432 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:01:33.432 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:01:33.434 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.436+0000 7f078fa44640 1 -- 172.21.15.129:0/2817630941 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07740a2f00 msgr2=0x7f07740a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:33.434 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.436+0000 7f078fa44640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07740a2f00 0x7f07740a5410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0780030040 tx=0x7f07800098b0 comp rx=0 tx=0).stop 2024-05-20T18:01:33.435 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.436+0000 7f078fa44640 1 -- 172.21.15.129:0/2817630941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f078810d8b0 msgr2=0x7f078807f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.437+0000 7f078fa44640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f078810d8b0 0x7f078807f8a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f077800d6a0 tx=0x7f077800db60 comp rx=0 tx=0).stop 2024-05-20T18:01:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.439+0000 7f078fa44640 1 -- 172.21.15.129:0/2817630941 shutdown_connections 2024-05-20T18:01:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.439+0000 7f078fa44640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07740a2f00 0x7f07740a5410 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.439+0000 7f078fa44640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f078810d8b0 0x7f078807f8a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.439+0000 7f078fa44640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f078810e370 0x7f0788131b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.439+0000 7f078fa44640 1 --2- 172.21.15.129:0/2817630941 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0788109320 0x7f078807f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.439+0000 7f078fa44640 1 -- 172.21.15.129:0/2817630941 >> 172.21.15.129:0/2817630941 conn(0x7f078806d780 msgr2=0x7f078810b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.439+0000 7f078fa44640 1 -- 172.21.15.129:0/2817630941 shutdown_connections 2024-05-20T18:01:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:33.440+0000 7f078fa44640 1 -- 172.21.15.129:0/2817630941 wait complete. 2024-05-20T18:01:33.663 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:01:33.663 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:01:34.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.189+0000 7f1d2372a640 1 -- 172.21.15.129:0/1800946154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d1c109300 msgr2=0x7f1d1c109710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:34.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.189+0000 7f1d2372a640 1 --2- 172.21.15.129:0/1800946154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d1c109300 0x7f1d1c109710 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c009900 tx=0x7f1d0c02a440 comp rx=0 tx=0).stop 2024-05-20T18:01:34.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.194+0000 7f1d2372a640 1 -- 172.21.15.129:0/1800946154 shutdown_connections 2024-05-20T18:01:34.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.194+0000 7f1d2372a640 1 --2- 172.21.15.129:0/1800946154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c10d890 0x7f1d1c10dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.194+0000 7f1d2372a640 1 --2- 172.21.15.129:0/1800946154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d1c10e350 0x7f1d1c073de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.194+0000 7f1d2372a640 1 --2- 172.21.15.129:0/1800946154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d1c109300 0x7f1d1c109710 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.194+0000 7f1d2372a640 1 -- 172.21.15.129:0/1800946154 >> 172.21.15.129:0/1800946154 conn(0x7f1d1c06d750 msgr2=0x7f1d1c06dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:34.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.195+0000 7f1d2372a640 1 -- 172.21.15.129:0/1800946154 shutdown_connections 2024-05-20T18:01:34.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.197+0000 7f1d2372a640 1 -- 172.21.15.129:0/1800946154 wait complete. 2024-05-20T18:01:34.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.199+0000 7f1d2372a640 1 Processor -- start 2024-05-20T18:01:34.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.200+0000 7f1d2372a640 1 -- start start 2024-05-20T18:01:34.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.200+0000 7f1d2372a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c109300 0x7f1d1c0778f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.200+0000 7f1d2372a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d1c10d890 0x7f1d1c077e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.200+0000 7f1d2372a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d1c10e350 0x7f1d1c1bef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.200+0000 7f1d2372a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c081bf0 con 0x7f1d1c10e350 2024-05-20T18:01:34.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.200+0000 7f1d2372a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c081a70 con 0x7f1d1c109300 2024-05-20T18:01:34.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.200+0000 7f1d2372a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c081d70 con 0x7f1d1c10d890 2024-05-20T18:01:34.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.201+0000 7f1d20cc5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d1c10d890 0x7f1d1c077e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.201+0000 7f1d20cc5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d1c10d890 0x7f1d1c077e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:39894/0 (socket says 172.21.15.129:39894) 2024-05-20T18:01:34.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.201+0000 7f1d20cc5640 1 -- 172.21.15.129:0/1320694073 learned_addr learned my addr 172.21.15.129:0/1320694073 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:34.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.201+0000 7f1d214c6640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c109300 0x7f1d1c0778f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.201+0000 7f1d20cc5640 1 -- 172.21.15.129:0/1320694073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c109300 msgr2=0x7f1d1c0778f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:34.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.201+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c109300 0x7f1d1c0778f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.202+0000 7f1d20cc5640 1 -- 172.21.15.129:0/1320694073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d1c10e350 msgr2=0x7f1d1c1bef90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T18:01:34.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.202+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d1c10e350 0x7f1d1c1bef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.202+0000 7f1d20cc5640 1 -- 172.21.15.129:0/1320694073 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1c1bf5d0 con 0x7f1d1c10d890 2024-05-20T18:01:34.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.205+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d1c10d890 0x7f1d1c077e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400ea80 tx=0x7f1d1400ef40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.210+0000 7f1d127fc640 1 -- 172.21.15.129:0/1320694073 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1400ce40 con 0x7f1d1c10d890 2024-05-20T18:01:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.210+0000 7f1d127fc640 1 -- 172.21.15.129:0/1320694073 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d14011950 con 0x7f1d1c10d890 2024-05-20T18:01:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.210+0000 7f1d127fc640 1 -- 172.21.15.129:0/1320694073 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d14010b10 con 0x7f1d1c10d890 2024-05-20T18:01:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.210+0000 7f1d2372a640 1 -- 172.21.15.129:0/1320694073 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d1c1bf8c0 con 0x7f1d1c10d890 2024-05-20T18:01:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.210+0000 7f1d2372a640 1 -- 172.21.15.129:0/1320694073 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d1c1bfdc0 con 0x7f1d1c10d890 2024-05-20T18:01:34.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.217+0000 7f1d2a5cb740 1 Processor -- start 2024-05-20T18:01:34.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.219+0000 7f1d127fc640 1 -- 172.21.15.129:0/1320694073 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f1d14010cb0 con 0x7f1d1c10d890 2024-05-20T18:01:34.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.220+0000 7f1d127fc640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1d080a2f00 0x7f1d080a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.221+0000 7f1d127fc640 1 -- 172.21.15.129:0/1320694073 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f1d140cf9a0 con 0x7f1d1c10d890 2024-05-20T18:01:34.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.222+0000 7f1d2a5cb740 1 -- start start 2024-05-20T18:01:34.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.222+0000 7f1d21cc7640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1d080a2f00 0x7f1d080a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.222+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e29653f390 0x55e2968ddd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.222+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 0x55e2968de270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.222+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 0x55e2968de7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.222+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e296843d00 con 0x55e2965164d0 2024-05-20T18:01:34.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.222+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e2967eb4c0 con 0x55e29651e6b0 2024-05-20T18:01:34.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.222+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e29683fcc0 con 0x55e29653f390 2024-05-20T18:01:34.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.224+0000 7f1d214c6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e29653f390 0x55e2968ddd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.224+0000 7f1d214c6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e29653f390 0x55e2968ddd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:39896/0 (socket says 172.21.15.129:39896) 2024-05-20T18:01:34.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.224+0000 7f1d214c6640 1 -- 172.21.15.129:0/3948079820 learned_addr learned my addr 172.21.15.129:0/3948079820 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:34.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.224+0000 7f1d21cc7640 1 --2- 172.21.15.129:0/3948079820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 0x55e2968de7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.224+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/3948079820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 0x55e2968de270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.226+0000 7f1d21cc7640 1 -- 172.21.15.129:0/3948079820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e29653f390 msgr2=0x55e2968ddd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:34.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.226+0000 7f1d21cc7640 1 --2- 172.21.15.129:0/3948079820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e29653f390 0x55e2968ddd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.226+0000 7f1d21cc7640 1 -- 172.21.15.129:0/3948079820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 msgr2=0x55e2968de270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:34.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.226+0000 7f1d21cc7640 1 --2- 172.21.15.129:0/3948079820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 0x55e2968de270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.226+0000 7f1d21cc7640 1 -- 172.21.15.129:0/3948079820 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2966217d0 con 0x55e29651e6b0 2024-05-20T18:01:34.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.227+0000 7f1d21cc7640 1 --2- 172.21.15.129:0/3948079820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 0x55e2968de7b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1d18011750 tx=0x7f1d18014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72d16475d6f12a21 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:34.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.231+0000 7f1d077fe640 1 -- 172.21.15.129:0/3948079820 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18020030 con 0x55e29651e6b0 2024-05-20T18:01:34.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.231+0000 7f1d077fe640 1 -- 172.21.15.129:0/3948079820 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d18012c80 con 0x55e29651e6b0 2024-05-20T18:01:34.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.233+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/3948079820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 msgr2=0x55e2968de7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:34.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.233+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/3948079820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 0x55e2968de7b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1d18011750 tx=0x7f1d18014000 comp rx=0 tx=0).stop 2024-05-20T18:01:34.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.235+0000 7f1d21cc7640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1d080a2f00 0x7f1d080a5410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800ba30 tx=0x7f1d1800be30 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:34.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.235+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/3948079820 shutdown_connections 2024-05-20T18:01:34.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.235+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/3948079820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 0x55e2968de7b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.235+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/3948079820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 0x55e2968de270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.235+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/3948079820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e29653f390 0x55e2968ddd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.235+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/3948079820 >> 172.21.15.129:0/3948079820 conn(0x55e2965334e0 msgr2=0x55e2968da4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:34.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.237+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/3948079820 shutdown_connections 2024-05-20T18:01:34.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.237+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/3948079820 wait complete. 2024-05-20T18:01:34.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.238+0000 7f1d2a5cb740 1 -- start start 2024-05-20T18:01:34.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.238+0000 7f1d2a5cb740 1 Processor -- start 2024-05-20T18:01:34.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.240+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 0x55e2968e83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.240+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 0x55e2968e8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.240+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e29653f390 0x55e2968ee5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.240+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e2968eb070 con 0x55e2965164d0 2024-05-20T18:01:34.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.241+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e2967feb90 con 0x55e29651e6b0 2024-05-20T18:01:34.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.241+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e2968eb1f0 con 0x55e29653f390 2024-05-20T18:01:34.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.241+0000 7f1d2a5cb740 10 client.0 _ll_register_callbacks cb 0x55e29683ada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:34.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.244+0000 7f1d20cc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 0x55e2968e8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.244+0000 7f1d21cc7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e29653f390 0x55e2968ee5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.244+0000 7f1d20cc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 0x55e2968e8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:46396/0 (socket says 172.21.15.129:46396) 2024-05-20T18:01:34.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.244+0000 7f1d20cc5640 1 -- 172.21.15.129:0/854625943 learned_addr learned my addr 172.21.15.129:0/854625943 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:34.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.250+0000 7f1d20cc5640 1 -- 172.21.15.129:0/854625943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e29653f390 msgr2=0x55e2968ee5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:34.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.250+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/854625943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e29653f390 0x55e2968ee5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.250+0000 7f1d20cc5640 1 -- 172.21.15.129:0/854625943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 msgr2=0x55e2968e83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:34.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.250+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/854625943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 0x55e2968e83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.250+0000 7f1d20cc5640 1 -- 172.21.15.129:0/854625943 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2965573f0 con 0x55e29651e6b0 2024-05-20T18:01:34.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.254+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/854625943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 0x55e2968e8930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400efd0 tx=0x7f1d1409bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba9bb3e21e11b394 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:34.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.254+0000 7f1d04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:01:34.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.258+0000 7f1d04ff9640 1 -- 172.21.15.129:0/854625943 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d140cd300 con 0x55e29651e6b0 2024-05-20T18:01:34.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.258+0000 7f1d04ff9640 1 -- 172.21.15.129:0/854625943 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f1d140cd4a0 con 0x55e29651e6b0 2024-05-20T18:01:34.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.258+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/854625943 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e296851430 con 0x55e29651e6b0 2024-05-20T18:01:34.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.263+0000 7f1d04ff9640 1 -- 172.21.15.129:0/854625943 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1d140ab050 con 0x55e29651e6b0 2024-05-20T18:01:34.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.264+0000 7f1d2a5cb740 10 client.25607 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:34.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.264+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/854625943 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55e2965170f0 con 0x55e29651e6b0 2024-05-20T18:01:34.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.276+0000 7f1d04ff9640 1 -- 172.21.15.129:0/854625943 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e296851430 con 0x55e29651e6b0 2024-05-20T18:01:34.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.276+0000 7f1d04ff9640 1 -- 172.21.15.129:0/854625943 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f1d1400f040 con 0x55e29651e6b0 2024-05-20T18:01:34.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.276+0000 7f1d2a5cb740 10 client.25607 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:34.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.276+0000 7f1d2a5cb740 10 client.25607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:34.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.276+0000 7f1d2a5cb740 20 client.25607 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:34.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.276+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/854625943 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2964fdf60 0x55e2968fb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.276+0000 7f1d2a5cb740 4 client.25607 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:01:34.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.276+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/854625943 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e29684b780 con 0x55e2964fdf60 2024-05-20T18:01:34.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.278+0000 7f1d214c6640 1 --2- 172.21.15.129:0/854625943 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2964fdf60 0x55e2968fb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.280+0000 7f1d214c6640 1 --2- 172.21.15.129:0/854625943 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2964fdf60 0x55e2968fb1f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=706b4faf68cd92ca server_cookie=eca4b7b431a1fae7 in_seq=0 out_seq=0 2024-05-20T18:01:34.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.281+0000 7f1d04ff9640 10 client.25607 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:34.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.285+0000 7f1d04ff9640 1 -- 172.21.15.129:0/854625943 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e29684b780 con 0x55e2964fdf60 2024-05-20T18:01:34.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.285+0000 7f1d04ff9640 10 client.25607 handle_command_reply: tid=0 2024-05-20T18:01:34.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.289+0000 7f1d2a5cb740 1 client.25607 shutdown 2024-05-20T18:01:34.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.289+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/854625943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 msgr2=0x55e2968e8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:34.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.289+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/854625943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 0x55e2968e8930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400efd0 tx=0x7f1d1409bf90 comp rx=0 tx=0).stop 2024-05-20T18:01:34.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.289+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/854625943 shutdown_connections 2024-05-20T18:01:34.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.289+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/854625943 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2964fdf60 0x55e2968fb1f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.289+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/854625943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29651e6b0 0x55e2968e8930 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400efd0 tx=0x7f1d1409bf90 comp rx=0 tx=0).stop 2024-05-20T18:01:34.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.289+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/854625943 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e29653f390 0x55e2968ee5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.289+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/854625943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 0x55e2968e83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.289+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/854625943 >> 172.21.15.129:0/854625943 conn(0x55e2965334e0 msgr2=0x55e2968dfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:34.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.295+0000 7f1d04ff9640 0 client.25607 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:34.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.295+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/854625943 shutdown_connections 2024-05-20T18:01:34.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.296+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/854625943 wait complete. 2024-05-20T18:01:34.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.296+0000 7f1d2a5cb740 20 client.25607 trim_cache size 0 max 16384 2024-05-20T18:01:34.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.351+0000 7f1d2a5cb740 1 Processor -- start 2024-05-20T18:01:34.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.351+0000 7f1d2a5cb740 1 -- start start 2024-05-20T18:01:34.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.351+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 0x55e2968ee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.351+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29653f390 0x55e2968e26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.351+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2964d33a0 0x55e2969009f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.351+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0000c910 con 0x55e2965164d0 2024-05-20T18:01:34.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.351+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0000f190 con 0x55e29653f390 2024-05-20T18:01:34.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.351+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0000cf80 con 0x55e2964d33a0 2024-05-20T18:01:34.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.352+0000 7f1d21cc7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2964d33a0 0x55e2969009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.352+0000 7f1d21cc7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2964d33a0 0x55e2969009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:39910/0 (socket says 172.21.15.129:39910) 2024-05-20T18:01:34.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.352+0000 7f1d21cc7640 1 -- 172.21.15.129:0/416353945 learned_addr learned my addr 172.21.15.129:0/416353945 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.352+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/416353945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29653f390 0x55e2968e26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.352+0000 7f1d21cc7640 1 -- 172.21.15.129:0/416353945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29653f390 msgr2=0x55e2968e26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.352+0000 7f1d21cc7640 1 --2- 172.21.15.129:0/416353945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29653f390 0x55e2968e26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.352+0000 7f1d21cc7640 1 -- 172.21.15.129:0/416353945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 msgr2=0x55e2968ee4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.352+0000 7f1d21cc7640 1 --2- 172.21.15.129:0/416353945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 0x55e2968ee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.352+0000 7f1d21cc7640 1 -- 172.21.15.129:0/416353945 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2968e3aa0 con 0x55e2964d33a0 2024-05-20T18:01:34.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.353+0000 7f1d21cc7640 1 --2- 172.21.15.129:0/416353945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2964d33a0 0x55e2969009f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1d18005280 tx=0x7f1d180052b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71068f218725405d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:34.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.360+0000 7f1d06ffd640 1 -- 172.21.15.129:0/416353945 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18033070 con 0x55e2964d33a0 2024-05-20T18:01:34.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.360+0000 7f1d06ffd640 1 -- 172.21.15.129:0/416353945 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d180054d0 con 0x55e2964d33a0 2024-05-20T18:01:34.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.361+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/416353945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2964d33a0 msgr2=0x55e2969009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:34.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.361+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/416353945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2964d33a0 0x55e2969009f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1d18005280 tx=0x7f1d180052b0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.364+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/416353945 shutdown_connections 2024-05-20T18:01:34.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.364+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/416353945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29653f390 0x55e2968e26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.364+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/416353945 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2964d33a0 0x55e2969009f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.364+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/416353945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2965164d0 0x55e2968ee4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.364+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/416353945 >> 172.21.15.129:0/416353945 conn(0x55e2965334e0 msgr2=0x55e2968e6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:34.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.368+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/416353945 shutdown_connections 2024-05-20T18:01:34.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.372+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/416353945 wait complete. 2024-05-20T18:01:34.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.372+0000 7f1d2a5cb740 1 -- start start 2024-05-20T18:01:34.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.372+0000 7f1d2a5cb740 1 Processor -- start 2024-05-20T18:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.373+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2964d33a0 0x55e2968ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.374+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2965164d0 0x55e2968efd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.374+0000 7f1d2a5cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29653f390 0x55e2968f0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.374+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0000d650 con 0x55e2964d33a0 2024-05-20T18:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.374+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0000eac0 con 0x55e29653f390 2024-05-20T18:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.374+0000 7f1d2a5cb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0000dd20 con 0x55e2965164d0 2024-05-20T18:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.374+0000 7f1d2a5cb740 10 client.0 _ll_register_callbacks cb 0x7f1d1409bc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:34.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.375+0000 7f1d214c6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2964d33a0 0x55e2968ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.375+0000 7f1d214c6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2964d33a0 0x55e2968ef7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40990/0 (socket says 172.21.15.129:40990) 2024-05-20T18:01:34.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.375+0000 7f1d214c6640 1 -- 172.21.15.129:0/531904857 learned_addr learned my addr 172.21.15.129:0/531904857 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:34.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.375+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2965164d0 0x55e2968efd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.375+0000 7f1d21cc7640 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29653f390 0x55e2968f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.376+0000 7f1d20cc5640 1 -- 172.21.15.129:0/531904857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29653f390 msgr2=0x55e2968f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:34.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.376+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29653f390 0x55e2968f0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.376+0000 7f1d20cc5640 1 -- 172.21.15.129:0/531904857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2964d33a0 msgr2=0x55e2968ef7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:34.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.376+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2964d33a0 0x55e2968ef7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.376+0000 7f1d20cc5640 1 -- 172.21.15.129:0/531904857 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e296961640 con 0x55e2965164d0 2024-05-20T18:01:34.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.377+0000 7f1d20cc5640 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2965164d0 0x55e2968efd20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55e2968e92a0 tx=0x7f1d14005b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b2338ef5e536e68 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.381+0000 7f1d04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.381+0000 7f1d04ff9640 1 -- 172.21.15.129:0/531904857 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d140117b0 con 0x55e2965164d0 2024-05-20T18:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.381+0000 7f1d04ff9640 1 -- 172.21.15.129:0/531904857 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f1d140cd300 con 0x55e2965164d0 2024-05-20T18:01:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.381+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/531904857 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d180054d0 con 0x55e2965164d0 2024-05-20T18:01:34.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.383+0000 7f1d04ff9640 1 -- 172.21.15.129:0/531904857 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f1d140cd500 con 0x55e2965164d0 2024-05-20T18:01:34.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.384+0000 7f1d2a5cb740 10 client.25645 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:34.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.384+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/531904857 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55e2969628b0 con 0x55e2965164d0 2024-05-20T18:01:34.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.385+0000 7f1d04ff9640 1 -- 172.21.15.129:0/531904857 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d180054d0 con 0x55e2965164d0 2024-05-20T18:01:34.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.387+0000 7f1d04ff9640 1 -- 172.21.15.129:0/531904857 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f1d14097ba0 con 0x55e2965164d0 2024-05-20T18:01:34.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.387+0000 7f1d2a5cb740 10 client.25645 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:34.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.387+0000 7f1d2a5cb740 10 client.25645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:34.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.387+0000 7f1d2a5cb740 20 client.25645 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:34.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.387+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e296962ec0 0x55e2969652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:34.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.387+0000 7f1d2a5cb740 4 client.25645 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:01:34.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.387+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/531904857 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e2967c86f0 con 0x55e296962ec0 2024-05-20T18:01:34.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.390+0000 7f1d214c6640 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e296962ec0 0x55e2969652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:34.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.394+0000 7f1d214c6640 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e296962ec0 0x55e2969652e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1b1f2db6cba5001 server_cookie=9b79fb125cbf75df in_seq=0 out_seq=0 2024-05-20T18:01:34.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.394+0000 7f1d04ff9640 10 client.25645 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:34.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.398+0000 7f1d04ff9640 1 -- 172.21.15.129:0/531904857 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e2967c86f0 con 0x55e296962ec0 2024-05-20T18:01:34.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.398+0000 7f1d04ff9640 10 client.25645 handle_command_reply: tid=0 2024-05-20T18:01:34.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.399+0000 7f1d2a5cb740 1 client.25645 shutdown 2024-05-20T18:01:34.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.399+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/531904857 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2965164d0 msgr2=0x55e2968efd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:34.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.399+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2965164d0 0x55e2968efd20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55e2968e92a0 tx=0x7f1d14005b70 comp rx=0 tx=0).stop 2024-05-20T18:01:34.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.407+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/531904857 shutdown_connections 2024-05-20T18:01:34.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.407+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e296962ec0 0x55e2969652e0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.407+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e29653f390 0x55e2968f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.407+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2965164d0 0x55e2968efd20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.407+0000 7f1d2a5cb740 1 --2- 172.21.15.129:0/531904857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2964d33a0 0x55e2968ef7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.407+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/531904857 >> 172.21.15.129:0/531904857 conn(0x55e2965334e0 msgr2=0x55e2968e19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:34.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.409+0000 7f1d04ff9640 0 client.25645 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:34.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.412+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/531904857 shutdown_connections 2024-05-20T18:01:34.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.412+0000 7f1d2a5cb740 1 -- 172.21.15.129:0/531904857 wait complete. 2024-05-20T18:01:34.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.413+0000 7f1d2a5cb740 20 client.25645 trim_cache size 0 max 16384 2024-05-20T18:01:34.416 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:01:34.416 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:01:34.416 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:01:34.416 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:01:34.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.421+0000 7f1d2372a640 1 -- 172.21.15.129:0/1320694073 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1d080a2f00 msgr2=0x7f1d080a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:34.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.422+0000 7f1d2372a640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1d080a2f00 0x7f1d080a5410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1d1800ba30 tx=0x7f1d1800be30 comp rx=0 tx=0).stop 2024-05-20T18:01:34.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.422+0000 7f1d2372a640 1 -- 172.21.15.129:0/1320694073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d1c10d890 msgr2=0x7f1d1c077e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:34.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.422+0000 7f1d2372a640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d1c10d890 0x7f1d1c077e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400ea80 tx=0x7f1d1400ef40 comp rx=0 tx=0).stop 2024-05-20T18:01:34.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.423+0000 7f1d2372a640 1 -- 172.21.15.129:0/1320694073 shutdown_connections 2024-05-20T18:01:34.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.423+0000 7f1d2372a640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1d080a2f00 0x7f1d080a5410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.423+0000 7f1d2372a640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d1c10e350 0x7f1d1c1bef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.423+0000 7f1d2372a640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1d1c10d890 0x7f1d1c077e30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.423+0000 7f1d2372a640 1 --2- 172.21.15.129:0/1320694073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c109300 0x7f1d1c0778f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:34.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.423+0000 7f1d2372a640 1 -- 172.21.15.129:0/1320694073 >> 172.21.15.129:0/1320694073 conn(0x7f1d1c06d750 msgr2=0x7f1d1c084380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:34.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.427+0000 7f1d2372a640 1 -- 172.21.15.129:0/1320694073 shutdown_connections 2024-05-20T18:01:34.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:34.428+0000 7f1d2372a640 1 -- 172.21.15.129:0/1320694073 wait complete. 2024-05-20T18:01:34.496 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:01:34.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:01:35.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.017+0000 7fc3a0e14640 1 -- 172.21.15.129:0/4083984352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c119770 msgr2=0x7fc39c11bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:35.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.017+0000 7fc3a0e14640 1 --2- 172.21.15.129:0/4083984352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c119770 0x7fc39c11bb90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc39001a910 tx=0x7fc39003b500 comp rx=0 tx=0).stop 2024-05-20T18:01:35.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.022+0000 7fc3a0e14640 1 -- 172.21.15.129:0/4083984352 shutdown_connections 2024-05-20T18:01:35.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.022+0000 7fc3a0e14640 1 --2- 172.21.15.129:0/4083984352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c11c0d0 0x7fc39c11e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:35.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.022+0000 7fc3a0e14640 1 --2- 172.21.15.129:0/4083984352 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c119770 0x7fc39c11bb90 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:35.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.022+0000 7fc3a0e14640 1 --2- 172.21.15.129:0/4083984352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc39c109060 0x7fc39c109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:35.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.022+0000 7fc3a0e14640 1 -- 172.21.15.129:0/4083984352 >> 172.21.15.129:0/4083984352 conn(0x7fc39c06d760 msgr2=0x7fc39c06dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:35.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.025+0000 7fc3a0e14640 1 -- 172.21.15.129:0/4083984352 shutdown_connections 2024-05-20T18:01:35.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.029+0000 7fc3a0e14640 1 -- 172.21.15.129:0/4083984352 wait complete. 2024-05-20T18:01:35.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.031+0000 7fc3a0e14640 1 Processor -- start 2024-05-20T18:01:35.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.031+0000 7fc3a0e14640 1 -- start start 2024-05-20T18:01:35.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.032+0000 7fc3a0e14640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c109060 0x7fc39c115430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:35.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.032+0000 7fc3a0e14640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c119770 0x7fc39c115970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:35.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.032+0000 7fc3a0e14640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc39c11c0d0 0x7fc39c115eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:35.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.032+0000 7fc3a0e14640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c113940 con 0x7fc39c109060 2024-05-20T18:01:35.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.032+0000 7fc3a0e14640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c1137c0 con 0x7fc39c119770 2024-05-20T18:01:35.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.032+0000 7fc3a0e14640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c113ac0 con 0x7fc39c11c0d0 2024-05-20T18:01:35.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.033+0000 7fc39a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c109060 0x7fc39c115430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:35.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.033+0000 7fc39a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c109060 0x7fc39c115430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41008/0 (socket says 172.21.15.129:41008) 2024-05-20T18:01:35.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.033+0000 7fc39a59c640 1 -- 172.21.15.129:0/1810751214 learned_addr learned my addr 172.21.15.129:0/1810751214 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:35.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.034+0000 7fc399d9b640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c119770 0x7fc39c115970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:35.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.034+0000 7fc39a59c640 1 -- 172.21.15.129:0/1810751214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc39c11c0d0 msgr2=0x7fc39c115eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:01:35.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.034+0000 7fc39a59c640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc39c11c0d0 0x7fc39c115eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:35.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.034+0000 7fc39a59c640 1 -- 172.21.15.129:0/1810751214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c119770 msgr2=0x7fc39c115970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:35.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.034+0000 7fc39a59c640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c119770 0x7fc39c115970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:35.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.034+0000 7fc39a59c640 1 -- 172.21.15.129:0/1810751214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39c1abfa0 con 0x7fc39c109060 2024-05-20T18:01:35.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.034+0000 7fc399d9b640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c119770 0x7fc39c115970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:01:35.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.034+0000 7fc39a59c640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c109060 0x7fc39c115430 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00e9f0 tx=0x7fc38c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:35.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.039+0000 7fc38b7fe640 1 -- 172.21.15.129:0/1810751214 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c00cd80 con 0x7fc39c109060 2024-05-20T18:01:35.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.039+0000 7fc3a0e14640 1 -- 172.21.15.129:0/1810751214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39c1ac1d0 con 0x7fc39c109060 2024-05-20T18:01:35.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.039+0000 7fc3a0e14640 1 -- 172.21.15.129:0/1810751214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc39c1ac720 con 0x7fc39c109060 2024-05-20T18:01:35.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.039+0000 7fc38b7fe640 1 -- 172.21.15.129:0/1810751214 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc38c010910 con 0x7fc39c109060 2024-05-20T18:01:35.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.039+0000 7fc38b7fe640 1 -- 172.21.15.129:0/1810751214 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c018990 con 0x7fc39c109060 2024-05-20T18:01:35.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.043+0000 7fc3a0e14640 1 -- 172.21.15.129:0/1810751214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc37c005290 con 0x7fc39c109060 2024-05-20T18:01:35.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.054+0000 7fc38b7fe640 1 -- 172.21.15.129:0/1810751214 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc38c018c70 con 0x7fc39c109060 2024-05-20T18:01:35.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.055+0000 7fc38b7fe640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc3840a2f00 0x7fc3840a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:35.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.056+0000 7fc399d9b640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc3840a2f00 0x7fc3840a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:35.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.056+0000 7fc38b7fe640 1 -- 172.21.15.129:0/1810751214 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fc38c0cf800 con 0x7fc39c109060 2024-05-20T18:01:35.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.069+0000 7fc38b7fe640 1 -- 172.21.15.129:0/1810751214 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc38c09c3b0 con 0x7fc39c109060 2024-05-20T18:01:35.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.135+0000 7fc399d9b640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc3840a2f00 0x7fc3840a5410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc39c11b5a0 tx=0x7fc390005fd0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:35.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:35 smithi154 ceph-mon[36482]: pgmap v94: 97 pgs: 97 active+clean; 19 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 642 MiB/s wr, 1.70k op/s 2024-05-20T18:01:35.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:35 smithi129 ceph-mon[32029]: pgmap v94: 97 pgs: 97 active+clean; 19 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 642 MiB/s wr, 1.70k op/s 2024-05-20T18:01:35.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:35 smithi204 ceph-mon[37076]: pgmap v94: 97 pgs: 97 active+clean; 19 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 642 MiB/s wr, 1.70k op/s 2024-05-20T18:01:35.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.773+0000 7fc3a0e14640 1 -- 172.21.15.129:0/1810751214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc37c005850 con 0x7fc39c109060 2024-05-20T18:01:35.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.777+0000 7fc38b7fe640 1 -- 172.21.15.129:0/1810751214 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11829 (secure 0 0 0) 0x7fc38c09bcb0 con 0x7fc39c109060 2024-05-20T18:01:35.781 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:01:35.781 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:59:17.074606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25229":{"gid":25229,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1860679091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1860679091},{"type":"v1","addr":"172.21.15.154:6837","nonce":1860679091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:01:35.781 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-05-20T18:01:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.790+0000 7fc3897fa640 1 -- 172.21.15.129:0/1810751214 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc3840a2f00 msgr2=0x7fc3840a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.790+0000 7fc3897fa640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc3840a2f00 0x7fc3840a5410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc39c11b5a0 tx=0x7fc390005fd0 comp rx=0 tx=0).stop 2024-05-20T18:01:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.790+0000 7fc3897fa640 1 -- 172.21.15.129:0/1810751214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c109060 msgr2=0x7fc39c115430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.790+0000 7fc3897fa640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c109060 0x7fc39c115430 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00e9f0 tx=0x7fc38c00eeb0 comp rx=0 tx=0).stop 2024-05-20T18:01:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.794+0000 7fc3897fa640 1 -- 172.21.15.129:0/1810751214 shutdown_connections 2024-05-20T18:01:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.794+0000 7fc3897fa640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc3840a2f00 0x7fc3840a5410 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.794+0000 7fc3897fa640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc39c119770 0x7fc39c115970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.794+0000 7fc3897fa640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc39c11c0d0 0x7fc39c115eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.794+0000 7fc3897fa640 1 --2- 172.21.15.129:0/1810751214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc39c109060 0x7fc39c115430 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.794+0000 7fc3897fa640 1 -- 172.21.15.129:0/1810751214 >> 172.21.15.129:0/1810751214 conn(0x7fc39c06d760 msgr2=0x7fc39c119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:35.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.800+0000 7fc3897fa640 1 -- 172.21.15.129:0/1810751214 shutdown_connections 2024-05-20T18:01:35.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:35.800+0000 7fc3897fa640 1 -- 172.21.15.129:0/1810751214 wait complete. 2024-05-20T18:01:35.871 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:01:36.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:36 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1810751214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:01:36.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:36 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1810751214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:01:36.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:36 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1810751214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:01:36.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.484+0000 7f78114e6640 1 -- 172.21.15.129:0/1733862077 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc035020 con 0x7f780c10f270 2024-05-20T18:01:36.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.485+0000 7f78134ea640 1 -- 172.21.15.129:0/1733862077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c10f270 msgr2=0x7f780c108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:36.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.485+0000 7f78134ea640 1 --2- 172.21.15.129:0/1733862077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c10f270 0x7f780c108b50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009900 tx=0x7f77fc02a440 comp rx=0 tx=0).stop 2024-05-20T18:01:36.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.492+0000 7f78134ea640 1 -- 172.21.15.129:0/1733862077 shutdown_connections 2024-05-20T18:01:36.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.492+0000 7f78134ea640 1 --2- 172.21.15.129:0/1733862077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c10f270 0x7f780c108b50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.492+0000 7f78134ea640 1 --2- 172.21.15.129:0/1733862077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f780c109090 0x7f780c109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.492+0000 7f78134ea640 1 --2- 172.21.15.129:0/1733862077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c10e890 0x7f780c10eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.492+0000 7f78134ea640 1 -- 172.21.15.129:0/1733862077 >> 172.21.15.129:0/1733862077 conn(0x7f780c06d580 msgr2=0x7f780c06d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:36.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.503+0000 7f78134ea640 1 -- 172.21.15.129:0/1733862077 shutdown_connections 2024-05-20T18:01:36.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.505+0000 7f78134ea640 1 -- 172.21.15.129:0/1733862077 wait complete. 2024-05-20T18:01:36.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.507+0000 7f78134ea640 1 Processor -- start 2024-05-20T18:01:36.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.507+0000 7f78134ea640 1 -- start start 2024-05-20T18:01:36.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.508+0000 7f78134ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c109090 0x7f780c1a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.508+0000 7f78134ea640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c10e890 0x7f780c1a8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.508+0000 7f78134ea640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f780c10f270 0x7f780c1acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.508+0000 7f78134ea640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f780c121b70 con 0x7f780c10e890 2024-05-20T18:01:36.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.508+0000 7f78134ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f780c1219f0 con 0x7f780c109090 2024-05-20T18:01:36.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.508+0000 7f78134ea640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f780c121cf0 con 0x7f780c10f270 2024-05-20T18:01:36.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.516+0000 7f7811ce7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c10e890 0x7f780c1a8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.516+0000 7f7811ce7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c10e890 0x7f780c1a8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41024/0 (socket says 172.21.15.129:41024) 2024-05-20T18:01:36.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.516+0000 7f7811ce7640 1 -- 172.21.15.129:0/1669198865 learned_addr learned my addr 172.21.15.129:0/1669198865 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:36.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.516+0000 7f78124e8640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c109090 0x7f780c1a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.516+0000 7f7812ce9640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f780c10f270 0x7f780c1acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.531+0000 7f7811ce7640 1 -- 172.21.15.129:0/1669198865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f780c10f270 msgr2=0x7f780c1acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.531+0000 7f7811ce7640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f780c10f270 0x7f780c1acd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.531+0000 7f7811ce7640 1 -- 172.21.15.129:0/1669198865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c109090 msgr2=0x7f780c1a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.531+0000 7f7811ce7640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c109090 0x7f780c1a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.531+0000 7f7811ce7640 1 -- 172.21.15.129:0/1669198865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780c1ad350 con 0x7f780c10e890 2024-05-20T18:01:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.532+0000 7f78124e8640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c109090 0x7f780c1a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:01:36.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.537+0000 7f7811ce7640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c10e890 0x7f780c1a8a50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f77fc004bb0 tx=0x7f77fc004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:36.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.540+0000 7f78037fe640 1 -- 172.21.15.129:0/1669198865 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc040070 con 0x7f780c10e890 2024-05-20T18:01:36.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.540+0000 7f78037fe640 1 -- 172.21.15.129:0/1669198865 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77fc031c80 con 0x7f780c10e890 2024-05-20T18:01:36.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.540+0000 7f78037fe640 1 -- 172.21.15.129:0/1669198865 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc035020 con 0x7f780c10e890 2024-05-20T18:01:36.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.542+0000 7f78134ea640 1 -- 172.21.15.129:0/1669198865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780c1ad5e0 con 0x7f780c10e890 2024-05-20T18:01:36.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.542+0000 7f78134ea640 1 -- 172.21.15.129:0/1669198865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f780c1ada80 con 0x7f780c10e890 2024-05-20T18:01:36.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.550+0000 7f781b5f7740 1 Processor -- start 2024-05-20T18:01:36.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.550+0000 7f781b5f7740 1 -- start start 2024-05-20T18:01:36.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.551+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a51f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.551+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c2368a4d0 0x564c23a524d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.551+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c236926b0 0x564c23a52a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.551+0000 7f781b5f7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564c239b7c40 con 0x564c236b3390 2024-05-20T18:01:36.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.551+0000 7f781b5f7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564c2395f580 con 0x564c236926b0 2024-05-20T18:01:36.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.551+0000 7f781b5f7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564c239b3c00 con 0x564c2368a4d0 2024-05-20T18:01:36.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.552+0000 7f7812ce9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a51f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.552+0000 7f7812ce9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a51f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41032/0 (socket says 172.21.15.129:41032) 2024-05-20T18:01:36.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.552+0000 7f7812ce9640 1 -- 172.21.15.129:0/2816820907 learned_addr learned my addr 172.21.15.129:0/2816820907 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:36.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.553+0000 7f7812ce9640 1 -- 172.21.15.129:0/2816820907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c2368a4d0 msgr2=0x564c23a524d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:01:36.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.553+0000 7f7812ce9640 1 --2- 172.21.15.129:0/2816820907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c2368a4d0 0x564c23a524d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.553+0000 7f7812ce9640 1 -- 172.21.15.129:0/2816820907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c236926b0 msgr2=0x564c23a52a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:01:36.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.553+0000 7f7812ce9640 1 --2- 172.21.15.129:0/2816820907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c236926b0 0x564c23a52a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.553+0000 7f7812ce9640 1 -- 172.21.15.129:0/2816820907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c236cb3f0 con 0x564c236b3390 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.553+0000 7f7812ce9640 1 --2- 172.21.15.129:0/2816820907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a51f90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f780800b7a0 tx=0x7f780800bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b258d86504882f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.554+0000 7f7800ff9640 1 -- 172.21.15.129:0/2816820907 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780800d950 con 0x564c236b3390 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.554+0000 7f7800ff9640 1 -- 172.21.15.129:0/2816820907 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7808012420 con 0x564c236b3390 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.554+0000 7f7800ff9640 1 -- 172.21.15.129:0/2816820907 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7808014610 con 0x564c236b3390 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.556+0000 7f781b5f7740 1 -- 172.21.15.129:0/2816820907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 msgr2=0x564c23a51f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.556+0000 7f781b5f7740 1 --2- 172.21.15.129:0/2816820907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a51f90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f780800b7a0 tx=0x7f780800bc60 comp rx=0 tx=0).stop 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.556+0000 7f781b5f7740 1 -- 172.21.15.129:0/2816820907 shutdown_connections 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.556+0000 7f781b5f7740 1 --2- 172.21.15.129:0/2816820907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c236926b0 0x564c23a52a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.556+0000 7f781b5f7740 1 --2- 172.21.15.129:0/2816820907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c2368a4d0 0x564c23a524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.556+0000 7f781b5f7740 1 --2- 172.21.15.129:0/2816820907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a51f90 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.556+0000 7f781b5f7740 1 -- 172.21.15.129:0/2816820907 >> 172.21.15.129:0/2816820907 conn(0x564c236a74e0 msgr2=0x564c23a4e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:36.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.563+0000 7f78037fe640 1 -- 172.21.15.129:0/1669198865 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f77fc043070 con 0x7f780c10e890 2024-05-20T18:01:36.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.563+0000 7f781b5f7740 1 -- 172.21.15.129:0/2816820907 shutdown_connections 2024-05-20T18:01:36.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.564+0000 7f781b5f7740 1 -- 172.21.15.129:0/2816820907 wait complete. 2024-05-20T18:01:36.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.564+0000 7f781b5f7740 1 -- start start 2024-05-20T18:01:36.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.565+0000 7f781b5f7740 1 Processor -- start 2024-05-20T18:01:36.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.566+0000 7f78037fe640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f77f80a2c80 0x7f77f80a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.567+0000 7f78037fe640 1 -- 172.21.15.129:0/1669198865 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f77fc0f0550 con 0x7f780c10e890 2024-05-20T18:01:36.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.567+0000 7f7812ce9640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f77f80a2c80 0x7f77f80a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.569+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c2368a4d0 0x564c23a5c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.569+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c236b3390 0x564c23a5caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.569+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 0x564c23a68880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.569+0000 7f781b5f7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564c23a5f330 con 0x564c2368a4d0 2024-05-20T18:01:36.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.569+0000 7f781b5f7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564c23972c40 con 0x564c236b3390 2024-05-20T18:01:36.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.569+0000 7f781b5f7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564c23a5f4b0 con 0x564c23671f60 2024-05-20T18:01:36.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.570+0000 7f781b5f7740 10 client.0 _ll_register_callbacks cb 0x564c236cabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:36.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.570+0000 7f7812ce9640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f77f80a2c80 0x7f77f80a5190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7808012e60 tx=0x7f7808006f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:36.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.575+0000 7f7812ce9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 0x564c23a68880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.575+0000 7f7812ce9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 0x564c23a68880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:39946/0 (socket says 172.21.15.129:39946) 2024-05-20T18:01:36.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.575+0000 7f7812ce9640 1 -- 172.21.15.129:0/3051501717 learned_addr learned my addr 172.21.15.129:0/3051501717 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:36.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.576+0000 7f7812ce9640 1 -- 172.21.15.129:0/3051501717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c236b3390 msgr2=0x564c23a5caf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:01:36.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.576+0000 7f7812ce9640 1 --2- 172.21.15.129:0/3051501717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c236b3390 0x564c23a5caf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.576+0000 7f7812ce9640 1 -- 172.21.15.129:0/3051501717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c2368a4d0 msgr2=0x564c23a5c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:36.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.576+0000 7f7812ce9640 1 --2- 172.21.15.129:0/3051501717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c2368a4d0 0x564c23a5c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.576+0000 7f7812ce9640 1 -- 172.21.15.129:0/3051501717 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c237957d0 con 0x564c23671f60 2024-05-20T18:01:36.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.577+0000 7f7812ce9640 1 --2- 172.21.15.129:0/3051501717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 0x564c23a68880 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x564c23a4f100 tx=0x7f780800bc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=980a0c38972ad954 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:36.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.578+0000 7f77f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:01:36.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.578+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3051501717 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780801db80 con 0x564c23671f60 2024-05-20T18:01:36.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.578+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3051501717 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f780801dd20 con 0x564c23671f60 2024-05-20T18:01:36.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.579+0000 7f781b5f7740 1 -- 172.21.15.129:0/3051501717 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7808012420 con 0x564c23671f60 2024-05-20T18:01:36.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.583+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3051501717 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7808033470 con 0x564c23671f60 2024-05-20T18:01:36.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.583+0000 7f781b5f7740 10 client.25654 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:36.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.583+0000 7f781b5f7740 1 -- 172.21.15.129:0/3051501717 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x564c23a6bad0 con 0x564c23671f60 2024-05-20T18:01:36.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.589+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3051501717 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7808012420 con 0x564c23671f60 2024-05-20T18:01:36.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.589+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3051501717 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f7808013070 con 0x564c23671f60 2024-05-20T18:01:36.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.595+0000 7f781b5f7740 10 client.25654 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:36.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.595+0000 7f781b5f7740 10 client.25654 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:36.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.595+0000 7f781b5f7740 20 client.25654 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:36.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.595+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3051501717 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564c236473a0 0x564c23a6ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.595+0000 7f781b5f7740 4 client.25654 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:01:36.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.595+0000 7f781b5f7740 1 -- 172.21.15.129:0/3051501717 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c239c5370 con 0x564c236473a0 2024-05-20T18:01:36.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.600+0000 7f78124e8640 1 --2- 172.21.15.129:0/3051501717 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564c236473a0 0x564c23a6ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.608+0000 7f78124e8640 1 --2- 172.21.15.129:0/3051501717 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564c236473a0 0x564c23a6ef40 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e5ab832727bd95b server_cookie=714d344c42160014 in_seq=0 out_seq=0 2024-05-20T18:01:36.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.609+0000 7f77f5ffb640 10 client.25654 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:36.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.611+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3051501717 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564c239c5370 con 0x564c236473a0 2024-05-20T18:01:36.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.611+0000 7f77f5ffb640 10 client.25654 handle_command_reply: tid=0 2024-05-20T18:01:36.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.613+0000 7f781b5f7740 1 client.25654 shutdown 2024-05-20T18:01:36.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.614+0000 7f781b5f7740 1 -- 172.21.15.129:0/3051501717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 msgr2=0x564c23a68880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:36.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.614+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3051501717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 0x564c23a68880 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x564c23a4f100 tx=0x7f780800bc60 comp rx=0 tx=0).stop 2024-05-20T18:01:36.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.614+0000 7f781b5f7740 1 -- 172.21.15.129:0/3051501717 shutdown_connections 2024-05-20T18:01:36.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.614+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3051501717 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564c236473a0 0x564c23a6ef40 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.614+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3051501717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c236b3390 0x564c23a5caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.614+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3051501717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 0x564c23a68880 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.614+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3051501717 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c2368a4d0 0x564c23a5c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.614+0000 7f781b5f7740 1 -- 172.21.15.129:0/3051501717 >> 172.21.15.129:0/3051501717 conn(0x564c236a74e0 msgr2=0x564c23a53cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:36.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.616+0000 7f77f5ffb640 0 client.25654 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:36.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.622+0000 7f781b5f7740 1 -- 172.21.15.129:0/3051501717 shutdown_connections 2024-05-20T18:01:36.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.622+0000 7f781b5f7740 1 -- 172.21.15.129:0/3051501717 wait complete. 2024-05-20T18:01:36.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.625+0000 7f781b5f7740 20 client.25654 trim_cache size 0 max 16384 2024-05-20T18:01:36.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.679+0000 7f781b5f7740 1 Processor -- start 2024-05-20T18:01:36.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.679+0000 7f781b5f7740 1 -- start start 2024-05-20T18:01:36.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.679+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 0x564c23a5c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.680+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c2368a4d0 0x564c23a687f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.680+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a71f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.680+0000 7f781b5f7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f780c1a9840 con 0x564c236b3390 2024-05-20T18:01:36.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.680+0000 7f781b5f7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f780c1c69f0 con 0x564c2368a4d0 2024-05-20T18:01:36.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.680+0000 7f781b5f7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f780c119170 con 0x564c23671f60 2024-05-20T18:01:36.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.683+0000 7f78124e8640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 0x564c23a5c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.683+0000 7f7811ce7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c2368a4d0 0x564c23a687f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.683+0000 7f7812ce9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a71f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.683+0000 7f7812ce9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a71f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41056/0 (socket says 172.21.15.129:41056) 2024-05-20T18:01:36.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.683+0000 7f7812ce9640 1 -- 172.21.15.129:0/1639055836 learned_addr learned my addr 172.21.15.129:0/1639055836 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:36.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.684+0000 7f7811ce7640 1 -- 172.21.15.129:0/1639055836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 msgr2=0x564c23a5c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:36.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.684+0000 7f7811ce7640 1 --2- 172.21.15.129:0/1639055836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 0x564c23a5c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.684+0000 7f7811ce7640 1 -- 172.21.15.129:0/1639055836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 msgr2=0x564c23a71f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:36.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.684+0000 7f7811ce7640 1 --2- 172.21.15.129:0/1639055836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a71f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.684+0000 7f7811ce7640 1 -- 172.21.15.129:0/1639055836 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c23a53580 con 0x564c2368a4d0 2024-05-20T18:01:36.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.685+0000 7f7811ce7640 1 --2- 172.21.15.129:0/1639055836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c2368a4d0 0x564c23a687f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77fc0edc90 tx=0x7f77fc0bc8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dc4dae496eb37db server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:36.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.697+0000 7f77f7fff640 1 -- 172.21.15.129:0/1639055836 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc0d5070 con 0x564c2368a4d0 2024-05-20T18:01:36.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.697+0000 7f77f7fff640 1 -- 172.21.15.129:0/1639055836 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77fc0bd160 con 0x564c2368a4d0 2024-05-20T18:01:36.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.699+0000 7f781b5f7740 1 -- 172.21.15.129:0/1639055836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c2368a4d0 msgr2=0x564c23a687f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:36.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.699+0000 7f781b5f7740 1 --2- 172.21.15.129:0/1639055836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c2368a4d0 0x564c23a687f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77fc0edc90 tx=0x7f77fc0bc8f0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.703+0000 7f781b5f7740 1 -- 172.21.15.129:0/1639055836 shutdown_connections 2024-05-20T18:01:36.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.704+0000 7f781b5f7740 1 --2- 172.21.15.129:0/1639055836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c2368a4d0 0x564c23a687f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.704+0000 7f781b5f7740 1 --2- 172.21.15.129:0/1639055836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a71f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.704+0000 7f781b5f7740 1 --2- 172.21.15.129:0/1639055836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c23671f60 0x564c23a5c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.704+0000 7f781b5f7740 1 -- 172.21.15.129:0/1639055836 >> 172.21.15.129:0/1639055836 conn(0x564c236a74e0 msgr2=0x564c23a5b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.707+0000 7f781b5f7740 1 -- 172.21.15.129:0/1639055836 shutdown_connections 2024-05-20T18:01:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.711+0000 7f781b5f7740 1 -- 172.21.15.129:0/1639055836 wait complete. 2024-05-20T18:01:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.712+0000 7f781b5f7740 1 -- start start 2024-05-20T18:01:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.712+0000 7f781b5f7740 1 Processor -- start 2024-05-20T18:01:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.712+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c23671f60 0x564c23a6ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.712+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c2368a4d0 0x564c23a6f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.712+0000 7f781b5f7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a6f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.712+0000 7f781b5f7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f780c119840 con 0x564c236b3390 2024-05-20T18:01:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.713+0000 7f781b5f7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f780c1c6320 con 0x564c23671f60 2024-05-20T18:01:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.713+0000 7f781b5f7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f780c119f10 con 0x564c2368a4d0 2024-05-20T18:01:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.713+0000 7f781b5f7740 10 client.0 _ll_register_callbacks cb 0x7f780800b5f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.717+0000 7f7812ce9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a6f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.717+0000 7f7812ce9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a6f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41072/0 (socket says 172.21.15.129:41072) 2024-05-20T18:01:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.717+0000 7f7812ce9640 1 -- 172.21.15.129:0/3222033612 learned_addr learned my addr 172.21.15.129:0/3222033612 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.717+0000 7f78124e8640 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c23671f60 0x564c23a6ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.717+0000 7f7811ce7640 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c2368a4d0 0x564c23a6f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.718+0000 7f78124e8640 1 -- 172.21.15.129:0/3222033612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c2368a4d0 msgr2=0x564c23a6f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.718+0000 7f78124e8640 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c2368a4d0 0x564c23a6f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.718+0000 7f78124e8640 1 -- 172.21.15.129:0/3222033612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 msgr2=0x564c23a6f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.718+0000 7f78124e8640 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a6f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.718+0000 7f78124e8640 1 -- 172.21.15.129:0/3222033612 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c23ad5090 con 0x564c23671f60 2024-05-20T18:01:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.718+0000 7f78124e8640 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c23671f60 0x564c23a6ec30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x564c23a5f590 tx=0x7f7804010b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e58f569cb4c4dc58 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.719+0000 7f77f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:01:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.719+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3222033612 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780400fa40 con 0x564c23671f60 2024-05-20T18:01:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.719+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3222033612 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f780400fbe0 con 0x564c23671f60 2024-05-20T18:01:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.719+0000 7f781b5f7740 1 -- 172.21.15.129:0/3222033612 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77fc0bd160 con 0x564c23671f60 2024-05-20T18:01:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.719+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3222033612 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f780400fde0 con 0x564c23671f60 2024-05-20T18:01:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.719+0000 7f781b5f7740 10 client.25622 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.719+0000 7f781b5f7740 1 -- 172.21.15.129:0/3222033612 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x564c23ad6330 con 0x564c23671f60 2024-05-20T18:01:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.720+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3222033612 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77fc0bd160 con 0x564c23671f60 2024-05-20T18:01:36.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.722+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3222033612 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f7804017ce0 con 0x564c23671f60 2024-05-20T18:01:36.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.722+0000 7f781b5f7740 10 client.25622 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:36.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.722+0000 7f781b5f7740 10 client.25622 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:36.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.722+0000 7f781b5f7740 20 client.25622 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:36.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.722+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564c23ad6a00 0x564c23ad8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:36.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.722+0000 7f781b5f7740 4 client.25622 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:01:36.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.722+0000 7f781b5f7740 1 -- 172.21.15.129:0/3222033612 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564c239bf6c0 con 0x564c23ad6a00 2024-05-20T18:01:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.723+0000 7f7811ce7640 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564c23ad6a00 0x564c23ad8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.724+0000 7f7811ce7640 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564c23ad6a00 0x564c23ad8e20 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88065bf892e91667 server_cookie=83abe78f07316ef1 in_seq=0 out_seq=0 2024-05-20T18:01:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.724+0000 7f77f5ffb640 10 client.25622 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:36.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.726+0000 7f77f5ffb640 1 -- 172.21.15.129:0/3222033612 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564c239bf6c0 con 0x564c23ad6a00 2024-05-20T18:01:36.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.726+0000 7f77f5ffb640 10 client.25622 handle_command_reply: tid=0 2024-05-20T18:01:36.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.726+0000 7f781b5f7740 1 client.25622 shutdown 2024-05-20T18:01:36.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.727+0000 7f781b5f7740 1 -- 172.21.15.129:0/3222033612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c23671f60 msgr2=0x564c23a6ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:36.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.727+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c23671f60 0x564c23a6ec30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x564c23a5f590 tx=0x7f7804010b10 comp rx=0 tx=0).stop 2024-05-20T18:01:36.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.729+0000 7f781b5f7740 1 -- 172.21.15.129:0/3222033612 shutdown_connections 2024-05-20T18:01:36.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.729+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564c23ad6a00 0x564c23ad8e20 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.729+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564c236b3390 0x564c23a6f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.729+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564c2368a4d0 0x564c23a6f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.729+0000 7f781b5f7740 1 --2- 172.21.15.129:0/3222033612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c23671f60 0x564c23a6ec30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.729+0000 7f781b5f7740 1 -- 172.21.15.129:0/3222033612 >> 172.21.15.129:0/3222033612 conn(0x564c236a74e0 msgr2=0x564c23a5a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:36.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.729+0000 7f77f5ffb640 0 client.25622 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:36.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.732+0000 7f781b5f7740 1 -- 172.21.15.129:0/3222033612 shutdown_connections 2024-05-20T18:01:36.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.732+0000 7f781b5f7740 1 -- 172.21.15.129:0/3222033612 wait complete. 2024-05-20T18:01:36.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.732+0000 7f781b5f7740 20 client.25622 trim_cache size 0 max 16384 2024-05-20T18:01:36.736 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:01:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.745+0000 7f78134ea640 1 -- 172.21.15.129:0/1669198865 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f77f80a2c80 msgr2=0x7f77f80a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.745+0000 7f78134ea640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f77f80a2c80 0x7f77f80a5190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7808012e60 tx=0x7f7808006f90 comp rx=0 tx=0).stop 2024-05-20T18:01:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.745+0000 7f78134ea640 1 -- 172.21.15.129:0/1669198865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c10e890 msgr2=0x7f780c1a8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.745+0000 7f78134ea640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c10e890 0x7f780c1a8a50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f77fc004bb0 tx=0x7f77fc004be0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.747+0000 7f78134ea640 1 -- 172.21.15.129:0/1669198865 shutdown_connections 2024-05-20T18:01:36.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.747+0000 7f78134ea640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f77f80a2c80 0x7f77f80a5190 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.747+0000 7f78134ea640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f780c10f270 0x7f780c1acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.747+0000 7f78134ea640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c10e890 0x7f780c1a8a50 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.747+0000 7f78134ea640 1 --2- 172.21.15.129:0/1669198865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c109090 0x7f780c1a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:36.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.748+0000 7f78134ea640 1 -- 172.21.15.129:0/1669198865 >> 172.21.15.129:0/1669198865 conn(0x7f780c06d580 msgr2=0x7f780c10a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:36.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.755+0000 7f78134ea640 1 -- 172.21.15.129:0/1669198865 shutdown_connections 2024-05-20T18:01:36.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:36.755+0000 7f78134ea640 1 -- 172.21.15.129:0/1669198865 wait complete. 2024-05-20T18:01:37.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:37 smithi154 ceph-mon[36482]: pgmap v95: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 869 MiB/s wr, 2.04k op/s 2024-05-20T18:01:37.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:37 smithi129 ceph-mon[32029]: pgmap v95: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 869 MiB/s wr, 2.04k op/s 2024-05-20T18:01:37.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:37 smithi204 ceph-mon[37076]: pgmap v95: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 869 MiB/s wr, 2.04k op/s 2024-05-20T18:01:37.837 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:01:37.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:01:37.837 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:01:38.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.455+0000 7fa31af38640 1 -- 172.21.15.129:0/3783538616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa31410e350 msgr2=0x7fa314079e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:38.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.455+0000 7fa31af38640 1 --2- 172.21.15.129:0/3783538616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa31410e350 0x7fa314079e10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00bc00 tx=0x7fa30c02e690 comp rx=0 tx=0).stop 2024-05-20T18:01:38.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.455+0000 7fa31af38640 1 -- 172.21.15.129:0/3783538616 shutdown_connections 2024-05-20T18:01:38.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.455+0000 7fa31af38640 1 --2- 172.21.15.129:0/3783538616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa31410e350 0x7fa314079e10 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.455+0000 7fa31af38640 1 --2- 172.21.15.129:0/3783538616 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa31410d890 0x7fa31410dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.455+0000 7fa31af38640 1 --2- 172.21.15.129:0/3783538616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa314109300 0x7fa314109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.455+0000 7fa31af38640 1 -- 172.21.15.129:0/3783538616 >> 172.21.15.129:0/3783538616 conn(0x7fa31406d750 msgr2=0x7fa31406dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.458+0000 7fa31af38640 1 -- 172.21.15.129:0/3783538616 shutdown_connections 2024-05-20T18:01:38.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.458+0000 7fa31af38640 1 -- 172.21.15.129:0/3783538616 wait complete. 2024-05-20T18:01:38.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.459+0000 7fa31af38640 1 Processor -- start 2024-05-20T18:01:38.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.460+0000 7fa31af38640 1 -- start start 2024-05-20T18:01:38.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.460+0000 7fa31af38640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa314109300 0x7fa3141b9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.460+0000 7fa31af38640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31410d890 0x7fa3141b9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.460+0000 7fa31af38640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa31410e350 0x7fa3141bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.460+0000 7fa31af38640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa314132c10 con 0x7fa31410e350 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.460+0000 7fa31af38640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa314132a90 con 0x7fa31410d890 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.460+0000 7fa31af38640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa314132d90 con 0x7fa314109300 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.460+0000 7fa313fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31410d890 0x7fa3141b9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.460+0000 7fa318cd4640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa314109300 0x7fa3141b9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.461+0000 7fa313fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31410d890 0x7fa3141b9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:46486/0 (socket says 172.21.15.129:46486) 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.461+0000 7fa313fff640 1 -- 172.21.15.129:0/489932385 learned_addr learned my addr 172.21.15.129:0/489932385 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.461+0000 7fa318cd4640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa314109300 0x7fa3141b9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:40002/0 (socket says 172.21.15.129:40002) 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.461+0000 7fa313fff640 1 -- 172.21.15.129:0/489932385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa314109300 msgr2=0x7fa3141b9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.461+0000 7fa313fff640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa314109300 0x7fa3141b9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.461+0000 7fa313fff640 1 -- 172.21.15.129:0/489932385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa31410e350 msgr2=0x7fa3141bdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:38.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.461+0000 7fa313fff640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa31410e350 0x7fa3141bdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.461+0000 7fa313fff640 1 -- 172.21.15.129:0/489932385 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3141be360 con 0x7fa31410d890 2024-05-20T18:01:38.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.462+0000 7fa313fff640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31410d890 0x7fa3141b9ab0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa30800fad0 tx=0x7fa30800fed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:38.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.463+0000 7fa311ffb640 1 -- 172.21.15.129:0/489932385 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30800dc90 con 0x7fa31410d890 2024-05-20T18:01:38.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.463+0000 7fa31af38640 1 -- 172.21.15.129:0/489932385 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3141be650 con 0x7fa31410d890 2024-05-20T18:01:38.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.464+0000 7fa31af38640 1 -- 172.21.15.129:0/489932385 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3141c5fd0 con 0x7fa31410d890 2024-05-20T18:01:38.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.468+0000 7fa321de6740 1 Processor -- start 2024-05-20T18:01:38.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.469+0000 7fa311ffb640 1 -- 172.21.15.129:0/489932385 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa30800de30 con 0x7fa31410d890 2024-05-20T18:01:38.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.469+0000 7fa311ffb640 1 -- 172.21.15.129:0/489932385 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa308017410 con 0x7fa31410d890 2024-05-20T18:01:38.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.480+0000 7fa311ffb640 1 -- 172.21.15.129:0/489932385 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa3080046c0 con 0x7fa31410d890 2024-05-20T18:01:38.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.481+0000 7fa311ffb640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa2fc0a2f10 0x7fa2fc0a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.482+0000 7fa311ffb640 1 -- 172.21.15.129:0/489932385 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa3080cf080 con 0x7fa31410d890 2024-05-20T18:01:38.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.509+0000 7fa321de6740 1 -- start start 2024-05-20T18:01:38.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.510+0000 7fa321de6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c0a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.510+0000 7fa321de6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c0ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.510+0000 7fa321de6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567684a730 0x555676c0b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.510+0000 7fa321de6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555676b6f990 con 0x55567684a730 2024-05-20T18:01:38.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.510+0000 7fa321de6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555676b175b0 con 0x555676842550 2024-05-20T18:01:38.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.510+0000 7fa321de6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x555676b6b950 con 0x55567686b410 2024-05-20T18:01:38.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.511+0000 7fa318cd4640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c0a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.511+0000 7fa313fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c0ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.511+0000 7fa313fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c0ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:46496/0 (socket says 172.21.15.129:46496) 2024-05-20T18:01:38.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.511+0000 7fa313fff640 1 -- 172.21.15.129:0/65420213 learned_addr learned my addr 172.21.15.129:0/65420213 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:38.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.514+0000 7fa3194d5640 1 --2- 172.21.15.129:0/65420213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567684a730 0x555676c0b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.515+0000 7fa3194d5640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa2fc0a2f10 0x7fa2fc0a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.518+0000 7fa318cd4640 1 -- 172.21.15.129:0/65420213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 msgr2=0x555676c0ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:38.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.518+0000 7fa318cd4640 1 --2- 172.21.15.129:0/65420213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c0ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.519+0000 7fa318cd4640 1 -- 172.21.15.129:0/65420213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567684a730 msgr2=0x555676c0b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:38.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.519+0000 7fa318cd4640 1 --2- 172.21.15.129:0/65420213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567684a730 0x555676c0b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.519+0000 7fa318cd4640 1 -- 172.21.15.129:0/65420213 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55567694b7e0 con 0x55567686b410 2024-05-20T18:01:38.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.519+0000 7fa318cd4640 1 --2- 172.21.15.129:0/65420213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c0a620 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa304009c30 tx=0x7fa304011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4096da0c5d307451 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:38.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.523+0000 7fa2faffd640 1 -- 172.21.15.129:0/65420213 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30401e070 con 0x55567686b410 2024-05-20T18:01:38.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.523+0000 7fa2faffd640 1 -- 172.21.15.129:0/65420213 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa304009de0 con 0x55567686b410 2024-05-20T18:01:38.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.526+0000 7fa3194d5640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa2fc0a2f10 0x7fa2fc0a5420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00bb50 tx=0x7fa30c00b840 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:38.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.529+0000 7fa2faffd640 1 -- 172.21.15.129:0/65420213 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa304019410 con 0x55567686b410 2024-05-20T18:01:38.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.530+0000 7fa321de6740 1 -- 172.21.15.129:0/65420213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 msgr2=0x555676c0a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:38.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.530+0000 7fa321de6740 1 --2- 172.21.15.129:0/65420213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c0a620 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa304009c30 tx=0x7fa304011810 comp rx=0 tx=0).stop 2024-05-20T18:01:38.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.533+0000 7fa321de6740 1 -- 172.21.15.129:0/65420213 shutdown_connections 2024-05-20T18:01:38.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.534+0000 7fa321de6740 1 --2- 172.21.15.129:0/65420213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c0ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.534+0000 7fa321de6740 1 --2- 172.21.15.129:0/65420213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567684a730 0x555676c0b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.534+0000 7fa321de6740 1 --2- 172.21.15.129:0/65420213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c0a620 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.534+0000 7fa321de6740 1 -- 172.21.15.129:0/65420213 >> 172.21.15.129:0/65420213 conn(0x55567685f560 msgr2=0x555676c06dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:38.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.540+0000 7fa321de6740 1 -- 172.21.15.129:0/65420213 shutdown_connections 2024-05-20T18:01:38.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.540+0000 7fa321de6740 1 -- 172.21.15.129:0/65420213 wait complete. 2024-05-20T18:01:38.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.541+0000 7fa321de6740 1 -- start start 2024-05-20T18:01:38.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.541+0000 7fa321de6740 1 Processor -- start 2024-05-20T18:01:38.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.545+0000 7fa321de6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c14d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.545+0000 7fa321de6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567684a730 0x555676c15250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.545+0000 7fa321de6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c1aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.545+0000 7fa321de6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555676c17970 con 0x55567684a730 2024-05-20T18:01:38.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.545+0000 7fa321de6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555676b2ab40 con 0x555676842550 2024-05-20T18:01:38.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.545+0000 7fa321de6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x555676c17af0 con 0x55567686b410 2024-05-20T18:01:38.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.546+0000 7fa321de6740 10 client.0 _ll_register_callbacks cb 0x555676b1f0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:38.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.547+0000 7fa313fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567684a730 0x555676c15250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.547+0000 7fa313fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567684a730 0x555676c15250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41104/0 (socket says 172.21.15.129:41104) 2024-05-20T18:01:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.547+0000 7fa313fff640 1 -- 172.21.15.129:0/3362700590 learned_addr learned my addr 172.21.15.129:0/3362700590 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.547+0000 7fa318cd4640 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c14d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.547+0000 7fa3194d5640 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c1aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.549+0000 7fa3194d5640 1 -- 172.21.15.129:0/3362700590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 msgr2=0x555676c14d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:38.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.549+0000 7fa3194d5640 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c14d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.549+0000 7fa3194d5640 1 -- 172.21.15.129:0/3362700590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567684a730 msgr2=0x555676c15250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:38.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.549+0000 7fa3194d5640 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567684a730 0x555676c15250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.549+0000 7fa3194d5640 1 -- 172.21.15.129:0/3362700590 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556769abf70 con 0x55567686b410 2024-05-20T18:01:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.550+0000 7fa3194d5640 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c1aed0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa30c043000 tx=0x7fa30c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae88b530e83f3362 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:38.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.552+0000 7fa2f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:01:38.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.552+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3362700590 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c03a040 con 0x55567686b410 2024-05-20T18:01:38.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.553+0000 7fa321de6740 1 -- 172.21.15.129:0/3362700590 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa304009de0 con 0x55567686b410 2024-05-20T18:01:38.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.553+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3362700590 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa30c004590 con 0x55567686b410 2024-05-20T18:01:38.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.556+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3362700590 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa30c059050 con 0x55567686b410 2024-05-20T18:01:38.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.556+0000 7fa321de6740 10 client.25672 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:38.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.556+0000 7fa321de6740 1 -- 172.21.15.129:0/3362700590 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555676843170 con 0x55567686b410 2024-05-20T18:01:38.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.557+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3362700590 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa304009de0 con 0x55567686b410 2024-05-20T18:01:38.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.557+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3362700590 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fa30c040040 con 0x55567686b410 2024-05-20T18:01:38.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.557+0000 7fa321de6740 10 client.25672 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:38.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.557+0000 7fa321de6740 10 client.25672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:38.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.557+0000 7fa321de6740 20 client.25672 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:38.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.557+0000 7fa321de6740 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55567682a1f0 0x555676c38680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.557+0000 7fa321de6740 4 client.25672 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:01:38.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.558+0000 7fa321de6740 1 -- 172.21.15.129:0/3362700590 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555676b7d0c0 con 0x55567682a1f0 2024-05-20T18:01:38.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.559+0000 7fa318cd4640 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55567682a1f0 0x555676c38680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.561+0000 7fa318cd4640 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55567682a1f0 0x555676c38680 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb301045943c07f8 server_cookie=4306497d176d807b in_seq=0 out_seq=0 2024-05-20T18:01:38.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.561+0000 7fa2f3fff640 10 client.25672 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.565+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3362700590 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555676b7d0c0 con 0x55567682a1f0 2024-05-20T18:01:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.565+0000 7fa2f3fff640 10 client.25672 handle_command_reply: tid=0 2024-05-20T18:01:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.565+0000 7fa321de6740 1 client.25672 shutdown 2024-05-20T18:01:38.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.571+0000 7fa321de6740 1 -- 172.21.15.129:0/3362700590 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 msgr2=0x555676c1aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:38.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.571+0000 7fa321de6740 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c1aed0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa30c043000 tx=0x7fa30c004a40 comp rx=0 tx=0).stop 2024-05-20T18:01:38.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.575+0000 7fa321de6740 1 -- 172.21.15.129:0/3362700590 shutdown_connections 2024-05-20T18:01:38.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.575+0000 7fa321de6740 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55567682a1f0 0x555676c38680 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.575+0000 7fa321de6740 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c1aed0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.575+0000 7fa321de6740 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567684a730 0x555676c15250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.575+0000 7fa321de6740 1 --2- 172.21.15.129:0/3362700590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c14d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.575+0000 7fa321de6740 1 -- 172.21.15.129:0/3362700590 >> 172.21.15.129:0/3362700590 conn(0x55567685f560 msgr2=0x555676c0a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:38.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.575+0000 7fa2f3fff640 0 client.25672 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:38.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.577+0000 7fa321de6740 1 -- 172.21.15.129:0/3362700590 shutdown_connections 2024-05-20T18:01:38.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.577+0000 7fa321de6740 1 -- 172.21.15.129:0/3362700590 wait complete. 2024-05-20T18:01:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.579+0000 7fa321de6740 20 client.25672 trim_cache size 0 max 16384 2024-05-20T18:01:38.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.675+0000 7fa321de6740 1 Processor -- start 2024-05-20T18:01:38.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.676+0000 7fa321de6740 1 -- start start 2024-05-20T18:01:38.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.677+0000 7fa321de6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555676842550 0x555676c0e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.677+0000 7fa321de6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55567684a730 0x555676c1ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.678+0000 7fa318cd4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555676842550 0x555676c0e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.678+0000 7fa318cd4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555676842550 0x555676c0e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41106/0 (socket says 172.21.15.129:41106) 2024-05-20T18:01:38.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.678+0000 7fa318cd4640 1 -- 172.21.15.129:0/3045772832 learned_addr learned my addr 172.21.15.129:0/3045772832 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.678+0000 7fa321de6740 1 --2- 172.21.15.129:0/3045772832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c1b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.678+0000 7fa321de6740 1 -- 172.21.15.129:0/3045772832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa30000c8c0 con 0x555676842550 2024-05-20T18:01:38.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.678+0000 7fa321de6740 1 -- 172.21.15.129:0/3045772832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa30000f140 con 0x55567684a730 2024-05-20T18:01:38.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.678+0000 7fa321de6740 1 -- 172.21.15.129:0/3045772832 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa30000cf30 con 0x55567686b410 2024-05-20T18:01:38.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.681+0000 7fa313fff640 1 --2- 172.21.15.129:0/3045772832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55567684a730 0x555676c1ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.681+0000 7fa3194d5640 1 --2- 172.21.15.129:0/3045772832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c1b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.682+0000 7fa318cd4640 1 -- 172.21.15.129:0/3045772832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 msgr2=0x555676c1b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.682+0000 7fa318cd4640 1 --2- 172.21.15.129:0/3045772832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c1b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.682+0000 7fa318cd4640 1 -- 172.21.15.129:0/3045772832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55567684a730 msgr2=0x555676c1ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.682+0000 7fa318cd4640 1 --2- 172.21.15.129:0/3045772832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55567684a730 0x555676c1ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.682+0000 7fa318cd4640 1 -- 172.21.15.129:0/3045772832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555676c15a70 con 0x555676842550 2024-05-20T18:01:38.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.684+0000 7fa318cd4640 1 --2- 172.21.15.129:0/3045772832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555676842550 0x555676c0e920 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x555676c07790 tx=0x7fa304009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6961f2af95bc217 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:38.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.685+0000 7fa2fa7fc640 1 -- 172.21.15.129:0/3045772832 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30401e040 con 0x555676842550 2024-05-20T18:01:38.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.685+0000 7fa2fa7fc640 1 -- 172.21.15.129:0/3045772832 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa30401f040 con 0x555676842550 2024-05-20T18:01:38.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.686+0000 7fa2fa7fc640 1 -- 172.21.15.129:0/3045772832 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa304004a80 con 0x555676842550 2024-05-20T18:01:38.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.688+0000 7fa321de6740 1 -- 172.21.15.129:0/3045772832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555676842550 msgr2=0x555676c0e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:38.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.688+0000 7fa321de6740 1 --2- 172.21.15.129:0/3045772832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555676842550 0x555676c0e920 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x555676c07790 tx=0x7fa304009970 comp rx=0 tx=0).stop 2024-05-20T18:01:38.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.700+0000 7fa321de6740 1 -- 172.21.15.129:0/3045772832 shutdown_connections 2024-05-20T18:01:38.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.701+0000 7fa321de6740 1 --2- 172.21.15.129:0/3045772832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55567684a730 0x555676c1ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.701+0000 7fa321de6740 1 --2- 172.21.15.129:0/3045772832 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567686b410 0x555676c1b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.701+0000 7fa321de6740 1 --2- 172.21.15.129:0/3045772832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555676842550 0x555676c0e920 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.701+0000 7fa321de6740 1 -- 172.21.15.129:0/3045772832 >> 172.21.15.129:0/3045772832 conn(0x55567685f560 msgr2=0x555676c0e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:38.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.705+0000 7fa321de6740 1 -- 172.21.15.129:0/3045772832 shutdown_connections 2024-05-20T18:01:38.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.705+0000 7fa321de6740 1 -- 172.21.15.129:0/3045772832 wait complete. 2024-05-20T18:01:38.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.706+0000 7fa321de6740 1 -- start start 2024-05-20T18:01:38.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.707+0000 7fa321de6740 1 Processor -- start 2024-05-20T18:01:38.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.709+0000 7fa321de6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c21600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.709+0000 7fa321de6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567684a730 0x555676c1e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.710+0000 7fa321de6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567686b410 0x555676c1eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.710+0000 7fa318cd4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c21600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.710+0000 7fa318cd4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c21600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:46542/0 (socket says 172.21.15.129:46542) 2024-05-20T18:01:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.710+0000 7fa318cd4640 1 -- 172.21.15.129:0/3794606956 learned_addr learned my addr 172.21.15.129:0/3794606956 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.710+0000 7fa321de6740 1 -- 172.21.15.129:0/3794606956 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa30000d600 con 0x55567686b410 2024-05-20T18:01:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.710+0000 7fa321de6740 1 -- 172.21.15.129:0/3794606956 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa30000ea70 con 0x555676842550 2024-05-20T18:01:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.710+0000 7fa321de6740 1 -- 172.21.15.129:0/3794606956 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa30000dcd0 con 0x55567684a730 2024-05-20T18:01:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.711+0000 7fa313fff640 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567684a730 0x555676c1e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.711+0000 7fa3194d5640 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567686b410 0x555676c1eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.711+0000 7fa321de6740 10 client.0 _ll_register_callbacks cb 0x7fa304005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.714+0000 7fa3194d5640 1 -- 172.21.15.129:0/3794606956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567684a730 msgr2=0x555676c1e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.714+0000 7fa3194d5640 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567684a730 0x555676c1e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.714+0000 7fa3194d5640 1 -- 172.21.15.129:0/3794606956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 msgr2=0x555676c21600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.714+0000 7fa3194d5640 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c21600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:38.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.714+0000 7fa3194d5640 1 -- 172.21.15.129:0/3794606956 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555676c1f3f0 con 0x55567686b410 2024-05-20T18:01:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.718+0000 7fa3194d5640 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567686b410 0x555676c1eeb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x555676c0b4b0 tx=0x7fa30c0416b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e7cf83be35f516 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.718+0000 7fa2f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:01:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.719+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3794606956 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c03a040 con 0x55567686b410 2024-05-20T18:01:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.720+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3794606956 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa30c004710 con 0x55567686b410 2024-05-20T18:01:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.720+0000 7fa321de6740 1 -- 172.21.15.129:0/3794606956 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa30401f040 con 0x55567686b410 2024-05-20T18:01:38.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.731+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3794606956 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa30c005bd0 con 0x55567686b410 2024-05-20T18:01:38.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.736+0000 7fa321de6740 10 client.15954 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:38.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.736+0000 7fa321de6740 1 -- 172.21.15.129:0/3794606956 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555676ca06a0 con 0x55567686b410 2024-05-20T18:01:38.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.739+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3794606956 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa30401f040 con 0x55567686b410 2024-05-20T18:01:38.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.740+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3794606956 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fa30c040070 con 0x55567686b410 2024-05-20T18:01:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.741+0000 7fa321de6740 10 client.15954 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.741+0000 7fa321de6740 10 client.15954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.741+0000 7fa321de6740 20 client.15954 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.741+0000 7fa321de6740 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556767ff630 0x555676ca2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.742+0000 7fa321de6740 4 client.15954 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:01:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.742+0000 7fa321de6740 1 -- 172.21.15.129:0/3794606956 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555676b77410 con 0x5556767ff630 2024-05-20T18:01:38.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.745+0000 7fa318cd4640 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556767ff630 0x555676ca2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:38.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.762+0000 7fa2f3fff640 10 client.15954 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:38.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:38.762+0000 7fa318cd4640 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556767ff630 0x555676ca2e10 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9df79e23df828ffc server_cookie=e623853eb3773dc4 in_seq=0 out_seq=0 2024-05-20T18:01:39.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.258+0000 7fa2f3fff640 1 -- 172.21.15.129:0/3794606956 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555676b77410 con 0x5556767ff630 2024-05-20T18:01:39.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.258+0000 7fa2f3fff640 10 client.15954 handle_command_reply: tid=0 2024-05-20T18:01:39.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.259+0000 7fa321de6740 1 client.15954 shutdown 2024-05-20T18:01:39.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.260+0000 7fa321de6740 1 -- 172.21.15.129:0/3794606956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567686b410 msgr2=0x555676c1eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:39.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.260+0000 7fa321de6740 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567686b410 0x555676c1eeb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x555676c0b4b0 tx=0x7fa30c0416b0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.260+0000 7fa321de6740 1 -- 172.21.15.129:0/3794606956 shutdown_connections 2024-05-20T18:01:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.260+0000 7fa321de6740 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556767ff630 0x555676ca2e10 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.260+0000 7fa321de6740 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55567686b410 0x555676c1eeb0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.260+0000 7fa321de6740 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55567684a730 0x555676c1e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.260+0000 7fa321de6740 1 --2- 172.21.15.129:0/3794606956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555676842550 0x555676c21600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.260+0000 7fa321de6740 1 -- 172.21.15.129:0/3794606956 >> 172.21.15.129:0/3794606956 conn(0x55567685f560 msgr2=0x555676c0a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.260+0000 7fa2f3fff640 0 client.15954 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.261+0000 7fa321de6740 1 -- 172.21.15.129:0/3794606956 shutdown_connections 2024-05-20T18:01:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.261+0000 7fa321de6740 1 -- 172.21.15.129:0/3794606956 wait complete. 2024-05-20T18:01:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.261+0000 7fa321de6740 20 client.15954 trim_cache size 0 max 16384 2024-05-20T18:01:39.261 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:01:39.261 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:01:39.261 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "acc2f1cc-4fb4-4b22-a614-4ad00679e3ee", 2024-05-20T18:01:39.261 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:01:39.261 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:01:39.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.272+0000 7fa31af38640 1 -- 172.21.15.129:0/489932385 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa2fc0a2f10 msgr2=0x7fa2fc0a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.272+0000 7fa31af38640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa2fc0a2f10 0x7fa2fc0a5420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00bb50 tx=0x7fa30c00b840 comp rx=0 tx=0).stop 2024-05-20T18:01:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.272+0000 7fa31af38640 1 -- 172.21.15.129:0/489932385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31410d890 msgr2=0x7fa3141b9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.272+0000 7fa31af38640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31410d890 0x7fa3141b9ab0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa30800fad0 tx=0x7fa30800fed0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.272+0000 7fa31af38640 1 -- 172.21.15.129:0/489932385 shutdown_connections 2024-05-20T18:01:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.272+0000 7fa31af38640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa2fc0a2f10 0x7fa2fc0a5420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.272+0000 7fa31af38640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31410d890 0x7fa3141b9ab0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.272+0000 7fa31af38640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa31410e350 0x7fa3141bdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.272+0000 7fa31af38640 1 --2- 172.21.15.129:0/489932385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa314109300 0x7fa3141b9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.272+0000 7fa31af38640 1 -- 172.21.15.129:0/489932385 >> 172.21.15.129:0/489932385 conn(0x7fa31406d750 msgr2=0x7fa31410b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:39.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.278+0000 7fa31af38640 1 -- 172.21.15.129:0/489932385 shutdown_connections 2024-05-20T18:01:39.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.279+0000 7fa31af38640 1 -- 172.21.15.129:0/489932385 wait complete. 2024-05-20T18:01:39.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:01:39.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:01:39.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:39 smithi129 ceph-mon[32029]: pgmap v96: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 774 MiB/s wr, 1.48k op/s 2024-05-20T18:01:39.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:39 smithi204 ceph-mon[37076]: pgmap v96: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 774 MiB/s wr, 1.48k op/s 2024-05-20T18:01:39.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:39 smithi154 ceph-mon[36482]: pgmap v96: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 774 MiB/s wr, 1.48k op/s 2024-05-20T18:01:39.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.771+0000 7fafc7fff640 1 -- 172.21.15.129:0/709409978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc00a5800 msgr2=0x7fafc00a5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:39.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.771+0000 7fafc7fff640 1 --2- 172.21.15.129:0/709409978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc00a5800 0x7fafc00a5c10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009900 tx=0x7fafb802a4d0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.779+0000 7fafc7fff640 1 -- 172.21.15.129:0/709409978 shutdown_connections 2024-05-20T18:01:39.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.779+0000 7fafc7fff640 1 --2- 172.21.15.129:0/709409978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc00b86d0 0x7fafc00baaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.779+0000 7fafc7fff640 1 --2- 172.21.15.129:0/709409978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafc00b5d70 0x7fafc00b8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.779+0000 7fafc7fff640 1 --2- 172.21.15.129:0/709409978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc00a5800 0x7fafc00a5c10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.779+0000 7fafc7fff640 1 -- 172.21.15.129:0/709409978 >> 172.21.15.129:0/709409978 conn(0x7fafc001aee0 msgr2=0x7fafc001b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:39.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.785+0000 7fafc7fff640 1 -- 172.21.15.129:0/709409978 shutdown_connections 2024-05-20T18:01:39.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.786+0000 7fafc7fff640 1 -- 172.21.15.129:0/709409978 wait complete. 2024-05-20T18:01:39.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.787+0000 7fafc7fff640 1 Processor -- start 2024-05-20T18:01:39.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.788+0000 7fafc7fff640 1 -- start start 2024-05-20T18:01:39.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.789+0000 7fafc7fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc00a5800 0x7fafc014a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:39.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.789+0000 7fafc7fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafc00b5d70 0x7fafc014ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:39.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.789+0000 7fafc7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc00b86d0 0x7fafc0144a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:39.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.789+0000 7fafc7fff640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fafc00be460 con 0x7fafc00b86d0 2024-05-20T18:01:39.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.789+0000 7fafc7fff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fafc00be2e0 con 0x7fafc00a5800 2024-05-20T18:01:39.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.789+0000 7fafc7fff640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fafc00be5e0 con 0x7fafc00b5d70 2024-05-20T18:01:39.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.790+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc00b86d0 0x7fafc0144a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:39.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.791+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc00b86d0 0x7fafc0144a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48870/0 (socket says 172.21.15.129:48870) 2024-05-20T18:01:39.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.791+0000 7fafc77fe640 1 -- 172.21.15.129:0/466078745 learned_addr learned my addr 172.21.15.129:0/466078745 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:39.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.792+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc00a5800 0x7fafc014a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.792+0000 7fafc67fc640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafc00b5d70 0x7fafc014ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.793+0000 7fafc6ffd640 1 -- 172.21.15.129:0/466078745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafc00b5d70 msgr2=0x7fafc014ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.793+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafc00b5d70 0x7fafc014ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.793+0000 7fafc6ffd640 1 -- 172.21.15.129:0/466078745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc00b86d0 msgr2=0x7fafc0144a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.793+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc00b86d0 0x7fafc0144a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.793+0000 7fafc6ffd640 1 -- 172.21.15.129:0/466078745 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc0145100 con 0x7fafc00a5800 2024-05-20T18:01:39.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.794+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc00a5800 0x7fafc014a850 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009850 tx=0x7fafb80055a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:39.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.795+0000 7fafcc8c8640 1 -- 172.21.15.129:0/466078745 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8041020 con 0x7fafc00a5800 2024-05-20T18:01:39.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.795+0000 7fafcc8c8640 1 -- 172.21.15.129:0/466078745 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb803dd70 con 0x7fafc00a5800 2024-05-20T18:01:39.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.795+0000 7fafcc8c8640 1 -- 172.21.15.129:0/466078745 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb803c3e0 con 0x7fafc00a5800 2024-05-20T18:01:39.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.799+0000 7fafc7fff640 1 -- 172.21.15.129:0/466078745 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc0145390 con 0x7fafc00a5800 2024-05-20T18:01:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.800+0000 7fafc7fff640 1 -- 172.21.15.129:0/466078745 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc0145860 con 0x7fafc00a5800 2024-05-20T18:01:39.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.806+0000 7fafd4773740 1 Processor -- start 2024-05-20T18:01:39.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.806+0000 7fafd4773740 1 -- start start 2024-05-20T18:01:39.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.812+0000 7fafd4773740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7906bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.813+0000 7fafd4773740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78ca44d0 0x55ca7906c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.813+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7906bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.813+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7906bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:46230/0 (socket says 172.21.15.129:46230) 2024-05-20T18:01:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.813+0000 7fafc77fe640 1 -- 172.21.15.129:0/1371383939 learned_addr learned my addr 172.21.15.129:0/1371383939 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.813+0000 7fafd4773740 1 --2- 172.21.15.129:0/1371383939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78cac6b0 0x55ca7906c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:39.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.813+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/1371383939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78ca44d0 0x55ca7906c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:39.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.814+0000 7fafd4773740 1 -- 172.21.15.129:0/1371383939 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ca78fd1d00 con 0x55ca78cac6b0 2024-05-20T18:01:39.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.814+0000 7fafd4773740 1 -- 172.21.15.129:0/1371383939 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ca78f794c0 con 0x55ca78ccd390 2024-05-20T18:01:39.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.814+0000 7fafd4773740 1 -- 172.21.15.129:0/1371383939 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ca78fcdcc0 con 0x55ca78ca44d0 2024-05-20T18:01:39.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.814+0000 7fafcc8c8640 1 -- 172.21.15.129:0/466078745 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fafb8044020 con 0x7fafc00a5800 2024-05-20T18:01:39.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.816+0000 7fafcc8c8640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fafb00a2f00 0x7fafb00a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.817+0000 7fafcc8c8640 1 -- 172.21.15.129:0/466078745 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fafb80f0430 con 0x7fafc00a5800 2024-05-20T18:01:39.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.818+0000 7fafc77fe640 1 -- 172.21.15.129:0/1371383939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78ca44d0 msgr2=0x55ca7906c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:39.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.818+0000 7fafc77fe640 1 --2- 172.21.15.129:0/1371383939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78ca44d0 0x55ca7906c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.818+0000 7fafc77fe640 1 -- 172.21.15.129:0/1371383939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78cac6b0 msgr2=0x55ca7906c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:39.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.818+0000 7fafc77fe640 1 --2- 172.21.15.129:0/1371383939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78cac6b0 0x55ca7906c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.818+0000 7fafc77fe640 1 -- 172.21.15.129:0/1371383939 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca78daf7d0 con 0x55ca78ccd390 2024-05-20T18:01:39.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.819+0000 7fafc77fe640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fafb00a2f00 0x7fafb00a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:39.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.819+0000 7fafc77fe640 1 --2- 172.21.15.129:0/1371383939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7906bcc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fafc80675f0 tx=0x7fafc8063020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=673bf25d845b99aa server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:39.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.825+0000 7fafc7fff640 1 -- 172.21.15.129:0/1371383939 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc806eb70 con 0x55ca78ccd390 2024-05-20T18:01:39.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.825+0000 7fafc7fff640 1 -- 172.21.15.129:0/1371383939 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc806ed10 con 0x55ca78ccd390 2024-05-20T18:01:39.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.825+0000 7fafc7fff640 1 -- 172.21.15.129:0/1371383939 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8079830 con 0x55ca78ccd390 2024-05-20T18:01:39.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.828+0000 7fafd4773740 1 -- 172.21.15.129:0/1371383939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 msgr2=0x55ca7906bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:39.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.828+0000 7fafd4773740 1 --2- 172.21.15.129:0/1371383939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7906bcc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fafc80675f0 tx=0x7fafc8063020 comp rx=0 tx=0).stop 2024-05-20T18:01:39.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.830+0000 7fafd4773740 1 -- 172.21.15.129:0/1371383939 shutdown_connections 2024-05-20T18:01:39.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.830+0000 7fafd4773740 1 --2- 172.21.15.129:0/1371383939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78cac6b0 0x55ca7906c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.830+0000 7fafd4773740 1 --2- 172.21.15.129:0/1371383939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78ca44d0 0x55ca7906c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.830+0000 7fafd4773740 1 --2- 172.21.15.129:0/1371383939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7906bcc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.830+0000 7fafd4773740 1 -- 172.21.15.129:0/1371383939 >> 172.21.15.129:0/1371383939 conn(0x55ca78cc14e0 msgr2=0x55ca79068880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:39.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.838+0000 7fafd4773740 1 -- 172.21.15.129:0/1371383939 shutdown_connections 2024-05-20T18:01:39.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.842+0000 7fafc77fe640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fafb00a2f00 0x7fafb00a5410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafc80675f0 tx=0x7fafc8063020 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:39.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.842+0000 7fafd4773740 1 -- 172.21.15.129:0/1371383939 wait complete. 2024-05-20T18:01:39.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.842+0000 7fafd4773740 1 -- start start 2024-05-20T18:01:39.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.843+0000 7fafd4773740 1 Processor -- start 2024-05-20T18:01:39.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.844+0000 7fafd4773740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78ca44d0 0x55ca790767b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:39.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.845+0000 7fafd4773740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78cac6b0 0x55ca79076cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:39.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.845+0000 7fafd4773740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7907c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:39.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.845+0000 7fafd4773740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ca790792b0 con 0x55ca78cac6b0 2024-05-20T18:01:39.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.845+0000 7fafd4773740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ca78f8cb90 con 0x55ca78ccd390 2024-05-20T18:01:39.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.845+0000 7fafd4773740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ca79079430 con 0x55ca78ca44d0 2024-05-20T18:01:39.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.845+0000 7fafd4773740 10 client.0 _ll_register_callbacks cb 0x55ca78fc8da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:39.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.846+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78ca44d0 0x55ca790767b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:39.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.847+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7907c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:39.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.847+0000 7fafc77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7907c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:46244/0 (socket says 172.21.15.129:46244) 2024-05-20T18:01:39.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.847+0000 7fafc77fe640 1 -- 172.21.15.129:0/3227054351 learned_addr learned my addr 172.21.15.129:0/3227054351 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:39.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.848+0000 7fafc67fc640 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78cac6b0 0x55ca79076cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:39.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.849+0000 7fafc77fe640 1 -- 172.21.15.129:0/3227054351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78ca44d0 msgr2=0x55ca790767b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:39.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.849+0000 7fafc77fe640 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78ca44d0 0x55ca790767b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.849+0000 7fafc77fe640 1 -- 172.21.15.129:0/3227054351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78cac6b0 msgr2=0x55ca79076cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:39.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.849+0000 7fafc77fe640 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78cac6b0 0x55ca79076cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.849+0000 7fafc77fe640 1 -- 172.21.15.129:0/3227054351 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca78ce53f0 con 0x55ca78ccd390 2024-05-20T18:01:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.855+0000 7fafc77fe640 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7907c790 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fafc802d830 tx=0x7fafc807d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b815bf42572b1abd server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.865+0000 7fafaaffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:01:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.865+0000 7fafaaffd640 1 -- 172.21.15.129:0/3227054351 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8096030 con 0x55ca78ccd390 2024-05-20T18:01:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.865+0000 7fafd4773740 1 -- 172.21.15.129:0/3227054351 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc806ed10 con 0x55ca78ccd390 2024-05-20T18:01:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.866+0000 7fafaaffd640 1 -- 172.21.15.129:0/3227054351 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fafc807b260 con 0x55ca78ccd390 2024-05-20T18:01:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.868+0000 7fafaaffd640 1 -- 172.21.15.129:0/3227054351 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fafc807bd60 con 0x55ca78ccd390 2024-05-20T18:01:39.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.868+0000 7fafd4773740 10 client.25652 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:39.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.868+0000 7fafd4773740 1 -- 172.21.15.129:0/3227054351 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ca78ca50f0 con 0x55ca78ccd390 2024-05-20T18:01:39.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.870+0000 7fafaaffd640 1 -- 172.21.15.129:0/3227054351 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc806ed10 con 0x55ca78ccd390 2024-05-20T18:01:39.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.873+0000 7fafaaffd640 1 -- 172.21.15.129:0/3227054351 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fafc806b040 con 0x55ca78ccd390 2024-05-20T18:01:39.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.873+0000 7fafd4773740 10 client.25652 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:39.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.873+0000 7fafd4773740 10 client.25652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:39.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.873+0000 7fafd4773740 20 client.25652 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:39.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.873+0000 7fafd4773740 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca78c8bf60 0x55ca79089240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:39.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.873+0000 7fafd4773740 4 client.25652 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:01:39.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.873+0000 7fafd4773740 1 -- 172.21.15.129:0/3227054351 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca78fdf430 con 0x55ca78c8bf60 2024-05-20T18:01:39.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.874+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca78c8bf60 0x55ca79089240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:39.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.875+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca78c8bf60 0x55ca79089240 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc9283a6fbbc587c server_cookie=5f0bd1d305cb49bb in_seq=0 out_seq=0 2024-05-20T18:01:39.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.875+0000 7fafaaffd640 10 client.25652 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:39.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.886+0000 7fafaaffd640 1 -- 172.21.15.129:0/3227054351 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ca78fdf430 con 0x55ca78c8bf60 2024-05-20T18:01:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.886+0000 7fafaaffd640 10 client.25652 handle_command_reply: tid=0 2024-05-20T18:01:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.886+0000 7fafd4773740 1 client.25652 shutdown 2024-05-20T18:01:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.887+0000 7fafd4773740 1 -- 172.21.15.129:0/3227054351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 msgr2=0x55ca7907c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.887+0000 7fafd4773740 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7907c790 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fafc802d830 tx=0x7fafc807d040 comp rx=0 tx=0).stop 2024-05-20T18:01:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.893+0000 7fafd4773740 1 -- 172.21.15.129:0/3227054351 shutdown_connections 2024-05-20T18:01:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.893+0000 7fafd4773740 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca78c8bf60 0x55ca79089240 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.893+0000 7fafd4773740 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7907c790 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.893+0000 7fafd4773740 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78cac6b0 0x55ca79076cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.893+0000 7fafd4773740 1 --2- 172.21.15.129:0/3227054351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78ca44d0 0x55ca790767b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:39.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.893+0000 7fafd4773740 1 -- 172.21.15.129:0/3227054351 >> 172.21.15.129:0/3227054351 conn(0x55ca78cc14e0 msgr2=0x55ca790678e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:39.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.898+0000 7fafaaffd640 0 client.25652 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:39.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.901+0000 7fafd4773740 1 -- 172.21.15.129:0/3227054351 shutdown_connections 2024-05-20T18:01:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.905+0000 7fafd4773740 1 -- 172.21.15.129:0/3227054351 wait complete. 2024-05-20T18:01:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.905+0000 7fafd4773740 20 client.25652 trim_cache size 0 max 16384 2024-05-20T18:01:40.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.995+0000 7fafd4773740 1 Processor -- start 2024-05-20T18:01:40.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.996+0000 7fafd4773740 1 -- start start 2024-05-20T18:01:40.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.998+0000 7fafd4773740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ca44d0 0x55ca7907c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:40.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.998+0000 7fafd4773740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78cac6b0 0x55ca79071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:40.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.998+0000 7fafd4773740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78ccd390 0x55ca79074e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:40.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.998+0000 7fafd4773740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fafc0157dc0 con 0x55ca78ccd390 2024-05-20T18:01:40.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.998+0000 7fafd4773740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fafc015a590 con 0x55ca78ca44d0 2024-05-20T18:01:40.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:39.999+0000 7fafd4773740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fafc0158380 con 0x55ca78cac6b0 2024-05-20T18:01:40.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.000+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ca44d0 0x55ca7907c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:40.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.000+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ca44d0 0x55ca7907c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:46254/0 (socket says 172.21.15.129:46254) 2024-05-20T18:01:40.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.000+0000 7fafc6ffd640 1 -- 172.21.15.129:0/4215271928 learned_addr learned my addr 172.21.15.129:0/4215271928 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:40.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.000+0000 7fafc67fc640 1 --2- 172.21.15.129:0/4215271928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78cac6b0 0x55ca79071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:40.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.001+0000 7fafc67fc640 1 -- 172.21.15.129:0/4215271928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ca44d0 msgr2=0x55ca7907c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:40.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.001+0000 7fafc67fc640 1 --2- 172.21.15.129:0/4215271928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ca44d0 0x55ca7907c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.001+0000 7fafc67fc640 1 -- 172.21.15.129:0/4215271928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78ccd390 msgr2=0x55ca79074e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:40.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.001+0000 7fafc67fc640 1 --2- 172.21.15.129:0/4215271928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78ccd390 0x55ca79074e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.001+0000 7fafc67fc640 1 -- 172.21.15.129:0/4215271928 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca79076e90 con 0x55ca78cac6b0 2024-05-20T18:01:40.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.003+0000 7fafc67fc640 1 --2- 172.21.15.129:0/4215271928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78cac6b0 0x55ca79071780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x55ca79069980 tx=0x7fafbc00bb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb3d3648127be9c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:40.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.003+0000 7fafb57fa640 1 -- 172.21.15.129:0/4215271928 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc018020 con 0x55ca78cac6b0 2024-05-20T18:01:40.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.003+0000 7fafb57fa640 1 -- 172.21.15.129:0/4215271928 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafbc00fc70 con 0x55ca78cac6b0 2024-05-20T18:01:40.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.004+0000 7fafb57fa640 1 -- 172.21.15.129:0/4215271928 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafbc014e50 con 0x55ca78cac6b0 2024-05-20T18:01:40.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.005+0000 7fafd4773740 1 -- 172.21.15.129:0/4215271928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78cac6b0 msgr2=0x55ca79071780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:40.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.005+0000 7fafd4773740 1 --2- 172.21.15.129:0/4215271928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78cac6b0 0x55ca79071780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x55ca79069980 tx=0x7fafbc00bb80 comp rx=0 tx=0).stop 2024-05-20T18:01:40.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.006+0000 7fafd4773740 1 -- 172.21.15.129:0/4215271928 shutdown_connections 2024-05-20T18:01:40.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.006+0000 7fafd4773740 1 --2- 172.21.15.129:0/4215271928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78ccd390 0x55ca79074e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.006+0000 7fafd4773740 1 --2- 172.21.15.129:0/4215271928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78cac6b0 0x55ca79071780 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.006+0000 7fafd4773740 1 --2- 172.21.15.129:0/4215271928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ca44d0 0x55ca7907c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.006+0000 7fafd4773740 1 -- 172.21.15.129:0/4215271928 >> 172.21.15.129:0/4215271928 conn(0x55ca78cc14e0 msgr2=0x55ca79070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:40.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.016+0000 7fafd4773740 1 -- 172.21.15.129:0/4215271928 shutdown_connections 2024-05-20T18:01:40.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.017+0000 7fafd4773740 1 -- 172.21.15.129:0/4215271928 wait complete. 2024-05-20T18:01:40.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.018+0000 7fafd4773740 1 -- start start 2024-05-20T18:01:40.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.019+0000 7fafd4773740 1 Processor -- start 2024-05-20T18:01:40.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.020+0000 7fafd4773740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78ca44d0 0x55ca79082ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:40.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.020+0000 7fafd4773740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7907d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:40.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.020+0000 7fafd4773740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78c613a0 0x55ca7907dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:40.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.021+0000 7fafd4773740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fafc0158a50 con 0x55ca78ca44d0 2024-05-20T18:01:40.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.021+0000 7fafd4773740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fafc0159ec0 con 0x55ca78ccd390 2024-05-20T18:01:40.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.021+0000 7fafd4773740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fafc0159120 con 0x55ca78c613a0 2024-05-20T18:01:40.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.021+0000 7fafd4773740 10 client.0 _ll_register_callbacks cb 0x7fafc808a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:40.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.022+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78ca44d0 0x55ca79082ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:40.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.022+0000 7fafc6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78ca44d0 0x55ca79082ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48908/0 (socket says 172.21.15.129:48908) 2024-05-20T18:01:40.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.022+0000 7fafc6ffd640 1 -- 172.21.15.129:0/2735244809 learned_addr learned my addr 172.21.15.129:0/2735244809 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:40.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.022+0000 7fafc77fe640 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78c613a0 0x55ca7907dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:40.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.023+0000 7fafc67fc640 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7907d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:40.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.024+0000 7fafc77fe640 1 -- 172.21.15.129:0/2735244809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 msgr2=0x55ca7907d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:40.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.024+0000 7fafc77fe640 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7907d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.024+0000 7fafc77fe640 1 -- 172.21.15.129:0/2735244809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78ca44d0 msgr2=0x55ca79082ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:40.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.024+0000 7fafc77fe640 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78ca44d0 0x55ca79082ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.024+0000 7fafc77fe640 1 -- 172.21.15.129:0/2735244809 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca79071e60 con 0x55ca78c613a0 2024-05-20T18:01:40.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.025+0000 7fafc77fe640 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78c613a0 0x55ca7907dae0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fafc8050f00 tx=0x7fafc806ef80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f55c43b1236a480 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:40.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.025+0000 7fafaaffd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:01:40.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.025+0000 7fafaaffd640 1 -- 172.21.15.129:0/2735244809 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8096070 con 0x55ca78c613a0 2024-05-20T18:01:40.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.026+0000 7fafd4773740 1 -- 172.21.15.129:0/2735244809 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafbc00fc70 con 0x55ca78c613a0 2024-05-20T18:01:40.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.026+0000 7fafaaffd640 1 -- 172.21.15.129:0/2735244809 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fafc8079920 con 0x55ca78c613a0 2024-05-20T18:01:40.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.028+0000 7fafaaffd640 1 -- 172.21.15.129:0/2735244809 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fafc807ada0 con 0x55ca78c613a0 2024-05-20T18:01:40.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.032+0000 7fafd4773740 10 client.25687 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:40.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.032+0000 7fafd4773740 1 -- 172.21.15.129:0/2735244809 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ca790f0110 con 0x55ca78c613a0 2024-05-20T18:01:40.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.033+0000 7fafaaffd640 1 -- 172.21.15.129:0/2735244809 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafbc00fc70 con 0x55ca78c613a0 2024-05-20T18:01:40.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.035+0000 7fafaaffd640 1 -- 172.21.15.129:0/2735244809 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fafc806b070 con 0x55ca78c613a0 2024-05-20T18:01:40.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.035+0000 7fafd4773740 10 client.25687 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:40.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.035+0000 7fafd4773740 10 client.25687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:40.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.035+0000 7fafd4773740 20 client.25687 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:40.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.036+0000 7fafd4773740 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca790f07c0 0x55ca790f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:40.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.036+0000 7fafd4773740 4 client.25687 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:01:40.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.036+0000 7fafd4773740 1 -- 172.21.15.129:0/2735244809 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ca78fd9780 con 0x55ca790f07c0 2024-05-20T18:01:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.037+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca790f07c0 0x55ca790f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:40.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.057+0000 7fafc6ffd640 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca790f07c0 0x55ca790f2be0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbd4ed80011898fb server_cookie=980df2e317558b2 in_seq=0 out_seq=0 2024-05-20T18:01:40.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.058+0000 7fafaaffd640 10 client.25687 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:40.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.065+0000 7fafaaffd640 1 -- 172.21.15.129:0/2735244809 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55ca78fd9780 con 0x55ca790f07c0 2024-05-20T18:01:40.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.065+0000 7fafaaffd640 10 client.25687 handle_command_reply: tid=0 2024-05-20T18:01:40.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.066+0000 7fafd4773740 1 client.25687 shutdown 2024-05-20T18:01:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.083+0000 7fafd4773740 1 -- 172.21.15.129:0/2735244809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78c613a0 msgr2=0x55ca7907dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.083+0000 7fafd4773740 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78c613a0 0x55ca7907dae0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fafc8050f00 tx=0x7fafc806ef80 comp rx=0 tx=0).stop 2024-05-20T18:01:40.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.086+0000 7fafd4773740 1 -- 172.21.15.129:0/2735244809 shutdown_connections 2024-05-20T18:01:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.086+0000 7fafd4773740 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca790f07c0 0x55ca790f2be0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.086+0000 7fafd4773740 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca78ccd390 0x55ca7907d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.086+0000 7fafd4773740 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca78c613a0 0x55ca7907dae0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.086+0000 7fafd4773740 1 --2- 172.21.15.129:0/2735244809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca78ca44d0 0x55ca79082ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.086+0000 7fafd4773740 1 -- 172.21.15.129:0/2735244809 >> 172.21.15.129:0/2735244809 conn(0x55ca78cc14e0 msgr2=0x55ca79074e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:40.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.089+0000 7fafaaffd640 0 client.25687 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:40.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.089+0000 7fafd4773740 1 -- 172.21.15.129:0/2735244809 shutdown_connections 2024-05-20T18:01:40.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.089+0000 7fafd4773740 1 -- 172.21.15.129:0/2735244809 wait complete. 2024-05-20T18:01:40.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.090+0000 7fafd4773740 20 client.25687 trim_cache size 0 max 16384 2024-05-20T18:01:40.095 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:01:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "scrub active (7 inodes in the stack)", 2024-05-20T18:01:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:01:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: "acc2f1cc-4fb4-4b22-a614-4ad00679e3ee": { 2024-05-20T18:01:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:01:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "acc2f1cc-4fb4-4b22-a614-4ad00679e3ee", 2024-05-20T18:01:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:01:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:01:40.095 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:01:40.095 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:01:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.097+0000 7fafb67fc640 1 -- 172.21.15.129:0/466078745 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fafb00a2f00 msgr2=0x7fafb00a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.098+0000 7fafb67fc640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fafb00a2f00 0x7fafb00a5410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafc80675f0 tx=0x7fafc8063020 comp rx=0 tx=0).stop 2024-05-20T18:01:40.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.098+0000 7fafb67fc640 1 -- 172.21.15.129:0/466078745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc00a5800 msgr2=0x7fafc014a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:40.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.098+0000 7fafb67fc640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc00a5800 0x7fafc014a850 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009850 tx=0x7fafb80055a0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.098+0000 7fafb67fc640 1 -- 172.21.15.129:0/466078745 shutdown_connections 2024-05-20T18:01:40.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.098+0000 7fafb67fc640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fafb00a2f00 0x7fafb00a5410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.098+0000 7fafb67fc640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc00b86d0 0x7fafc0144a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.099+0000 7fafb67fc640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fafc00b5d70 0x7fafc014ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.099+0000 7fafb67fc640 1 --2- 172.21.15.129:0/466078745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc00a5800 0x7fafc014a850 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:40.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.099+0000 7fafb67fc640 1 -- 172.21.15.129:0/466078745 >> 172.21.15.129:0/466078745 conn(0x7fafc001aee0 msgr2=0x7fafc00b75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:40.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.099+0000 7fafb67fc640 1 -- 172.21.15.129:0/466078745 shutdown_connections 2024-05-20T18:01:40.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:01:40.100+0000 7fafb67fc640 1 -- 172.21.15.129:0/466078745 wait complete. 2024-05-20T18:01:40.149 INFO:tasks.cephfs.filesystem:scrub status for tag:acc2f1cc-4fb4-4b22-a614-4ad00679e3ee - {'path': '/', 'tag': 'acc2f1cc-4fb4-4b22-a614-4ad00679e3ee', 'options': 'recursive,force'} 2024-05-20T18:01:41.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:41 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:01:41.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:41 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:41.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:41 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:01:41.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:41 smithi129 ceph-mon[32029]: pgmap v97: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 887 MiB/s wr, 1.49k op/s 2024-05-20T18:01:41.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:41 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:01:41.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:41 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:41.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:41 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:01:41.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:41 smithi204 ceph-mon[37076]: pgmap v97: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 887 MiB/s wr, 1.49k op/s 2024-05-20T18:01:41.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:41 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:01:41.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:41 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:41.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:41 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:01:41.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:41 smithi154 ceph-mon[36482]: pgmap v97: 97 pgs: 97 active+clean; 20 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 887 MiB/s wr, 1.49k op/s 2024-05-20T18:01:42.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:42 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:42.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:42 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:42.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:42 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:01:43.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:43 smithi154 ceph-mon[36482]: pgmap v98: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 811 MiB/s wr, 1.21k op/s 2024-05-20T18:01:43.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:43 smithi204 ceph-mon[37076]: pgmap v98: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 811 MiB/s wr, 1.21k op/s 2024-05-20T18:01:43.681 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:43 smithi129 ceph-mon[32029]: pgmap v98: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 811 MiB/s wr, 1.21k op/s 2024-05-20T18:01:45.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:45 smithi154 ceph-mon[36482]: pgmap v99: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 682 MiB/s wr, 1.08k op/s 2024-05-20T18:01:45.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:45 smithi129 ceph-mon[32029]: pgmap v99: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 682 MiB/s wr, 1.08k op/s 2024-05-20T18:01:45.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:45 smithi204 ceph-mon[37076]: pgmap v99: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 682 MiB/s wr, 1.08k op/s 2024-05-20T18:01:46.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:46 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:01:46.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:46 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:01:46.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:46 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:01:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.874+0000 7fa71d6d6640 1 -- 172.21.15.129:0/3813573687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71810e890 msgr2=0x7fa71810eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:46.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.874+0000 7fa71d6d6640 1 --2- 172.21.15.129:0/3813573687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71810e890 0x7fa71810eca0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa708009900 tx=0x7fa70802a400 comp rx=0 tx=0).stop 2024-05-20T18:01:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.886+0000 7fa71d6d6640 1 -- 172.21.15.129:0/3813573687 shutdown_connections 2024-05-20T18:01:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.886+0000 7fa71d6d6640 1 --2- 172.21.15.129:0/3813573687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa718109090 0x7fa718109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.886+0000 7fa71d6d6640 1 --2- 172.21.15.129:0/3813573687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa71810f270 0x7fa718108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.886+0000 7fa71d6d6640 1 --2- 172.21.15.129:0/3813573687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71810e890 0x7fa71810eca0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.886+0000 7fa71d6d6640 1 -- 172.21.15.129:0/3813573687 >> 172.21.15.129:0/3813573687 conn(0x7fa71806e690 msgr2=0x7fa71806eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.887+0000 7fa71d6d6640 1 -- 172.21.15.129:0/3813573687 shutdown_connections 2024-05-20T18:01:46.889 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.890+0000 7fa71d6d6640 1 -- 172.21.15.129:0/3813573687 wait complete. 2024-05-20T18:01:46.889 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.892+0000 7fa71d6d6640 1 Processor -- start 2024-05-20T18:01:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.892+0000 7fa71d6d6640 1 -- start start 2024-05-20T18:01:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.893+0000 7fa71d6d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa718109090 0x7fa7181a83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.893+0000 7fa71d6d6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa71810e890 0x7fa7181a8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.893+0000 7fa71d6d6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71810f270 0x7fa7181acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.893+0000 7fa71d6d6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa718121ad0 con 0x7fa71810e890 2024-05-20T18:01:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.893+0000 7fa71d6d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa718121950 con 0x7fa718109090 2024-05-20T18:01:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.893+0000 7fa71d6d6640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa718121c50 con 0x7fa71810f270 2024-05-20T18:01:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.897+0000 7fa71ced5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71810f270 0x7fa7181acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.897+0000 7fa717fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa718109090 0x7fa7181a83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.897+0000 7fa71ced5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71810f270 0x7fa7181acbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:59120/0 (socket says 172.21.15.129:59120) 2024-05-20T18:01:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.897+0000 7fa71ced5640 1 -- 172.21.15.129:0/409951498 learned_addr learned my addr 172.21.15.129:0/409951498 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.900+0000 7fa71ced5640 1 -- 172.21.15.129:0/409951498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa718109090 msgr2=0x7fa7181a83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.900+0000 7fa71ced5640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa718109090 0x7fa7181a83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.900+0000 7fa71ced5640 1 -- 172.21.15.129:0/409951498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa71810e890 msgr2=0x7fa7181a8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.900+0000 7fa71ced5640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa71810e890 0x7fa7181a8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.900+0000 7fa71ced5640 1 -- 172.21.15.129:0/409951498 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7181ad210 con 0x7fa71810f270 2024-05-20T18:01:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.901+0000 7fa71ced5640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71810f270 0x7fa7181acbd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa71000b970 tx=0x7fa71000be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.902+0000 7fa7157fa640 1 -- 172.21.15.129:0/409951498 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa710019070 con 0x7fa71810f270 2024-05-20T18:01:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.902+0000 7fa7157fa640 1 -- 172.21.15.129:0/409951498 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa71000cb40 con 0x7fa71810f270 2024-05-20T18:01:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.902+0000 7fa7157fa640 1 -- 172.21.15.129:0/409951498 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa710007650 con 0x7fa71810f270 2024-05-20T18:01:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.902+0000 7fa71d6d6640 1 -- 172.21.15.129:0/409951498 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7181ad500 con 0x7fa71810f270 2024-05-20T18:01:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.902+0000 7fa71d6d6640 1 -- 172.21.15.129:0/409951498 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7181ada00 con 0x7fa71810f270 2024-05-20T18:01:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.908+0000 7fa7257e9740 1 Processor -- start 2024-05-20T18:01:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.910+0000 7fa7157fa640 1 -- 172.21.15.129:0/409951498 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa71001c070 con 0x7fa71810f270 2024-05-20T18:01:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.911+0000 7fa7257e9740 1 -- start start 2024-05-20T18:01:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.911+0000 7fa7257e9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a651b0 0x5591d2e04280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.911+0000 7fa7257e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a3c2f0 0x5591d2e047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.911+0000 7fa7257e9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e04d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.911+0000 7fa7257e9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5591d2d69340 con 0x5591d2a651b0 2024-05-20T18:01:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.911+0000 7fa7257e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5591d2d11290 con 0x5591d2a3c2f0 2024-05-20T18:01:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.911+0000 7fa7257e9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5591d2d65300 con 0x5591d2a444d0 2024-05-20T18:01:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.915+0000 7fa717fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a3c2f0 0x5591d2e047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.915+0000 7fa7177fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e04d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.915+0000 7fa717fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a3c2f0 0x5591d2e047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:46282/0 (socket says 172.21.15.129:46282) 2024-05-20T18:01:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.915+0000 7fa717fff640 1 -- 172.21.15.129:0/1056931216 learned_addr learned my addr 172.21.15.129:0/1056931216 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.915+0000 7fa7177fe640 1 --2- 172.21.15.129:0/1056931216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a651b0 0x5591d2e04280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.915+0000 7fa7157fa640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa7000a2e80 0x7fa7000a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.916+0000 7fa7157fa640 1 -- 172.21.15.129:0/409951498 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa7100cedf0 con 0x7fa71810f270 2024-05-20T18:01:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.916+0000 7fa7177fe640 1 -- 172.21.15.129:0/1056931216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a3c2f0 msgr2=0x5591d2e047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.916+0000 7fa7177fe640 1 --2- 172.21.15.129:0/1056931216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a3c2f0 0x5591d2e047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.917+0000 7fa7177fe640 1 -- 172.21.15.129:0/1056931216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a651b0 msgr2=0x5591d2e04280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.917+0000 7fa7177fe640 1 --2- 172.21.15.129:0/1056931216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a651b0 0x5591d2e04280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.917+0000 7fa7177fe640 1 -- 172.21.15.129:0/1056931216 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d2d358e0 con 0x5591d2a444d0 2024-05-20T18:01:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.919+0000 7fa71ced5640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa7000a2e80 0x7fa7000a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.920+0000 7fa7177fe640 1 --2- 172.21.15.129:0/1056931216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e04d00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa70c005d00 tx=0x7fa70c00d3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ea30441064d82fc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.921+0000 7fa6fe7fc640 1 -- 172.21.15.129:0/1056931216 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70c00a620 con 0x5591d2a444d0 2024-05-20T18:01:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.921+0000 7fa6fe7fc640 1 -- 172.21.15.129:0/1056931216 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa70c00aca0 con 0x5591d2a444d0 2024-05-20T18:01:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.922+0000 7fa7257e9740 1 -- 172.21.15.129:0/1056931216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 msgr2=0x5591d2e04d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.922+0000 7fa7257e9740 1 --2- 172.21.15.129:0/1056931216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e04d00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa70c005d00 tx=0x7fa70c00d3a0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.923+0000 7fa71ced5640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa7000a2e80 0x7fa7000a5390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa7100cc6f0 tx=0x7fa710004b50 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.926+0000 7fa7257e9740 1 -- 172.21.15.129:0/1056931216 shutdown_connections 2024-05-20T18:01:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.926+0000 7fa7257e9740 1 --2- 172.21.15.129:0/1056931216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a3c2f0 0x5591d2e047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.926+0000 7fa7257e9740 1 --2- 172.21.15.129:0/1056931216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e04d00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.926+0000 7fa7257e9740 1 --2- 172.21.15.129:0/1056931216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a651b0 0x5591d2e04280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.926+0000 7fa7257e9740 1 -- 172.21.15.129:0/1056931216 >> 172.21.15.129:0/1056931216 conn(0x5591d2a59300 msgr2=0x5591d2e00a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.927+0000 7fa7257e9740 1 -- 172.21.15.129:0/1056931216 shutdown_connections 2024-05-20T18:01:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.927+0000 7fa7257e9740 1 -- 172.21.15.129:0/1056931216 wait complete. 2024-05-20T18:01:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.928+0000 7fa7257e9740 1 -- start start 2024-05-20T18:01:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.928+0000 7fa7257e9740 1 Processor -- start 2024-05-20T18:01:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.929+0000 7fa7257e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a3c2f0 0x5591d2e0e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.929+0000 7fa7257e9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e0ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.929+0000 7fa7257e9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a651b0 0x5591d2e14940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.929+0000 7fa7257e9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5591d2e11400 con 0x5591d2a651b0 2024-05-20T18:01:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.929+0000 7fa7257e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5591d2d24790 con 0x5591d2a3c2f0 2024-05-20T18:01:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.929+0000 7fa7257e9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5591d2e11580 con 0x5591d2a444d0 2024-05-20T18:01:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.930+0000 7fa7257e9740 10 client.0 _ll_register_callbacks cb 0x5591d2d5fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:46.929 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.930+0000 7fa717fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e0ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:46.929 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.930+0000 7fa717fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e0ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:59148/0 (socket says 172.21.15.129:59148) 2024-05-20T18:01:46.929 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.930+0000 7fa717fff640 1 -- 172.21.15.129:0/3205028321 learned_addr learned my addr 172.21.15.129:0/3205028321 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:46.929 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.930+0000 7fa7177fe640 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a3c2f0 0x5591d2e0e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:46.929 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.930+0000 7fa7177fe640 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a651b0 0x5591d2e14940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:46.929 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.931+0000 7fa717fff640 1 -- 172.21.15.129:0/3205028321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a3c2f0 msgr2=0x5591d2e0e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:46.930 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.931+0000 7fa717fff640 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a3c2f0 0x5591d2e0e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.930 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.931+0000 7fa717fff640 1 -- 172.21.15.129:0/3205028321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a651b0 msgr2=0x5591d2e14940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:46.930 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.931+0000 7fa717fff640 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a651b0 0x5591d2e14940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.930 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.931+0000 7fa717fff640 1 -- 172.21.15.129:0/3205028321 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591d2a7d210 con 0x5591d2a444d0 2024-05-20T18:01:46.930 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.932+0000 7fa717fff640 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e0ee80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa70802a9f0 tx=0x7fa708005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71095c79f708e575 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:46.932 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.934+0000 7fa6f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:01:46.932 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.934+0000 7fa6f77fe640 1 -- 172.21.15.129:0/3205028321 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708035040 con 0x5591d2a444d0 2024-05-20T18:01:46.932 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.934+0000 7fa6f77fe640 1 -- 172.21.15.129:0/3205028321 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa7080046c0 con 0x5591d2a444d0 2024-05-20T18:01:46.932 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.934+0000 7fa7257e9740 1 -- 172.21.15.129:0/3205028321 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa70c00aca0 con 0x5591d2a444d0 2024-05-20T18:01:46.935 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.934+0000 7fa6f77fe640 1 -- 172.21.15.129:0/3205028321 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa7080048c0 con 0x5591d2a444d0 2024-05-20T18:01:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.939+0000 7fa7257e9740 10 client.25699 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.939+0000 7fa7257e9740 1 -- 172.21.15.129:0/3205028321 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5591d2b49620 con 0x5591d2a444d0 2024-05-20T18:01:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.941+0000 7fa6f77fe640 1 -- 172.21.15.129:0/3205028321 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa70c00aca0 con 0x5591d2a444d0 2024-05-20T18:01:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.941+0000 7fa6f77fe640 1 -- 172.21.15.129:0/3205028321 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fa7080315b0 con 0x5591d2a444d0 2024-05-20T18:01:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.943+0000 7fa7257e9740 10 client.25699 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.943+0000 7fa7257e9740 10 client.25699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.943+0000 7fa7257e9740 20 client.25699 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.943+0000 7fa7257e9740 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591d2a23d80 0x5591d2e20c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.943+0000 7fa7257e9740 4 client.25699 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:01:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.943+0000 7fa7257e9740 1 -- 172.21.15.129:0/3205028321 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591d2d76a70 con 0x5591d2a23d80 2024-05-20T18:01:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.946+0000 7fa7177fe640 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591d2a23d80 0x5591d2e20c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.954+0000 7fa6f77fe640 10 client.25699 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:46.952 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.954+0000 7fa7177fe640 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591d2a23d80 0x5591d2e20c20 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df1e5a580704dc70 server_cookie=a59c36dccf3d684d in_seq=0 out_seq=0 2024-05-20T18:01:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.958+0000 7fa6f77fe640 1 -- 172.21.15.129:0/3205028321 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5591d2d76a70 con 0x5591d2a23d80 2024-05-20T18:01:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.958+0000 7fa6f77fe640 10 client.25699 handle_command_reply: tid=0 2024-05-20T18:01:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.958+0000 7fa7257e9740 1 client.25699 shutdown 2024-05-20T18:01:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.959+0000 7fa7257e9740 1 -- 172.21.15.129:0/3205028321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 msgr2=0x5591d2e0ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.959+0000 7fa7257e9740 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e0ee80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa70802a9f0 tx=0x7fa708005270 comp rx=0 tx=0).stop 2024-05-20T18:01:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.959+0000 7fa7257e9740 1 -- 172.21.15.129:0/3205028321 shutdown_connections 2024-05-20T18:01:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.959+0000 7fa7257e9740 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591d2a23d80 0x5591d2e20c20 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.959+0000 7fa7257e9740 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a651b0 0x5591d2e14940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.959+0000 7fa7257e9740 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e0ee80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.959+0000 7fa7257e9740 1 --2- 172.21.15.129:0/3205028321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a3c2f0 0x5591d2e0e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.959+0000 7fa6f77fe640 0 client.25699 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.959+0000 7fa7257e9740 1 -- 172.21.15.129:0/3205028321 >> 172.21.15.129:0/3205028321 conn(0x5591d2a59300 msgr2=0x5591d2e05fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:46.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.960+0000 7fa7257e9740 1 -- 172.21.15.129:0/3205028321 shutdown_connections 2024-05-20T18:01:46.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.960+0000 7fa7257e9740 1 -- 172.21.15.129:0/3205028321 wait complete. 2024-05-20T18:01:46.963 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:46.965+0000 7fa7257e9740 20 client.25699 trim_cache size 0 max 16384 2024-05-20T18:01:47.018 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.021+0000 7fa7257e9740 1 Processor -- start 2024-05-20T18:01:47.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.024+0000 7fa7257e9740 1 -- start start 2024-05-20T18:01:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.025+0000 7fa7257e9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a3c2f0 0x5591d2e14840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.025+0000 7fa7257e9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e06640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.025+0000 7fa7257e9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a651b0 0x5591d2e26390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.025+0000 7fa7257e9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa71804f720 con 0x5591d2a3c2f0 2024-05-20T18:01:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.025+0000 7fa7257e9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7181c67a0 con 0x5591d2a651b0 2024-05-20T18:01:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.025+0000 7fa7257e9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa718119070 con 0x5591d2a444d0 2024-05-20T18:01:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.028+0000 7fa717fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e06640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.028+0000 7fa7177fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a3c2f0 0x5591d2e14840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.028+0000 7fa7177fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a3c2f0 0x5591d2e14840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48962/0 (socket says 172.21.15.129:48962) 2024-05-20T18:01:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.028+0000 7fa7177fe640 1 -- 172.21.15.129:0/3483649556 learned_addr learned my addr 172.21.15.129:0/3483649556 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.028+0000 7fa7177fe640 1 --2- 172.21.15.129:0/3483649556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a651b0 0x5591d2e26390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.029+0000 7fa717fff640 1 -- 172.21.15.129:0/3483649556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a651b0 msgr2=0x5591d2e26390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.029+0000 7fa717fff640 1 --2- 172.21.15.129:0/3483649556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a651b0 0x5591d2e26390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.029+0000 7fa717fff640 1 -- 172.21.15.129:0/3483649556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a3c2f0 msgr2=0x5591d2e14840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.029+0000 7fa717fff640 1 --2- 172.21.15.129:0/3483649556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a3c2f0 0x5591d2e14840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.029+0000 7fa717fff640 1 -- 172.21.15.129:0/3483649556 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d2e01420 con 0x5591d2a444d0 2024-05-20T18:01:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.029+0000 7fa717fff640 1 --2- 172.21.15.129:0/3483649556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e06640 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa708005ab0 tx=0x7fa70802a400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9bd601ef8f06143 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.030+0000 7fa6fdffb640 1 -- 172.21.15.129:0/3483649556 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708052070 con 0x5591d2a444d0 2024-05-20T18:01:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.030+0000 7fa6fdffb640 1 -- 172.21.15.129:0/3483649556 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa708040070 con 0x5591d2a444d0 2024-05-20T18:01:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.030+0000 7fa6fdffb640 1 -- 172.21.15.129:0/3483649556 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa708035040 con 0x5591d2a444d0 2024-05-20T18:01:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.037+0000 7fa7257e9740 1 -- 172.21.15.129:0/3483649556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 msgr2=0x5591d2e06640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.037+0000 7fa7257e9740 1 --2- 172.21.15.129:0/3483649556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e06640 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa708005ab0 tx=0x7fa70802a400 comp rx=0 tx=0).stop 2024-05-20T18:01:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.041+0000 7fa7257e9740 1 -- 172.21.15.129:0/3483649556 shutdown_connections 2024-05-20T18:01:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.041+0000 7fa7257e9740 1 --2- 172.21.15.129:0/3483649556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a651b0 0x5591d2e26390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.041+0000 7fa7257e9740 1 --2- 172.21.15.129:0/3483649556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a444d0 0x5591d2e06640 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.041+0000 7fa7257e9740 1 --2- 172.21.15.129:0/3483649556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a3c2f0 0x5591d2e14840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.041+0000 7fa7257e9740 1 -- 172.21.15.129:0/3483649556 >> 172.21.15.129:0/3483649556 conn(0x5591d2a59300 msgr2=0x5591d2e05f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.042+0000 7fa7257e9740 1 -- 172.21.15.129:0/3483649556 shutdown_connections 2024-05-20T18:01:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.042+0000 7fa7257e9740 1 -- 172.21.15.129:0/3483649556 wait complete. 2024-05-20T18:01:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.042+0000 7fa7257e9740 1 -- start start 2024-05-20T18:01:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.043+0000 7fa7257e9740 1 Processor -- start 2024-05-20T18:01:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.044+0000 7fa7257e9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a3c2f0 0x5591d2e1b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.044+0000 7fa7257e9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a444d0 0x5591d2e167e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.044+0000 7fa7177fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a3c2f0 0x5591d2e1b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.044+0000 7fa7177fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a3c2f0 0x5591d2e1b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:59168/0 (socket says 172.21.15.129:59168) 2024-05-20T18:01:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.045+0000 7fa717fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a444d0 0x5591d2e167e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.045+0000 7fa7177fe640 1 -- 172.21.15.129:0/1904232154 learned_addr learned my addr 172.21.15.129:0/1904232154 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:01:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.045+0000 7fa7257e9740 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a651b0 0x5591d2e16dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.045+0000 7fa7257e9740 1 -- 172.21.15.129:0/1904232154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa718119700 con 0x5591d2a444d0 2024-05-20T18:01:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.045+0000 7fa7257e9740 1 -- 172.21.15.129:0/1904232154 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7181c60d0 con 0x5591d2a651b0 2024-05-20T18:01:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.045+0000 7fa7257e9740 1 -- 172.21.15.129:0/1904232154 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa718119dd0 con 0x5591d2a3c2f0 2024-05-20T18:01:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.046+0000 7fa7257e9740 10 client.0 _ll_register_callbacks cb 0x7fa708009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:01:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.046+0000 7fa7177fe640 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a651b0 0x5591d2e16dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.047+0000 7fa7177fe640 1 -- 172.21.15.129:0/1904232154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a3c2f0 msgr2=0x5591d2e1b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.047+0000 7fa7177fe640 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a3c2f0 0x5591d2e1b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.047+0000 7fa7177fe640 1 -- 172.21.15.129:0/1904232154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a444d0 msgr2=0x5591d2e167e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:01:47.046 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.047+0000 7fa7177fe640 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a444d0 0x5591d2e167e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:47.046 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.047+0000 7fa7177fe640 1 -- 172.21.15.129:0/1904232154 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591d2e17520 con 0x5591d2a651b0 2024-05-20T18:01:47.046 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.048+0000 7fa7177fe640 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a651b0 0x5591d2e16dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa70c006010 tx=0x7fa708004490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd4276bb57fba7bc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:01:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.054+0000 7fa6f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:01:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.054+0000 7fa6f77fe640 1 -- 172.21.15.129:0/1904232154 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70c010e40 con 0x5591d2a651b0 2024-05-20T18:01:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.054+0000 7fa6f77fe640 1 -- 172.21.15.129:0/1904232154 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa70c027040 con 0x5591d2a651b0 2024-05-20T18:01:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.055+0000 7fa7257e9740 1 -- 172.21.15.129:0/1904232154 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa708040070 con 0x5591d2a651b0 2024-05-20T18:01:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.055+0000 7fa6f77fe640 1 -- 172.21.15.129:0/1904232154 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa70c0103d0 con 0x5591d2a651b0 2024-05-20T18:01:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.066+0000 7fa7257e9740 10 client.25673 fetch_fsmap learned FSMap version 22 2024-05-20T18:01:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.066+0000 7fa7257e9740 1 -- 172.21.15.129:0/1904232154 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5591d2e88ee0 con 0x5591d2a651b0 2024-05-20T18:01:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.067+0000 7fa6f77fe640 1 -- 172.21.15.129:0/1904232154 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa70c005500 con 0x5591d2a651b0 2024-05-20T18:01:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.067+0000 7fa6f77fe640 1 -- 172.21.15.129:0/1904232154 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fa70c00a890 con 0x5591d2a651b0 2024-05-20T18:01:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.067+0000 7fa7257e9740 10 client.25673 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:01:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.067+0000 7fa7257e9740 10 client.25673 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:01:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.067+0000 7fa7257e9740 20 client.25673 populate_metadata read hostname 'smithi129' 2024-05-20T18:01:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.067+0000 7fa7257e9740 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591d29f91c0 0x5591d2e8b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:01:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.068+0000 7fa7257e9740 4 client.25673 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:01:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.068+0000 7fa7257e9740 1 -- 172.21.15.129:0/1904232154 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fa708040070 con 0x5591d29f91c0 2024-05-20T18:01:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.068+0000 7fa717fff640 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591d29f91c0 0x5591d2e8b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:01:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.069+0000 7fa6f77fe640 10 client.25673 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:47.069 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:47.069+0000 7fa717fff640 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591d29f91c0 0x5591d2e8b360 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77649bb87ec62f8b server_cookie=3de7a86d70277af3 in_seq=0 out_seq=0 2024-05-20T18:01:47.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:47 smithi154 ceph-mon[36482]: pgmap v100: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 920 MiB/s wr, 1.68k op/s 2024-05-20T18:01:47.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:47 smithi129 ceph-mon[32029]: pgmap v100: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 920 MiB/s wr, 1.68k op/s 2024-05-20T18:01:47.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:47 smithi204 ceph-mon[37076]: pgmap v100: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 920 MiB/s wr, 1.68k op/s 2024-05-20T18:01:49.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:49 smithi154 ceph-mon[36482]: pgmap v101: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 801 MiB/s wr, 1.64k op/s 2024-05-20T18:01:49.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:49 smithi129 ceph-mon[32029]: pgmap v101: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 801 MiB/s wr, 1.64k op/s 2024-05-20T18:01:49.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:49 smithi204 ceph-mon[37076]: pgmap v101: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 801 MiB/s wr, 1.64k op/s 2024-05-20T18:01:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.610+0000 7fa6f77fe640 1 -- 172.21.15.129:0/1904232154 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fa708040070 con 0x5591d29f91c0 2024-05-20T18:01:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.610+0000 7fa6f77fe640 10 client.25673 handle_command_reply: tid=0 2024-05-20T18:01:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.610+0000 7fa7257e9740 1 client.25673 shutdown 2024-05-20T18:01:51.616 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.617+0000 7fa7257e9740 1 -- 172.21.15.129:0/1904232154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a651b0 msgr2=0x5591d2e16dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.617+0000 7fa7257e9740 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a651b0 0x5591d2e16dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa70c006010 tx=0x7fa708004490 comp rx=0 tx=0).stop 2024-05-20T18:01:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.618+0000 7fa7257e9740 1 -- 172.21.15.129:0/1904232154 shutdown_connections 2024-05-20T18:01:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.618+0000 7fa7257e9740 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591d29f91c0 0x5591d2e8b360 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.618+0000 7fa7257e9740 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591d2a651b0 0x5591d2e16dc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.618+0000 7fa7257e9740 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591d2a444d0 0x5591d2e167e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.618+0000 7fa7257e9740 1 --2- 172.21.15.129:0/1904232154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591d2a3c2f0 0x5591d2e1b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.618+0000 7fa6f77fe640 0 client.25673 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:01:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.618+0000 7fa7257e9740 1 -- 172.21.15.129:0/1904232154 >> 172.21.15.129:0/1904232154 conn(0x5591d2a59300 msgr2=0x5591d2e04010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.618+0000 7fa7257e9740 1 -- 172.21.15.129:0/1904232154 shutdown_connections 2024-05-20T18:01:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.618+0000 7fa7257e9740 1 -- 172.21.15.129:0/1904232154 wait complete. 2024-05-20T18:01:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.619+0000 7fa7257e9740 20 client.25673 trim_cache size 0 max 16384 2024-05-20T18:01:51.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:01:51.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:01:51.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:01:51.618 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:01:51.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.628+0000 7fa71d6d6640 1 -- 172.21.15.129:0/409951498 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa7000a2e80 msgr2=0x7fa7000a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:51.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.628+0000 7fa71d6d6640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa7000a2e80 0x7fa7000a5390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa7100cc6f0 tx=0x7fa710004b50 comp rx=0 tx=0).stop 2024-05-20T18:01:51.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.628+0000 7fa71d6d6640 1 -- 172.21.15.129:0/409951498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71810f270 msgr2=0x7fa7181acbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:01:51.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.628+0000 7fa71d6d6640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71810f270 0x7fa7181acbd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa71000b970 tx=0x7fa71000be30 comp rx=0 tx=0).stop 2024-05-20T18:01:51.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.634+0000 7fa71d6d6640 1 -- 172.21.15.129:0/409951498 shutdown_connections 2024-05-20T18:01:51.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.634+0000 7fa71d6d6640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa7000a2e80 0x7fa7000a5390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:51.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.634+0000 7fa71d6d6640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa71810f270 0x7fa7181acbd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:51.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.634+0000 7fa71d6d6640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa71810e890 0x7fa7181a8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:51.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.634+0000 7fa71d6d6640 1 --2- 172.21.15.129:0/409951498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa718109090 0x7fa7181a83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:01:51.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.634+0000 7fa71d6d6640 1 -- 172.21.15.129:0/409951498 >> 172.21.15.129:0/409951498 conn(0x7fa71806e690 msgr2=0x7fa71810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:01:51.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.638+0000 7fa71d6d6640 1 -- 172.21.15.129:0/409951498 shutdown_connections 2024-05-20T18:01:51.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:01:51.642+0000 7fa71d6d6640 1 -- 172.21.15.129:0/409951498 wait complete. 2024-05-20T18:01:51.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:51 smithi154 ceph-mon[36482]: pgmap v102: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 903 MiB/s wr, 1.82k op/s 2024-05-20T18:01:51.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:51 smithi129 ceph-mon[32029]: pgmap v102: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 903 MiB/s wr, 1.82k op/s 2024-05-20T18:01:51.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:51 smithi204 ceph-mon[37076]: pgmap v102: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 903 MiB/s wr, 1.82k op/s 2024-05-20T18:01:53.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:53 smithi154 ceph-mon[36482]: pgmap v103: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 788 MiB/s wr, 1.61k op/s 2024-05-20T18:01:53.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:53 smithi129 ceph-mon[32029]: pgmap v103: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 788 MiB/s wr, 1.61k op/s 2024-05-20T18:01:53.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:53 smithi204 ceph-mon[37076]: pgmap v103: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 788 MiB/s wr, 1.61k op/s 2024-05-20T18:01:54.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:54 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:01:54.667 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:54 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:01:54.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:54 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:01:55.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:55 smithi154 ceph-mon[36482]: pgmap v104: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 694 MiB/s wr, 1.45k op/s 2024-05-20T18:01:55.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:55 smithi129 ceph-mon[32029]: pgmap v104: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 694 MiB/s wr, 1.45k op/s 2024-05-20T18:01:55.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:55 smithi204 ceph-mon[37076]: pgmap v104: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 694 MiB/s wr, 1.45k op/s 2024-05-20T18:01:57.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:57 smithi154 ceph-mon[36482]: pgmap v105: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 922 MiB/s wr, 1.99k op/s 2024-05-20T18:01:57.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:57 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:01:57.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:57 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:01:57.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:57 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:01:57.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:57 smithi129 ceph-mon[32029]: pgmap v105: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 922 MiB/s wr, 1.99k op/s 2024-05-20T18:01:57.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:57 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:01:57.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:57 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:01:57.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:57 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:01:57.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:57 smithi204 ceph-mon[37076]: pgmap v105: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 922 MiB/s wr, 1.99k op/s 2024-05-20T18:01:57.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:57 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:01:57.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:57 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:01:57.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:57 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:01:59.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:01:59 smithi154 ceph-mon[36482]: pgmap v106: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 792 MiB/s wr, 1.65k op/s 2024-05-20T18:01:59.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:01:59 smithi129 ceph-mon[32029]: pgmap v106: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 792 MiB/s wr, 1.65k op/s 2024-05-20T18:01:59.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:01:59 smithi204 ceph-mon[37076]: pgmap v106: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 792 MiB/s wr, 1.65k op/s 2024-05-20T18:02:01.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:01 smithi154 ceph-mon[36482]: pgmap v107: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 MiB/s wr, 1.71k op/s 2024-05-20T18:02:01.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:01 smithi129 ceph-mon[32029]: pgmap v107: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 MiB/s wr, 1.71k op/s 2024-05-20T18:02:01.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:01 smithi204 ceph-mon[37076]: pgmap v107: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 MiB/s wr, 1.71k op/s 2024-05-20T18:02:03.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:03 smithi154 ceph-mon[36482]: pgmap v108: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 789 MiB/s wr, 1.52k op/s 2024-05-20T18:02:03.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:03 smithi129 ceph-mon[32029]: pgmap v108: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 789 MiB/s wr, 1.52k op/s 2024-05-20T18:02:03.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:03 smithi204 ceph-mon[37076]: pgmap v108: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 789 MiB/s wr, 1.52k op/s 2024-05-20T18:02:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.002+0000 7f3162f6c640 1 -- 172.21.15.129:0/1480761374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c10dac0 msgr2=0x7f315c10ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.002+0000 7f3162f6c640 1 --2- 172.21.15.129:0/1480761374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c10dac0 0x7f315c10ded0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3154009ca0 tx=0x7f315402a8f0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.006+0000 7f3162f6c640 1 -- 172.21.15.129:0/1480761374 shutdown_connections 2024-05-20T18:02:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.006+0000 7f3162f6c640 1 --2- 172.21.15.129:0/1480761374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f315c10e410 0x7f315c113d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.006+0000 7f3162f6c640 1 --2- 172.21.15.129:0/1480761374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315c1142a0 0x7f315c1166c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.006+0000 7f3162f6c640 1 --2- 172.21.15.129:0/1480761374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c10dac0 0x7f315c10ded0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.006+0000 7f3162f6c640 1 -- 172.21.15.129:0/1480761374 >> 172.21.15.129:0/1480761374 conn(0x7f315c06ea20 msgr2=0x7f315c06ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:05.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.009+0000 7f3162f6c640 1 -- 172.21.15.129:0/1480761374 shutdown_connections 2024-05-20T18:02:05.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.010+0000 7f3162f6c640 1 -- 172.21.15.129:0/1480761374 wait complete. 2024-05-20T18:02:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.012+0000 7f3162f6c640 1 Processor -- start 2024-05-20T18:02:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.012+0000 7f3162f6c640 1 -- start start 2024-05-20T18:02:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.013+0000 7f3162f6c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c10dac0 0x7f315c1a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.013+0000 7f3162f6c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f315c10e410 0x7f315c1a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.013+0000 7f3162f6c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315c1142a0 0x7f315c1a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.013+0000 7f3162f6c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f315c11a400 con 0x7f315c10dac0 2024-05-20T18:02:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.013+0000 7f3162f6c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f315c11a280 con 0x7f315c10e410 2024-05-20T18:02:05.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.013+0000 7f3162f6c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f315c11a580 con 0x7f315c1142a0 2024-05-20T18:02:05.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.015+0000 7f316276b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315c1142a0 0x7f315c1a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.015+0000 7f316276b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315c1142a0 0x7f315c1a5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:33958/0 (socket says 172.21.15.129:33958) 2024-05-20T18:02:05.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.015+0000 7f316276b640 1 -- 172.21.15.129:0/3926489454 learned_addr learned my addr 172.21.15.129:0/3926489454 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:05.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.016+0000 7f3161769640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f315c10e410 0x7f315c1a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.016+0000 7f3161f6a640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c10dac0 0x7f315c1a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.019+0000 7f3161f6a640 1 -- 172.21.15.129:0/3926489454 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315c1142a0 msgr2=0x7f315c1a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:05.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.019+0000 7f3161f6a640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315c1142a0 0x7f315c1a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.019+0000 7f3161f6a640 1 -- 172.21.15.129:0/3926489454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f315c10e410 msgr2=0x7f315c1a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:05.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.019+0000 7f3161f6a640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f315c10e410 0x7f315c1a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.019+0000 7f3161f6a640 1 -- 172.21.15.129:0/3926489454 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c1a5b80 con 0x7f315c10dac0 2024-05-20T18:02:05.018 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.020+0000 7f3161f6a640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c10dac0 0x7f315c1a0d20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3154009ca0 tx=0x7f3154003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.021+0000 7f3152ffd640 1 -- 172.21.15.129:0/3926489454 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154051070 con 0x7f315c10dac0 2024-05-20T18:02:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.021+0000 7f3162f6c640 1 -- 172.21.15.129:0/3926489454 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315c1a5e10 con 0x7f315c10dac0 2024-05-20T18:02:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.021+0000 7f3162f6c640 1 -- 172.21.15.129:0/3926489454 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f315c1a6360 con 0x7f315c10dac0 2024-05-20T18:02:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.025+0000 7f316b07a740 1 Processor -- start 2024-05-20T18:02:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.026+0000 7f3152ffd640 1 -- 172.21.15.129:0/3926489454 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315402d070 con 0x7f315c10dac0 2024-05-20T18:02:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.026+0000 7f3152ffd640 1 -- 172.21.15.129:0/3926489454 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154004860 con 0x7f315c10dac0 2024-05-20T18:02:05.028 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.027+0000 7f316b07a740 1 -- start start 2024-05-20T18:02:05.028 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.027+0000 7f316b07a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28851b0 0x559dd2c25a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.028 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.027+0000 7f316b07a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd285c2f0 0x559dd2c240f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.028 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.027+0000 7f316b07a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28644d0 0x559dd2c24630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.028 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.027+0000 7f316b07a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559dd2b89340 con 0x559dd28644d0 2024-05-20T18:02:05.028 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.027+0000 7f316b07a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559dd2b31290 con 0x559dd28851b0 2024-05-20T18:02:05.028 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.027+0000 7f316b07a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x559dd2b85300 con 0x559dd285c2f0 2024-05-20T18:02:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.030+0000 7f316276b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28851b0 0x559dd2c25a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.030+0000 7f3161769640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28644d0 0x559dd2c24630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.030+0000 7f3161769640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28644d0 0x559dd2c24630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44316/0 (socket says 172.21.15.129:44316) 2024-05-20T18:02:05.029 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.030+0000 7f3161769640 1 -- 172.21.15.129:0/3254924318 learned_addr learned my addr 172.21.15.129:0/3254924318 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:05.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.033+0000 7f3152ffd640 1 -- 172.21.15.129:0/3926489454 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f3154004040 con 0x7f315c10dac0 2024-05-20T18:02:05.033 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.035+0000 7f3152ffd640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31480a2f00 0x7f31480a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.036+0000 7f3152ffd640 1 -- 172.21.15.129:0/3926489454 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f31540f8240 con 0x7f315c10dac0 2024-05-20T18:02:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.036+0000 7f3161769640 1 -- 172.21.15.129:0/3254924318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd285c2f0 msgr2=0x559dd2c240f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.036+0000 7f3161f6a640 1 --2- 172.21.15.129:0/3254924318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd285c2f0 0x559dd2c240f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.036+0000 7f3161769640 1 --2- 172.21.15.129:0/3254924318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd285c2f0 0x559dd2c240f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.036+0000 7f3161769640 1 -- 172.21.15.129:0/3254924318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28851b0 msgr2=0x559dd2c25a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.036+0000 7f3161769640 1 --2- 172.21.15.129:0/3254924318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28851b0 0x559dd2c25a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.036+0000 7f3161769640 1 -- 172.21.15.129:0/3254924318 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dd2b558e0 con 0x559dd28644d0 2024-05-20T18:02:05.038 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.038+0000 7f3161769640 1 --2- 172.21.15.129:0/3254924318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28644d0 0x559dd2c24630 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f314c00ecd0 tx=0x7f314c00e6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47c2b3ad606c871 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.038+0000 7f316276b640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31480a2f00 0x7f31480a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.040+0000 7f3162f6c640 1 -- 172.21.15.129:0/3254924318 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c01c070 con 0x559dd28644d0 2024-05-20T18:02:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.040+0000 7f3162f6c640 1 -- 172.21.15.129:0/3254924318 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f314c017b60 con 0x559dd28644d0 2024-05-20T18:02:05.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.040+0000 7f3162f6c640 1 -- 172.21.15.129:0/3254924318 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c016b50 con 0x559dd28644d0 2024-05-20T18:02:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.049+0000 7f316b07a740 1 -- 172.21.15.129:0/3254924318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28644d0 msgr2=0x559dd2c24630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:05.047 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.049+0000 7f316b07a740 1 --2- 172.21.15.129:0/3254924318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28644d0 0x559dd2c24630 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f314c00ecd0 tx=0x7f314c00e6e0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.055+0000 7f316b07a740 1 -- 172.21.15.129:0/3254924318 shutdown_connections 2024-05-20T18:02:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.055+0000 7f316b07a740 1 --2- 172.21.15.129:0/3254924318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28644d0 0x559dd2c24630 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.055+0000 7f316b07a740 1 --2- 172.21.15.129:0/3254924318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd285c2f0 0x559dd2c240f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.055+0000 7f316b07a740 1 --2- 172.21.15.129:0/3254924318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28851b0 0x559dd2c25a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.055+0000 7f316b07a740 1 -- 172.21.15.129:0/3254924318 >> 172.21.15.129:0/3254924318 conn(0x559dd2879300 msgr2=0x559dd2c20d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.055+0000 7f316276b640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31480a2f00 0x7f31480a5410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x559dd2b4ad20 tx=0x7f3158004750 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:05.054 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.056+0000 7f316b07a740 1 -- 172.21.15.129:0/3254924318 shutdown_connections 2024-05-20T18:02:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.057+0000 7f316b07a740 1 -- 172.21.15.129:0/3254924318 wait complete. 2024-05-20T18:02:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.058+0000 7f316b07a740 1 -- start start 2024-05-20T18:02:05.056 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.058+0000 7f316b07a740 1 Processor -- start 2024-05-20T18:02:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.059+0000 7f316b07a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd285c2f0 0x559dd2c305b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.059+0000 7f316b07a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28644d0 0x559dd2c30af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.059+0000 7f316b07a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28851b0 0x559dd2c2ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.059+0000 7f316b07a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559dd2c313e0 con 0x559dd28851b0 2024-05-20T18:02:05.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.059+0000 7f316b07a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559dd2b44790 con 0x559dd28644d0 2024-05-20T18:02:05.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.059+0000 7f316b07a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x559dd2c31560 con 0x559dd285c2f0 2024-05-20T18:02:05.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.059+0000 7f316b07a740 10 client.0 _ll_register_callbacks cb 0x559dd2b7fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.062+0000 7f316276b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28851b0 0x559dd2c2ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.062+0000 7f316276b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28851b0 0x559dd2c2ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44324/0 (socket says 172.21.15.129:44324) 2024-05-20T18:02:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.062+0000 7f316276b640 1 -- 172.21.15.129:0/4247714528 learned_addr learned my addr 172.21.15.129:0/4247714528 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:05.060 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.062+0000 7f3161f6a640 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd285c2f0 0x559dd2c305b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.063+0000 7f316276b640 1 -- 172.21.15.129:0/4247714528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd285c2f0 msgr2=0x559dd2c305b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.063+0000 7f316276b640 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd285c2f0 0x559dd2c305b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.063+0000 7f316276b640 1 -- 172.21.15.129:0/4247714528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28644d0 msgr2=0x559dd2c30af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.063+0000 7f316276b640 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28644d0 0x559dd2c30af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.063+0000 7f316276b640 1 -- 172.21.15.129:0/4247714528 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dd289d210 con 0x559dd28851b0 2024-05-20T18:02:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.064+0000 7f316276b640 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28851b0 0x559dd2c2ea70 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f315800fae0 tx=0x7f3158008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83939cac11d9a03a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:05.062 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.064+0000 7f31457fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:02:05.063 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.065+0000 7f31457fa640 1 -- 172.21.15.129:0/4247714528 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315800ee10 con 0x559dd28851b0 2024-05-20T18:02:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.066+0000 7f316b07a740 1 -- 172.21.15.129:0/4247714528 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f314c017b60 con 0x559dd28851b0 2024-05-20T18:02:05.064 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.066+0000 7f31457fa640 1 -- 172.21.15.129:0/4247714528 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f315801d420 con 0x559dd28851b0 2024-05-20T18:02:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.072+0000 7f31457fa640 1 -- 172.21.15.129:0/4247714528 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f315801d620 con 0x559dd28851b0 2024-05-20T18:02:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.076+0000 7f316b07a740 10 client.15987 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.076+0000 7f316b07a740 1 -- 172.21.15.129:0/4247714528 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x559dd2969620 con 0x559dd28851b0 2024-05-20T18:02:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.077+0000 7f31457fa640 1 -- 172.21.15.129:0/4247714528 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f314c017b60 con 0x559dd28851b0 2024-05-20T18:02:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.078+0000 7f31457fa640 1 -- 172.21.15.129:0/4247714528 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f3158027a30 con 0x559dd28851b0 2024-05-20T18:02:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.078+0000 7f316b07a740 10 client.15987 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.078+0000 7f316b07a740 10 client.15987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.078+0000 7f316b07a740 20 client.15987 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.078+0000 7f316b07a740 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559dd2843d80 0x559dd2c521e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.078+0000 7f316b07a740 4 client.15987 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.078+0000 7f316b07a740 1 -- 172.21.15.129:0/4247714528 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559dd2b96a70 con 0x559dd2843d80 2024-05-20T18:02:05.078 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.080+0000 7f3161f6a640 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559dd2843d80 0x559dd2c521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.085+0000 7f3161f6a640 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559dd2843d80 0x559dd2c521e0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a24cda583e0d776 server_cookie=257e203a60f3a885 in_seq=0 out_seq=0 2024-05-20T18:02:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.086+0000 7f31457fa640 10 client.15987 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.088+0000 7f31457fa640 1 -- 172.21.15.129:0/4247714528 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559dd2b96a70 con 0x559dd2843d80 2024-05-20T18:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.088+0000 7f31457fa640 10 client.15987 handle_command_reply: tid=0 2024-05-20T18:02:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.088+0000 7f316b07a740 1 client.15987 shutdown 2024-05-20T18:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.089+0000 7f316b07a740 1 -- 172.21.15.129:0/4247714528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28851b0 msgr2=0x559dd2c2ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:05.087 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.089+0000 7f316b07a740 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28851b0 0x559dd2c2ea70 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f315800fae0 tx=0x7f3158008040 comp rx=0 tx=0).stop 2024-05-20T18:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.091+0000 7f316b07a740 1 -- 172.21.15.129:0/4247714528 shutdown_connections 2024-05-20T18:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.091+0000 7f316b07a740 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559dd2843d80 0x559dd2c521e0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.091+0000 7f316b07a740 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28644d0 0x559dd2c30af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.091+0000 7f316b07a740 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd28851b0 0x559dd2c2ea70 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.091+0000 7f316b07a740 1 --2- 172.21.15.129:0/4247714528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd285c2f0 0x559dd2c305b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.090 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.091+0000 7f316b07a740 1 -- 172.21.15.129:0/4247714528 >> 172.21.15.129:0/4247714528 conn(0x559dd2879300 msgr2=0x559dd2c26bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:05.090 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.091+0000 7f31457fa640 0 client.15987 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:05.092 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.094+0000 7f316b07a740 1 -- 172.21.15.129:0/4247714528 shutdown_connections 2024-05-20T18:02:05.096 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.094+0000 7f316b07a740 1 -- 172.21.15.129:0/4247714528 wait complete. 2024-05-20T18:02:05.096 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.097+0000 7f316b07a740 20 client.15987 trim_cache size 0 max 16384 2024-05-20T18:02:05.140 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.141+0000 7f316b07a740 1 Processor -- start 2024-05-20T18:02:05.140 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.141+0000 7f316b07a740 1 -- start start 2024-05-20T18:02:05.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.142+0000 7f316b07a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c2d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.142+0000 7f316b07a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd28644d0 0x559dd2c2d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.142+0000 7f316b07a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28851b0 0x559dd2c579e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.142+0000 7f316b07a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f314000c8f0 con 0x559dd285c2f0 2024-05-20T18:02:05.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.142+0000 7f316b07a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f314000f170 con 0x559dd28851b0 2024-05-20T18:02:05.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.142+0000 7f316b07a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f314000cf60 con 0x559dd28644d0 2024-05-20T18:02:05.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.142+0000 7f3161f6a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c2d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.143+0000 7f3161f6a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c2d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44336/0 (socket says 172.21.15.129:44336) 2024-05-20T18:02:05.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.143+0000 7f3161f6a640 1 -- 172.21.15.129:0/2439452144 learned_addr learned my addr 172.21.15.129:0/2439452144 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:05.142 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.143+0000 7f3161f6a640 1 -- 172.21.15.129:0/2439452144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd28644d0 msgr2=0x559dd2c2d880 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:02:05.142 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.143+0000 7f3161f6a640 1 --2- 172.21.15.129:0/2439452144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd28644d0 0x559dd2c2d880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.142 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.143+0000 7f3161f6a640 1 -- 172.21.15.129:0/2439452144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28851b0 msgr2=0x559dd2c579e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:02:05.142 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.143+0000 7f3161f6a640 1 --2- 172.21.15.129:0/2439452144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28851b0 0x559dd2c579e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.142 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.143+0000 7f3161f6a640 1 -- 172.21.15.129:0/2439452144 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dd2c24f90 con 0x559dd285c2f0 2024-05-20T18:02:05.142 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.144+0000 7f3161f6a640 1 --2- 172.21.15.129:0/2439452144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c2d340 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f31540c4730 tx=0x7f3154043040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb159cf84395a87d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:05.143 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.145+0000 7f31477fe640 1 -- 172.21.15.129:0/2439452144 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315402f340 con 0x559dd285c2f0 2024-05-20T18:02:05.143 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.145+0000 7f31477fe640 1 -- 172.21.15.129:0/2439452144 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31540dc030 con 0x559dd285c2f0 2024-05-20T18:02:05.143 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.145+0000 7f31477fe640 1 -- 172.21.15.129:0/2439452144 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154004330 con 0x559dd285c2f0 2024-05-20T18:02:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.146+0000 7f316b07a740 1 -- 172.21.15.129:0/2439452144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 msgr2=0x559dd2c2d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.146+0000 7f316b07a740 1 --2- 172.21.15.129:0/2439452144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c2d340 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f31540c4730 tx=0x7f3154043040 comp rx=0 tx=0).stop 2024-05-20T18:02:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.146+0000 7f316b07a740 1 -- 172.21.15.129:0/2439452144 shutdown_connections 2024-05-20T18:02:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.146+0000 7f316b07a740 1 --2- 172.21.15.129:0/2439452144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28851b0 0x559dd2c579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.146+0000 7f316b07a740 1 --2- 172.21.15.129:0/2439452144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd28644d0 0x559dd2c2d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.147+0000 7f316b07a740 1 --2- 172.21.15.129:0/2439452144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c2d340 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.147+0000 7f316b07a740 1 -- 172.21.15.129:0/2439452144 >> 172.21.15.129:0/2439452144 conn(0x559dd2879300 msgr2=0x559dd2c29c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.149+0000 7f316b07a740 1 -- 172.21.15.129:0/2439452144 shutdown_connections 2024-05-20T18:02:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.149+0000 7f316b07a740 1 -- 172.21.15.129:0/2439452144 wait complete. 2024-05-20T18:02:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.150+0000 7f316b07a740 1 -- start start 2024-05-20T18:02:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.151+0000 7f316b07a740 1 Processor -- start 2024-05-20T18:02:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.152+0000 7f316b07a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c33be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.152+0000 7f316b07a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28644d0 0x559dd2c34120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.152+0000 7f3161f6a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c33be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.152+0000 7f316b07a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd28851b0 0x559dd2cb2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.152+0000 7f3161f6a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c33be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44346/0 (socket says 172.21.15.129:44346) 2024-05-20T18:02:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.152+0000 7f3161f6a640 1 -- 172.21.15.129:0/4203156652 learned_addr learned my addr 172.21.15.129:0/4203156652 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.152+0000 7f316b07a740 1 -- 172.21.15.129:0/4203156652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f314000d630 con 0x559dd285c2f0 2024-05-20T18:02:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.152+0000 7f316b07a740 1 -- 172.21.15.129:0/4203156652 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f314000eaa0 con 0x559dd28644d0 2024-05-20T18:02:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.152+0000 7f316b07a740 1 -- 172.21.15.129:0/4203156652 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f314000dd00 con 0x559dd28851b0 2024-05-20T18:02:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.153+0000 7f316b07a740 10 client.0 _ll_register_callbacks cb 0x7f315800b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.153+0000 7f3161769640 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28644d0 0x559dd2c34120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.153+0000 7f3161f6a640 1 -- 172.21.15.129:0/4203156652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd28851b0 msgr2=0x559dd2cb2560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:02:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.153+0000 7f3161f6a640 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd28851b0 0x559dd2cb2560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.153+0000 7f3161f6a640 1 -- 172.21.15.129:0/4203156652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28644d0 msgr2=0x559dd2c34120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.153+0000 7f3161f6a640 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28644d0 0x559dd2c34120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.153+0000 7f3161f6a640 1 -- 172.21.15.129:0/4203156652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dd2cb2aa0 con 0x559dd285c2f0 2024-05-20T18:02:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.153+0000 7f3161769640 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28644d0 0x559dd2c34120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:02:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.153+0000 7f3161f6a640 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c33be0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f31540cce30 tx=0x7f31540f5990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8259200f495815e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.153+0000 7f31457fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:02:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.155+0000 7f31457fa640 1 -- 172.21.15.129:0/4203156652 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31540c47a0 con 0x559dd285c2f0 2024-05-20T18:02:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.155+0000 7f31457fa640 1 -- 172.21.15.129:0/4203156652 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f31540c0e10 con 0x559dd285c2f0 2024-05-20T18:02:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.155+0000 7f316b07a740 1 -- 172.21.15.129:0/4203156652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31540dc030 con 0x559dd285c2f0 2024-05-20T18:02:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.155+0000 7f31457fa640 1 -- 172.21.15.129:0/4203156652 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f31540c8aa0 con 0x559dd285c2f0 2024-05-20T18:02:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.155+0000 7f316b07a740 10 client.15993 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.155+0000 7f316b07a740 1 -- 172.21.15.129:0/4203156652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x559dd2cb9d80 con 0x559dd285c2f0 2024-05-20T18:02:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.157+0000 7f31457fa640 1 -- 172.21.15.129:0/4203156652 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31540dc030 con 0x559dd285c2f0 2024-05-20T18:02:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.157+0000 7f31457fa640 1 -- 172.21.15.129:0/4203156652 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f3154031ce0 con 0x559dd285c2f0 2024-05-20T18:02:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.157+0000 7f316b07a740 10 client.15993 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.157+0000 7f316b07a740 10 client.15993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.157+0000 7f316b07a740 20 client.15993 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.157+0000 7f316b07a740 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559dd28191c0 0x559dd2cbc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.157+0000 7f316b07a740 4 client.15993 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:02:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.157+0000 7f316b07a740 1 -- 172.21.15.129:0/4203156652 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559dd2b90dc0 con 0x559dd28191c0 2024-05-20T18:02:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.159+0000 7f3161769640 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559dd28191c0 0x559dd2cbc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:05.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.161+0000 7f31457fa640 10 client.15993 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:05.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.161+0000 7f3161769640 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559dd28191c0 0x559dd2cbc490 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=492ae61b9a27e819 server_cookie=c125b21e1f10bdd in_seq=0 out_seq=0 2024-05-20T18:02:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.231+0000 7f31457fa640 1 -- 172.21.15.129:0/4203156652 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559dd2b90dc0 con 0x559dd28191c0 2024-05-20T18:02:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.231+0000 7f31457fa640 10 client.15993 handle_command_reply: tid=0 2024-05-20T18:02:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.231+0000 7f316b07a740 1 client.15993 shutdown 2024-05-20T18:02:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.232+0000 7f316b07a740 1 -- 172.21.15.129:0/4203156652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 msgr2=0x559dd2c33be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.232+0000 7f316b07a740 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c33be0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f31540cce30 tx=0x7f31540f5990 comp rx=0 tx=0).stop 2024-05-20T18:02:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.233+0000 7f316b07a740 1 -- 172.21.15.129:0/4203156652 shutdown_connections 2024-05-20T18:02:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.233+0000 7f316b07a740 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559dd28191c0 0x559dd2cbc490 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.233+0000 7f316b07a740 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559dd28644d0 0x559dd2c34120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.233+0000 7f316b07a740 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559dd28851b0 0x559dd2cb2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.233+0000 7f316b07a740 1 --2- 172.21.15.129:0/4203156652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559dd285c2f0 0x559dd2c33be0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.233+0000 7f316b07a740 1 -- 172.21.15.129:0/4203156652 >> 172.21.15.129:0/4203156652 conn(0x559dd2879300 msgr2=0x559dd2c29c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.233+0000 7f31457fa640 0 client.15993 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.233+0000 7f316b07a740 1 -- 172.21.15.129:0/4203156652 shutdown_connections 2024-05-20T18:02:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.233+0000 7f316b07a740 1 -- 172.21.15.129:0/4203156652 wait complete. 2024-05-20T18:02:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.234+0000 7f316b07a740 20 client.15993 trim_cache size 0 max 16384 2024-05-20T18:02:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:02:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:02:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:02:05.233 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:02:05.233 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 -- 172.21.15.129:0/3926489454 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31480a2f00 msgr2=0x7f31480a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31480a2f00 0x7f31480a5410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x559dd2b4ad20 tx=0x7f3158004750 comp rx=0 tx=0).stop 2024-05-20T18:02:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 -- 172.21.15.129:0/3926489454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c10dac0 msgr2=0x7f315c1a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c10dac0 0x7f315c1a0d20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3154009ca0 tx=0x7f3154003f10 comp rx=0 tx=0).stop 2024-05-20T18:02:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 -- 172.21.15.129:0/3926489454 shutdown_connections 2024-05-20T18:02:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f31480a2f00 0x7f31480a5410 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f315c10e410 0x7f315c1a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f315c1142a0 0x7f315c1a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 --2- 172.21.15.129:0/3926489454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c10dac0 0x7f315c1a0d20 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 -- 172.21.15.129:0/3926489454 >> 172.21.15.129:0/3926489454 conn(0x7f315c06ea20 msgr2=0x7f315c115590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 -- 172.21.15.129:0/3926489454 shutdown_connections 2024-05-20T18:02:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:05.235+0000 7f3150ff9640 1 -- 172.21.15.129:0/3926489454 wait complete. 2024-05-20T18:02:05.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:05 smithi154 ceph-mon[36482]: pgmap v109: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 728 MiB/s wr, 1.45k op/s 2024-05-20T18:02:05.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:05 smithi129 ceph-mon[32029]: pgmap v109: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 728 MiB/s wr, 1.45k op/s 2024-05-20T18:02:05.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:05 smithi204 ceph-mon[37076]: pgmap v109: 97 pgs: 97 active+clean; 20 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 728 MiB/s wr, 1.45k op/s 2024-05-20T18:02:07.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:07 smithi154 ceph-mon[36482]: pgmap v110: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 930 MiB/s wr, 2.00k op/s 2024-05-20T18:02:07.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:07 smithi129 ceph-mon[32029]: pgmap v110: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 930 MiB/s wr, 2.00k op/s 2024-05-20T18:02:07.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:07 smithi204 ceph-mon[37076]: pgmap v110: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 930 MiB/s wr, 2.00k op/s 2024-05-20T18:02:09.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:09 smithi154 ceph-mon[36482]: pgmap v111: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 813 MiB/s wr, 1.98k op/s 2024-05-20T18:02:09.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:09 smithi129 ceph-mon[32029]: pgmap v111: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 813 MiB/s wr, 1.98k op/s 2024-05-20T18:02:09.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:09 smithi204 ceph-mon[37076]: pgmap v111: 97 pgs: 97 active+clean; 20 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 813 MiB/s wr, 1.98k op/s 2024-05-20T18:02:10.150 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:02:10.150 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:02:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.503+0000 7faa7bfff640 1 -- 172.21.15.129:0/1089005496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa7c109090 msgr2=0x7faa7c109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.503+0000 7faa7bfff640 1 --2- 172.21.15.129:0/1089005496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa7c109090 0x7faa7c109520 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009900 tx=0x7faa6c02a440 comp rx=0 tx=0).stop 2024-05-20T18:02:10.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.504+0000 7faa7bfff640 1 -- 172.21.15.129:0/1089005496 shutdown_connections 2024-05-20T18:02:10.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.504+0000 7faa7bfff640 1 --2- 172.21.15.129:0/1089005496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c10f270 0x7faa7c108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.504+0000 7faa7bfff640 1 --2- 172.21.15.129:0/1089005496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa7c109090 0x7faa7c109520 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.504+0000 7faa7bfff640 1 --2- 172.21.15.129:0/1089005496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa7c10e890 0x7faa7c10eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.504+0000 7faa7bfff640 1 -- 172.21.15.129:0/1089005496 >> 172.21.15.129:0/1089005496 conn(0x7faa7c06d580 msgr2=0x7faa7c06d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:10.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.507+0000 7faa7bfff640 1 -- 172.21.15.129:0/1089005496 shutdown_connections 2024-05-20T18:02:10.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.510+0000 7faa7bfff640 1 -- 172.21.15.129:0/1089005496 wait complete. 2024-05-20T18:02:10.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.511+0000 7faa7bfff640 1 Processor -- start 2024-05-20T18:02:10.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.512+0000 7faa7bfff640 1 -- start start 2024-05-20T18:02:10.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.512+0000 7faa7bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c109090 0x7faa7c1a8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.512+0000 7faa7bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa7c10e890 0x7faa7c1a8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.513+0000 7faa7bfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa7c10f270 0x7faa7c1acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.513+0000 7faa7bfff640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c121b70 con 0x7faa7c10e890 2024-05-20T18:02:10.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.513+0000 7faa7bfff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c1219f0 con 0x7faa7c109090 2024-05-20T18:02:10.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.513+0000 7faa7bfff640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c121cf0 con 0x7faa7c10f270 2024-05-20T18:02:10.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.514+0000 7faa7affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c109090 0x7faa7c1a8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.514+0000 7faa7affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c109090 0x7faa7c1a8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:56796/0 (socket says 172.21.15.129:56796) 2024-05-20T18:02:10.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.514+0000 7faa7affd640 1 -- 172.21.15.129:0/308400212 learned_addr learned my addr 172.21.15.129:0/308400212 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:10.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.516+0000 7faa7a7fc640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa7c10e890 0x7faa7c1a8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.516+0000 7faa7affd640 1 -- 172.21.15.129:0/308400212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa7c10f270 msgr2=0x7faa7c1acd50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:02:10.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.516+0000 7faa7affd640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa7c10f270 0x7faa7c1acd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.516+0000 7faa7affd640 1 -- 172.21.15.129:0/308400212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa7c10e890 msgr2=0x7faa7c1a8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:10.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.516+0000 7faa7affd640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa7c10e890 0x7faa7c1a8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.516+0000 7faa7affd640 1 -- 172.21.15.129:0/308400212 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7c1ad390 con 0x7faa7c109090 2024-05-20T18:02:10.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.516+0000 7faa7a7fc640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa7c10e890 0x7faa7c1a8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:02:10.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.518+0000 7faa7affd640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c109090 0x7faa7c1a8530 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7faa7400ba50 tx=0x7faa7400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:10.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.527+0000 7faa80a3b640 1 -- 172.21.15.129:0/308400212 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74019070 con 0x7faa7c109090 2024-05-20T18:02:10.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.527+0000 7faa7bfff640 1 -- 172.21.15.129:0/308400212 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa7c1ad680 con 0x7faa7c109090 2024-05-20T18:02:10.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.527+0000 7faa7bfff640 1 -- 172.21.15.129:0/308400212 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa7c1b5000 con 0x7faa7c109090 2024-05-20T18:02:10.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.527+0000 7faa80a3b640 1 -- 172.21.15.129:0/308400212 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa7400cc10 con 0x7faa7c109090 2024-05-20T18:02:10.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.527+0000 7faa80a3b640 1 -- 172.21.15.129:0/308400212 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa740076a0 con 0x7faa7c109090 2024-05-20T18:02:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.534+0000 7faa889c2740 1 Processor -- start 2024-05-20T18:02:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.534+0000 7faa889c2740 1 -- start start 2024-05-20T18:02:10.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.535+0000 7faa889c2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be3e390 0x56412c1dcf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.535+0000 7faa889c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 0x56412c1dd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.535+0000 7faa889c2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 0x56412c1dd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.535+0000 7faa889c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56412c142d00 con 0x56412be154d0 2024-05-20T18:02:10.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.535+0000 7faa889c2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56412c0ea4c0 con 0x56412be3e390 2024-05-20T18:02:10.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.535+0000 7faa889c2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56412c13ecc0 con 0x56412be1d6b0 2024-05-20T18:02:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.537+0000 7faa7a7fc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 0x56412c1dd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.537+0000 7faa7b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be3e390 0x56412c1dcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.537+0000 7faa7a7fc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 0x56412c1dd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:57864/0 (socket says 172.21.15.129:57864) 2024-05-20T18:02:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.537+0000 7faa7a7fc640 1 -- 172.21.15.129:0/3108435958 learned_addr learned my addr 172.21.15.129:0/3108435958 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:10.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.542+0000 7faa7b7fe640 1 -- 172.21.15.129:0/3108435958 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 msgr2=0x56412c1dd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:10.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.543+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/3108435958 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 0x56412c1dd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.543+0000 7faa7b7fe640 1 -- 172.21.15.129:0/3108435958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 msgr2=0x56412c1dd470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:02:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.544+0000 7faa80a3b640 1 -- 172.21.15.129:0/308400212 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7faa7401c070 con 0x7faa7c109090 2024-05-20T18:02:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.545+0000 7faa7affd640 1 --2- 172.21.15.129:0/3108435958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 0x56412c1dd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.545+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/3108435958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 0x56412c1dd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.545+0000 7faa7b7fe640 1 -- 172.21.15.129:0/3108435958 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56412bf207d0 con 0x56412be3e390 2024-05-20T18:02:10.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.545+0000 7faa7affd640 1 --2- 172.21.15.129:0/3108435958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 0x56412c1dd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:02:10.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.547+0000 7faa80a3b640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7faa640a2ec0 0x7faa640a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.547+0000 7faa80a3b640 1 -- 172.21.15.129:0/308400212 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7faa740ce8c0 con 0x7faa7c109090 2024-05-20T18:02:10.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.548+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/3108435958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be3e390 0x56412c1dcf30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faa6c02a960 tx=0x7faa6c009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b11941af1df5048 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:10.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.550+0000 7faa7affd640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7faa640a2ec0 0x7faa640a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.552+0000 7faa61ffb640 1 -- 172.21.15.129:0/3108435958 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c03e070 con 0x56412be3e390 2024-05-20T18:02:10.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.552+0000 7faa61ffb640 1 -- 172.21.15.129:0/3108435958 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa6c005530 con 0x56412be3e390 2024-05-20T18:02:10.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.552+0000 7faa61ffb640 1 -- 172.21.15.129:0/3108435958 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c035040 con 0x56412be3e390 2024-05-20T18:02:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.554+0000 7faa889c2740 1 -- 172.21.15.129:0/3108435958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be3e390 msgr2=0x56412c1dcf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:10.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.554+0000 7faa889c2740 1 --2- 172.21.15.129:0/3108435958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be3e390 0x56412c1dcf30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faa6c02a960 tx=0x7faa6c009670 comp rx=0 tx=0).stop 2024-05-20T18:02:10.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.556+0000 7faa7affd640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7faa640a2ec0 0x7faa640a53d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x56412c1daa20 tx=0x7faa7400c650 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:10.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.557+0000 7faa889c2740 1 -- 172.21.15.129:0/3108435958 shutdown_connections 2024-05-20T18:02:10.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.557+0000 7faa889c2740 1 --2- 172.21.15.129:0/3108435958 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 0x56412c1dd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.557+0000 7faa889c2740 1 --2- 172.21.15.129:0/3108435958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 0x56412c1dd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.557+0000 7faa889c2740 1 --2- 172.21.15.129:0/3108435958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be3e390 0x56412c1dcf30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.557+0000 7faa889c2740 1 -- 172.21.15.129:0/3108435958 >> 172.21.15.129:0/3108435958 conn(0x56412be324e0 msgr2=0x56412c1d96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:10.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.557+0000 7faa889c2740 1 -- 172.21.15.129:0/3108435958 shutdown_connections 2024-05-20T18:02:10.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.557+0000 7faa889c2740 1 -- 172.21.15.129:0/3108435958 wait complete. 2024-05-20T18:02:10.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.558+0000 7faa889c2740 1 -- start start 2024-05-20T18:02:10.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.558+0000 7faa889c2740 1 Processor -- start 2024-05-20T18:02:10.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.559+0000 7faa889c2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be154d0 0x56412c1e8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.559+0000 7faa889c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be1d6b0 0x56412c1e94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.559+0000 7faa889c2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be3e390 0x56412c1e7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.559+0000 7faa889c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56412c1ea220 con 0x56412be1d6b0 2024-05-20T18:02:10.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.559+0000 7faa889c2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56412c0fdb90 con 0x56412be3e390 2024-05-20T18:02:10.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.559+0000 7faa889c2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56412c1ea3a0 con 0x56412be154d0 2024-05-20T18:02:10.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.560+0000 7faa7b7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be154d0 0x56412c1e8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.560+0000 7faa7b7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be154d0 0x56412c1e8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:57874/0 (socket says 172.21.15.129:57874) 2024-05-20T18:02:10.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.560+0000 7faa7b7fe640 1 -- 172.21.15.129:0/3185233761 learned_addr learned my addr 172.21.15.129:0/3185233761 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:10.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.560+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be3e390 0x56412c1e7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.561+0000 7faa889c2740 10 client.0 _ll_register_callbacks cb 0x56412c139da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:10.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.562+0000 7faa7b7fe640 1 -- 172.21.15.129:0/3185233761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be3e390 msgr2=0x56412c1e7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:10.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.562+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be3e390 0x56412c1e7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.562+0000 7faa7b7fe640 1 -- 172.21.15.129:0/3185233761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be1d6b0 msgr2=0x56412c1e94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:10.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.562+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be1d6b0 0x56412c1e94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.562+0000 7faa7b7fe640 1 -- 172.21.15.129:0/3185233761 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56412be563f0 con 0x56412be154d0 2024-05-20T18:02:10.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.574+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be154d0 0x56412c1e8f90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faa6c004500 tx=0x7faa6c0056e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ba08a182314dfc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:10.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.575+0000 7faa5affd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:02:10.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.575+0000 7faa5affd640 1 -- 172.21.15.129:0/3185233761 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c035070 con 0x56412be154d0 2024-05-20T18:02:10.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.575+0000 7faa5affd640 1 -- 172.21.15.129:0/3185233761 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7faa6c005260 con 0x56412be154d0 2024-05-20T18:02:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.575+0000 7faa889c2740 1 -- 172.21.15.129:0/3185233761 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa6c005530 con 0x56412be154d0 2024-05-20T18:02:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.576+0000 7faa5affd640 1 -- 172.21.15.129:0/3185233761 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7faa6c03e070 con 0x56412be154d0 2024-05-20T18:02:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.576+0000 7faa889c2740 10 client.25714 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.576+0000 7faa889c2740 1 -- 172.21.15.129:0/3185233761 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56412be160f0 con 0x56412be154d0 2024-05-20T18:02:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.577+0000 7faa5affd640 1 -- 172.21.15.129:0/3185233761 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa6c005530 con 0x56412be154d0 2024-05-20T18:02:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.577+0000 7faa5affd640 1 -- 172.21.15.129:0/3185233761 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7faa6c0315b0 con 0x56412be154d0 2024-05-20T18:02:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.577+0000 7faa889c2740 10 client.25714 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.577+0000 7faa889c2740 10 client.25714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.577+0000 7faa889c2740 20 client.25714 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.577+0000 7faa889c2740 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56412bdfcf60 0x56412c1f9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.577+0000 7faa889c2740 4 client.25714 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:10.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.577+0000 7faa889c2740 1 -- 172.21.15.129:0/3185233761 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56412c150430 con 0x56412bdfcf60 2024-05-20T18:02:10.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.581+0000 7faa7a7fc640 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56412bdfcf60 0x56412c1f9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.590+0000 7faa7a7fc640 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56412bdfcf60 0x56412c1f9b10 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58837d8fdf445d04 server_cookie=af93051e7c20d79a in_seq=0 out_seq=0 2024-05-20T18:02:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.590+0000 7faa5affd640 10 client.25714 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.593+0000 7faa5affd640 1 -- 172.21.15.129:0/3185233761 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56412c150430 con 0x56412bdfcf60 2024-05-20T18:02:10.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.594+0000 7faa5affd640 10 client.25714 handle_command_reply: tid=0 2024-05-20T18:02:10.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.594+0000 7faa889c2740 1 client.25714 shutdown 2024-05-20T18:02:10.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.595+0000 7faa889c2740 1 -- 172.21.15.129:0/3185233761 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be154d0 msgr2=0x56412c1e8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:10.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.595+0000 7faa889c2740 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be154d0 0x56412c1e8f90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faa6c004500 tx=0x7faa6c0056e0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.598+0000 7faa889c2740 1 -- 172.21.15.129:0/3185233761 shutdown_connections 2024-05-20T18:02:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.598+0000 7faa889c2740 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56412bdfcf60 0x56412c1f9b10 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.598+0000 7faa889c2740 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be3e390 0x56412c1e7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.598+0000 7faa889c2740 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be1d6b0 0x56412c1e94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.598+0000 7faa889c2740 1 --2- 172.21.15.129:0/3185233761 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be154d0 0x56412c1e8f90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.598+0000 7faa889c2740 1 -- 172.21.15.129:0/3185233761 >> 172.21.15.129:0/3185233761 conn(0x56412be324e0 msgr2=0x56412c1dcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.598+0000 7faa5affd640 0 client.25714 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.598+0000 7faa889c2740 1 -- 172.21.15.129:0/3185233761 shutdown_connections 2024-05-20T18:02:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.598+0000 7faa889c2740 1 -- 172.21.15.129:0/3185233761 wait complete. 2024-05-20T18:02:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.599+0000 7faa889c2740 20 client.25714 trim_cache size 0 max 16384 2024-05-20T18:02:10.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.643+0000 7faa889c2740 1 Processor -- start 2024-05-20T18:02:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.643+0000 7faa889c2740 1 -- start start 2024-05-20T18:02:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.644+0000 7faa889c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 0x56412c1e6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.644+0000 7faa889c2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be1d6b0 0x56412c1fcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.644+0000 7faa889c2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be3e390 0x56412c1fd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.644+0000 7faa889c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa6800c8f0 con 0x56412be154d0 2024-05-20T18:02:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.644+0000 7faa889c2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa6800f170 con 0x56412be1d6b0 2024-05-20T18:02:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.644+0000 7faa889c2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faa6800cf60 con 0x56412be3e390 2024-05-20T18:02:10.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.645+0000 7faa7b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 0x56412c1e6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.645+0000 7faa7b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 0x56412c1e6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33390/0 (socket says 172.21.15.129:33390) 2024-05-20T18:02:10.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.645+0000 7faa7b7fe640 1 -- 172.21.15.129:0/1483587452 learned_addr learned my addr 172.21.15.129:0/1483587452 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:10.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.646+0000 7faa7a7fc640 1 --2- 172.21.15.129:0/1483587452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be1d6b0 0x56412c1fcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.648+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/1483587452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be3e390 0x56412c1fd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.650+0000 7faa7a7fc640 1 -- 172.21.15.129:0/1483587452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be3e390 msgr2=0x56412c1fd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.650+0000 7faa7a7fc640 1 --2- 172.21.15.129:0/1483587452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be3e390 0x56412c1fd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.650+0000 7faa7a7fc640 1 -- 172.21.15.129:0/1483587452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 msgr2=0x56412c1e6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.650+0000 7faa7a7fc640 1 --2- 172.21.15.129:0/1483587452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 0x56412c1e6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.650+0000 7faa7a7fc640 1 -- 172.21.15.129:0/1483587452 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56412c1ddc80 con 0x56412be1d6b0 2024-05-20T18:02:10.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.652+0000 7faa7a7fc640 1 --2- 172.21.15.129:0/1483587452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be1d6b0 0x56412c1fcb40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x56412c0fdce0 tx=0x7faa70006c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc75eef513622618 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:10.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.654+0000 7faa617fa640 1 -- 172.21.15.129:0/1483587452 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7001a3d0 con 0x56412be1d6b0 2024-05-20T18:02:10.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.654+0000 7faa617fa640 1 -- 172.21.15.129:0/1483587452 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa7001aa50 con 0x56412be1d6b0 2024-05-20T18:02:10.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.656+0000 7faa889c2740 1 -- 172.21.15.129:0/1483587452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be1d6b0 msgr2=0x56412c1fcb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:10.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.656+0000 7faa889c2740 1 --2- 172.21.15.129:0/1483587452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be1d6b0 0x56412c1fcb40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x56412c0fdce0 tx=0x7faa70006c40 comp rx=0 tx=0).stop 2024-05-20T18:02:10.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.658+0000 7faa889c2740 1 -- 172.21.15.129:0/1483587452 shutdown_connections 2024-05-20T18:02:10.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.658+0000 7faa889c2740 1 --2- 172.21.15.129:0/1483587452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be1d6b0 0x56412c1fcb40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.658+0000 7faa889c2740 1 --2- 172.21.15.129:0/1483587452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be3e390 0x56412c1fd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.658+0000 7faa889c2740 1 --2- 172.21.15.129:0/1483587452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be154d0 0x56412c1e6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.658+0000 7faa889c2740 1 -- 172.21.15.129:0/1483587452 >> 172.21.15.129:0/1483587452 conn(0x56412be324e0 msgr2=0x56412c1e5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:10.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.658+0000 7faa889c2740 1 -- 172.21.15.129:0/1483587452 shutdown_connections 2024-05-20T18:02:10.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.658+0000 7faa889c2740 1 -- 172.21.15.129:0/1483587452 wait complete. 2024-05-20T18:02:10.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.658+0000 7faa889c2740 1 -- start start 2024-05-20T18:02:10.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.659+0000 7faa889c2740 1 Processor -- start 2024-05-20T18:02:10.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.659+0000 7faa889c2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be154d0 0x56412c26b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.659+0000 7faa889c2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 0x56412c26b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.660+0000 7faa889c2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be3e390 0x56412c265600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.660+0000 7faa889c2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa6800d630 con 0x56412be3e390 2024-05-20T18:02:10.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.660+0000 7faa889c2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa6800eaa0 con 0x56412be154d0 2024-05-20T18:02:10.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.660+0000 7faa889c2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faa6800dd00 con 0x56412be1d6b0 2024-05-20T18:02:10.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.660+0000 7faa889c2740 10 client.0 _ll_register_callbacks cb 0x7faa7000a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:10.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.663+0000 7faa7a7fc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 0x56412c26b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.663+0000 7faa7a7fc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 0x56412c26b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:57882/0 (socket says 172.21.15.129:57882) 2024-05-20T18:02:10.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.663+0000 7faa7a7fc640 1 -- 172.21.15.129:0/2971540001 learned_addr learned my addr 172.21.15.129:0/2971540001 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:10.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.663+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be3e390 0x56412c265600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.663+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be154d0 0x56412c26b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.665+0000 7faa7b7fe640 1 -- 172.21.15.129:0/2971540001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 msgr2=0x56412c26b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:10.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.665+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 0x56412c26b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.665+0000 7faa7b7fe640 1 -- 172.21.15.129:0/2971540001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be154d0 msgr2=0x56412c26b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:10.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.665+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be154d0 0x56412c26b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.665+0000 7faa7b7fe640 1 -- 172.21.15.129:0/2971540001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56412c265c10 con 0x56412be3e390 2024-05-20T18:02:10.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.665+0000 7faa7b7fe640 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be3e390 0x56412c265600 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0045d0 tx=0x7faa6c005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48340eeb82d8661 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.665+0000 7faa5affd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:02:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.666+0000 7faa5affd640 1 -- 172.21.15.129:0/2971540001 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c035070 con 0x56412be3e390 2024-05-20T18:02:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.666+0000 7faa5affd640 1 -- 172.21.15.129:0/2971540001 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7faa6c02add0 con 0x56412be3e390 2024-05-20T18:02:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.668+0000 7faa889c2740 1 -- 172.21.15.129:0/2971540001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa7001aa50 con 0x56412be3e390 2024-05-20T18:02:10.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.670+0000 7faa5affd640 1 -- 172.21.15.129:0/2971540001 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7faa6c005260 con 0x56412be3e390 2024-05-20T18:02:10.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.673+0000 7faa889c2740 10 client.16002 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:10.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.673+0000 7faa889c2740 1 -- 172.21.15.129:0/2971540001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56412c254610 con 0x56412be3e390 2024-05-20T18:02:10.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.679+0000 7faa5affd640 1 -- 172.21.15.129:0/2971540001 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa7001aa50 con 0x56412be3e390 2024-05-20T18:02:10.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.681+0000 7faa5affd640 1 -- 172.21.15.129:0/2971540001 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7faa6c068020 con 0x56412be3e390 2024-05-20T18:02:10.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.681+0000 7faa889c2740 10 client.16002 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:10.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.681+0000 7faa889c2740 10 client.16002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:10.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.681+0000 7faa889c2740 20 client.16002 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:10.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.681+0000 7faa889c2740 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56412bdd23a0 0x56412c256b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:10.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.681+0000 7faa889c2740 4 client.16002 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:02:10.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.681+0000 7faa889c2740 1 -- 172.21.15.129:0/2971540001 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56412c14a780 con 0x56412bdd23a0 2024-05-20T18:02:10.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.685+0000 7faa7a7fc640 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56412bdd23a0 0x56412c256b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.691+0000 7faa7a7fc640 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56412bdd23a0 0x56412c256b10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=252823cec7edae9a server_cookie=bb663d5c7cf6dfa7 in_seq=0 out_seq=0 2024-05-20T18:02:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.692+0000 7faa5affd640 10 client.16002 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.696+0000 7faa5affd640 1 -- 172.21.15.129:0/2971540001 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56412c14a780 con 0x56412bdd23a0 2024-05-20T18:02:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.696+0000 7faa5affd640 10 client.16002 handle_command_reply: tid=0 2024-05-20T18:02:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.696+0000 7faa889c2740 1 client.16002 shutdown 2024-05-20T18:02:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.699+0000 7faa889c2740 1 -- 172.21.15.129:0/2971540001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be3e390 msgr2=0x56412c265600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.699+0000 7faa889c2740 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be3e390 0x56412c265600 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0045d0 tx=0x7faa6c005e30 comp rx=0 tx=0).stop 2024-05-20T18:02:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.702+0000 7faa889c2740 1 -- 172.21.15.129:0/2971540001 shutdown_connections 2024-05-20T18:02:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.702+0000 7faa889c2740 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56412bdd23a0 0x56412c256b10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.702+0000 7faa889c2740 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56412be3e390 0x56412c265600 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.702+0000 7faa889c2740 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56412be1d6b0 0x56412c26b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.702+0000 7faa889c2740 1 --2- 172.21.15.129:0/2971540001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56412be154d0 0x56412c26b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.702+0000 7faa889c2740 1 -- 172.21.15.129:0/2971540001 >> 172.21.15.129:0/2971540001 conn(0x56412be324e0 msgr2=0x56412c1e8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.702+0000 7faa5affd640 0 client.16002 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.706+0000 7faa889c2740 1 -- 172.21.15.129:0/2971540001 shutdown_connections 2024-05-20T18:02:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.708+0000 7faa889c2740 1 -- 172.21.15.129:0/2971540001 wait complete. 2024-05-20T18:02:10.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.711+0000 7faa889c2740 20 client.16002 trim_cache size 0 max 16384 2024-05-20T18:02:10.713 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:02:10.713 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:02:10.713 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:02:10.713 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:02:10.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.716+0000 7faa7bfff640 1 -- 172.21.15.129:0/308400212 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7faa640a2ec0 msgr2=0x7faa640a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:10.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.716+0000 7faa7bfff640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7faa640a2ec0 0x7faa640a53d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x56412c1daa20 tx=0x7faa7400c650 comp rx=0 tx=0).stop 2024-05-20T18:02:10.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.716+0000 7faa7bfff640 1 -- 172.21.15.129:0/308400212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c109090 msgr2=0x7faa7c1a8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:10.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.716+0000 7faa7bfff640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c109090 0x7faa7c1a8530 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7faa7400ba50 tx=0x7faa7400bf10 comp rx=0 tx=0).stop 2024-05-20T18:02:10.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.717+0000 7faa7bfff640 1 -- 172.21.15.129:0/308400212 shutdown_connections 2024-05-20T18:02:10.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.717+0000 7faa7bfff640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7faa640a2ec0 0x7faa640a53d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.717+0000 7faa7bfff640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa7c10f270 0x7faa7c1acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.717+0000 7faa7bfff640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa7c10e890 0x7faa7c1a8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.717+0000 7faa7bfff640 1 --2- 172.21.15.129:0/308400212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c109090 0x7faa7c1a8530 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.717+0000 7faa7bfff640 1 -- 172.21.15.129:0/308400212 >> 172.21.15.129:0/308400212 conn(0x7faa7c06d580 msgr2=0x7faa7c10a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:10.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.719+0000 7faa7bfff640 1 -- 172.21.15.129:0/308400212 shutdown_connections 2024-05-20T18:02:10.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.720+0000 7faa7bfff640 1 -- 172.21.15.129:0/308400212 wait complete. 2024-05-20T18:02:10.749 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:02:10.749 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:02:10.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.981+0000 7f9bab52d640 1 -- 172.21.15.129:0/1845927846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ba41077f0 msgr2=0x7f9ba4100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:10.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.981+0000 7f9bab52d640 1 --2- 172.21.15.129:0/1845927846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ba41077f0 0x7f9ba4100900 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f9ba001a910 tx=0x7f9ba003b4e0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.991+0000 7f9bab52d640 1 -- 172.21.15.129:0/1845927846 shutdown_connections 2024-05-20T18:02:10.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.991+0000 7f9bab52d640 1 --2- 172.21.15.129:0/1845927846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ba4106090 0x7f9ba41064a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.991+0000 7f9bab52d640 1 --2- 172.21.15.129:0/1845927846 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ba4107c00 0x7f9ba4100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.991+0000 7f9bab52d640 1 --2- 172.21.15.129:0/1845927846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ba41077f0 0x7f9ba4100900 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:10.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.991+0000 7f9bab52d640 1 -- 172.21.15.129:0/1845927846 >> 172.21.15.129:0/1845927846 conn(0x7f9ba40fc7b0 msgr2=0x7f9ba40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:11.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.997+0000 7f9bab52d640 1 -- 172.21.15.129:0/1845927846 shutdown_connections 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.997+0000 7f9bab52d640 1 -- 172.21.15.129:0/1845927846 wait complete. 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.998+0000 7f9bab52d640 1 Processor -- start 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.998+0000 7f9bab52d640 1 -- start start 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.999+0000 7f9bab52d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ba4106090 0x7f9ba4116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.999+0000 7f9bab52d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ba41077f0 0x7f9ba4112370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.999+0000 7f9bab52d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ba4107c00 0x7f9ba4112910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.999+0000 7f9bab52d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba410a8d0 con 0x7f9ba4107c00 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.999+0000 7f9bab52d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba410a750 con 0x7f9ba41077f0 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.999+0000 7f9bab52d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba410aa50 con 0x7f9ba4106090 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.999+0000 7f9baad2c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ba4107c00 0x7f9ba4112910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.999+0000 7f9baad2c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ba4107c00 0x7f9ba4112910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33422/0 (socket says 172.21.15.129:33422) 2024-05-20T18:02:11.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:10.999+0000 7f9baad2c640 1 -- 172.21.15.129:0/578876674 learned_addr learned my addr 172.21.15.129:0/578876674 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.000+0000 7f9baad2c640 1 -- 172.21.15.129:0/578876674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ba4106090 msgr2=0x7f9ba4116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.000+0000 7f9baad2c640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ba4106090 0x7f9ba4116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.000+0000 7f9baad2c640 1 -- 172.21.15.129:0/578876674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ba41077f0 msgr2=0x7f9ba4112370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.000+0000 7f9baad2c640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ba41077f0 0x7f9ba4112370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.000+0000 7f9baad2c640 1 -- 172.21.15.129:0/578876674 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba4113130 con 0x7f9ba4107c00 2024-05-20T18:02:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.000+0000 7f9baad2c640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ba4107c00 0x7f9ba4112910 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400d750 tx=0x7f9b9400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.001+0000 7f9b9b7fe640 1 -- 172.21.15.129:0/578876674 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94017070 con 0x7f9ba4107c00 2024-05-20T18:02:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.002+0000 7f9b9b7fe640 1 -- 172.21.15.129:0/578876674 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b94005500 con 0x7f9ba4107c00 2024-05-20T18:02:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.002+0000 7f9b9b7fe640 1 -- 172.21.15.129:0/578876674 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94004d90 con 0x7f9ba4107c00 2024-05-20T18:02:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.002+0000 7f9bab52d640 1 -- 172.21.15.129:0/578876674 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba41a3860 con 0x7f9ba4107c00 2024-05-20T18:02:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.002+0000 7f9bab52d640 1 -- 172.21.15.129:0/578876674 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba41a3de0 con 0x7f9ba4107c00 2024-05-20T18:02:11.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.010+0000 7f9b9b7fe640 1 -- 172.21.15.129:0/578876674 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9b94004f30 con 0x7f9ba4107c00 2024-05-20T18:02:11.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.011+0000 7f9b9b7fe640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b900a2f00 0x7f9b900a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.012+0000 7f9b9b7fe640 1 -- 172.21.15.129:0/578876674 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f9b940cee90 con 0x7f9ba4107c00 2024-05-20T18:02:11.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.013+0000 7f9baa52b640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b900a2f00 0x7f9b900a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.013+0000 7f9bab52d640 1 -- 172.21.15.129:0/578876674 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba4101b20 con 0x7f9ba4107c00 2024-05-20T18:02:11.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.017+0000 7f9baa52b640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b900a2f00 0x7f9b900a5410 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9ba001a6d0 tx=0x7f9ba0000bc0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:11.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.026+0000 7f9b9b7fe640 1 -- 172.21.15.129:0/578876674 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9b9409ba40 con 0x7f9ba4107c00 2024-05-20T18:02:11.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.583+0000 7f9bab52d640 1 -- 172.21.15.129:0/578876674 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ba41a4040 con 0x7f9ba4107c00 2024-05-20T18:02:11.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.585+0000 7f9b9b7fe640 1 -- 172.21.15.129:0/578876674 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11829 (secure 0 0 0) 0x7f9b9409b340 con 0x7f9ba4107c00 2024-05-20T18:02:11.584 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:02:11.584 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:59:17.074606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25229":{"gid":25229,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1860679091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1860679091},{"type":"v1","addr":"172.21.15.154:6837","nonce":1860679091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:02:11.584 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-05-20T18:02:11.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.591+0000 7f9bab52d640 1 -- 172.21.15.129:0/578876674 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b900a2f00 msgr2=0x7f9b900a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:11.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.591+0000 7f9bab52d640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b900a2f00 0x7f9b900a5410 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9ba001a6d0 tx=0x7f9ba0000bc0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.591+0000 7f9bab52d640 1 -- 172.21.15.129:0/578876674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ba4107c00 msgr2=0x7f9ba4112910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:11.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.591+0000 7f9bab52d640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ba4107c00 0x7f9ba4112910 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400d750 tx=0x7f9b9400dc10 comp rx=0 tx=0).stop 2024-05-20T18:02:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.594+0000 7f9bab52d640 1 -- 172.21.15.129:0/578876674 shutdown_connections 2024-05-20T18:02:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.594+0000 7f9bab52d640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b900a2f00 0x7f9b900a5410 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.594+0000 7f9bab52d640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ba41077f0 0x7f9ba4112370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.594+0000 7f9bab52d640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ba4107c00 0x7f9ba4112910 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.594+0000 7f9bab52d640 1 --2- 172.21.15.129:0/578876674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ba4106090 0x7f9ba4116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.594+0000 7f9bab52d640 1 -- 172.21.15.129:0/578876674 >> 172.21.15.129:0/578876674 conn(0x7f9ba40fc7b0 msgr2=0x7f9ba4104060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:11.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.596+0000 7f9bab52d640 1 -- 172.21.15.129:0/578876674 shutdown_connections 2024-05-20T18:02:11.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.596+0000 7f9bab52d640 1 -- 172.21.15.129:0/578876674 wait complete. 2024-05-20T18:02:11.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:02:11.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:11 smithi154 ceph-mon[36482]: pgmap v112: 97 pgs: 97 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 905 MiB/s wr, 3.10k op/s 2024-05-20T18:02:11.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:11 smithi129 ceph-mon[32029]: pgmap v112: 97 pgs: 97 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 905 MiB/s wr, 3.10k op/s 2024-05-20T18:02:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.927+0000 7f85c17ef640 1 -- 172.21.15.129:0/1406541146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85bc109740 msgr2=0x7f85bc079e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.927+0000 7f85c17ef640 1 --2- 172.21.15.129:0/1406541146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85bc109740 0x7f85bc079e70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f85ac009900 tx=0x7f85ac02a440 comp rx=0 tx=0).stop 2024-05-20T18:02:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.928+0000 7f85c17ef640 1 -- 172.21.15.129:0/1406541146 shutdown_connections 2024-05-20T18:02:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.928+0000 7f85c17ef640 1 --2- 172.21.15.129:0/1406541146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85bc10e210 0x7f85bc1090b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.928+0000 7f85c17ef640 1 --2- 172.21.15.129:0/1406541146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85bc109740 0x7f85bc079e70 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.928+0000 7f85c17ef640 1 --2- 172.21.15.129:0/1406541146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85bc10de00 0x7f85bc108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.928+0000 7f85c17ef640 1 -- 172.21.15.129:0/1406541146 >> 172.21.15.129:0/1406541146 conn(0x7f85bc06d580 msgr2=0x7f85bc06d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:11.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.930+0000 7f85c17ef640 1 -- 172.21.15.129:0/1406541146 shutdown_connections 2024-05-20T18:02:11.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:11 smithi204 ceph-mon[37076]: pgmap v112: 97 pgs: 97 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 905 MiB/s wr, 3.10k op/s 2024-05-20T18:02:11.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.930+0000 7f85c17ef640 1 -- 172.21.15.129:0/1406541146 wait complete. 2024-05-20T18:02:11.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.932+0000 7f85c17ef640 1 Processor -- start 2024-05-20T18:02:11.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.933+0000 7f85c17ef640 1 -- start start 2024-05-20T18:02:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.933+0000 7f85c17ef640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85bc10de00 0x7f85bc07eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.933+0000 7f85c17ef640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85bc10e210 0x7f85bc07f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.934+0000 7f85c17ef640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85bc07fa50 0x7f85bc1c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.934+0000 7f85c17ef640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc07d7e0 con 0x7f85bc10de00 2024-05-20T18:02:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.934+0000 7f85c17ef640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc07d660 con 0x7f85bc10e210 2024-05-20T18:02:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.934+0000 7f85c17ef640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc07d960 con 0x7f85bc07fa50 2024-05-20T18:02:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.934+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85bc10e210 0x7f85bc07f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.934+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85bc10e210 0x7f85bc07f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:56876/0 (socket says 172.21.15.129:56876) 2024-05-20T18:02:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.934+0000 7f85bb7fe640 1 -- 172.21.15.129:0/3426164337 learned_addr learned my addr 172.21.15.129:0/3426164337 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.934+0000 7f85bbfff640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85bc10de00 0x7f85bc07eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.935+0000 7f85bb7fe640 1 -- 172.21.15.129:0/3426164337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85bc07fa50 msgr2=0x7f85bc1c4e70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:02:11.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.935+0000 7f85bb7fe640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85bc07fa50 0x7f85bc1c4e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.935+0000 7f85bb7fe640 1 -- 172.21.15.129:0/3426164337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85bc10de00 msgr2=0x7f85bc07eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.935+0000 7f85bb7fe640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85bc10de00 0x7f85bc07eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.935+0000 7f85bb7fe640 1 -- 172.21.15.129:0/3426164337 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85bc1c5690 con 0x7f85bc10e210 2024-05-20T18:02:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.935+0000 7f85bb7fe640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85bc10e210 0x7f85bc07f450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f85b000b470 tx=0x7f85b000b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.936+0000 7f85b97fa640 1 -- 172.21.15.129:0/3426164337 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0015020 con 0x7f85bc10e210 2024-05-20T18:02:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.936+0000 7f85b97fa640 1 -- 172.21.15.129:0/3426164337 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85b0011d70 con 0x7f85bc10e210 2024-05-20T18:02:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.936+0000 7f85b97fa640 1 -- 172.21.15.129:0/3426164337 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b000f340 con 0x7f85bc10e210 2024-05-20T18:02:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.936+0000 7f85c17ef640 1 -- 172.21.15.129:0/3426164337 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85bc1c5980 con 0x7f85bc10e210 2024-05-20T18:02:11.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.936+0000 7f85c17ef640 1 -- 172.21.15.129:0/3426164337 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85bc1c5e80 con 0x7f85bc10e210 2024-05-20T18:02:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.942+0000 7f85c99d0740 1 Processor -- start 2024-05-20T18:02:11.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.944+0000 7f85c99d0740 1 -- start start 2024-05-20T18:02:11.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.944+0000 7f85b97fa640 1 -- 172.21.15.129:0/3426164337 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f85b0018020 con 0x7f85bc10e210 2024-05-20T18:02:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.945+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 0x55971e911f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.945+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e54a4d0 0x55971e9124d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.945+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e5526b0 0x55971e912a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.945+0000 7f85c99d0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55971e877c40 con 0x55971e5526b0 2024-05-20T18:02:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.945+0000 7f85c99d0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55971e81f580 con 0x55971e54a4d0 2024-05-20T18:02:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.945+0000 7f85c99d0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55971e873c00 con 0x55971e573390 2024-05-20T18:02:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.946+0000 7f85c0fee640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 0x55971e911f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.946+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e5526b0 0x55971e912a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.946+0000 7f85c0fee640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 0x55971e911f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:57902/0 (socket says 172.21.15.129:57902) 2024-05-20T18:02:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.946+0000 7f85c0fee640 1 -- 172.21.15.129:0/2633113491 learned_addr learned my addr 172.21.15.129:0/2633113491 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.947+0000 7f85bbfff640 1 --2- 172.21.15.129:0/2633113491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e54a4d0 0x55971e9124d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.949+0000 7f85c0fee640 1 -- 172.21.15.129:0/2633113491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e54a4d0 msgr2=0x55971e9124d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.949+0000 7f85c0fee640 1 --2- 172.21.15.129:0/2633113491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e54a4d0 0x55971e9124d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.949+0000 7f85c0fee640 1 -- 172.21.15.129:0/2633113491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e5526b0 msgr2=0x55971e912a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:11.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.949+0000 7f85c0fee640 1 --2- 172.21.15.129:0/2633113491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e5526b0 0x55971e912a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.949+0000 7f85c0fee640 1 -- 172.21.15.129:0/2633113491 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55971e58b3f0 con 0x55971e573390 2024-05-20T18:02:11.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.949+0000 7f85b97fa640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f85a40a2d60 0x7f85a40a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.949+0000 7f85b97fa640 1 -- 172.21.15.129:0/3426164337 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f85b00ceca0 con 0x7f85bc10e210 2024-05-20T18:02:11.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.951+0000 7f85c0fee640 1 --2- 172.21.15.129:0/2633113491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 0x55971e911f90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f85ac02a960 tx=0x7f85ac009670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21745f18385101b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:11.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.951+0000 7f85bbfff640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f85a40a2d60 0x7f85a40a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.951+0000 7f85a27fc640 1 -- 172.21.15.129:0/2633113491 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac03e070 con 0x55971e573390 2024-05-20T18:02:11.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.952+0000 7f85a27fc640 1 -- 172.21.15.129:0/2633113491 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85ac005530 con 0x55971e573390 2024-05-20T18:02:11.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.952+0000 7f85a27fc640 1 -- 172.21.15.129:0/2633113491 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac035040 con 0x55971e573390 2024-05-20T18:02:11.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.953+0000 7f85c99d0740 1 -- 172.21.15.129:0/2633113491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 msgr2=0x55971e911f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:11.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.953+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2633113491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 0x55971e911f90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f85ac02a960 tx=0x7f85ac009670 comp rx=0 tx=0).stop 2024-05-20T18:02:11.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.953+0000 7f85c99d0740 1 -- 172.21.15.129:0/2633113491 shutdown_connections 2024-05-20T18:02:11.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.953+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2633113491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e54a4d0 0x55971e9124d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.953+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2633113491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e5526b0 0x55971e912a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.953+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2633113491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 0x55971e911f90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.953+0000 7f85c99d0740 1 -- 172.21.15.129:0/2633113491 >> 172.21.15.129:0/2633113491 conn(0x55971e5674e0 msgr2=0x55971e90e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:11.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.955+0000 7f85bbfff640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f85a40a2d60 0x7f85a40a5270 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55971e913550 tx=0x7f85b400d040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:11.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.955+0000 7f85c99d0740 1 -- 172.21.15.129:0/2633113491 shutdown_connections 2024-05-20T18:02:11.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.955+0000 7f85c99d0740 1 -- 172.21.15.129:0/2633113491 wait complete. 2024-05-20T18:02:11.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.956+0000 7f85c99d0740 1 -- start start 2024-05-20T18:02:11.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.956+0000 7f85c99d0740 1 Processor -- start 2024-05-20T18:02:11.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.957+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e54a4d0 0x55971e91c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.958+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e5526b0 0x55971e91cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.958+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e573390 0x55971e928810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.958+0000 7f85c99d0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55971e91f2d0 con 0x55971e54a4d0 2024-05-20T18:02:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.958+0000 7f85c99d0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55971e832c40 con 0x55971e573390 2024-05-20T18:02:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.958+0000 7f85c99d0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55971e91f450 con 0x55971e5526b0 2024-05-20T18:02:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.958+0000 7f85c99d0740 10 client.0 _ll_register_callbacks cb 0x55971e58abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.958+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e54a4d0 0x55971e91c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.958+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e54a4d0 0x55971e91c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33460/0 (socket says 172.21.15.129:33460) 2024-05-20T18:02:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.958+0000 7f85bbfff640 1 -- 172.21.15.129:0/498072509 learned_addr learned my addr 172.21.15.129:0/498072509 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.958+0000 7f85c0fee640 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e573390 0x55971e928810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.959+0000 7f85bb7fe640 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e5526b0 0x55971e91cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.960+0000 7f85bbfff640 1 -- 172.21.15.129:0/498072509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e5526b0 msgr2=0x55971e91cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.960+0000 7f85bbfff640 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e5526b0 0x55971e91cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.960+0000 7f85bbfff640 1 -- 172.21.15.129:0/498072509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e573390 msgr2=0x55971e928810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.960+0000 7f85bbfff640 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e573390 0x55971e928810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.960+0000 7f85bbfff640 1 -- 172.21.15.129:0/498072509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55971e6557d0 con 0x55971e54a4d0 2024-05-20T18:02:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.960+0000 7f85bbfff640 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e54a4d0 0x55971e91c580 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f85b400acc0 tx=0x7f85b4004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c036e283a5ca1a2a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:11.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.960+0000 7f859b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:02:11.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.961+0000 7f859b7fe640 1 -- 172.21.15.129:0/498072509 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b40225f0 con 0x55971e54a4d0 2024-05-20T18:02:11.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.961+0000 7f859b7fe640 1 -- 172.21.15.129:0/498072509 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f85b4022790 con 0x55971e54a4d0 2024-05-20T18:02:11.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.961+0000 7f85c99d0740 1 -- 172.21.15.129:0/498072509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85ac005530 con 0x55971e54a4d0 2024-05-20T18:02:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.962+0000 7f859b7fe640 1 -- 172.21.15.129:0/498072509 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f85b4030050 con 0x55971e54a4d0 2024-05-20T18:02:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.962+0000 7f85c99d0740 10 client.16020 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.962+0000 7f85c99d0740 1 -- 172.21.15.129:0/498072509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55971e92b8b0 con 0x55971e54a4d0 2024-05-20T18:02:11.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.963+0000 7f859b7fe640 1 -- 172.21.15.129:0/498072509 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85b4027070 con 0x55971e54a4d0 2024-05-20T18:02:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.964+0000 7f859b7fe640 1 -- 172.21.15.129:0/498072509 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f85b4016ad0 con 0x55971e54a4d0 2024-05-20T18:02:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.964+0000 7f85c99d0740 10 client.16020 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.964+0000 7f85c99d0740 10 client.16020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.964+0000 7f85c99d0740 20 client.16020 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.965+0000 7f85c99d0740 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55971e531f60 0x55971e92ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.965+0000 7f85c99d0740 4 client.16020 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.965+0000 7f85c99d0740 1 -- 172.21.15.129:0/498072509 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f85ac005530 con 0x55971e531f60 2024-05-20T18:02:11.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.967+0000 7f85bb7fe640 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55971e531f60 0x55971e92ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.967+0000 7f859b7fe640 10 client.16020 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:11.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.967+0000 7f85bb7fe640 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55971e531f60 0x55971e92ed80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d1daab898f16a97 server_cookie=287bf40fa1b927ce in_seq=0 out_seq=0 2024-05-20T18:02:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.968+0000 7f859b7fe640 1 -- 172.21.15.129:0/498072509 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f85ac005530 con 0x55971e531f60 2024-05-20T18:02:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.968+0000 7f859b7fe640 10 client.16020 handle_command_reply: tid=0 2024-05-20T18:02:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.968+0000 7f85c99d0740 1 client.16020 shutdown 2024-05-20T18:02:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.969+0000 7f85c99d0740 1 -- 172.21.15.129:0/498072509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e54a4d0 msgr2=0x55971e91c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.969+0000 7f85c99d0740 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e54a4d0 0x55971e91c580 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f85b400acc0 tx=0x7f85b4004a80 comp rx=0 tx=0).stop 2024-05-20T18:02:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.969+0000 7f85c99d0740 1 -- 172.21.15.129:0/498072509 shutdown_connections 2024-05-20T18:02:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.969+0000 7f85c99d0740 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55971e531f60 0x55971e92ed80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.969+0000 7f85c99d0740 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e573390 0x55971e928810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.969+0000 7f85c99d0740 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e5526b0 0x55971e91cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.969+0000 7f85c99d0740 1 --2- 172.21.15.129:0/498072509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e54a4d0 0x55971e91c580 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.969+0000 7f85c99d0740 1 -- 172.21.15.129:0/498072509 >> 172.21.15.129:0/498072509 conn(0x55971e5674e0 msgr2=0x55971e913cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.970+0000 7f859b7fe640 0 client.16020 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.970+0000 7f85c99d0740 1 -- 172.21.15.129:0/498072509 shutdown_connections 2024-05-20T18:02:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.970+0000 7f85c99d0740 1 -- 172.21.15.129:0/498072509 wait complete. 2024-05-20T18:02:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.970+0000 7f85c99d0740 20 client.16020 trim_cache size 0 max 16384 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.998+0000 7f85c99d0740 1 Processor -- start 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.998+0000 7f85c99d0740 1 -- start start 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.998+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e54a4d0 0x55971e9283f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.998+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e5526b0 0x55971e912570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.998+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e573390 0x55971e915c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.999+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e5526b0 0x55971e912570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.999+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e5526b0 0x55971e912570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:56914/0 (socket says 172.21.15.129:56914) 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.999+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e54a4d0 0x55971e9283f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.999+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e54a4d0 0x55971e9283f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:57918/0 (socket says 172.21.15.129:57918) 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.999+0000 7f85bb7fe640 1 -- 172.21.15.129:0/2278392276 learned_addr learned my addr 172.21.15.129:0/2278392276 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.999+0000 7f85c99d0740 1 -- 172.21.15.129:0/2278392276 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f85a800c7e0 con 0x55971e573390 2024-05-20T18:02:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.999+0000 7f85c99d0740 1 -- 172.21.15.129:0/2278392276 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85a800efb0 con 0x55971e5526b0 2024-05-20T18:02:11.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:11.999+0000 7f85c99d0740 1 -- 172.21.15.129:0/2278392276 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f85a800cda0 con 0x55971e54a4d0 2024-05-20T18:02:11.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.000+0000 7f85bbfff640 1 -- 172.21.15.129:0/2278392276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e5526b0 msgr2=0x55971e912570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:11.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.000+0000 7f85bbfff640 1 --2- 172.21.15.129:0/2278392276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e5526b0 0x55971e912570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.000+0000 7f85bbfff640 1 -- 172.21.15.129:0/2278392276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e573390 msgr2=0x55971e915c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:11.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.000+0000 7f85bbfff640 1 --2- 172.21.15.129:0/2278392276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e573390 0x55971e915c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:11.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.000+0000 7f85bbfff640 1 -- 172.21.15.129:0/2278392276 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55971e91a8c0 con 0x55971e54a4d0 2024-05-20T18:02:11.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.001+0000 7f85bbfff640 1 --2- 172.21.15.129:0/2278392276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e54a4d0 0x55971e9283f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f85b4000f80 tx=0x7f85b40173c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f128449d50e79b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:12.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.001+0000 7f85a1ffb640 1 -- 172.21.15.129:0/2278392276 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4038030 con 0x55971e54a4d0 2024-05-20T18:02:12.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.002+0000 7f85a1ffb640 1 -- 172.21.15.129:0/2278392276 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85b4027040 con 0x55971e54a4d0 2024-05-20T18:02:12.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.002+0000 7f85a1ffb640 1 -- 172.21.15.129:0/2278392276 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4017c70 con 0x55971e54a4d0 2024-05-20T18:02:12.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.003+0000 7f85c99d0740 1 -- 172.21.15.129:0/2278392276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e54a4d0 msgr2=0x55971e9283f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:12.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.003+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2278392276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e54a4d0 0x55971e9283f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f85b4000f80 tx=0x7f85b40173c0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.003+0000 7f85c99d0740 1 -- 172.21.15.129:0/2278392276 shutdown_connections 2024-05-20T18:02:12.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.003+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2278392276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e5526b0 0x55971e912570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.003+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2278392276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e573390 0x55971e915c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.003+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2278392276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e54a4d0 0x55971e9283f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.003+0000 7f85c99d0740 1 -- 172.21.15.129:0/2278392276 >> 172.21.15.129:0/2278392276 conn(0x55971e5674e0 msgr2=0x55971e911e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:12.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.006+0000 7f85c99d0740 1 -- 172.21.15.129:0/2278392276 shutdown_connections 2024-05-20T18:02:12.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.006+0000 7f85c99d0740 1 -- 172.21.15.129:0/2278392276 wait complete. 2024-05-20T18:02:12.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.006+0000 7f85c99d0740 1 -- start start 2024-05-20T18:02:12.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.007+0000 7f85c99d0740 1 Processor -- start 2024-05-20T18:02:12.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.008+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e54a4d0 0x55971e9153f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:12.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.008+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e5526b0 0x55971e915930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:12.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.008+0000 7f85c99d0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 0x55971e915e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:12.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.008+0000 7f85c99d0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f85a800d470 con 0x55971e5526b0 2024-05-20T18:02:12.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.008+0000 7f85c99d0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85a800e8e0 con 0x55971e54a4d0 2024-05-20T18:02:12.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.008+0000 7f85c99d0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f85a800db40 con 0x55971e573390 2024-05-20T18:02:12.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.008+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e5526b0 0x55971e915930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:12.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.008+0000 7f85bb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e5526b0 0x55971e915930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33470/0 (socket says 172.21.15.129:33470) 2024-05-20T18:02:12.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.008+0000 7f85bb7fe640 1 -- 172.21.15.129:0/2657729738 learned_addr learned my addr 172.21.15.129:0/2657729738 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:12.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.008+0000 7f85c0fee640 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 0x55971e915e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:12.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.008+0000 7f85c99d0740 10 client.0 _ll_register_callbacks cb 0x7f85b40099f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:12.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.009+0000 7f85bbfff640 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e54a4d0 0x55971e9153f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:12.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.009+0000 7f85c0fee640 1 -- 172.21.15.129:0/2657729738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e54a4d0 msgr2=0x55971e9153f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:12.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.009+0000 7f85c0fee640 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e54a4d0 0x55971e9153f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.009+0000 7f85c0fee640 1 -- 172.21.15.129:0/2657729738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e5526b0 msgr2=0x55971e915930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:12.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.009+0000 7f85c0fee640 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e5526b0 0x55971e915930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.009+0000 7f85c0fee640 1 -- 172.21.15.129:0/2657729738 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55971e9229f0 con 0x55971e573390 2024-05-20T18:02:12.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.011+0000 7f85c0fee640 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 0x55971e915e70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f85ac009900 tx=0x7f85ac0048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fdfed02ad052958 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:12.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.012+0000 7f859b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:02:12.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.012+0000 7f859b7fe640 1 -- 172.21.15.129:0/2657729738 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac03e070 con 0x55971e573390 2024-05-20T18:02:12.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.012+0000 7f859b7fe640 1 -- 172.21.15.129:0/2657729738 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f85ac004410 con 0x55971e573390 2024-05-20T18:02:12.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.012+0000 7f85c99d0740 1 -- 172.21.15.129:0/2657729738 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85b4027040 con 0x55971e573390 2024-05-20T18:02:12.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.014+0000 7f859b7fe640 1 -- 172.21.15.129:0/2657729738 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f85ac035040 con 0x55971e573390 2024-05-20T18:02:12.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.014+0000 7f85c99d0740 10 client.25732 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:12.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.014+0000 7f85c99d0740 1 -- 172.21.15.129:0/2657729738 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55971e9971e0 con 0x55971e573390 2024-05-20T18:02:12.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.016+0000 7f859b7fe640 1 -- 172.21.15.129:0/2657729738 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85ac005530 con 0x55971e573390 2024-05-20T18:02:12.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.016+0000 7f859b7fe640 1 -- 172.21.15.129:0/2657729738 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f85ac003840 con 0x55971e573390 2024-05-20T18:02:12.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.016+0000 7f85c99d0740 10 client.25732 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:12.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.017+0000 7f85c99d0740 10 client.25732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:12.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.017+0000 7f85c99d0740 20 client.25732 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:12.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.017+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55971e5073a0 0x55971e9993c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:12.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.017+0000 7f85c99d0740 4 client.25732 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:02:12.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.017+0000 7f85c99d0740 1 -- 172.21.15.129:0/2657729738 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f85b4027040 con 0x55971e5073a0 2024-05-20T18:02:12.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.017+0000 7f85bbfff640 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55971e5073a0 0x55971e9993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:12.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.019+0000 7f85bbfff640 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55971e5073a0 0x55971e9993c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8717e894c8ef8e8a server_cookie=1f6f86ea2847612f in_seq=0 out_seq=0 2024-05-20T18:02:12.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.019+0000 7f859b7fe640 10 client.25732 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:12.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.020+0000 7f859b7fe640 1 -- 172.21.15.129:0/2657729738 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f85b4027040 con 0x55971e5073a0 2024-05-20T18:02:12.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.020+0000 7f859b7fe640 10 client.25732 handle_command_reply: tid=0 2024-05-20T18:02:12.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.022+0000 7f85c99d0740 1 client.25732 shutdown 2024-05-20T18:02:12.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.022+0000 7f85c99d0740 1 -- 172.21.15.129:0/2657729738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 msgr2=0x55971e915e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:12.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.022+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 0x55971e915e70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f85ac009900 tx=0x7f85ac0048f0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.023+0000 7f85c99d0740 1 -- 172.21.15.129:0/2657729738 shutdown_connections 2024-05-20T18:02:12.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.023+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55971e5073a0 0x55971e9993c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.023+0000 7f859b7fe640 0 client.25732 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:12.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.023+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55971e573390 0x55971e915e70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.023+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55971e5526b0 0x55971e915930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.023+0000 7f85c99d0740 1 --2- 172.21.15.129:0/2657729738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971e54a4d0 0x55971e9153f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.023+0000 7f85c99d0740 1 -- 172.21.15.129:0/2657729738 >> 172.21.15.129:0/2657729738 conn(0x55971e5674e0 msgr2=0x55971e9141c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:12.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.024+0000 7f85c99d0740 1 -- 172.21.15.129:0/2657729738 shutdown_connections 2024-05-20T18:02:12.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.024+0000 7f85c99d0740 1 -- 172.21.15.129:0/2657729738 wait complete. 2024-05-20T18:02:12.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.025+0000 7f85c99d0740 20 client.25732 trim_cache size 0 max 16384 2024-05-20T18:02:12.026 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:02:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.030+0000 7f85c17ef640 1 -- 172.21.15.129:0/3426164337 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f85a40a2d60 msgr2=0x7f85a40a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.030+0000 7f85c17ef640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f85a40a2d60 0x7f85a40a5270 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55971e913550 tx=0x7f85b400d040 comp rx=0 tx=0).stop 2024-05-20T18:02:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.030+0000 7f85c17ef640 1 -- 172.21.15.129:0/3426164337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85bc10e210 msgr2=0x7f85bc07f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.030+0000 7f85c17ef640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85bc10e210 0x7f85bc07f450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f85b000b470 tx=0x7f85b000b930 comp rx=0 tx=0).stop 2024-05-20T18:02:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.030+0000 7f85c17ef640 1 -- 172.21.15.129:0/3426164337 shutdown_connections 2024-05-20T18:02:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.030+0000 7f85c17ef640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f85a40a2d60 0x7f85a40a5270 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.030+0000 7f85c17ef640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85bc10e210 0x7f85bc07f450 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.030+0000 7f85c17ef640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85bc07fa50 0x7f85bc1c4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.030+0000 7f85c17ef640 1 --2- 172.21.15.129:0/3426164337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85bc10de00 0x7f85bc07eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.030+0000 7f85c17ef640 1 -- 172.21.15.129:0/3426164337 >> 172.21.15.129:0/3426164337 conn(0x7f85bc06d580 msgr2=0x7f85bc10a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:12.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.031+0000 7f85c17ef640 1 -- 172.21.15.129:0/3426164337 shutdown_connections 2024-05-20T18:02:12.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:12.031+0000 7f85c17ef640 1 -- 172.21.15.129:0/3426164337 wait complete. 2024-05-20T18:02:12.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:12 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/578876674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:02:12.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:12 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/578876674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:02:12.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:12 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/578876674' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:02:13.054 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:02:13.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:02:13.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:02:13.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.289+0000 7f9759a99640 1 -- 172.21.15.129:0/3435785594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9754105940 msgr2=0x7f9754105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:13.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.289+0000 7f9759a99640 1 --2- 172.21.15.129:0/3435785594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9754105940 0x7f9754105d70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f974800aa10 tx=0x7f974802b720 comp rx=0 tx=0).stop 2024-05-20T18:02:13.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.289+0000 7f9759a99640 1 -- 172.21.15.129:0/3435785594 shutdown_connections 2024-05-20T18:02:13.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.289+0000 7f9759a99640 1 --2- 172.21.15.129:0/3435785594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9754105940 0x7f9754105d70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.289+0000 7f9759a99640 1 --2- 172.21.15.129:0/3435785594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9754106400 0x7f975410dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.289+0000 7f9759a99640 1 --2- 172.21.15.129:0/3435785594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97541013b0 0x7f97541017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.290+0000 7f9759a99640 1 -- 172.21.15.129:0/3435785594 >> 172.21.15.129:0/3435785594 conn(0x7f9754076830 msgr2=0x7f9754076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:13.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.290+0000 7f9759a99640 1 -- 172.21.15.129:0/3435785594 shutdown_connections 2024-05-20T18:02:13.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.290+0000 7f9759a99640 1 -- 172.21.15.129:0/3435785594 wait complete. 2024-05-20T18:02:13.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.292+0000 7f9759a99640 1 Processor -- start 2024-05-20T18:02:13.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.292+0000 7f9759a99640 1 -- start start 2024-05-20T18:02:13.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.293+0000 7f9759a99640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97541013b0 0x7f9754197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.293+0000 7f9759a99640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9754105940 0x7f9754197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.293+0000 7f9759a99640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9754106400 0x7f9754197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.293+0000 7f9759a99640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9754110c30 con 0x7f97541013b0 2024-05-20T18:02:13.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.294+0000 7f9752ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9754105940 0x7f9754197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:13.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.294+0000 7f9759a99640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9754110ab0 con 0x7f9754105940 2024-05-20T18:02:13.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.294+0000 7f9759a99640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9754110db0 con 0x7f9754106400 2024-05-20T18:02:13.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.294+0000 7f9752ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9754105940 0x7f9754197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:56928/0 (socket says 172.21.15.129:56928) 2024-05-20T18:02:13.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.294+0000 7f9752ffd640 1 -- 172.21.15.129:0/935798389 learned_addr learned my addr 172.21.15.129:0/935798389 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:13.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.294+0000 7f9752ffd640 1 -- 172.21.15.129:0/935798389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9754106400 msgr2=0x7f9754197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:13.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.294+0000 7f9752ffd640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9754106400 0x7f9754197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.294+0000 7f9752ffd640 1 -- 172.21.15.129:0/935798389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97541013b0 msgr2=0x7f9754197410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T18:02:13.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.294+0000 7f9752ffd640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97541013b0 0x7f9754197410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.294+0000 7f9752ffd640 1 -- 172.21.15.129:0/935798389 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97541abf50 con 0x7f9754105940 2024-05-20T18:02:13.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.294+0000 7f9752ffd640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9754105940 0x7f9754197950 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f974802bc40 tx=0x7f9748030c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:13.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.296+0000 7f9750ff9640 1 -- 172.21.15.129:0/935798389 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974803f070 con 0x7f9754105940 2024-05-20T18:02:13.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.296+0000 7f9750ff9640 1 -- 172.21.15.129:0/935798389 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9748003cd0 con 0x7f9754105940 2024-05-20T18:02:13.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.296+0000 7f9750ff9640 1 -- 172.21.15.129:0/935798389 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9748036040 con 0x7f9754105940 2024-05-20T18:02:13.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.297+0000 7f9759a99640 1 -- 172.21.15.129:0/935798389 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97541ac1e0 con 0x7f9754105940 2024-05-20T18:02:13.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.300+0000 7f9759a99640 1 -- 172.21.15.129:0/935798389 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97541ac680 con 0x7f9754105940 2024-05-20T18:02:13.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.311+0000 7f9760a1d740 1 Processor -- start 2024-05-20T18:02:13.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.311+0000 7f9760a1d740 1 -- start start 2024-05-20T18:02:13.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.313+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978bd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.314+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974f5550 0x5653978bdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.314+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974fd730 0x5653978be0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.314+0000 7f9760a1d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x565397822990 con 0x56539751e410 2024-05-20T18:02:13.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.319+0000 7f9760a1d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5653977ca5b0 con 0x5653974f5550 2024-05-20T18:02:13.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.319+0000 7f9760a1d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56539781e950 con 0x5653974fd730 2024-05-20T18:02:13.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.320+0000 7f97537fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974f5550 0x5653978bdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:13.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.321+0000 7f97537fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974f5550 0x5653978bdb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:56942/0 (socket says 172.21.15.129:56942) 2024-05-20T18:02:13.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.321+0000 7f97537fe640 1 -- 172.21.15.129:0/1445430641 learned_addr learned my addr 172.21.15.129:0/1445430641 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:13.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.324+0000 7f9753fff640 1 --2- 172.21.15.129:0/1445430641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978bd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:13.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.328+0000 7f97537fe640 1 -- 172.21.15.129:0/1445430641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974fd730 msgr2=0x5653978be0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:02:13.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.328+0000 7f97537fe640 1 --2- 172.21.15.129:0/1445430641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974fd730 0x5653978be0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.328+0000 7f97537fe640 1 -- 172.21.15.129:0/1445430641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 msgr2=0x5653978bd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:13.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.328+0000 7f97537fe640 1 --2- 172.21.15.129:0/1445430641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978bd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.328+0000 7f97537fe640 1 -- 172.21.15.129:0/1445430641 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653975fe7e0 con 0x5653974f5550 2024-05-20T18:02:13.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.332+0000 7f9750ff9640 1 -- 172.21.15.129:0/935798389 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9748042070 con 0x7f9754105940 2024-05-20T18:02:13.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.333+0000 7f97537fe640 1 --2- 172.21.15.129:0/1445430641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974f5550 0x5653978bdb60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9744009bd0 tx=0x7f974400e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de6011a670d0c238 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:13.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.333+0000 7f9750ff9640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f973c0a2ed0 0x7f973c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.334+0000 7f9750ff9640 1 -- 172.21.15.129:0/935798389 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f97480ef560 con 0x7f9754105940 2024-05-20T18:02:13.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.335+0000 7f9753fff640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f973c0a2ed0 0x7f973c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:13.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.336+0000 7f9739ffb640 1 -- 172.21.15.129:0/1445430641 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744005020 con 0x5653974f5550 2024-05-20T18:02:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.336+0000 7f9739ffb640 1 -- 172.21.15.129:0/1445430641 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97440047a0 con 0x5653974f5550 2024-05-20T18:02:13.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.336+0000 7f9739ffb640 1 -- 172.21.15.129:0/1445430641 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744010690 con 0x5653974f5550 2024-05-20T18:02:13.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.338+0000 7f9760a1d740 1 -- 172.21.15.129:0/1445430641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974f5550 msgr2=0x5653978bdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:13.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.338+0000 7f9760a1d740 1 --2- 172.21.15.129:0/1445430641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974f5550 0x5653978bdb60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9744009bd0 tx=0x7f974400e720 comp rx=0 tx=0).stop 2024-05-20T18:02:13.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.339+0000 7f9753fff640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f973c0a2ed0 0x7f973c0a53e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f974c005c40 tx=0x7f974c009460 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:13.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.339+0000 7f9760a1d740 1 -- 172.21.15.129:0/1445430641 shutdown_connections 2024-05-20T18:02:13.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.339+0000 7f9760a1d740 1 --2- 172.21.15.129:0/1445430641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974f5550 0x5653978bdb60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.339+0000 7f9760a1d740 1 --2- 172.21.15.129:0/1445430641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974fd730 0x5653978be0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.339+0000 7f9760a1d740 1 --2- 172.21.15.129:0/1445430641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978bd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.339+0000 7f9760a1d740 1 -- 172.21.15.129:0/1445430641 >> 172.21.15.129:0/1445430641 conn(0x565397512560 msgr2=0x5653978b9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.344+0000 7f9760a1d740 1 -- 172.21.15.129:0/1445430641 shutdown_connections 2024-05-20T18:02:13.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.347+0000 7f9760a1d740 1 -- 172.21.15.129:0/1445430641 wait complete. 2024-05-20T18:02:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.348+0000 7f9760a1d740 1 -- start start 2024-05-20T18:02:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.349+0000 7f9760a1d740 1 Processor -- start 2024-05-20T18:02:13.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.350+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974f5550 0x5653978c9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.350+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974fd730 0x5653978c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.350+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.350+0000 7f9760a1d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5653978ca970 con 0x56539751e410 2024-05-20T18:02:13.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.350+0000 7f9760a1d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5653977ddb40 con 0x5653974fd730 2024-05-20T18:02:13.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.350+0000 7f9760a1d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5653978caaf0 con 0x5653974f5550 2024-05-20T18:02:13.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.350+0000 7f9760a1d740 10 client.0 _ll_register_callbacks cb 0x5653977d20f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:13.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.350+0000 7f9752ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974fd730 0x5653978c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:13.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.350+0000 7f9752ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974fd730 0x5653978c9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:56946/0 (socket says 172.21.15.129:56946) 2024-05-20T18:02:13.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.350+0000 7f9752ffd640 1 -- 172.21.15.129:0/2964620968 learned_addr learned my addr 172.21.15.129:0/2964620968 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:13.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.350+0000 7f97537fe640 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974f5550 0x5653978c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:13.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.353+0000 7f9753fff640 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:13.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.358+0000 7f97537fe640 1 -- 172.21.15.129:0/2964620968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974fd730 msgr2=0x5653978c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:13.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.358+0000 7f97537fe640 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974fd730 0x5653978c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.358+0000 7f97537fe640 1 -- 172.21.15.129:0/2964620968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 msgr2=0x5653978c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:13.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.358+0000 7f97537fe640 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.358+0000 7f97537fe640 1 -- 172.21.15.129:0/2964620968 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56539765ef70 con 0x5653974f5550 2024-05-20T18:02:13.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.371+0000 7f97537fe640 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974f5550 0x5653978c9720 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x5653978bebe0 tx=0x7f9744009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d6441900c89ae23 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:13.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.373+0000 7f9732ffd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:02:13.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.376+0000 7f9732ffd640 1 -- 172.21.15.129:0/2964620968 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974401d040 con 0x5653974f5550 2024-05-20T18:02:13.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.376+0000 7f9760a1d740 1 -- 172.21.15.129:0/2964620968 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97440047a0 con 0x5653974f5550 2024-05-20T18:02:13.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.376+0000 7f9732ffd640 1 -- 172.21.15.129:0/2964620968 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f9744005020 con 0x5653974f5550 2024-05-20T18:02:13.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.384+0000 7f9732ffd640 1 -- 172.21.15.129:0/2964620968 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f974402f050 con 0x5653974f5550 2024-05-20T18:02:13.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.384+0000 7f9760a1d740 10 client.25738 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:13.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.384+0000 7f9760a1d740 1 -- 172.21.15.129:0/2964620968 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5653974f6170 con 0x5653974f5550 2024-05-20T18:02:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.387+0000 7f9732ffd640 1 -- 172.21.15.129:0/2964620968 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97440047a0 con 0x5653974f5550 2024-05-20T18:02:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.387+0000 7f9732ffd640 1 -- 172.21.15.129:0/2964620968 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f9744018b00 con 0x5653974f5550 2024-05-20T18:02:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.387+0000 7f9760a1d740 10 client.25738 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.387+0000 7f9760a1d740 10 client.25738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.387+0000 7f9760a1d740 20 client.25738 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.388+0000 7f9760a1d740 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5653974dd1f0 0x5653978eb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.388+0000 7f9760a1d740 4 client.25738 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:13.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.388+0000 7f9760a1d740 1 -- 172.21.15.129:0/2964620968 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653978300c0 con 0x5653974dd1f0 2024-05-20T18:02:13.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.390+0000 7f9752ffd640 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5653974dd1f0 0x5653978eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:13.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.401+0000 7f9752ffd640 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5653974dd1f0 0x5653978eb560 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74563a4cc0c400d6 server_cookie=7ded0245aeae43a6 in_seq=0 out_seq=0 2024-05-20T18:02:13.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.401+0000 7f9732ffd640 10 client.25738 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.404+0000 7f9732ffd640 1 -- 172.21.15.129:0/2964620968 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5653978300c0 con 0x5653974dd1f0 2024-05-20T18:02:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.404+0000 7f9732ffd640 10 client.25738 handle_command_reply: tid=0 2024-05-20T18:02:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.404+0000 7f9760a1d740 1 client.25738 shutdown 2024-05-20T18:02:13.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.405+0000 7f9760a1d740 1 -- 172.21.15.129:0/2964620968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974f5550 msgr2=0x5653978c9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:13.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.405+0000 7f9760a1d740 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974f5550 0x5653978c9720 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x5653978bebe0 tx=0x7f9744009ef0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.406+0000 7f9760a1d740 1 -- 172.21.15.129:0/2964620968 shutdown_connections 2024-05-20T18:02:13.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.406+0000 7f9760a1d740 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5653974dd1f0 0x5653978eb560 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.406+0000 7f9760a1d740 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974fd730 0x5653978c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.406+0000 7f9760a1d740 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.406+0000 7f9760a1d740 1 --2- 172.21.15.129:0/2964620968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974f5550 0x5653978c9720 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x5653978bebe0 tx=0x7f9744009ef0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.406+0000 7f9760a1d740 1 -- 172.21.15.129:0/2964620968 >> 172.21.15.129:0/2964620968 conn(0x565397512560 msgr2=0x5653978bf340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:13.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.406+0000 7f9732ffd640 0 client.25738 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:13.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.406+0000 7f9760a1d740 1 -- 172.21.15.129:0/2964620968 shutdown_connections 2024-05-20T18:02:13.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.408+0000 7f9760a1d740 1 -- 172.21.15.129:0/2964620968 wait complete. 2024-05-20T18:02:13.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.408+0000 7f9760a1d740 20 client.25738 trim_cache size 0 max 16384 2024-05-20T18:02:13.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.485+0000 7f9760a1d740 1 Processor -- start 2024-05-20T18:02:13.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.491+0000 7f9760a1d740 1 -- start start 2024-05-20T18:02:13.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.492+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974fd730 0x5653978c12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.492+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978c1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.492+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974b2630 0x5653978c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.492+0000 7f9760a1d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9754198940 con 0x56539751e410 2024-05-20T18:02:13.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.492+0000 7f9760a1d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97541ad230 con 0x5653974fd730 2024-05-20T18:02:13.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.492+0000 7f9760a1d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9754072560 con 0x5653974b2630 2024-05-20T18:02:13.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.492+0000 7f9753fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974b2630 0x5653978c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:13.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.492+0000 7f9753fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974b2630 0x5653978c4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:57968/0 (socket says 172.21.15.129:57968) 2024-05-20T18:02:13.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.492+0000 7f9753fff640 1 -- 172.21.15.129:0/430743660 learned_addr learned my addr 172.21.15.129:0/430743660 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:13.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.493+0000 7f9753fff640 1 -- 172.21.15.129:0/430743660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974fd730 msgr2=0x5653978c12d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:02:13.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.493+0000 7f9753fff640 1 --2- 172.21.15.129:0/430743660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974fd730 0x5653978c12d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.493+0000 7f9753fff640 1 -- 172.21.15.129:0/430743660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 msgr2=0x5653978c1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:13.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.493+0000 7f9753fff640 1 --2- 172.21.15.129:0/430743660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978c1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.493+0000 7f9753fff640 1 -- 172.21.15.129:0/430743660 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653978ba9c0 con 0x5653974b2630 2024-05-20T18:02:13.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.494+0000 7f9753fff640 1 --2- 172.21.15.129:0/430743660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974b2630 0x5653978c4de0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f974c012610 tx=0x7f974c013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=135d9c7c0023055c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:13.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.499+0000 7f97397fa640 1 -- 172.21.15.129:0/430743660 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c003b70 con 0x5653974b2630 2024-05-20T18:02:13.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.504+0000 7f97397fa640 1 -- 172.21.15.129:0/430743660 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f974c024030 con 0x5653974b2630 2024-05-20T18:02:13.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.504+0000 7f97397fa640 1 -- 172.21.15.129:0/430743660 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c0119c0 con 0x5653974b2630 2024-05-20T18:02:13.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.506+0000 7f9760a1d740 1 -- 172.21.15.129:0/430743660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974b2630 msgr2=0x5653978c4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:13.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.506+0000 7f9760a1d740 1 --2- 172.21.15.129:0/430743660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974b2630 0x5653978c4de0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f974c012610 tx=0x7f974c013000 comp rx=0 tx=0).stop 2024-05-20T18:02:13.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.509+0000 7f9760a1d740 1 -- 172.21.15.129:0/430743660 shutdown_connections 2024-05-20T18:02:13.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.509+0000 7f9760a1d740 1 --2- 172.21.15.129:0/430743660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974b2630 0x5653978c4de0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.509+0000 7f9760a1d740 1 --2- 172.21.15.129:0/430743660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978c1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.509+0000 7f9760a1d740 1 --2- 172.21.15.129:0/430743660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974fd730 0x5653978c12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.509+0000 7f9760a1d740 1 -- 172.21.15.129:0/430743660 >> 172.21.15.129:0/430743660 conn(0x565397512560 msgr2=0x5653978c7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:13.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.520+0000 7f9760a1d740 1 -- 172.21.15.129:0/430743660 shutdown_connections 2024-05-20T18:02:13.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.520+0000 7f9760a1d740 1 -- 172.21.15.129:0/430743660 wait complete. 2024-05-20T18:02:13.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.520+0000 7f9760a1d740 1 -- start start 2024-05-20T18:02:13.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.521+0000 7f9760a1d740 1 Processor -- start 2024-05-20T18:02:13.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.522+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974b2630 0x5653978cc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.522+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974fd730 0x5653978ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.522+0000 7f9760a1d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978cd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.522+0000 7f9760a1d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9754072c30 con 0x56539751e410 2024-05-20T18:02:13.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.522+0000 7f9760a1d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97541acb60 con 0x5653974b2630 2024-05-20T18:02:13.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.522+0000 7f9760a1d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9754073300 con 0x5653974fd730 2024-05-20T18:02:13.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.522+0000 7f9760a1d740 10 client.0 _ll_register_callbacks cb 0x7f9744009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:13.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.523+0000 7f9752ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974fd730 0x5653978ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:13.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.523+0000 7f9752ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974fd730 0x5653978ccec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:57974/0 (socket says 172.21.15.129:57974) 2024-05-20T18:02:13.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.523+0000 7f9752ffd640 1 -- 172.21.15.129:0/1514990074 learned_addr learned my addr 172.21.15.129:0/1514990074 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:13.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.524+0000 7f9752ffd640 1 -- 172.21.15.129:0/1514990074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974b2630 msgr2=0x5653978cc980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:02:13.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.524+0000 7f9752ffd640 1 --2- 172.21.15.129:0/1514990074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974b2630 0x5653978cc980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.524+0000 7f9752ffd640 1 -- 172.21.15.129:0/1514990074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 msgr2=0x5653978cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:13.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.524+0000 7f9752ffd640 1 --2- 172.21.15.129:0/1514990074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.524+0000 7f9752ffd640 1 -- 172.21.15.129:0/1514990074 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653979518d0 con 0x5653974fd730 2024-05-20T18:02:13.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.530+0000 7f9752ffd640 1 --2- 172.21.15.129:0/1514990074 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974fd730 0x5653978ccec0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9748003fa0 tx=0x7f97480bb430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22b240e73ebd9ea9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:13.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.531+0000 7f9732ffd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:02:13.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.531+0000 7f9732ffd640 1 -- 172.21.15.129:0/1514990074 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97480bba10 con 0x5653974fd730 2024-05-20T18:02:13.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.532+0000 7f9760a1d740 1 -- 172.21.15.129:0/1514990074 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f974c024030 con 0x5653974fd730 2024-05-20T18:02:13.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.532+0000 7f9732ffd640 1 -- 172.21.15.129:0/1514990074 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f97480b8080 con 0x5653974fd730 2024-05-20T18:02:13.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.532+0000 7f9732ffd640 1 -- 172.21.15.129:0/1514990074 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f97480bfdc0 con 0x5653974fd730 2024-05-20T18:02:13.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.533+0000 7f9760a1d740 10 client.25744 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:13.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.533+0000 7f9760a1d740 1 -- 172.21.15.129:0/1514990074 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x565397952b50 con 0x5653974fd730 2024-05-20T18:02:13.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.537+0000 7f9732ffd640 1 -- 172.21.15.129:0/1514990074 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f974c024030 con 0x5653974fd730 2024-05-20T18:02:13.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.537+0000 7f9732ffd640 1 -- 172.21.15.129:0/1514990074 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f974800b040 con 0x5653974fd730 2024-05-20T18:02:13.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.537+0000 7f9760a1d740 10 client.25744 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:13.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.537+0000 7f9760a1d740 10 client.25744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:13.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.537+0000 7f9760a1d740 20 client.25744 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:13.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.537+0000 7f9760a1d740 1 --2- 172.21.15.129:0/1514990074 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5653979532b0 0x5653979556d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:13.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.537+0000 7f9760a1d740 4 client.25744 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:02:13.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.537+0000 7f9760a1d740 1 -- 172.21.15.129:0/1514990074 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56539782a410 con 0x5653979532b0 2024-05-20T18:02:13.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.542+0000 7f97537fe640 1 --2- 172.21.15.129:0/1514990074 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5653979532b0 0x5653979556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:13.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.563+0000 7f9732ffd640 10 client.25744 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:13.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.563+0000 7f97537fe640 1 --2- 172.21.15.129:0/1514990074 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5653979532b0 0x5653979556d0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44de7970c074f942 server_cookie=13f44ecac8f0fdcd in_seq=0 out_seq=0 2024-05-20T18:02:13.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.571+0000 7f9732ffd640 1 -- 172.21.15.129:0/1514990074 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56539782a410 con 0x5653979532b0 2024-05-20T18:02:13.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.571+0000 7f9732ffd640 10 client.25744 handle_command_reply: tid=0 2024-05-20T18:02:13.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.571+0000 7f9760a1d740 1 client.25744 shutdown 2024-05-20T18:02:13.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.576+0000 7f9760a1d740 1 -- 172.21.15.129:0/1514990074 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974fd730 msgr2=0x5653978ccec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:13.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.576+0000 7f9760a1d740 1 --2- 172.21.15.129:0/1514990074 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974fd730 0x5653978ccec0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9748003fa0 tx=0x7f97480bb430 comp rx=0 tx=0).stop 2024-05-20T18:02:13.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.576+0000 7f9760a1d740 1 -- 172.21.15.129:0/1514990074 shutdown_connections 2024-05-20T18:02:13.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.576+0000 7f9760a1d740 1 --2- 172.21.15.129:0/1514990074 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5653979532b0 0x5653979556d0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.576+0000 7f9760a1d740 1 --2- 172.21.15.129:0/1514990074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56539751e410 0x5653978cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.576+0000 7f9760a1d740 1 --2- 172.21.15.129:0/1514990074 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5653974fd730 0x5653978ccec0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.576+0000 7f9760a1d740 1 --2- 172.21.15.129:0/1514990074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653974b2630 0x5653978cc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.576+0000 7f9760a1d740 1 -- 172.21.15.129:0/1514990074 >> 172.21.15.129:0/1514990074 conn(0x565397512560 msgr2=0x5653978c7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:13.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.586+0000 7f9732ffd640 0 client.25744 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:13.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.588+0000 7f9760a1d740 1 -- 172.21.15.129:0/1514990074 shutdown_connections 2024-05-20T18:02:13.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.588+0000 7f9760a1d740 1 -- 172.21.15.129:0/1514990074 wait complete. 2024-05-20T18:02:13.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.588+0000 7f9760a1d740 20 client.25744 trim_cache size 0 max 16384 2024-05-20T18:02:13.587 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:02:13.587 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:02:13.587 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "d2826b22-dca1-45ee-acfa-c6584d3f4b94", 2024-05-20T18:02:13.587 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:02:13.587 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:02:13.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.593+0000 7f9759a99640 1 -- 172.21.15.129:0/935798389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f973c0a2ed0 msgr2=0x7f973c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:13.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.594+0000 7f9759a99640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f973c0a2ed0 0x7f973c0a53e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f974c005c40 tx=0x7f974c009460 comp rx=0 tx=0).stop 2024-05-20T18:02:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.594+0000 7f9759a99640 1 -- 172.21.15.129:0/935798389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9754105940 msgr2=0x7f9754197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.594+0000 7f9759a99640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9754105940 0x7f9754197950 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f974802bc40 tx=0x7f9748030c60 comp rx=0 tx=0).stop 2024-05-20T18:02:13.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.595+0000 7f9759a99640 1 -- 172.21.15.129:0/935798389 shutdown_connections 2024-05-20T18:02:13.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.595+0000 7f9759a99640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f973c0a2ed0 0x7f973c0a53e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.595+0000 7f9759a99640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9754105940 0x7f9754197950 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.596+0000 7f9759a99640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9754106400 0x7f9754197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.596+0000 7f9759a99640 1 --2- 172.21.15.129:0/935798389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97541013b0 0x7f9754197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:13.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.596+0000 7f9759a99640 1 -- 172.21.15.129:0/935798389 >> 172.21.15.129:0/935798389 conn(0x7f9754076830 msgr2=0x7f97540ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:13.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.596+0000 7f9759a99640 1 -- 172.21.15.129:0/935798389 shutdown_connections 2024-05-20T18:02:13.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:13.597+0000 7f9759a99640 1 -- 172.21.15.129:0/935798389 wait complete. 2024-05-20T18:02:13.668 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:02:13.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:02:13.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:13 smithi154 ceph-mon[36482]: pgmap v113: 97 pgs: 97 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 709 MiB/s wr, 2.88k op/s 2024-05-20T18:02:13.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:13 smithi129 ceph-mon[32029]: pgmap v113: 97 pgs: 97 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 709 MiB/s wr, 2.88k op/s 2024-05-20T18:02:13.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:13 smithi204 ceph-mon[37076]: pgmap v113: 97 pgs: 97 active+clean; 22 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 709 MiB/s wr, 2.88k op/s 2024-05-20T18:02:14.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.020+0000 7f5ad2022640 1 -- 172.21.15.129:0/2729349636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5acc109660 msgr2=0x7f5acc115a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:14.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.020+0000 7f5ad2022640 1 --2- 172.21.15.129:0/2729349636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5acc109660 0x7f5acc115a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009ca0 tx=0x7f5ac402a8e0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.022+0000 7f5ad2022640 1 -- 172.21.15.129:0/2729349636 shutdown_connections 2024-05-20T18:02:14.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.022+0000 7f5ad2022640 1 --2- 172.21.15.129:0/2729349636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5acc109660 0x7f5acc115a30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.022+0000 7f5ad2022640 1 --2- 172.21.15.129:0/2729349636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5acc10f3b0 0x7f5acc108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.022+0000 7f5ad2022640 1 --2- 172.21.15.129:0/2729349636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5acc10ef10 0x7f5acc108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.022+0000 7f5ad2022640 1 -- 172.21.15.129:0/2729349636 >> 172.21.15.129:0/2729349636 conn(0x7f5acc075b10 msgr2=0x7f5acc075f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:14.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.027+0000 7f5ad2022640 1 -- 172.21.15.129:0/2729349636 shutdown_connections 2024-05-20T18:02:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.031+0000 7f5ad2022640 1 -- 172.21.15.129:0/2729349636 wait complete. 2024-05-20T18:02:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.032+0000 7f5ad2022640 1 Processor -- start 2024-05-20T18:02:14.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.035+0000 7f5ad2022640 1 -- start start 2024-05-20T18:02:14.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.035+0000 7f5ad2022640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5acc109660 0x7f5acc0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.035+0000 7f5ad2022640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5acc10ef10 0x7f5acc06c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.035+0000 7f5ad2022640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5acc10f3b0 0x7f5acc06ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.035+0000 7f5ad2022640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc1192d0 con 0x7f5acc10f3b0 2024-05-20T18:02:14.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.035+0000 7f5ad2022640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc119150 con 0x7f5acc109660 2024-05-20T18:02:14.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.035+0000 7f5ad2022640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc119450 con 0x7f5acc10ef10 2024-05-20T18:02:14.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.036+0000 7f5ad081f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5acc10ef10 0x7f5acc06c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.036+0000 7f5ad081f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5acc10ef10 0x7f5acc06c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:57986/0 (socket says 172.21.15.129:57986) 2024-05-20T18:02:14.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.036+0000 7f5ad081f640 1 -- 172.21.15.129:0/3824707294 learned_addr learned my addr 172.21.15.129:0/3824707294 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:14.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.036+0000 7f5ad1020640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5acc109660 0x7f5acc0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.036+0000 7f5ad081f640 1 -- 172.21.15.129:0/3824707294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5acc109660 msgr2=0x7f5acc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:14.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.036+0000 7f5ad081f640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5acc109660 0x7f5acc0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.036+0000 7f5ad081f640 1 -- 172.21.15.129:0/3824707294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5acc10f3b0 msgr2=0x7f5acc06ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:14.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.036+0000 7f5ad081f640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5acc10f3b0 0x7f5acc06ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.036+0000 7f5ad081f640 1 -- 172.21.15.129:0/3824707294 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5acc06d640 con 0x7f5acc10ef10 2024-05-20T18:02:14.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.036+0000 7f5ad081f640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5acc10ef10 0x7f5acc06c860 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800b450 tx=0x7f5ac800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:14.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.043+0000 7f5ac27fc640 1 -- 172.21.15.129:0/3824707294 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8015020 con 0x7f5acc10ef10 2024-05-20T18:02:14.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.043+0000 7f5ac27fc640 1 -- 172.21.15.129:0/3824707294 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac8011d70 con 0x7f5acc10ef10 2024-05-20T18:02:14.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.043+0000 7f5ac27fc640 1 -- 172.21.15.129:0/3824707294 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac800f340 con 0x7f5acc10ef10 2024-05-20T18:02:14.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.044+0000 7f5ad2022640 1 -- 172.21.15.129:0/3824707294 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5acc0712d0 con 0x7f5acc10ef10 2024-05-20T18:02:14.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.044+0000 7f5ad2022640 1 -- 172.21.15.129:0/3824707294 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5acc0717d0 con 0x7f5acc10ef10 2024-05-20T18:02:14.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.052+0000 7f5ac27fc640 1 -- 172.21.15.129:0/3824707294 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5ac8018020 con 0x7f5acc10ef10 2024-05-20T18:02:14.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.056+0000 7f5ada20a740 1 Processor -- start 2024-05-20T18:02:14.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.056+0000 7f5ada20a740 1 -- start start 2024-05-20T18:02:14.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.056+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x559679433f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.056+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x559679434470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.057+0000 7f5ad1821640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x559679433f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.058+0000 7f5ad1821640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x559679433f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33538/0 (socket says 172.21.15.129:33538) 2024-05-20T18:02:14.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.063+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x5596794349b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.063+0000 7f5ada20a740 1 -- 172.21.15.129:0/412330743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559679399d00 con 0x559679095390 2024-05-20T18:02:14.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.063+0000 7f5ada20a740 1 -- 172.21.15.129:0/412330743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5596793414c0 con 0x55967906c4d0 2024-05-20T18:02:14.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.063+0000 7f5ada20a740 1 -- 172.21.15.129:0/412330743 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x559679395cc0 con 0x5596790746b0 2024-05-20T18:02:14.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.063+0000 7f5ad1821640 1 -- 172.21.15.129:0/412330743 learned_addr learned my addr 172.21.15.129:0/412330743 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:14.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.064+0000 7f5ad081f640 1 --2- 172.21.15.129:0/412330743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x5596794349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.064+0000 7f5ad1020640 1 --2- 172.21.15.129:0/412330743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x559679434470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.065+0000 7f5ad1821640 1 -- 172.21.15.129:0/412330743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 msgr2=0x5596794349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:14.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.065+0000 7f5ad1821640 1 --2- 172.21.15.129:0/412330743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x5596794349b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.065+0000 7f5ad1821640 1 -- 172.21.15.129:0/412330743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 msgr2=0x559679434470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.065+0000 7f5ad1821640 1 --2- 172.21.15.129:0/412330743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x559679434470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.065+0000 7f5ad1821640 1 -- 172.21.15.129:0/412330743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596791777d0 con 0x559679095390 2024-05-20T18:02:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.065+0000 7f5ad081f640 1 --2- 172.21.15.129:0/412330743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x5596794349b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:02:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.065+0000 7f5ac27fc640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ab80a2f00 0x7f5ab80a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.066+0000 7f5ac27fc640 1 -- 172.21.15.129:0/3824707294 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5ac80cecf0 con 0x7f5acc10ef10 2024-05-20T18:02:14.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.068+0000 7f5ad1821640 1 --2- 172.21.15.129:0/412330743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x559679433f30 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009890 tx=0x7f5ac40098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5a5076534bab5b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:14.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.070+0000 7f5ad1020640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ab80a2f00 0x7f5ab80a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.070+0000 7f5ab77fe640 1 -- 172.21.15.129:0/412330743 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4031670 con 0x559679095390 2024-05-20T18:02:14.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.071+0000 7f5ab77fe640 1 -- 172.21.15.129:0/412330743 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac4036040 con 0x559679095390 2024-05-20T18:02:14.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.072+0000 7f5ada20a740 1 -- 172.21.15.129:0/412330743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 msgr2=0x559679433f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:14.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.072+0000 7f5ada20a740 1 --2- 172.21.15.129:0/412330743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x559679433f30 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009890 tx=0x7f5ac40098c0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.072+0000 7f5ada20a740 1 -- 172.21.15.129:0/412330743 shutdown_connections 2024-05-20T18:02:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.072+0000 7f5ada20a740 1 --2- 172.21.15.129:0/412330743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x559679434470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.072+0000 7f5ada20a740 1 --2- 172.21.15.129:0/412330743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x5596794349b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.072+0000 7f5ada20a740 1 --2- 172.21.15.129:0/412330743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x559679433f30 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.072+0000 7f5ada20a740 1 -- 172.21.15.129:0/412330743 >> 172.21.15.129:0/412330743 conn(0x5596790894e0 msgr2=0x5596794306e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:14.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.074+0000 7f5ad1020640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ab80a2f00 0x7f5ab80a5410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5596794354f0 tx=0x7f5abc005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:14.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.077+0000 7f5ada20a740 1 -- 172.21.15.129:0/412330743 shutdown_connections 2024-05-20T18:02:14.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.080+0000 7f5ada20a740 1 -- 172.21.15.129:0/412330743 wait complete. 2024-05-20T18:02:14.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.080+0000 7f5ada20a740 1 -- start start 2024-05-20T18:02:14.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.080+0000 7f5ada20a740 1 Processor -- start 2024-05-20T18:02:14.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.081+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967943e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.081+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x55967943eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.081+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x5596794447b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.081+0000 7f5ada20a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559679441270 con 0x559679095390 2024-05-20T18:02:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.081+0000 7f5ada20a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559679354b90 con 0x55967906c4d0 2024-05-20T18:02:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.081+0000 7f5ada20a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5596794413f0 con 0x5596790746b0 2024-05-20T18:02:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.081+0000 7f5ada20a740 10 client.0 _ll_register_callbacks cb 0x559679390da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.082+0000 7f5ad1020640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967943e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.082+0000 7f5ad1020640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967943e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:56996/0 (socket says 172.21.15.129:56996) 2024-05-20T18:02:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.082+0000 7f5ad1020640 1 -- 172.21.15.129:0/2261970291 learned_addr learned my addr 172.21.15.129:0/2261970291 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.083+0000 7f5ad1020640 1 -- 172.21.15.129:0/2261970291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 msgr2=0x55967943eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.083+0000 7f5ad081f640 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x55967943eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.083+0000 7f5ad1821640 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x5596794447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.083+0000 7f5ad1020640 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x55967943eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.083+0000 7f5ad1020640 1 -- 172.21.15.129:0/2261970291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 msgr2=0x5596794447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:14.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.083+0000 7f5ad1020640 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x5596794447b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.083+0000 7f5ad1020640 1 -- 172.21.15.129:0/2261970291 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596790ad3f0 con 0x55967906c4d0 2024-05-20T18:02:14.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.083+0000 7f5ad1821640 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x5596794447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:02:14.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.084+0000 7f5ad1020640 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967943e5f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009fd0 tx=0x7f5abc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c109fab5e830af76 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:14.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.085+0000 7f5ab4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:02:14.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.085+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/2261970291 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc005220 con 0x55967906c4d0 2024-05-20T18:02:14.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.085+0000 7f5ada20a740 1 -- 172.21.15.129:0/2261970291 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ac4036040 con 0x55967906c4d0 2024-05-20T18:02:14.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.085+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/2261970291 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5abc0053c0 con 0x55967906c4d0 2024-05-20T18:02:14.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.086+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/2261970291 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5abc0055c0 con 0x55967906c4d0 2024-05-20T18:02:14.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.086+0000 7f5ada20a740 10 client.25730 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:14.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.086+0000 7f5ada20a740 1 -- 172.21.15.129:0/2261970291 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55967906d0f0 con 0x55967906c4d0 2024-05-20T18:02:14.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.088+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/2261970291 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac4036040 con 0x55967906c4d0 2024-05-20T18:02:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.089+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/2261970291 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f5abc028400 con 0x55967906c4d0 2024-05-20T18:02:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.089+0000 7f5ada20a740 10 client.25730 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.089+0000 7f5ada20a740 10 client.25730 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.089+0000 7f5ada20a740 20 client.25730 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.089+0000 7f5ada20a740 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559679053f60 0x559679450f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.089+0000 7f5ada20a740 4 client.25730 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.089+0000 7f5ada20a740 1 -- 172.21.15.129:0/2261970291 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596793a7430 con 0x559679053f60 2024-05-20T18:02:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.089+0000 7f5ad081f640 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559679053f60 0x559679450f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.090+0000 7f5ab4ff9640 10 client.25730 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.090+0000 7f5ad081f640 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559679053f60 0x559679450f40 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=816bb52786d7e437 server_cookie=73f885d13252a243 in_seq=0 out_seq=0 2024-05-20T18:02:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.094+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/2261970291 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5596793a7430 con 0x559679053f60 2024-05-20T18:02:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.094+0000 7f5ab4ff9640 10 client.25730 handle_command_reply: tid=0 2024-05-20T18:02:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.094+0000 7f5ada20a740 1 client.25730 shutdown 2024-05-20T18:02:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.094+0000 7f5ada20a740 1 -- 172.21.15.129:0/2261970291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 msgr2=0x55967943e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:14.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.094+0000 7f5ada20a740 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967943e5f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009fd0 tx=0x7f5abc0046c0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.095+0000 7f5ada20a740 1 -- 172.21.15.129:0/2261970291 shutdown_connections 2024-05-20T18:02:14.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.095+0000 7f5ada20a740 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559679053f60 0x559679450f40 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.095+0000 7f5ada20a740 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x5596794447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.095+0000 7f5ada20a740 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x55967943eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.095+0000 7f5ada20a740 1 --2- 172.21.15.129:0/2261970291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967943e5f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.095+0000 7f5ada20a740 1 -- 172.21.15.129:0/2261970291 >> 172.21.15.129:0/2261970291 conn(0x5596790894e0 msgr2=0x559679433d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.099+0000 7f5ab4ff9640 0 client.25730 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:14.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.101+0000 7f5ada20a740 1 -- 172.21.15.129:0/2261970291 shutdown_connections 2024-05-20T18:02:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.102+0000 7f5ada20a740 1 -- 172.21.15.129:0/2261970291 wait complete. 2024-05-20T18:02:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.103+0000 7f5ada20a740 20 client.25730 trim_cache size 0 max 16384 2024-05-20T18:02:14.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.155+0000 7f5ada20a740 1 Processor -- start 2024-05-20T18:02:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.158+0000 7f5ada20a740 1 -- start start 2024-05-20T18:02:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.158+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967943d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.158+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596790746b0 0x559679456700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.158+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559679095390 0x559679456c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.158+0000 7f5ada20a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc1c4070 con 0x5596790746b0 2024-05-20T18:02:14.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.158+0000 7f5ada20a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc1c68f0 con 0x55967906c4d0 2024-05-20T18:02:14.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.158+0000 7f5ada20a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc1c46e0 con 0x559679095390 2024-05-20T18:02:14.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.160+0000 7f5ad1020640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967943d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.160+0000 7f5ad081f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596790746b0 0x559679456700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.160+0000 7f5ad081f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596790746b0 0x559679456700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33554/0 (socket says 172.21.15.129:33554) 2024-05-20T18:02:14.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.160+0000 7f5ad081f640 1 -- 172.21.15.129:0/4279969560 learned_addr learned my addr 172.21.15.129:0/4279969560 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:14.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.166+0000 7f5ad081f640 1 -- 172.21.15.129:0/4279969560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559679095390 msgr2=0x559679456c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:02:14.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.166+0000 7f5ad081f640 1 --2- 172.21.15.129:0/4279969560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559679095390 0x559679456c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.166+0000 7f5ad081f640 1 -- 172.21.15.129:0/4279969560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 msgr2=0x55967943d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:14.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.166+0000 7f5ad081f640 1 --2- 172.21.15.129:0/4279969560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967943d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.166+0000 7f5ad081f640 1 -- 172.21.15.129:0/4279969560 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559679435520 con 0x5596790746b0 2024-05-20T18:02:14.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.175+0000 7f5ad081f640 1 --2- 172.21.15.129:0/4279969560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596790746b0 0x559679456700 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80109b0 tx=0x7f5ac80d10c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c4a57877862cea server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:14.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.188+0000 7f5ab6ffd640 1 -- 172.21.15.129:0/4279969560 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac80aac70 con 0x5596790746b0 2024-05-20T18:02:14.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.188+0000 7f5ab6ffd640 1 -- 172.21.15.129:0/4279969560 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac809b9b0 con 0x5596790746b0 2024-05-20T18:02:14.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.188+0000 7f5ab6ffd640 1 -- 172.21.15.129:0/4279969560 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8010570 con 0x5596790746b0 2024-05-20T18:02:14.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.189+0000 7f5ada20a740 1 -- 172.21.15.129:0/4279969560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596790746b0 msgr2=0x559679456700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:14.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.189+0000 7f5ada20a740 1 --2- 172.21.15.129:0/4279969560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596790746b0 0x559679456700 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5ac80109b0 tx=0x7f5ac80d10c0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.191+0000 7f5ada20a740 1 -- 172.21.15.129:0/4279969560 shutdown_connections 2024-05-20T18:02:14.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.191+0000 7f5ada20a740 1 --2- 172.21.15.129:0/4279969560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559679095390 0x559679456c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.191+0000 7f5ada20a740 1 --2- 172.21.15.129:0/4279969560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596790746b0 0x559679456700 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.191+0000 7f5ada20a740 1 --2- 172.21.15.129:0/4279969560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967943d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.191+0000 7f5ada20a740 1 -- 172.21.15.129:0/4279969560 >> 172.21.15.129:0/4279969560 conn(0x5596790894e0 msgr2=0x55967943f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:14.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.194+0000 7f5ada20a740 1 -- 172.21.15.129:0/4279969560 shutdown_connections 2024-05-20T18:02:14.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.195+0000 7f5ada20a740 1 -- 172.21.15.129:0/4279969560 wait complete. 2024-05-20T18:02:14.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.196+0000 7f5ada20a740 1 -- start start 2024-05-20T18:02:14.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.196+0000 7f5ada20a740 1 Processor -- start 2024-05-20T18:02:14.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.199+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967944b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.199+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x559679448480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.199+0000 7f5ada20a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x5596794489c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.199+0000 7f5ada20a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc1c4db0 con 0x559679095390 2024-05-20T18:02:14.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.199+0000 7f5ada20a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc1c6220 con 0x55967906c4d0 2024-05-20T18:02:14.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.199+0000 7f5ada20a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc1c5480 con 0x5596790746b0 2024-05-20T18:02:14.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.200+0000 7f5ada20a740 10 client.0 _ll_register_callbacks cb 0x7f5abc009c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:14.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.202+0000 7f5ad1821640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x5596794489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.202+0000 7f5ad1821640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x5596794489c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33570/0 (socket says 172.21.15.129:33570) 2024-05-20T18:02:14.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.202+0000 7f5ad1821640 1 -- 172.21.15.129:0/218868599 learned_addr learned my addr 172.21.15.129:0/218868599 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:14.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.203+0000 7f5ad1020640 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967944b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.203+0000 7f5ad081f640 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x559679448480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.208+0000 7f5ad081f640 1 -- 172.21.15.129:0/218868599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 msgr2=0x55967944b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:14.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.208+0000 7f5ad081f640 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967944b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.208+0000 7f5ad081f640 1 -- 172.21.15.129:0/218868599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 msgr2=0x5596794489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:14.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.208+0000 7f5ad081f640 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x5596794489c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.208+0000 7f5ad081f640 1 -- 172.21.15.129:0/218868599 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559679449130 con 0x5596790746b0 2024-05-20T18:02:14.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.213+0000 7f5ad081f640 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x559679448480 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x559679430c70 tx=0x7f5ac8004760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11466209028344f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:14.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.213+0000 7f5ab4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:02:14.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.215+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/218868599 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac80a5060 con 0x5596790746b0 2024-05-20T18:02:14.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.215+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/218868599 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5ac80b4170 con 0x5596790746b0 2024-05-20T18:02:14.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.218+0000 7f5ada20a740 1 -- 172.21.15.129:0/218868599 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ac809b9b0 con 0x5596790746b0 2024-05-20T18:02:14.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.223+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/218868599 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5ac809ac10 con 0x5596790746b0 2024-05-20T18:02:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.227+0000 7f5ada20a740 10 client.25753 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.227+0000 7f5ada20a740 1 -- 172.21.15.129:0/218868599 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5596794b91f0 con 0x5596790746b0 2024-05-20T18:02:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.231+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/218868599 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac809b9b0 con 0x5596790746b0 2024-05-20T18:02:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.231+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/218868599 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f5ac8097920 con 0x5596790746b0 2024-05-20T18:02:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.231+0000 7f5ada20a740 10 client.25753 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.231+0000 7f5ada20a740 10 client.25753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.231+0000 7f5ada20a740 20 client.25753 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.231+0000 7f5ada20a740 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596790293a0 0x5596794bb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.231+0000 7f5ada20a740 4 client.25753 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:02:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.231+0000 7f5ada20a740 1 -- 172.21.15.129:0/218868599 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5596793a1780 con 0x5596790293a0 2024-05-20T18:02:14.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.233+0000 7f5ad1020640 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596790293a0 0x5596794bb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:14.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.236+0000 7f5ad1020640 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596790293a0 0x5596794bb920 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43cdd0fc6b70a8fd server_cookie=c9c0aef6eb3e57d6 in_seq=0 out_seq=0 2024-05-20T18:02:14.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.236+0000 7f5ab4ff9640 10 client.25753 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:14.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.238+0000 7f5ab4ff9640 1 -- 172.21.15.129:0/218868599 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5596793a1780 con 0x5596790293a0 2024-05-20T18:02:14.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.238+0000 7f5ab4ff9640 10 client.25753 handle_command_reply: tid=0 2024-05-20T18:02:14.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.239+0000 7f5ada20a740 1 client.25753 shutdown 2024-05-20T18:02:14.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.239+0000 7f5ada20a740 1 -- 172.21.15.129:0/218868599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 msgr2=0x559679448480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:14.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.239+0000 7f5ada20a740 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x559679448480 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x559679430c70 tx=0x7f5ac8004760 comp rx=0 tx=0).stop 2024-05-20T18:02:14.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.241+0000 7f5ada20a740 1 -- 172.21.15.129:0/218868599 shutdown_connections 2024-05-20T18:02:14.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.241+0000 7f5ada20a740 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596790293a0 0x5596794bb920 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.241+0000 7f5ada20a740 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559679095390 0x5596794489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.241+0000 7f5ada20a740 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596790746b0 0x559679448480 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.241+0000 7f5ada20a740 1 --2- 172.21.15.129:0/218868599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55967906c4d0 0x55967944b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.241+0000 7f5ada20a740 1 -- 172.21.15.129:0/218868599 >> 172.21.15.129:0/218868599 conn(0x5596790894e0 msgr2=0x55967943d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:14.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.241+0000 7f5ab4ff9640 0 client.25753 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:14.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.246+0000 7f5ada20a740 1 -- 172.21.15.129:0/218868599 shutdown_connections 2024-05-20T18:02:14.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.248+0000 7f5ada20a740 1 -- 172.21.15.129:0/218868599 wait complete. 2024-05-20T18:02:14.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.249+0000 7f5ada20a740 20 client.25753 trim_cache size 0 max 16384 2024-05-20T18:02:14.248 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:02:14.248 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "scrub active (0 inodes in the stack)", 2024-05-20T18:02:14.248 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:02:14.248 INFO:teuthology.orchestra.run.smithi129.stdout: "d2826b22-dca1-45ee-acfa-c6584d3f4b94": { 2024-05-20T18:02:14.248 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:02:14.249 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "d2826b22-dca1-45ee-acfa-c6584d3f4b94", 2024-05-20T18:02:14.249 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:02:14.249 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:02:14.249 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:02:14.249 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:02:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.255+0000 7f5ad2022640 1 -- 172.21.15.129:0/3824707294 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ab80a2f00 msgr2=0x7f5ab80a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.255+0000 7f5ad2022640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ab80a2f00 0x7f5ab80a5410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5596794354f0 tx=0x7f5abc005f90 comp rx=0 tx=0).stop 2024-05-20T18:02:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.255+0000 7f5ad2022640 1 -- 172.21.15.129:0/3824707294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5acc10ef10 msgr2=0x7f5acc06c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:14.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.255+0000 7f5ad2022640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5acc10ef10 0x7f5acc06c860 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800b450 tx=0x7f5ac800b910 comp rx=0 tx=0).stop 2024-05-20T18:02:14.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.259+0000 7f5ad2022640 1 -- 172.21.15.129:0/3824707294 shutdown_connections 2024-05-20T18:02:14.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.259+0000 7f5ad2022640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ab80a2f00 0x7f5ab80a5410 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.259+0000 7f5ad2022640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5acc10f3b0 0x7f5acc06ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.259+0000 7f5ad2022640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5acc10ef10 0x7f5acc06c860 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.259+0000 7f5ad2022640 1 --2- 172.21.15.129:0/3824707294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5acc109660 0x7f5acc0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:14.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.259+0000 7f5ad2022640 1 -- 172.21.15.129:0/3824707294 >> 172.21.15.129:0/3824707294 conn(0x7f5acc075b10 msgr2=0x7f5acc107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:14.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.259+0000 7f5ad2022640 1 -- 172.21.15.129:0/3824707294 shutdown_connections 2024-05-20T18:02:14.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:14.260+0000 7f5ad2022640 1 -- 172.21.15.129:0/3824707294 wait complete. 2024-05-20T18:02:14.322 INFO:tasks.cephfs.filesystem:scrub status for tag:d2826b22-dca1-45ee-acfa-c6584d3f4b94 - {'path': '/', 'tag': 'd2826b22-dca1-45ee-acfa-c6584d3f4b94', 'options': 'recursive,force'} 2024-05-20T18:02:14.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:14 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:02:14.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:14 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:14.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:14 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:02:14.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:14 smithi129 ceph-mon[32029]: pgmap v114: 97 pgs: 97 active+clean; 22 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 706 MiB/s wr, 3.44k op/s 2024-05-20T18:02:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:14 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:02:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:14 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:14 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:02:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:14 smithi154 ceph-mon[36482]: pgmap v114: 97 pgs: 97 active+clean; 22 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 706 MiB/s wr, 3.44k op/s 2024-05-20T18:02:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:14 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:02:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:14 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:14 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:02:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:14 smithi204 ceph-mon[37076]: pgmap v114: 97 pgs: 97 active+clean; 22 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 706 MiB/s wr, 3.44k op/s 2024-05-20T18:02:17.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:16 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:17.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:16 smithi154 ceph-mon[36482]: pgmap v115: 97 pgs: 97 active+clean; 23 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 852 MiB/s wr, 5.28k op/s 2024-05-20T18:02:17.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:16 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:17.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:16 smithi129 ceph-mon[32029]: pgmap v115: 97 pgs: 97 active+clean; 23 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 852 MiB/s wr, 5.28k op/s 2024-05-20T18:02:17.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:16 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:17.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:16 smithi204 ceph-mon[37076]: pgmap v115: 97 pgs: 97 active+clean; 23 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 852 MiB/s wr, 5.28k op/s 2024-05-20T18:02:18.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.582+0000 7f2953fff640 1 -- 172.21.15.129:0/2145324781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f295410e890 msgr2=0x7f295410eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:18.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.582+0000 7f2953fff640 1 --2- 172.21.15.129:0/2145324781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f295410e890 0x7f295410eca0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f293c01a910 tx=0x7f293c03b450 comp rx=0 tx=0).stop 2024-05-20T18:02:18.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.582+0000 7f2953fff640 1 -- 172.21.15.129:0/2145324781 shutdown_connections 2024-05-20T18:02:18.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.582+0000 7f2953fff640 1 --2- 172.21.15.129:0/2145324781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2954109090 0x7f2954109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.582+0000 7f2953fff640 1 --2- 172.21.15.129:0/2145324781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295410f270 0x7f2954108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.582+0000 7f2953fff640 1 --2- 172.21.15.129:0/2145324781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f295410e890 0x7f295410eca0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.582+0000 7f2953fff640 1 -- 172.21.15.129:0/2145324781 >> 172.21.15.129:0/2145324781 conn(0x7f295406e690 msgr2=0x7f295406eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.586+0000 7f2953fff640 1 -- 172.21.15.129:0/2145324781 shutdown_connections 2024-05-20T18:02:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.587+0000 7f2953fff640 1 -- 172.21.15.129:0/2145324781 wait complete. 2024-05-20T18:02:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.589+0000 7f2953fff640 1 Processor -- start 2024-05-20T18:02:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.589+0000 7f2953fff640 1 -- start start 2024-05-20T18:02:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.590+0000 7f2953fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2954109090 0x7f29541a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.590+0000 7f2953fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f295410e890 0x7f29541a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.590+0000 7f2953fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295410f270 0x7f295419fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.590+0000 7f2953fff640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f29541132b0 con 0x7f295410f270 2024-05-20T18:02:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.590+0000 7f2953fff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2954113130 con 0x7f295410e890 2024-05-20T18:02:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.590+0000 7f2953fff640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2954113430 con 0x7f2954109090 2024-05-20T18:02:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.590+0000 7f29537fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295410f270 0x7f295419fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.590+0000 7f29537fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295410f270 0x7f295419fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33584/0 (socket says 172.21.15.129:33584) 2024-05-20T18:02:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.590+0000 7f29537fe640 1 -- 172.21.15.129:0/104873553 learned_addr learned my addr 172.21.15.129:0/104873553 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.590+0000 7f2952ffd640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2954109090 0x7f29541a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.591+0000 7f2952ffd640 1 -- 172.21.15.129:0/104873553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f295410e890 msgr2=0x7f29541a5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.591+0000 7f2952ffd640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f295410e890 0x7f29541a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.591+0000 7f2952ffd640 1 -- 172.21.15.129:0/104873553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295410f270 msgr2=0x7f295419fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.591+0000 7f2952ffd640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295410f270 0x7f295419fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.591+0000 7f2952ffd640 1 -- 172.21.15.129:0/104873553 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29541a0120 con 0x7f2954109090 2024-05-20T18:02:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.591+0000 7f2952ffd640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2954109090 0x7f29541a5830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f293c01a860 tx=0x7f293c004b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.593+0000 7f294bfff640 1 -- 172.21.15.129:0/104873553 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c04f070 con 0x7f2954109090 2024-05-20T18:02:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.593+0000 7f294bfff640 1 -- 172.21.15.129:0/104873553 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293c0409f0 con 0x7f2954109090 2024-05-20T18:02:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.595+0000 7f294bfff640 1 -- 172.21.15.129:0/104873553 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c046030 con 0x7f2954109090 2024-05-20T18:02:18.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.598+0000 7f2953fff640 1 -- 172.21.15.129:0/104873553 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29541a03b0 con 0x7f2954109090 2024-05-20T18:02:18.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.598+0000 7f2953fff640 1 -- 172.21.15.129:0/104873553 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29541a0820 con 0x7f2954109090 2024-05-20T18:02:18.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.603+0000 7f29603b6740 1 Processor -- start 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.603+0000 7f29603b6740 1 -- start start 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.603+0000 7f29603b6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f961b0 0x559977335270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.603+0000 7f29603b6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 0x5599773357b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.603+0000 7f29603b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 0x559977335cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.603+0000 7f29603b6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55997729a340 con 0x559976f961b0 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.603+0000 7f29603b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559977242290 con 0x559976f754d0 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.603+0000 7f29603b6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x559977296300 con 0x559976f6d2f0 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.604+0000 7f2952ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 0x5599773357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.604+0000 7f2952ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 0x5599773357b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:58042/0 (socket says 172.21.15.129:58042) 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.604+0000 7f2952ffd640 1 -- 172.21.15.129:0/2043803114 learned_addr learned my addr 172.21.15.129:0/2043803114 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.604+0000 7f29527fc640 1 --2- 172.21.15.129:0/2043803114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 0x559977335cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.620 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.604+0000 7f29537fe640 1 --2- 172.21.15.129:0/2043803114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f961b0 0x559977335270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.606+0000 7f29537fe640 1 -- 172.21.15.129:0/2043803114 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 msgr2=0x5599773357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:18.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.606+0000 7f29537fe640 1 --2- 172.21.15.129:0/2043803114 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 0x5599773357b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.606+0000 7f29537fe640 1 -- 172.21.15.129:0/2043803114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 msgr2=0x559977335cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:18.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.606+0000 7f29537fe640 1 --2- 172.21.15.129:0/2043803114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 0x559977335cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.606+0000 7f29537fe640 1 -- 172.21.15.129:0/2043803114 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599772668e0 con 0x559976f961b0 2024-05-20T18:02:18.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.608+0000 7f29537fe640 1 --2- 172.21.15.129:0/2043803114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f961b0 0x559977335270 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f294c0099c0 tx=0x7f294c009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796a4356f92ec0ef server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:18.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.609+0000 7f294bfff640 1 -- 172.21.15.129:0/104873553 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f293c052070 con 0x7f2954109090 2024-05-20T18:02:18.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.609+0000 7f29497fa640 1 -- 172.21.15.129:0/2043803114 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c017070 con 0x559976f961b0 2024-05-20T18:02:18.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.610+0000 7f294bfff640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29400a2d60 0x7f29400a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.610+0000 7f29497fa640 1 -- 172.21.15.129:0/2043803114 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f294c013470 con 0x559976f961b0 2024-05-20T18:02:18.621 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.611+0000 7f2952ffd640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29400a2d60 0x7f29400a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.611+0000 7f294bfff640 1 -- 172.21.15.129:0/104873553 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f293c0efbd0 con 0x7f2954109090 2024-05-20T18:02:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.612+0000 7f29603b6740 1 -- 172.21.15.129:0/2043803114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f961b0 msgr2=0x559977335270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.612+0000 7f29603b6740 1 --2- 172.21.15.129:0/2043803114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f961b0 0x559977335270 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f294c0099c0 tx=0x7f294c009e80 comp rx=0 tx=0).stop 2024-05-20T18:02:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.614+0000 7f29603b6740 1 -- 172.21.15.129:0/2043803114 shutdown_connections 2024-05-20T18:02:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.614+0000 7f29603b6740 1 --2- 172.21.15.129:0/2043803114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 0x559977335cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.614+0000 7f29603b6740 1 --2- 172.21.15.129:0/2043803114 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 0x5599773357b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.614+0000 7f29603b6740 1 --2- 172.21.15.129:0/2043803114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f961b0 0x559977335270 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.614+0000 7f29603b6740 1 -- 172.21.15.129:0/2043803114 >> 172.21.15.129:0/2043803114 conn(0x559976f8a300 msgr2=0x559977331a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.615+0000 7f2952ffd640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29400a2d60 0x7f29400a5270 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f293c0b89c0 tx=0x7f293c0dc040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.618+0000 7f29603b6740 1 -- 172.21.15.129:0/2043803114 shutdown_connections 2024-05-20T18:02:18.622 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.619+0000 7f29603b6740 1 -- 172.21.15.129:0/2043803114 wait complete. 2024-05-20T18:02:18.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.619+0000 7f29603b6740 1 -- start start 2024-05-20T18:02:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.623+0000 7f29603b6740 1 Processor -- start 2024-05-20T18:02:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.624+0000 7f29603b6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f6d2f0 0x55997733f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.624+0000 7f29603b6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f754d0 0x55997733fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.624+0000 7f29603b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 0x55997736d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.624+0000 7f29603b6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559977364450 con 0x559976f6d2f0 2024-05-20T18:02:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.624+0000 7f29603b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559977255790 con 0x559976f961b0 2024-05-20T18:02:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.624+0000 7f29603b6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5599773645d0 con 0x559976f754d0 2024-05-20T18:02:18.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.624+0000 7f29603b6740 10 client.0 _ll_register_callbacks cb 0x559977290a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.626+0000 7f29537fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 0x55997736d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.626+0000 7f29527fc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f754d0 0x55997733fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.626+0000 7f29537fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 0x55997736d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:57036/0 (socket says 172.21.15.129:57036) 2024-05-20T18:02:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.626+0000 7f29537fe640 1 -- 172.21.15.129:0/408808430 learned_addr learned my addr 172.21.15.129:0/408808430 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.627+0000 7f29537fe640 1 -- 172.21.15.129:0/408808430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f754d0 msgr2=0x55997733fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.627+0000 7f29537fe640 1 --2- 172.21.15.129:0/408808430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f754d0 0x55997733fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.627+0000 7f29537fe640 1 -- 172.21.15.129:0/408808430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f6d2f0 msgr2=0x55997733f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.627+0000 7f29537fe640 1 --2- 172.21.15.129:0/408808430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f6d2f0 0x55997733f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.627+0000 7f29537fe640 1 -- 172.21.15.129:0/408808430 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559976fae210 con 0x559976f961b0 2024-05-20T18:02:18.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.629+0000 7f29537fe640 1 --2- 172.21.15.129:0/408808430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 0x55997736d950 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f294c0047b0 tx=0x7f294c009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=110d9428e0be21ff server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:18.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.634+0000 7f293a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:02:18.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.634+0000 7f293a7fc640 1 -- 172.21.15.129:0/408808430 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c017050 con 0x559976f961b0 2024-05-20T18:02:18.633 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.634+0000 7f29603b6740 1 -- 172.21.15.129:0/408808430 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f294c013470 con 0x559976f961b0 2024-05-20T18:02:18.633 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.634+0000 7f293a7fc640 1 -- 172.21.15.129:0/408808430 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f294c004020 con 0x559976f961b0 2024-05-20T18:02:18.633 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.635+0000 7f293a7fc640 1 -- 172.21.15.129:0/408808430 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f294c004220 con 0x559976f961b0 2024-05-20T18:02:18.635 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.637+0000 7f29603b6740 10 client.25745 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:18.635 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.637+0000 7f29603b6740 1 -- 172.21.15.129:0/408808430 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55997707a620 con 0x559976f961b0 2024-05-20T18:02:18.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.637+0000 7f293a7fc640 1 -- 172.21.15.129:0/408808430 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f294c013470 con 0x559976f961b0 2024-05-20T18:02:18.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.637+0000 7f293a7fc640 1 -- 172.21.15.129:0/408808430 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f294c023070 con 0x559976f961b0 2024-05-20T18:02:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.637+0000 7f29603b6740 10 client.25745 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.638+0000 7f29603b6740 10 client.25745 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.638+0000 7f29603b6740 20 client.25745 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.638+0000 7f29603b6740 1 --2- 172.21.15.129:0/408808430 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559976f54d80 0x559977373d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.638+0000 7f29603b6740 4 client.25745 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.638+0000 7f29603b6740 1 -- 172.21.15.129:0/408808430 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599772a7a70 con 0x559976f54d80 2024-05-20T18:02:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.638+0000 7f29527fc640 1 --2- 172.21.15.129:0/408808430 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559976f54d80 0x559977373d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.639+0000 7f29527fc640 1 --2- 172.21.15.129:0/408808430 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559976f54d80 0x559977373d70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72290280fc2479a2 server_cookie=4aa7b9ed68a19a54 in_seq=0 out_seq=0 2024-05-20T18:02:18.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.639+0000 7f293a7fc640 10 client.25745 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.642+0000 7f293a7fc640 1 -- 172.21.15.129:0/408808430 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5599772a7a70 con 0x559976f54d80 2024-05-20T18:02:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.643+0000 7f293a7fc640 10 client.25745 handle_command_reply: tid=0 2024-05-20T18:02:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.643+0000 7f29603b6740 1 client.25745 shutdown 2024-05-20T18:02:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.644+0000 7f29603b6740 1 -- 172.21.15.129:0/408808430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 msgr2=0x55997736d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.644+0000 7f29603b6740 1 --2- 172.21.15.129:0/408808430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 0x55997736d950 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f294c0047b0 tx=0x7f294c009630 comp rx=0 tx=0).stop 2024-05-20T18:02:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.645+0000 7f29603b6740 1 -- 172.21.15.129:0/408808430 shutdown_connections 2024-05-20T18:02:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.645+0000 7f29603b6740 1 --2- 172.21.15.129:0/408808430 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559976f54d80 0x559977373d70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.645+0000 7f29603b6740 1 --2- 172.21.15.129:0/408808430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 0x55997736d950 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.645+0000 7f29603b6740 1 --2- 172.21.15.129:0/408808430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f754d0 0x55997733fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.645+0000 7f29603b6740 1 --2- 172.21.15.129:0/408808430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f6d2f0 0x55997733f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.645+0000 7f29603b6740 1 -- 172.21.15.129:0/408808430 >> 172.21.15.129:0/408808430 conn(0x559976f8a300 msgr2=0x559977335000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.645+0000 7f293a7fc640 0 client.25745 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:18.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.649+0000 7f29603b6740 1 -- 172.21.15.129:0/408808430 shutdown_connections 2024-05-20T18:02:18.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.649+0000 7f29603b6740 1 -- 172.21.15.129:0/408808430 wait complete. 2024-05-20T18:02:18.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.649+0000 7f29603b6740 20 client.25745 trim_cache size 0 max 16384 2024-05-20T18:02:18.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.693+0000 7f29603b6740 1 Processor -- start 2024-05-20T18:02:18.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.695+0000 7f29603b6740 1 -- start start 2024-05-20T18:02:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.696+0000 7f29603b6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 0x55997733f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.696+0000 7f29603b6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f754d0 0x559977339440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.696+0000 7f29603b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 0x559977336010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29603b6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f295410a1e0 con 0x559976f754d0 2024-05-20T18:02:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29603b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2954082da0 con 0x559976f961b0 2024-05-20T18:02:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29603b6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2954080bb0 con 0x559976f6d2f0 2024-05-20T18:02:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29537fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 0x55997733f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29537fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 0x55997733f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:58068/0 (socket says 172.21.15.129:58068) 2024-05-20T18:02:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29537fe640 1 -- 172.21.15.129:0/428164704 learned_addr learned my addr 172.21.15.129:0/428164704 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29537fe640 1 --2- 172.21.15.129:0/428164704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 0x559977336010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29537fe640 1 -- 172.21.15.129:0/428164704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 msgr2=0x559977336010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29537fe640 1 --2- 172.21.15.129:0/428164704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 0x559977336010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29537fe640 1 -- 172.21.15.129:0/428164704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f754d0 msgr2=0x559977339440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29537fe640 1 --2- 172.21.15.129:0/428164704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f754d0 0x559977339440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.697+0000 7f29537fe640 1 -- 172.21.15.129:0/428164704 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559977340510 con 0x559976f6d2f0 2024-05-20T18:02:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.702+0000 7f29537fe640 1 --2- 172.21.15.129:0/428164704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 0x55997733f200 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f294c00ef40 tx=0x7f294c009110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4bb6af4e5ebb485 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.703+0000 7f2948ff9640 1 -- 172.21.15.129:0/428164704 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c017070 con 0x559976f6d2f0 2024-05-20T18:02:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.703+0000 7f2948ff9640 1 -- 172.21.15.129:0/428164704 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f294c0043c0 con 0x559976f6d2f0 2024-05-20T18:02:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.705+0000 7f29603b6740 1 -- 172.21.15.129:0/428164704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 msgr2=0x55997733f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.705+0000 7f29603b6740 1 --2- 172.21.15.129:0/428164704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 0x55997733f200 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f294c00ef40 tx=0x7f294c009110 comp rx=0 tx=0).stop 2024-05-20T18:02:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.705+0000 7f29603b6740 1 -- 172.21.15.129:0/428164704 shutdown_connections 2024-05-20T18:02:18.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.705+0000 7f29603b6740 1 --2- 172.21.15.129:0/428164704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f961b0 0x559977336010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.705+0000 7f29603b6740 1 --2- 172.21.15.129:0/428164704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f754d0 0x559977339440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.705+0000 7f29603b6740 1 --2- 172.21.15.129:0/428164704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f6d2f0 0x55997733f200 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.705+0000 7f29603b6740 1 -- 172.21.15.129:0/428164704 >> 172.21.15.129:0/428164704 conn(0x559976f8a300 msgr2=0x559977338d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:18.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.705+0000 7f29603b6740 1 -- 172.21.15.129:0/428164704 shutdown_connections 2024-05-20T18:02:18.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.705+0000 7f29603b6740 1 -- 172.21.15.129:0/428164704 wait complete. 2024-05-20T18:02:18.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.706+0000 7f29603b6740 1 -- start start 2024-05-20T18:02:18.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.706+0000 7f29603b6740 1 Processor -- start 2024-05-20T18:02:18.705 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.707+0000 7f29603b6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 0x559977369e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.705 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.707+0000 7f29603b6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f961b0 0x559977364870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.706 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.707+0000 7f29603b6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f2a1c0 0x559977364db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.706 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.707+0000 7f29603b6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2954081260 con 0x559976f2a1c0 2024-05-20T18:02:18.706 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.707+0000 7f29603b6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f29540826d0 con 0x559976f754d0 2024-05-20T18:02:18.706 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.707+0000 7f29603b6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2954081930 con 0x559976f961b0 2024-05-20T18:02:18.706 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.707+0000 7f29603b6740 10 client.0 _ll_register_callbacks cb 0x7f294c006f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.711+0000 7f29537fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 0x559977369e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.711+0000 7f29527fc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f961b0 0x559977364870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.711+0000 7f29537fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 0x559977369e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:57060/0 (socket says 172.21.15.129:57060) 2024-05-20T18:02:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.711+0000 7f29537fe640 1 -- 172.21.15.129:0/2252147357 learned_addr learned my addr 172.21.15.129:0/2252147357 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:18.715 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.717+0000 7f29537fe640 1 -- 172.21.15.129:0/2252147357 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f961b0 msgr2=0x559977364870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:18.715 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.717+0000 7f29537fe640 1 --2- 172.21.15.129:0/2252147357 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f961b0 0x559977364870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.715 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.717+0000 7f29537fe640 1 -- 172.21.15.129:0/2252147357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f2a1c0 msgr2=0x559977364db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:18.715 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.717+0000 7f29537fe640 1 --2- 172.21.15.129:0/2252147357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f2a1c0 0x559977364db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.717+0000 7f29537fe640 1 -- 172.21.15.129:0/2252147357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599773653c0 con 0x559976f754d0 2024-05-20T18:02:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.721+0000 7f29537fe640 1 --2- 172.21.15.129:0/2252147357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 0x559977369e00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f294c009460 tx=0x7f294c00eec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53f4db1910e67cf1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.721+0000 7f293a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:02:18.722 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.724+0000 7f293a7fc640 1 -- 172.21.15.129:0/2252147357 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c017040 con 0x559976f754d0 2024-05-20T18:02:18.722 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.724+0000 7f29603b6740 1 -- 172.21.15.129:0/2252147357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5599772a1dc0 con 0x559976f754d0 2024-05-20T18:02:18.722 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.724+0000 7f293a7fc640 1 -- 172.21.15.129:0/2252147357 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f294c013b40 con 0x559976f754d0 2024-05-20T18:02:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.729+0000 7f293a7fc640 1 -- 172.21.15.129:0/2252147357 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f294c044050 con 0x559976f754d0 2024-05-20T18:02:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.729+0000 7f29603b6740 10 client.25751 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.729+0000 7f29603b6740 1 -- 172.21.15.129:0/2252147357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5599773daf40 con 0x559976f754d0 2024-05-20T18:02:18.730 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.732+0000 7f293a7fc640 1 -- 172.21.15.129:0/2252147357 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5599772a1dc0 con 0x559976f754d0 2024-05-20T18:02:18.730 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.732+0000 7f293a7fc640 1 -- 172.21.15.129:0/2252147357 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f294c023040 con 0x559976f754d0 2024-05-20T18:02:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.732+0000 7f29603b6740 10 client.25751 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.732+0000 7f29603b6740 10 client.25751 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.732+0000 7f29603b6740 20 client.25751 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.733+0000 7f29603b6740 1 --2- 172.21.15.129:0/2252147357 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5599773db630 0x5599773dda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.733+0000 7f29603b6740 4 client.25751 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:02:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.733+0000 7f29603b6740 1 -- 172.21.15.129:0/2252147357 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55997721f8e0 con 0x5599773db630 2024-05-20T18:02:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.735+0000 7f29527fc640 1 --2- 172.21.15.129:0/2252147357 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5599773db630 0x5599773dda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.739+0000 7f29527fc640 1 --2- 172.21.15.129:0/2252147357 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5599773db630 0x5599773dda50 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84b68e20c7820a91 server_cookie=8d855a8425e0549 in_seq=0 out_seq=0 2024-05-20T18:02:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:18.739+0000 7f293a7fc640 10 client.25751 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:19.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:18 smithi154 ceph-mon[36482]: pgmap v116: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 702 MiB/s wr, 5.37k op/s 2024-05-20T18:02:19.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:18 smithi129 ceph-mon[32029]: pgmap v116: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 702 MiB/s wr, 5.37k op/s 2024-05-20T18:02:19.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:18 smithi204 ceph-mon[37076]: pgmap v116: 97 pgs: 97 active+clean; 24 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 702 MiB/s wr, 5.37k op/s 2024-05-20T18:02:21.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:20 smithi154 ceph-mon[36482]: pgmap v117: 97 pgs: 97 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 776 MiB/s wr, 5.59k op/s 2024-05-20T18:02:21.143 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:20 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:02:21.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:20 smithi129 ceph-mon[32029]: pgmap v117: 97 pgs: 97 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 776 MiB/s wr, 5.59k op/s 2024-05-20T18:02:21.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:20 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:02:21.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:20 smithi204 ceph-mon[37076]: pgmap v117: 97 pgs: 97 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 776 MiB/s wr, 5.59k op/s 2024-05-20T18:02:21.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:20 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:02:23.120 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:22 smithi129 ceph-mon[32029]: pgmap v118: 97 pgs: 97 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 599 MiB/s wr, 4.30k op/s 2024-05-20T18:02:23.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:22 smithi154 ceph-mon[36482]: pgmap v118: 97 pgs: 97 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 599 MiB/s wr, 4.30k op/s 2024-05-20T18:02:23.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:22 smithi204 ceph-mon[37076]: pgmap v118: 97 pgs: 97 active+clean; 25 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 599 MiB/s wr, 4.30k op/s 2024-05-20T18:02:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f293a7fc640 1 -- 172.21.15.129:0/2252147357 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55997721f8e0 con 0x5599773db630 2024-05-20T18:02:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f293a7fc640 10 client.25751 handle_command_reply: tid=0 2024-05-20T18:02:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f29603b6740 1 client.25751 shutdown 2024-05-20T18:02:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f29603b6740 1 -- 172.21.15.129:0/2252147357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 msgr2=0x559977369e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f29603b6740 1 --2- 172.21.15.129:0/2252147357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 0x559977369e00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f294c009460 tx=0x7f294c00eec0 comp rx=0 tx=0).stop 2024-05-20T18:02:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f29603b6740 1 -- 172.21.15.129:0/2252147357 shutdown_connections 2024-05-20T18:02:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f29603b6740 1 --2- 172.21.15.129:0/2252147357 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5599773db630 0x5599773dda50 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f29603b6740 1 --2- 172.21.15.129:0/2252147357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559976f2a1c0 0x559977364db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f29603b6740 1 --2- 172.21.15.129:0/2252147357 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559976f961b0 0x559977364870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f29603b6740 1 --2- 172.21.15.129:0/2252147357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559976f754d0 0x559977369e00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f29603b6740 1 -- 172.21.15.129:0/2252147357 >> 172.21.15.129:0/2252147357 conn(0x559976f8a300 msgr2=0x5599773312d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.818+0000 7f293a7fc640 0 client.25751 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.819+0000 7f29603b6740 1 -- 172.21.15.129:0/2252147357 shutdown_connections 2024-05-20T18:02:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.819+0000 7f29603b6740 1 -- 172.21.15.129:0/2252147357 wait complete. 2024-05-20T18:02:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.819+0000 7f29603b6740 20 client.25751 trim_cache size 0 max 16384 2024-05-20T18:02:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:02:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:02:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:02:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:02:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.822+0000 7f2953fff640 1 -- 172.21.15.129:0/104873553 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29400a2d60 msgr2=0x7f29400a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.822+0000 7f2953fff640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29400a2d60 0x7f29400a5270 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f293c0b89c0 tx=0x7f293c0dc040 comp rx=0 tx=0).stop 2024-05-20T18:02:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.822+0000 7f2953fff640 1 -- 172.21.15.129:0/104873553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2954109090 msgr2=0x7f29541a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.822+0000 7f2953fff640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2954109090 0x7f29541a5830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f293c01a860 tx=0x7f293c004b20 comp rx=0 tx=0).stop 2024-05-20T18:02:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.823+0000 7f2953fff640 1 -- 172.21.15.129:0/104873553 shutdown_connections 2024-05-20T18:02:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.823+0000 7f2953fff640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f29400a2d60 0x7f29400a5270 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.823+0000 7f2953fff640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f295410e890 0x7f29541a5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.823+0000 7f2953fff640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f295410f270 0x7f295419fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.823+0000 7f2953fff640 1 --2- 172.21.15.129:0/104873553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2954109090 0x7f29541a5830 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.823+0000 7f2953fff640 1 -- 172.21.15.129:0/104873553 >> 172.21.15.129:0/104873553 conn(0x7f295406e690 msgr2=0x7f295410c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.823+0000 7f2953fff640 1 -- 172.21.15.129:0/104873553 shutdown_connections 2024-05-20T18:02:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:24.823+0000 7f2953fff640 1 -- 172.21.15.129:0/104873553 wait complete. 2024-05-20T18:02:25.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:24 smithi154 ceph-mon[36482]: pgmap v119: 97 pgs: 97 active+clean; 25 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 643 MiB/s wr, 4.40k op/s 2024-05-20T18:02:25.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:24 smithi129 ceph-mon[32029]: pgmap v119: 97 pgs: 97 active+clean; 25 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 643 MiB/s wr, 4.40k op/s 2024-05-20T18:02:25.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:24 smithi204 ceph-mon[37076]: pgmap v119: 97 pgs: 97 active+clean; 25 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 643 MiB/s wr, 4.40k op/s 2024-05-20T18:02:27.168 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:26 smithi129 ceph-mon[32029]: pgmap v120: 97 pgs: 97 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 767 MiB/s wr, 4.68k op/s 2024-05-20T18:02:27.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:26 smithi204 ceph-mon[37076]: pgmap v120: 97 pgs: 97 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 767 MiB/s wr, 4.68k op/s 2024-05-20T18:02:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:26 smithi154 ceph-mon[36482]: pgmap v120: 97 pgs: 97 active+clean; 27 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 767 MiB/s wr, 4.68k op/s 2024-05-20T18:02:29.163 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:28 smithi129 ceph-mon[32029]: pgmap v121: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 639 MiB/s wr, 2.92k op/s 2024-05-20T18:02:29.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:28 smithi204 ceph-mon[37076]: pgmap v121: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 639 MiB/s wr, 2.92k op/s 2024-05-20T18:02:29.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:28 smithi154 ceph-mon[36482]: pgmap v121: 97 pgs: 97 active+clean; 28 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 639 MiB/s wr, 2.92k op/s 2024-05-20T18:02:31.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:30 smithi204 ceph-mon[37076]: pgmap v122: 97 pgs: 97 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 788 MiB/s wr, 2.93k op/s 2024-05-20T18:02:31.193 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:30 smithi154 ceph-mon[36482]: pgmap v122: 97 pgs: 97 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 788 MiB/s wr, 2.93k op/s 2024-05-20T18:02:31.417 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:30 smithi129 ceph-mon[32029]: pgmap v122: 97 pgs: 97 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 788 MiB/s wr, 2.93k op/s 2024-05-20T18:02:33.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:32 smithi154 ceph-mon[36482]: pgmap v123: 97 pgs: 97 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 621 MiB/s wr, 2.23k op/s 2024-05-20T18:02:33.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:32 smithi129 ceph-mon[32029]: pgmap v123: 97 pgs: 97 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 621 MiB/s wr, 2.23k op/s 2024-05-20T18:02:33.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:32 smithi204 ceph-mon[37076]: pgmap v123: 97 pgs: 97 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 621 MiB/s wr, 2.23k op/s 2024-05-20T18:02:35.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:34 smithi129 ceph-mon[32029]: pgmap v124: 97 pgs: 97 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 670 MiB/s wr, 2.45k op/s 2024-05-20T18:02:35.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:34 smithi154 ceph-mon[36482]: pgmap v124: 97 pgs: 97 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 670 MiB/s wr, 2.45k op/s 2024-05-20T18:02:35.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:34 smithi204 ceph-mon[37076]: pgmap v124: 97 pgs: 97 active+clean; 29 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 670 MiB/s wr, 2.45k op/s 2024-05-20T18:02:37.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:36 smithi154 ceph-mon[36482]: pgmap v125: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 877 MiB/s wr, 2.83k op/s 2024-05-20T18:02:37.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:36 smithi129 ceph-mon[32029]: pgmap v125: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 877 MiB/s wr, 2.83k op/s 2024-05-20T18:02:37.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:36 smithi204 ceph-mon[37076]: pgmap v125: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 877 MiB/s wr, 2.83k op/s 2024-05-20T18:02:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.205+0000 7f4c6359e640 1 -- 172.21.15.129:0/1904788718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c64109090 msgr2=0x7f4c64109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.205+0000 7f4c6359e640 1 --2- 172.21.15.129:0/1904788718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c64109090 0x7f4c64109520 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009900 tx=0x7f4c5802a440 comp rx=0 tx=0).stop 2024-05-20T18:02:38.208 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.206+0000 7f4c6359e640 1 -- 172.21.15.129:0/1904788718 shutdown_connections 2024-05-20T18:02:38.208 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.206+0000 7f4c6359e640 1 --2- 172.21.15.129:0/1904788718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c64109090 0x7f4c64109520 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.210 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.206+0000 7f4c6359e640 1 --2- 172.21.15.129:0/1904788718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c6410f270 0x7f4c64108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.210 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.206+0000 7f4c6359e640 1 --2- 172.21.15.129:0/1904788718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c6410e890 0x7f4c6410eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.210 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.206+0000 7f4c6359e640 1 -- 172.21.15.129:0/1904788718 >> 172.21.15.129:0/1904788718 conn(0x7f4c6406e690 msgr2=0x7f4c6406eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.212+0000 7f4c6359e640 1 -- 172.21.15.129:0/1904788718 shutdown_connections 2024-05-20T18:02:38.221 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.220+0000 7f4c6359e640 1 -- 172.21.15.129:0/1904788718 wait complete. 2024-05-20T18:02:38.222 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.222+0000 7f4c6359e640 1 Processor -- start 2024-05-20T18:02:38.223 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.223+0000 7f4c6359e640 1 -- start start 2024-05-20T18:02:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.225+0000 7f4c6359e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c64109090 0x7f4c64119400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.243 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.231+0000 7f4c6359e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c6410e890 0x7f4c64119940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.243 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.230+0000 7f4c6259c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c64109090 0x7f4c64119400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.243 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.245+0000 7f4c6359e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c6410f270 0x7f4c64119e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.243 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.245+0000 7f4c6359e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4c64111970 con 0x7f4c6410e890 2024-05-20T18:02:38.243 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.245+0000 7f4c6359e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c641117f0 con 0x7f4c6410f270 2024-05-20T18:02:38.243 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.245+0000 7f4c6359e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c64111af0 con 0x7f4c64109090 2024-05-20T18:02:38.243 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.245+0000 7f4c6259c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c64109090 0x7f4c64119400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:37518/0 (socket says 172.21.15.129:37518) 2024-05-20T18:02:38.243 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.245+0000 7f4c6259c640 1 -- 172.21.15.129:0/1382495118 learned_addr learned my addr 172.21.15.129:0/1382495118 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:38.244 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.246+0000 7f4c62d9d640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c6410f270 0x7f4c64119e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.248+0000 7f4c62d9d640 1 -- 172.21.15.129:0/1382495118 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c64109090 msgr2=0x7f4c64119400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.248+0000 7f4c62d9d640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c64109090 0x7f4c64119400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.248+0000 7f4c62d9d640 1 -- 172.21.15.129:0/1382495118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c6410e890 msgr2=0x7f4c64119940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.248+0000 7f4c62d9d640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c6410e890 0x7f4c64119940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.248+0000 7f4c62d9d640 1 -- 172.21.15.129:0/1382495118 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c641bcd80 con 0x7f4c6410f270 2024-05-20T18:02:38.251 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.252+0000 7f4c62d9d640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c6410f270 0x7f4c64119e80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c58033040 tx=0x7f4c58005230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:38.258 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.260+0000 7f4c537fe640 1 -- 172.21.15.129:0/1382495118 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c58040030 con 0x7f4c6410f270 2024-05-20T18:02:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.261+0000 7f4c6359e640 1 -- 172.21.15.129:0/1382495118 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c641bcfb0 con 0x7f4c6410f270 2024-05-20T18:02:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.261+0000 7f4c6359e640 1 -- 172.21.15.129:0/1382495118 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c641bd410 con 0x7f4c6410f270 2024-05-20T18:02:38.267 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.269+0000 7f4c6fa18740 1 Processor -- start 2024-05-20T18:02:38.268 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.270+0000 7f4c6fa18740 1 -- start start 2024-05-20T18:02:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.271+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a225270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.270 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.272+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e5d2f0 0x56199a2257b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.270 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.272+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e654d0 0x56199a225cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.272+0000 7f4c61d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a225270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.272+0000 7f4c61d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a225270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:37520/0 (socket says 172.21.15.129:37520) 2024-05-20T18:02:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.272+0000 7f4c61d9b640 1 -- 172.21.15.129:0/3725928326 learned_addr learned my addr 172.21.15.129:0/3725928326 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.273+0000 7f4c6fa18740 1 -- 172.21.15.129:0/3725928326 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56199a18a340 con 0x561999e654d0 2024-05-20T18:02:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.273+0000 7f4c6fa18740 1 -- 172.21.15.129:0/3725928326 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56199a132290 con 0x561999e5d2f0 2024-05-20T18:02:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.273+0000 7f4c6fa18740 1 -- 172.21.15.129:0/3725928326 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56199a186300 con 0x561999e861b0 2024-05-20T18:02:38.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.277+0000 7f4c6259c640 1 --2- 172.21.15.129:0/3725928326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e5d2f0 0x56199a2257b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.279+0000 7f4c537fe640 1 -- 172.21.15.129:0/1382495118 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c580055b0 con 0x7f4c6410f270 2024-05-20T18:02:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.280+0000 7f4c537fe640 1 -- 172.21.15.129:0/1382495118 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5803c870 con 0x7f4c6410f270 2024-05-20T18:02:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.281+0000 7f4c61d9b640 1 --2- 172.21.15.129:0/3725928326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e654d0 0x56199a225cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.281+0000 7f4c61d9b640 1 -- 172.21.15.129:0/3725928326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e5d2f0 msgr2=0x56199a2257b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.281+0000 7f4c61d9b640 1 --2- 172.21.15.129:0/3725928326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e5d2f0 0x56199a2257b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.281+0000 7f4c61d9b640 1 -- 172.21.15.129:0/3725928326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e654d0 msgr2=0x56199a225cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.281+0000 7f4c61d9b640 1 --2- 172.21.15.129:0/3725928326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e654d0 0x56199a225cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.282 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.281+0000 7f4c61d9b640 1 -- 172.21.15.129:0/3725928326 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56199a1568e0 con 0x561999e861b0 2024-05-20T18:02:38.282 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.282+0000 7f4c537fe640 1 -- 172.21.15.129:0/1382495118 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f4c58043030 con 0x7f4c6410f270 2024-05-20T18:02:38.282 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.283+0000 7f4c61d9b640 1 --2- 172.21.15.129:0/3725928326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a225270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400dfa0 tx=0x7f4c5400d4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61713278178daae2 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.290+0000 7f4c537fe640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4c4c0a2f30 0x7f4c4c0a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.291+0000 7f4c537fe640 1 -- 172.21.15.129:0/1382495118 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f4c580f0fb0 con 0x7f4c6410f270 2024-05-20T18:02:38.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.292+0000 7f4c50ff9640 1 -- 172.21.15.129:0/3725928326 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c54005020 con 0x561999e861b0 2024-05-20T18:02:38.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.292+0000 7f4c50ff9640 1 -- 172.21.15.129:0/3725928326 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c540047a0 con 0x561999e861b0 2024-05-20T18:02:38.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.293+0000 7f4c6259c640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4c4c0a2f30 0x7f4c4c0a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.294+0000 7f4c6fa18740 1 -- 172.21.15.129:0/3725928326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 msgr2=0x56199a225270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.294+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/3725928326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a225270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400dfa0 tx=0x7f4c5400d4a0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.298+0000 7f4c6fa18740 1 -- 172.21.15.129:0/3725928326 shutdown_connections 2024-05-20T18:02:38.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.298+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/3725928326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e5d2f0 0x56199a2257b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.298+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/3725928326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e654d0 0x56199a225cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.298+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/3725928326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a225270 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.298+0000 7f4c6fa18740 1 -- 172.21.15.129:0/3725928326 >> 172.21.15.129:0/3725928326 conn(0x561999e7a300 msgr2=0x56199a221a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:38.302 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.303+0000 7f4c6fa18740 1 -- 172.21.15.129:0/3725928326 shutdown_connections 2024-05-20T18:02:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.308+0000 7f4c6fa18740 1 -- 172.21.15.129:0/3725928326 wait complete. 2024-05-20T18:02:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.309+0000 7f4c6fa18740 1 -- start start 2024-05-20T18:02:38.307 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.309+0000 7f4c6fa18740 1 Processor -- start 2024-05-20T18:02:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.310+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e5d2f0 0x56199a22f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.310+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e654d0 0x56199a22fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.310+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a235950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.310+0000 7f4c6fa18740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56199a2323f0 con 0x561999e5d2f0 2024-05-20T18:02:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.310+0000 7f4c6fa18740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56199a145790 con 0x561999e654d0 2024-05-20T18:02:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.310+0000 7f4c6fa18740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56199a232570 con 0x561999e861b0 2024-05-20T18:02:38.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.310+0000 7f4c6fa18740 10 client.0 _ll_register_callbacks cb 0x56199a180a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:38.310 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.311+0000 7f4c62d9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a235950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.315+0000 7f4c62d9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a235950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:37530/0 (socket says 172.21.15.129:37530) 2024-05-20T18:02:38.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.315+0000 7f4c62d9d640 1 -- 172.21.15.129:0/2095975895 learned_addr learned my addr 172.21.15.129:0/2095975895 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:38.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.315+0000 7f4c61d9b640 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e654d0 0x56199a22fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.315+0000 7f4c6259c640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4c4c0a2f30 0x7f4c4c0a5440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x56199a226620 tx=0x7f4c5c007760 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:38.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.315+0000 7f4c6259c640 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e5d2f0 0x56199a22f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.322+0000 7f4c62d9d640 1 -- 172.21.15.129:0/2095975895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e654d0 msgr2=0x56199a22fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.322+0000 7f4c62d9d640 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e654d0 0x56199a22fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.322+0000 7f4c62d9d640 1 -- 172.21.15.129:0/2095975895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e5d2f0 msgr2=0x56199a22f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.322+0000 7f4c62d9d640 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e5d2f0 0x56199a22f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.322+0000 7f4c62d9d640 1 -- 172.21.15.129:0/2095975895 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561999e9e210 con 0x561999e861b0 2024-05-20T18:02:38.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.331+0000 7f4c62d9d640 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a235950 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4c580bd470 tx=0x7f4c580bcec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e3b22e2bff40858 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:38.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.331+0000 7f4c49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:02:38.335 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.337+0000 7f4c49ffb640 1 -- 172.21.15.129:0/2095975895 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c580d6060 con 0x561999e861b0 2024-05-20T18:02:38.335 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.337+0000 7f4c49ffb640 1 -- 172.21.15.129:0/2095975895 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f4c580ee990 con 0x561999e861b0 2024-05-20T18:02:38.335 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.337+0000 7f4c6fa18740 1 -- 172.21.15.129:0/2095975895 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c540047a0 con 0x561999e861b0 2024-05-20T18:02:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.338+0000 7f4c49ffb640 1 -- 172.21.15.129:0/2095975895 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4c5804a030 con 0x561999e861b0 2024-05-20T18:02:38.337 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.340+0000 7f4c6fa18740 10 client.25777 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.340+0000 7f4c6fa18740 1 -- 172.21.15.129:0/2095975895 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561999f6a620 con 0x561999e861b0 2024-05-20T18:02:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.340+0000 7f4c49ffb640 1 -- 172.21.15.129:0/2095975895 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c540047a0 con 0x561999e861b0 2024-05-20T18:02:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.341+0000 7f4c49ffb640 1 -- 172.21.15.129:0/2095975895 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f4c580c5cc0 con 0x561999e861b0 2024-05-20T18:02:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.342+0000 7f4c6fa18740 10 client.25777 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.342+0000 7f4c6fa18740 10 client.25777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.342+0000 7f4c6fa18740 20 client.25777 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.342+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561999e44d80 0x56199a241a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.343+0000 7f4c6fa18740 4 client.25777 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.343+0000 7f4c6fa18740 1 -- 172.21.15.129:0/2095975895 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56199a197a70 con 0x561999e44d80 2024-05-20T18:02:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.349+0000 7f4c6259c640 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561999e44d80 0x56199a241a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.367 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.369+0000 7f4c6259c640 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561999e44d80 0x56199a241a50 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=742c584045fc438e server_cookie=4f0dd00eeb772906 in_seq=0 out_seq=0 2024-05-20T18:02:38.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.369+0000 7f4c49ffb640 10 client.25777 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.374+0000 7f4c49ffb640 1 -- 172.21.15.129:0/2095975895 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56199a197a70 con 0x561999e44d80 2024-05-20T18:02:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.374+0000 7f4c49ffb640 10 client.25777 handle_command_reply: tid=0 2024-05-20T18:02:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.374+0000 7f4c6fa18740 1 client.25777 shutdown 2024-05-20T18:02:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.392+0000 7f4c6fa18740 1 -- 172.21.15.129:0/2095975895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 msgr2=0x56199a235950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.392+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a235950 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4c580bd470 tx=0x7f4c580bcec0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.396+0000 7f4c6fa18740 1 -- 172.21.15.129:0/2095975895 shutdown_connections 2024-05-20T18:02:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.396+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561999e44d80 0x56199a241a50 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.396+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e654d0 0x56199a22fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.396+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e861b0 0x56199a235950 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.396+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/2095975895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e5d2f0 0x56199a22f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.396+0000 7f4c49ffb640 0 client.25777 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.396+0000 7f4c6fa18740 1 -- 172.21.15.129:0/2095975895 >> 172.21.15.129:0/2095975895 conn(0x561999e7a300 msgr2=0x56199a226f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:38.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.396+0000 7f4c6fa18740 1 -- 172.21.15.129:0/2095975895 shutdown_connections 2024-05-20T18:02:38.397 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.399+0000 7f4c6fa18740 1 -- 172.21.15.129:0/2095975895 wait complete. 2024-05-20T18:02:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.403+0000 7f4c6fa18740 20 client.25777 trim_cache size 0 max 16384 2024-05-20T18:02:38.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.482+0000 7f4c6fa18740 1 Processor -- start 2024-05-20T18:02:38.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.489+0000 7f4c6fa18740 1 -- start start 2024-05-20T18:02:38.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.489+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e5d2f0 0x56199a2358c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.489+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e654d0 0x56199a2275c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.489+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e861b0 0x56199a247210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.489+0000 7f4c6fa18740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4400c880 con 0x561999e5d2f0 2024-05-20T18:02:38.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.489+0000 7f4c6fa18740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4400f100 con 0x561999e861b0 2024-05-20T18:02:38.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.489+0000 7f4c6fa18740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4400cef0 con 0x561999e654d0 2024-05-20T18:02:38.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.489+0000 7f4c62d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e861b0 0x56199a247210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.490+0000 7f4c62d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e861b0 0x56199a247210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:58314/0 (socket says 172.21.15.129:58314) 2024-05-20T18:02:38.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.490+0000 7f4c62d9d640 1 -- 172.21.15.129:0/1977545112 learned_addr learned my addr 172.21.15.129:0/1977545112 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:38.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.490+0000 7f4c62d9d640 1 -- 172.21.15.129:0/1977545112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e654d0 msgr2=0x56199a2275c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:02:38.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.490+0000 7f4c62d9d640 1 --2- 172.21.15.129:0/1977545112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e654d0 0x56199a2275c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.490+0000 7f4c62d9d640 1 -- 172.21.15.129:0/1977545112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e5d2f0 msgr2=0x56199a2358c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:02:38.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.490+0000 7f4c62d9d640 1 --2- 172.21.15.129:0/1977545112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e5d2f0 0x56199a2358c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.490+0000 7f4c62d9d640 1 -- 172.21.15.129:0/1977545112 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56199a227b00 con 0x561999e861b0 2024-05-20T18:02:38.489 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.490+0000 7f4c62d9d640 1 --2- 172.21.15.129:0/1977545112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e861b0 0x56199a247210 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009900 tx=0x7f4c58048000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef6475c59b6d13 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:38.492 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.494+0000 7f4c4bfff640 1 -- 172.21.15.129:0/1977545112 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c580c6150 con 0x561999e861b0 2024-05-20T18:02:38.492 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.494+0000 7f4c4bfff640 1 -- 172.21.15.129:0/1977545112 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c580c1890 con 0x561999e861b0 2024-05-20T18:02:38.492 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.494+0000 7f4c4bfff640 1 -- 172.21.15.129:0/1977545112 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5802ac40 con 0x561999e861b0 2024-05-20T18:02:38.495 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.497+0000 7f4c6fa18740 1 -- 172.21.15.129:0/1977545112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e861b0 msgr2=0x56199a247210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:38.495 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.497+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/1977545112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e861b0 0x56199a247210 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009900 tx=0x7f4c58048000 comp rx=0 tx=0).stop 2024-05-20T18:02:38.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.506+0000 7f4c6fa18740 1 -- 172.21.15.129:0/1977545112 shutdown_connections 2024-05-20T18:02:38.505 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.506+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/1977545112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e861b0 0x56199a247210 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.505 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.506+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/1977545112 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e654d0 0x56199a2275c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.505 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.506+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/1977545112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e5d2f0 0x56199a2358c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.505 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.507+0000 7f4c6fa18740 1 -- 172.21.15.129:0/1977545112 >> 172.21.15.129:0/1977545112 conn(0x561999e7a300 msgr2=0x56199a226eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:38.508 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.511+0000 7f4c6fa18740 1 -- 172.21.15.129:0/1977545112 shutdown_connections 2024-05-20T18:02:38.509 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.511+0000 7f4c6fa18740 1 -- 172.21.15.129:0/1977545112 wait complete. 2024-05-20T18:02:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.512+0000 7f4c6fa18740 1 -- start start 2024-05-20T18:02:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.512+0000 7f4c6fa18740 1 Processor -- start 2024-05-20T18:02:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.513+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e5d2f0 0x56199a2a20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.513+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e654d0 0x56199a2a2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.513+0000 7f4c6fa18740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e861b0 0x56199a237380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.514+0000 7f4c6fa18740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4400d5c0 con 0x561999e861b0 2024-05-20T18:02:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.514+0000 7f4c6fa18740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4400ea30 con 0x561999e5d2f0 2024-05-20T18:02:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.514+0000 7f4c6fa18740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4400dc90 con 0x561999e654d0 2024-05-20T18:02:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.514+0000 7f4c6fa18740 10 client.0 _ll_register_callbacks cb 0x7f4c58005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:38.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.517+0000 7f4c61d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e654d0 0x56199a2a2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.517+0000 7f4c61d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e654d0 0x56199a2a2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:37544/0 (socket says 172.21.15.129:37544) 2024-05-20T18:02:38.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.517+0000 7f4c61d9b640 1 -- 172.21.15.129:0/469731592 learned_addr learned my addr 172.21.15.129:0/469731592 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:38.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.517+0000 7f4c6259c640 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e5d2f0 0x56199a2a20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.518 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.518+0000 7f4c62d9d640 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e861b0 0x56199a237380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.518 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.518+0000 7f4c61d9b640 1 -- 172.21.15.129:0/469731592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e5d2f0 msgr2=0x56199a2a20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:38.518 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.518+0000 7f4c61d9b640 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e5d2f0 0x56199a2a20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.518 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.518+0000 7f4c61d9b640 1 -- 172.21.15.129:0/469731592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e861b0 msgr2=0x56199a237380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:38.518 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.518+0000 7f4c61d9b640 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e861b0 0x56199a237380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:38.519 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.518+0000 7f4c61d9b640 1 -- 172.21.15.129:0/469731592 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56199a2379c0 con 0x561999e654d0 2024-05-20T18:02:38.519 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.519+0000 7f4c61d9b640 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e654d0 0x56199a2a2620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400dfd0 tx=0x7f4c54005230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fed50242a9e47a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:38.522 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.524+0000 7f4c49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:02:38.523 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.524+0000 7f4c49ffb640 1 -- 172.21.15.129:0/469731592 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c54005850 con 0x561999e654d0 2024-05-20T18:02:38.523 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.524+0000 7f4c6fa18740 1 -- 172.21.15.129:0/469731592 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c580c1890 con 0x561999e654d0 2024-05-20T18:02:38.523 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.525+0000 7f4c49ffb640 1 -- 172.21.15.129:0/469731592 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f4c54012850 con 0x561999e654d0 2024-05-20T18:02:38.523 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.525+0000 7f4c49ffb640 1 -- 172.21.15.129:0/469731592 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4c54012a50 con 0x561999e654d0 2024-05-20T18:02:38.523 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.525+0000 7f4c6fa18740 10 client.25780 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:38.523 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.525+0000 7f4c6fa18740 1 -- 172.21.15.129:0/469731592 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56199a2a9770 con 0x561999e654d0 2024-05-20T18:02:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.530+0000 7f4c49ffb640 1 -- 172.21.15.129:0/469731592 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c54004c80 con 0x561999e654d0 2024-05-20T18:02:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.530+0000 7f4c49ffb640 1 -- 172.21.15.129:0/469731592 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f4c54025020 con 0x561999e654d0 2024-05-20T18:02:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.530+0000 7f4c6fa18740 10 client.25780 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.530+0000 7f4c6fa18740 10 client.25780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.530+0000 7f4c6fa18740 20 client.25780 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.530+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561999e1a1c0 0x56199a2abe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.530+0000 7f4c6fa18740 4 client.25780 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:02:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.530+0000 7f4c6fa18740 1 -- 172.21.15.129:0/469731592 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f4c580c1890 con 0x561999e1a1c0 2024-05-20T18:02:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.537+0000 7f4c6259c640 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561999e1a1c0 0x56199a2abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.541+0000 7f4c49ffb640 10 client.25780 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:38.542+0000 7f4c6259c640 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561999e1a1c0 0x56199a2abe30 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84cfee47cb0bcaf1 server_cookie=626396f36709a868 in_seq=0 out_seq=0 2024-05-20T18:02:39.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:38 smithi154 ceph-mon[36482]: pgmap v126: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 720 MiB/s wr, 2.00k op/s 2024-05-20T18:02:39.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:38 smithi129 ceph-mon[32029]: pgmap v126: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 720 MiB/s wr, 2.00k op/s 2024-05-20T18:02:39.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:38 smithi204 ceph-mon[37076]: pgmap v126: 97 pgs: 97 active+clean; 29 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 720 MiB/s wr, 2.00k op/s 2024-05-20T18:02:41.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:41 smithi154 ceph-mon[36482]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 908 MiB/s wr, 2.58k op/s 2024-05-20T18:02:41.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:41 smithi129 ceph-mon[32029]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 908 MiB/s wr, 2.58k op/s 2024-05-20T18:02:41.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:41 smithi204 ceph-mon[37076]: pgmap v127: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 908 MiB/s wr, 2.58k op/s 2024-05-20T18:02:43.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:43 smithi154 ceph-mon[36482]: pgmap v128: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 691 MiB/s wr, 1.96k op/s 2024-05-20T18:02:43.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:43 smithi129 ceph-mon[32029]: pgmap v128: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 691 MiB/s wr, 1.96k op/s 2024-05-20T18:02:43.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:43 smithi204 ceph-mon[37076]: pgmap v128: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 691 MiB/s wr, 1.96k op/s 2024-05-20T18:02:44.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:02:44.324 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:02:44.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.748+0000 7f0e7da91640 1 -- 172.21.15.129:0/13826081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e78119770 msgr2=0x7f0e7811bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:44.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.748+0000 7f0e7da91640 1 --2- 172.21.15.129:0/13826081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e78119770 0x7f0e7811bb90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009930 tx=0x7f0e6c02a510 comp rx=0 tx=0).stop 2024-05-20T18:02:44.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.761+0000 7f0e7da91640 1 -- 172.21.15.129:0/13826081 shutdown_connections 2024-05-20T18:02:44.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.761+0000 7f0e7da91640 1 --2- 172.21.15.129:0/13826081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e7811c0d0 0x7f0e7811e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.761+0000 7f0e7da91640 1 --2- 172.21.15.129:0/13826081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e78119770 0x7f0e7811bb90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.761+0000 7f0e7da91640 1 --2- 172.21.15.129:0/13826081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e78109060 0x7f0e78109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.761+0000 7f0e7da91640 1 -- 172.21.15.129:0/13826081 >> 172.21.15.129:0/13826081 conn(0x7f0e7806d750 msgr2=0x7f0e7806dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:44.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.765+0000 7f0e7da91640 1 -- 172.21.15.129:0/13826081 shutdown_connections 2024-05-20T18:02:44.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.765+0000 7f0e7da91640 1 -- 172.21.15.129:0/13826081 wait complete. 2024-05-20T18:02:44.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.767+0000 7f0e7da91640 1 Processor -- start 2024-05-20T18:02:44.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.767+0000 7f0e7da91640 1 -- start start 2024-05-20T18:02:44.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.768+0000 7f0e7da91640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e78109060 0x7f0e781a8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:44.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.768+0000 7f0e7da91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e78119770 0x7f0e781a8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:44.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.769+0000 7f0e7da91640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e7811c0d0 0x7f0e781acda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:44.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.769+0000 7f0e7da91640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0e78121c20 con 0x7f0e7811c0d0 2024-05-20T18:02:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.769+0000 7f0e7da91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e78121aa0 con 0x7f0e78119770 2024-05-20T18:02:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.769+0000 7f0e7da91640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0e78121da0 con 0x7f0e78109060 2024-05-20T18:02:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.769+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e78109060 0x7f0e781a8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.769+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e78109060 0x7f0e781a8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:51524/0 (socket says 172.21.15.129:51524) 2024-05-20T18:02:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.769+0000 7f0e777fe640 1 -- 172.21.15.129:0/3507043329 learned_addr learned my addr 172.21.15.129:0/3507043329 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.769+0000 7f0e77fff640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e7811c0d0 0x7f0e781acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.770+0000 7f0e777fe640 1 -- 172.21.15.129:0/3507043329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e78119770 msgr2=0x7f0e781a8ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:02:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.770+0000 7f0e777fe640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e78119770 0x7f0e781a8ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.770+0000 7f0e777fe640 1 -- 172.21.15.129:0/3507043329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e7811c0d0 msgr2=0x7f0e781acda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:44.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.770+0000 7f0e777fe640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e7811c0d0 0x7f0e781acda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.770+0000 7f0e777fe640 1 -- 172.21.15.129:0/3507043329 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e781ad3e0 con 0x7f0e78109060 2024-05-20T18:02:44.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.771+0000 7f0e777fe640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e78109060 0x7f0e781a8580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800e9f0 tx=0x7f0e6800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:44.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.775+0000 7f0e74ff9640 1 -- 172.21.15.129:0/3507043329 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6800cd80 con 0x7f0e78109060 2024-05-20T18:02:44.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.775+0000 7f0e74ff9640 1 -- 172.21.15.129:0/3507043329 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e68010910 con 0x7f0e78109060 2024-05-20T18:02:44.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.777+0000 7f0e74ff9640 1 -- 172.21.15.129:0/3507043329 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68018990 con 0x7f0e78109060 2024-05-20T18:02:44.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.779+0000 7f0e7da91640 1 -- 172.21.15.129:0/3507043329 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e781ad6d0 con 0x7f0e78109060 2024-05-20T18:02:44.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.779+0000 7f0e7da91640 1 -- 172.21.15.129:0/3507043329 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e781adba0 con 0x7f0e78109060 2024-05-20T18:02:44.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.784+0000 7f0e84a10740 1 Processor -- start 2024-05-20T18:02:44.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.809+0000 7f0e84a10740 1 -- start start 2024-05-20T18:02:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.810+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4e8390 0x55bedb886d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.810+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 0x55bedb887270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.811+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4c76b0 0x55bedb8877b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.811+0000 7f0e84a10740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bedb7ecd00 con 0x55bedb4e8390 2024-05-20T18:02:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.811+0000 7f0e84a10740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bedb7944c0 con 0x55bedb4c76b0 2024-05-20T18:02:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.811+0000 7f0e84a10740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55bedb7e8cc0 con 0x55bedb4bf4d0 2024-05-20T18:02:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.811+0000 7f0e74ff9640 1 -- 172.21.15.129:0/3507043329 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f0e68018b30 con 0x7f0e78109060 2024-05-20T18:02:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.811+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 0x55bedb887270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.812+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 0x55bedb887270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:51526/0 (socket says 172.21.15.129:51526) 2024-05-20T18:02:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.812+0000 7f0e777fe640 1 -- 172.21.15.129:0/3582829297 learned_addr learned my addr 172.21.15.129:0/3582829297 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.814+0000 7f0e777fe640 1 -- 172.21.15.129:0/3582829297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4c76b0 msgr2=0x55bedb8877b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:02:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.814+0000 7f0e777fe640 1 --2- 172.21.15.129:0/3582829297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4c76b0 0x55bedb8877b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.814+0000 7f0e777fe640 1 -- 172.21.15.129:0/3582829297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4e8390 msgr2=0x55bedb886d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.814+0000 7f0e777fe640 1 --2- 172.21.15.129:0/3582829297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4e8390 0x55bedb886d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.814+0000 7f0e777fe640 1 -- 172.21.15.129:0/3582829297 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bedb5ca7d0 con 0x55bedb4bf4d0 2024-05-20T18:02:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.815+0000 7f0e74ff9640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0e600a2de0 0x7f0e600a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:44.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.815+0000 7f0e74ff9640 1 -- 172.21.15.129:0/3507043329 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f0e680cf7a0 con 0x7f0e78109060 2024-05-20T18:02:44.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.815+0000 7f0e777fe640 1 --2- 172.21.15.129:0/3582829297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 0x55bedb887270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0e680ccdc0 tx=0x7f0e680d5c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87f1ac23f6f48565 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:44.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.816+0000 7f0e5dffb640 1 -- 172.21.15.129:0/3582829297 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68097ef0 con 0x55bedb4bf4d0 2024-05-20T18:02:44.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.816+0000 7f0e5dffb640 1 -- 172.21.15.129:0/3582829297 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e6809c350 con 0x55bedb4bf4d0 2024-05-20T18:02:44.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.816+0000 7f0e5dffb640 1 -- 172.21.15.129:0/3582829297 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6809b5b0 con 0x55bedb4bf4d0 2024-05-20T18:02:44.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.817+0000 7f0e77fff640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0e600a2de0 0x7f0e600a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:44.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.818+0000 7f0e84a10740 1 -- 172.21.15.129:0/3582829297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 msgr2=0x55bedb887270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:44.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.818+0000 7f0e84a10740 1 --2- 172.21.15.129:0/3582829297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 0x55bedb887270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0e680ccdc0 tx=0x7f0e680d5c50 comp rx=0 tx=0).stop 2024-05-20T18:02:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.822+0000 7f0e84a10740 1 -- 172.21.15.129:0/3582829297 shutdown_connections 2024-05-20T18:02:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.822+0000 7f0e84a10740 1 --2- 172.21.15.129:0/3582829297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4c76b0 0x55bedb8877b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.822+0000 7f0e84a10740 1 --2- 172.21.15.129:0/3582829297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 0x55bedb887270 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.822+0000 7f0e84a10740 1 --2- 172.21.15.129:0/3582829297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4e8390 0x55bedb886d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.822+0000 7f0e84a10740 1 -- 172.21.15.129:0/3582829297 >> 172.21.15.129:0/3582829297 conn(0x55bedb4dc4e0 msgr2=0x55bedb8834e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:44.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.823+0000 7f0e77fff640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0e600a2de0 0x7f0e600a52f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0e700092f0 tx=0x7f0e70009280 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:44.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.826+0000 7f0e84a10740 1 -- 172.21.15.129:0/3582829297 shutdown_connections 2024-05-20T18:02:44.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.830+0000 7f0e84a10740 1 -- 172.21.15.129:0/3582829297 wait complete. 2024-05-20T18:02:44.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.830+0000 7f0e84a10740 1 -- start start 2024-05-20T18:02:44.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.831+0000 7f0e84a10740 1 Processor -- start 2024-05-20T18:02:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.832+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 0x55bedb8913d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.832+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4c76b0 0x55bedb891910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.832+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 0x55bedb8975d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.832+0000 7f0e84a10740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bedb894020 con 0x55bedb4c76b0 2024-05-20T18:02:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.832+0000 7f0e84a10740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bedb7a7b90 con 0x55bedb4e8390 2024-05-20T18:02:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.832+0000 7f0e84a10740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55bedb8941a0 con 0x55bedb4bf4d0 2024-05-20T18:02:44.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.832+0000 7f0e84a10740 10 client.0 _ll_register_callbacks cb 0x55bedb7e3da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.833+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 0x55bedb8913d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.833+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 0x55bedb8975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.834+0000 7f0e77fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 0x55bedb8975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:54626/0 (socket says 172.21.15.129:54626) 2024-05-20T18:02:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.834+0000 7f0e77fff640 1 -- 172.21.15.129:0/2185775109 learned_addr learned my addr 172.21.15.129:0/2185775109 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.836+0000 7f0e777fe640 1 -- 172.21.15.129:0/2185775109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 msgr2=0x55bedb8975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.836+0000 7f0e777fe640 1 --2- 172.21.15.129:0/2185775109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 0x55bedb8975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.836+0000 7f0e777fe640 1 -- 172.21.15.129:0/2185775109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4c76b0 msgr2=0x55bedb891910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:02:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.836+0000 7f0e777fe640 1 --2- 172.21.15.129:0/2185775109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4c76b0 0x55bedb891910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:44.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.836+0000 7f0e777fe640 1 -- 172.21.15.129:0/2185775109 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bedb5003f0 con 0x55bedb4bf4d0 2024-05-20T18:02:44.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.842+0000 7f0e777fe640 1 --2- 172.21.15.129:0/2185775109 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 0x55bedb8913d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0e680ccdc0 tx=0x7f0e68009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a1947333abeae48 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:44.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.843+0000 7f0e56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:02:44.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.846+0000 7f0e56ffd640 1 -- 172.21.15.129:0/2185775109 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e680b1030 con 0x55bedb4bf4d0 2024-05-20T18:02:44.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.846+0000 7f0e84a10740 1 -- 172.21.15.129:0/2185775109 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e6809c350 con 0x55bedb4bf4d0 2024-05-20T18:02:44.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.846+0000 7f0e56ffd640 1 -- 172.21.15.129:0/2185775109 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f0e68016330 con 0x55bedb4bf4d0 2024-05-20T18:02:44.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.852+0000 7f0e56ffd640 1 -- 172.21.15.129:0/2185775109 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0e6809b790 con 0x55bedb4bf4d0 2024-05-20T18:02:44.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.852+0000 7f0e84a10740 10 client.25792 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:44.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.852+0000 7f0e84a10740 1 -- 172.21.15.129:0/2185775109 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55bedb4c00f0 con 0x55bedb4bf4d0 2024-05-20T18:02:44.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.853+0000 7f0e56ffd640 1 -- 172.21.15.129:0/2185775109 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e6809c350 con 0x55bedb4bf4d0 2024-05-20T18:02:44.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.853+0000 7f0e56ffd640 1 -- 172.21.15.129:0/2185775109 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f0e680a4430 con 0x55bedb4bf4d0 2024-05-20T18:02:44.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.853+0000 7f0e84a10740 10 client.25792 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:44.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.853+0000 7f0e84a10740 10 client.25792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:44.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.853+0000 7f0e84a10740 20 client.25792 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:44.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.853+0000 7f0e84a10740 1 --2- 172.21.15.129:0/2185775109 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bedb4a6f60 0x55bedb8a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:44.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.853+0000 7f0e84a10740 4 client.25792 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:44.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.853+0000 7f0e84a10740 1 -- 172.21.15.129:0/2185775109 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bedb7fa430 con 0x55bedb4a6f60 2024-05-20T18:02:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.861+0000 7f0e76ffd640 1 --2- 172.21.15.129:0/2185775109 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bedb4a6f60 0x55bedb8a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:44.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.878+0000 7f0e76ffd640 1 --2- 172.21.15.129:0/2185775109 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bedb4a6f60 0x55bedb8a3a50 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b50aa79548d8b89c server_cookie=13ae488fdf82ff77 in_seq=0 out_seq=0 2024-05-20T18:02:44.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:44.878+0000 7f0e56ffd640 10 client.25792 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:45.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:45 smithi154 ceph-mon[36482]: pgmap v129: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 742 MiB/s wr, 2.01k op/s 2024-05-20T18:02:45.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:45 smithi129 ceph-mon[32029]: pgmap v129: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 742 MiB/s wr, 2.01k op/s 2024-05-20T18:02:45.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:45 smithi204 ceph-mon[37076]: pgmap v129: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 742 MiB/s wr, 2.01k op/s 2024-05-20T18:02:45.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.635+0000 7f0e56ffd640 1 -- 172.21.15.129:0/2185775109 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bedb7fa430 con 0x55bedb4a6f60 2024-05-20T18:02:45.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.635+0000 7f0e56ffd640 10 client.25792 handle_command_reply: tid=0 2024-05-20T18:02:45.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.635+0000 7f0e84a10740 1 client.25792 shutdown 2024-05-20T18:02:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:02:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:02:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:02:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:02:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c49ffb640 1 -- 172.21.15.129:0/469731592 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f4c580c1890 con 0x561999e1a1c0 2024-05-20T18:02:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c49ffb640 10 client.25780 handle_command_reply: tid=0 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c6fa18740 1 client.25780 shutdown 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c6fa18740 1 -- 172.21.15.129:0/469731592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e654d0 msgr2=0x56199a2a2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e654d0 0x56199a2a2620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4c5400dfd0 tx=0x7f4c54005230 comp rx=0 tx=0).stop 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c6fa18740 1 -- 172.21.15.129:0/469731592 shutdown_connections 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561999e1a1c0 0x56199a2abe30 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561999e861b0 0x56199a237380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561999e654d0 0x56199a2a2620 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c6fa18740 1 --2- 172.21.15.129:0/469731592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561999e5d2f0 0x56199a2a20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c6fa18740 1 -- 172.21.15.129:0/469731592 >> 172.21.15.129:0/469731592 conn(0x561999e7a300 msgr2=0x56199a230da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.640+0000 7f4c49ffb640 0 client.25780 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.641+0000 7f4c6fa18740 1 -- 172.21.15.129:0/469731592 shutdown_connections 2024-05-20T18:02:45.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.641+0000 7f4c6fa18740 1 -- 172.21.15.129:0/469731592 wait complete. 2024-05-20T18:02:45.643 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.641+0000 7f4c6fa18740 20 client.25780 trim_cache size 0 max 16384 2024-05-20T18:02:45.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.641+0000 7f0e84a10740 1 -- 172.21.15.129:0/2185775109 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 msgr2=0x55bedb8913d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:45.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.641+0000 7f0e84a10740 1 --2- 172.21.15.129:0/2185775109 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 0x55bedb8913d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0e680ccdc0 tx=0x7f0e68009310 comp rx=0 tx=0).stop 2024-05-20T18:02:45.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.641+0000 7f0e84a10740 1 -- 172.21.15.129:0/2185775109 shutdown_connections 2024-05-20T18:02:45.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.641+0000 7f0e84a10740 1 --2- 172.21.15.129:0/2185775109 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bedb4a6f60 0x55bedb8a3a50 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.642+0000 7f0e84a10740 1 --2- 172.21.15.129:0/2185775109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 0x55bedb8975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.642+0000 7f0e84a10740 1 --2- 172.21.15.129:0/2185775109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4c76b0 0x55bedb891910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.642+0000 7f0e84a10740 1 --2- 172.21.15.129:0/2185775109 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb4bf4d0 0x55bedb8913d0 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0e680ccdc0 tx=0x7f0e68009310 comp rx=0 tx=0).stop 2024-05-20T18:02:45.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.642+0000 7f0e84a10740 1 -- 172.21.15.129:0/2185775109 >> 172.21.15.129:0/2185775109 conn(0x55bedb4dc4e0 msgr2=0x55bedb886d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:45.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.643+0000 7f0e56ffd640 0 client.25792 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:45.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.643+0000 7f0e84a10740 1 -- 172.21.15.129:0/2185775109 shutdown_connections 2024-05-20T18:02:45.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.643+0000 7f0e84a10740 1 -- 172.21.15.129:0/2185775109 wait complete. 2024-05-20T18:02:45.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.644+0000 7f0e84a10740 20 client.25792 trim_cache size 0 max 16384 2024-05-20T18:02:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.651+0000 7f4c6359e640 1 -- 172.21.15.129:0/1382495118 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4c4c0a2f30 msgr2=0x7f4c4c0a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.651+0000 7f4c6359e640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4c4c0a2f30 0x7f4c4c0a5440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x56199a226620 tx=0x7f4c5c007760 comp rx=0 tx=0).stop 2024-05-20T18:02:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.651+0000 7f4c6359e640 1 -- 172.21.15.129:0/1382495118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c6410f270 msgr2=0x7f4c64119e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.651+0000 7f4c6359e640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c6410f270 0x7f4c64119e80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c58033040 tx=0x7f4c58005230 comp rx=0 tx=0).stop 2024-05-20T18:02:45.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.658+0000 7f4c6359e640 1 -- 172.21.15.129:0/1382495118 shutdown_connections 2024-05-20T18:02:45.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.658+0000 7f4c6359e640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4c4c0a2f30 0x7f4c4c0a5440 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.658+0000 7f4c6359e640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c6410f270 0x7f4c64119e80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.658+0000 7f4c6359e640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c6410e890 0x7f4c64119940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.658+0000 7f4c6359e640 1 --2- 172.21.15.129:0/1382495118 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c64109090 0x7f4c64119400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.658+0000 7f4c6359e640 1 -- 172.21.15.129:0/1382495118 >> 172.21.15.129:0/1382495118 conn(0x7f4c6406e690 msgr2=0x7f4c6410c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:45.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.658+0000 7f4c6359e640 1 -- 172.21.15.129:0/1382495118 shutdown_connections 2024-05-20T18:02:45.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:45.658+0000 7f4c6359e640 1 -- 172.21.15.129:0/1382495118 wait complete. 2024-05-20T18:02:45.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.778+0000 7f0e84a10740 1 Processor -- start 2024-05-20T18:02:45.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.778+0000 7f0e84a10740 1 -- start start 2024-05-20T18:02:45.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.786+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4c76b0 0x55bedb890070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:45.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.787+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 0x55bedb882ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:45.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.787+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb47c3a0 0x55bedb8830e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:45.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.787+0000 7f0e84a10740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781bb5e0 con 0x55bedb4c76b0 2024-05-20T18:02:45.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.787+0000 7f0e84a10740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781bde60 con 0x55bedb4e8390 2024-05-20T18:02:45.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.787+0000 7f0e84a10740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781bbc50 con 0x55bedb47c3a0 2024-05-20T18:02:45.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.787+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4c76b0 0x55bedb890070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:45.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.788+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4c76b0 0x55bedb890070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40868/0 (socket says 172.21.15.129:40868) 2024-05-20T18:02:45.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.788+0000 7f0e777fe640 1 -- 172.21.15.129:0/461736708 learned_addr learned my addr 172.21.15.129:0/461736708 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:45.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.791+0000 7f0e76ffd640 1 --2- 172.21.15.129:0/461736708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 0x55bedb882ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:45.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.791+0000 7f0e77fff640 1 --2- 172.21.15.129:0/461736708 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb47c3a0 0x55bedb8830e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:45.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.792+0000 7f0e777fe640 1 -- 172.21.15.129:0/461736708 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb47c3a0 msgr2=0x55bedb8830e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:45.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.792+0000 7f0e777fe640 1 --2- 172.21.15.129:0/461736708 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb47c3a0 0x55bedb8830e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.792+0000 7f0e777fe640 1 -- 172.21.15.129:0/461736708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 msgr2=0x55bedb882ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:45.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.792+0000 7f0e777fe640 1 --2- 172.21.15.129:0/461736708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 0x55bedb882ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.792+0000 7f0e777fe640 1 -- 172.21.15.129:0/461736708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bedb8920b0 con 0x55bedb4c76b0 2024-05-20T18:02:45.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.795+0000 7f0e76ffd640 1 --2- 172.21.15.129:0/461736708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 0x55bedb882ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:02:45.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.797+0000 7f0e777fe640 1 --2- 172.21.15.129:0/461736708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4c76b0 0x55bedb890070 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800c6a0 tx=0x7f0e680d5c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2055fa13d7d2e89a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:45.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.799+0000 7f0e5d7fa640 1 -- 172.21.15.129:0/461736708 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6809ba20 con 0x55bedb4c76b0 2024-05-20T18:02:45.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.800+0000 7f0e5d7fa640 1 -- 172.21.15.129:0/461736708 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e680b4e50 con 0x55bedb4c76b0 2024-05-20T18:02:45.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.802+0000 7f0e84a10740 1 -- 172.21.15.129:0/461736708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4c76b0 msgr2=0x55bedb890070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:45.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.802+0000 7f0e84a10740 1 --2- 172.21.15.129:0/461736708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4c76b0 0x55bedb890070 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800c6a0 tx=0x7f0e680d5c50 comp rx=0 tx=0).stop 2024-05-20T18:02:45.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.802+0000 7f0e84a10740 1 -- 172.21.15.129:0/461736708 shutdown_connections 2024-05-20T18:02:45.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.802+0000 7f0e84a10740 1 --2- 172.21.15.129:0/461736708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb4e8390 0x55bedb882ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.802+0000 7f0e84a10740 1 --2- 172.21.15.129:0/461736708 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb47c3a0 0x55bedb8830e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.802+0000 7f0e84a10740 1 --2- 172.21.15.129:0/461736708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4c76b0 0x55bedb890070 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.802+0000 7f0e84a10740 1 -- 172.21.15.129:0/461736708 >> 172.21.15.129:0/461736708 conn(0x55bedb4dc4e0 msgr2=0x55bedb88f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:45.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.803+0000 7f0e84a10740 1 -- 172.21.15.129:0/461736708 shutdown_connections 2024-05-20T18:02:45.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.803+0000 7f0e84a10740 1 -- 172.21.15.129:0/461736708 wait complete. 2024-05-20T18:02:45.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.804+0000 7f0e84a10740 1 -- start start 2024-05-20T18:02:45.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.804+0000 7f0e84a10740 1 Processor -- start 2024-05-20T18:02:45.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.808+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb47c3a0 0x55bedb89a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:45.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.809+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4e8390 0x55bedb89ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:45.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.809+0000 7f0e84a10740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb89b1b0 0x55bedb909b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:45.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.809+0000 7f0e84a10740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781bc320 con 0x55bedb4e8390 2024-05-20T18:02:45.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.809+0000 7f0e84a10740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781bd790 con 0x55bedb89b1b0 2024-05-20T18:02:45.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.810+0000 7f0e84a10740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781bc9f0 con 0x55bedb47c3a0 2024-05-20T18:02:45.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.810+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb47c3a0 0x55bedb89a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:45.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.810+0000 7f0e777fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb47c3a0 0x55bedb89a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:51534/0 (socket says 172.21.15.129:51534) 2024-05-20T18:02:45.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.810+0000 7f0e777fe640 1 -- 172.21.15.129:0/1014707163 learned_addr learned my addr 172.21.15.129:0/1014707163 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:45.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.810+0000 7f0e77fff640 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb89b1b0 0x55bedb909b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:45.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.810+0000 7f0e84a10740 10 client.0 _ll_register_callbacks cb 0x7f0e680d5a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:45.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.814+0000 7f0e77fff640 1 -- 172.21.15.129:0/1014707163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb47c3a0 msgr2=0x55bedb89a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:45.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.814+0000 7f0e77fff640 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb47c3a0 0x55bedb89a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.814+0000 7f0e77fff640 1 -- 172.21.15.129:0/1014707163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4e8390 msgr2=0x55bedb89ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:45.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.813+0000 7f0e76ffd640 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4e8390 0x55bedb89ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:45.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.814+0000 7f0e77fff640 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4e8390 0x55bedb89ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.814+0000 7f0e77fff640 1 -- 172.21.15.129:0/1014707163 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bedb90a100 con 0x55bedb89b1b0 2024-05-20T18:02:45.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.815+0000 7f0e76ffd640 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4e8390 0x55bedb89ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:02:45.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.815+0000 7f0e77fff640 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb89b1b0 0x55bedb909b70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009b80 tx=0x7f0e70004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a729d66dbad4bccf server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:45.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.825+0000 7f0e56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:02:45.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.825+0000 7f0e56ffd640 1 -- 172.21.15.129:0/1014707163 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e700117a0 con 0x55bedb89b1b0 2024-05-20T18:02:45.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.825+0000 7f0e56ffd640 1 -- 172.21.15.129:0/1014707163 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f0e70011940 con 0x55bedb89b1b0 2024-05-20T18:02:45.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.825+0000 7f0e84a10740 1 -- 172.21.15.129:0/1014707163 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bedb7f4780 con 0x55bedb89b1b0 2024-05-20T18:02:45.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.829+0000 7f0e56ffd640 1 -- 172.21.15.129:0/1014707163 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f0e70011b40 con 0x55bedb89b1b0 2024-05-20T18:02:45.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.829+0000 7f0e84a10740 10 client.25775 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:45.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.829+0000 7f0e84a10740 1 -- 172.21.15.129:0/1014707163 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55bedb90b3d0 con 0x55bedb89b1b0 2024-05-20T18:02:45.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.832+0000 7f0e56ffd640 1 -- 172.21.15.129:0/1014707163 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bedb7f4780 con 0x55bedb89b1b0 2024-05-20T18:02:45.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.832+0000 7f0e56ffd640 1 -- 172.21.15.129:0/1014707163 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f0e7000f040 con 0x55bedb89b1b0 2024-05-20T18:02:45.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.832+0000 7f0e84a10740 10 client.25775 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:45.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.833+0000 7f0e84a10740 10 client.25775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:45.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.833+0000 7f0e84a10740 20 client.25775 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:45.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.833+0000 7f0e84a10740 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bedb90bb50 0x55bedb90df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:45.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.833+0000 7f0e84a10740 4 client.25775 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:02:45.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.833+0000 7f0e84a10740 1 -- 172.21.15.129:0/1014707163 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bedb7716f0 con 0x55bedb90bb50 2024-05-20T18:02:45.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.836+0000 7f0e777fe640 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bedb90bb50 0x55bedb90df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:45.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.839+0000 7f0e777fe640 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bedb90bb50 0x55bedb90df70 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bb05f04419b22db server_cookie=560ce71579fc2b94 in_seq=0 out_seq=0 2024-05-20T18:02:45.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.840+0000 7f0e56ffd640 10 client.25775 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:45.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.845+0000 7f0e56ffd640 1 -- 172.21.15.129:0/1014707163 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bedb7716f0 con 0x55bedb90bb50 2024-05-20T18:02:45.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.845+0000 7f0e56ffd640 10 client.25775 handle_command_reply: tid=0 2024-05-20T18:02:45.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.845+0000 7f0e84a10740 1 client.25775 shutdown 2024-05-20T18:02:45.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.848+0000 7f0e84a10740 1 -- 172.21.15.129:0/1014707163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb89b1b0 msgr2=0x55bedb909b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:45.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.848+0000 7f0e84a10740 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb89b1b0 0x55bedb909b70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009b80 tx=0x7f0e70004ab0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.850+0000 7f0e84a10740 1 -- 172.21.15.129:0/1014707163 shutdown_connections 2024-05-20T18:02:45.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.850+0000 7f0e84a10740 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55bedb90bb50 0x55bedb90df70 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.850+0000 7f0e84a10740 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bedb89b1b0 0x55bedb909b70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.850+0000 7f0e84a10740 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bedb4e8390 0x55bedb89ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.850+0000 7f0e84a10740 1 --2- 172.21.15.129:0/1014707163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bedb47c3a0 0x55bedb89a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.850+0000 7f0e84a10740 1 -- 172.21.15.129:0/1014707163 >> 172.21.15.129:0/1014707163 conn(0x55bedb4dc4e0 msgr2=0x55bedb8927c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:45.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.851+0000 7f0e56ffd640 0 client.25775 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:45.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.852+0000 7f0e84a10740 1 -- 172.21.15.129:0/1014707163 shutdown_connections 2024-05-20T18:02:45.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.854+0000 7f0e84a10740 1 -- 172.21.15.129:0/1014707163 wait complete. 2024-05-20T18:02:45.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.857+0000 7f0e84a10740 20 client.25775 trim_cache size 0 max 16384 2024-05-20T18:02:45.860 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:02:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:02:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:02:45.861 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:02:45.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.870+0000 7f0e7da91640 1 -- 172.21.15.129:0/3507043329 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0e600a2de0 msgr2=0x7f0e600a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:45.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.870+0000 7f0e7da91640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0e600a2de0 0x7f0e600a52f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0e700092f0 tx=0x7f0e70009280 comp rx=0 tx=0).stop 2024-05-20T18:02:45.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.870+0000 7f0e7da91640 1 -- 172.21.15.129:0/3507043329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e78109060 msgr2=0x7f0e781a8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:45.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.870+0000 7f0e7da91640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e78109060 0x7f0e781a8580 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800e9f0 tx=0x7f0e6800eeb0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.871+0000 7f0e7da91640 1 -- 172.21.15.129:0/3507043329 shutdown_connections 2024-05-20T18:02:45.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.871+0000 7f0e7da91640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0e600a2de0 0x7f0e600a52f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.871+0000 7f0e7da91640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e78119770 0x7f0e781a8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.871+0000 7f0e7da91640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e7811c0d0 0x7f0e781acda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.871+0000 7f0e7da91640 1 --2- 172.21.15.129:0/3507043329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e78109060 0x7f0e781a8580 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800e9f0 tx=0x7f0e6800eeb0 comp rx=0 tx=0).stop 2024-05-20T18:02:45.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.871+0000 7f0e7da91640 1 -- 172.21.15.129:0/3507043329 >> 172.21.15.129:0/3507043329 conn(0x7f0e7806d750 msgr2=0x7f0e78119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:45.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.878+0000 7f0e7da91640 1 -- 172.21.15.129:0/3507043329 shutdown_connections 2024-05-20T18:02:45.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:45.879+0000 7f0e7da91640 1 -- 172.21.15.129:0/3507043329 wait complete. 2024-05-20T18:02:45.948 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:02:45.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:02:46.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.432+0000 7ff5d5605640 1 -- 172.21.15.129:0/254596086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d01017d0 msgr2=0x7ff5d0101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:46.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.432+0000 7ff5d5605640 1 --2- 172.21.15.129:0/254596086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d01017d0 0x7ff5d0101be0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0009900 tx=0x7ff5c002a420 comp rx=0 tx=0).stop 2024-05-20T18:02:46.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.440+0000 7ff5d5605640 1 -- 172.21.15.129:0/254596086 shutdown_connections 2024-05-20T18:02:46.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.440+0000 7ff5d5605640 1 --2- 172.21.15.129:0/254596086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5d0106c50 0x7ff5d01070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:46.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.440+0000 7ff5d5605640 1 --2- 172.21.15.129:0/254596086 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5d0106220 0x7ff5d0106630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:46.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.440+0000 7ff5d5605640 1 --2- 172.21.15.129:0/254596086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d01017d0 0x7ff5d0101be0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:46.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.440+0000 7ff5d5605640 1 -- 172.21.15.129:0/254596086 >> 172.21.15.129:0/254596086 conn(0x7ff5d00fcec0 msgr2=0x7ff5d00ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:46.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.443+0000 7ff5d5605640 1 -- 172.21.15.129:0/254596086 shutdown_connections 2024-05-20T18:02:46.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.443+0000 7ff5d5605640 1 -- 172.21.15.129:0/254596086 wait complete. 2024-05-20T18:02:46.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.445+0000 7ff5d5605640 1 Processor -- start 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.445+0000 7ff5d5605640 1 -- start start 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.446+0000 7ff5d5605640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d01017d0 0x7ff5d01a5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.446+0000 7ff5d5605640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5d0106220 0x7ff5d01a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.446+0000 7ff5d5605640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5d0106c50 0x7ff5d019fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.446+0000 7ff5d5605640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d0119290 con 0x7ff5d01017d0 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.446+0000 7ff5d5605640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d0119110 con 0x7ff5d0106220 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.446+0000 7ff5d5605640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d0119410 con 0x7ff5d0106c50 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.447+0000 7ff5cf7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5d0106220 0x7ff5d01a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.447+0000 7ff5d4e04640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5d0106c50 0x7ff5d019fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.447+0000 7ff5d4e04640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5d0106c50 0x7ff5d019fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:51548/0 (socket says 172.21.15.129:51548) 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.447+0000 7ff5d4e04640 1 -- 172.21.15.129:0/76388503 learned_addr learned my addr 172.21.15.129:0/76388503 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.448+0000 7ff5cffff640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d01017d0 0x7ff5d01a5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:46.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.448+0000 7ff5d4e04640 1 -- 172.21.15.129:0/76388503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5d0106220 msgr2=0x7ff5d01a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:46.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.448+0000 7ff5d4e04640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5d0106220 0x7ff5d01a5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:46.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.448+0000 7ff5d4e04640 1 -- 172.21.15.129:0/76388503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d01017d0 msgr2=0x7ff5d01a5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:46.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.448+0000 7ff5d4e04640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d01017d0 0x7ff5d01a5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:46.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.449+0000 7ff5d4e04640 1 -- 172.21.15.129:0/76388503 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d01a00d0 con 0x7ff5d0106c50 2024-05-20T18:02:46.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.451+0000 7ff5d4e04640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5d0106c50 0x7ff5d019fa30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff5c400c970 tx=0x7ff5c400ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:46.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.476+0000 7ff5cd7fa640 1 -- 172.21.15.129:0/76388503 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4007bb0 con 0x7ff5d0106c50 2024-05-20T18:02:46.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.476+0000 7ff5d5605640 1 -- 172.21.15.129:0/76388503 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d01a03c0 con 0x7ff5d0106c50 2024-05-20T18:02:46.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.476+0000 7ff5d5605640 1 -- 172.21.15.129:0/76388503 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d006b570 con 0x7ff5d0106c50 2024-05-20T18:02:46.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.476+0000 7ff5cd7fa640 1 -- 172.21.15.129:0/76388503 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5c4007d50 con 0x7ff5d0106c50 2024-05-20T18:02:46.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.476+0000 7ff5cd7fa640 1 -- 172.21.15.129:0/76388503 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c400f450 con 0x7ff5d0106c50 2024-05-20T18:02:46.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.481+0000 7ff5d5605640 1 -- 172.21.15.129:0/76388503 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5bc005290 con 0x7ff5d0106c50 2024-05-20T18:02:46.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.488+0000 7ff5cd7fa640 1 -- 172.21.15.129:0/76388503 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ff5c40046c0 con 0x7ff5d0106c50 2024-05-20T18:02:46.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.491+0000 7ff5cd7fa640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff5b80a2f00 0x7ff5b80a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:46.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.491+0000 7ff5cd7fa640 1 -- 172.21.15.129:0/76388503 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7ff5c40cee90 con 0x7ff5d0106c50 2024-05-20T18:02:46.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.494+0000 7ff5cffff640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff5b80a2f00 0x7ff5b80a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:46.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.495+0000 7ff5cffff640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff5b80a2f00 0x7ff5b80a5410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0005ab0 tx=0x7ff5c002a920 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:46.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:46.496+0000 7ff5cd7fa640 1 -- 172.21.15.129:0/76388503 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff5c4014030 con 0x7ff5d0106c50 2024-05-20T18:02:47.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.201+0000 7ff5d5605640 1 -- 172.21.15.129:0/76388503 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5bc005580 con 0x7ff5d0106c50 2024-05-20T18:02:47.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.206+0000 7ff5cd7fa640 1 -- 172.21.15.129:0/76388503 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11829 (secure 0 0 0) 0x7ff5c409b4f0 con 0x7ff5d0106c50 2024-05-20T18:02:47.211 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:02:47.211 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T17:59:17.074606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25229":{"gid":25229,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/1860679091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":1860679091},{"type":"v1","addr":"172.21.15.154:6837","nonce":1860679091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:02:47.212 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 22 2024-05-20T18:02:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.219+0000 7ff5d5605640 1 -- 172.21.15.129:0/76388503 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff5b80a2f00 msgr2=0x7ff5b80a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.219+0000 7ff5d5605640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff5b80a2f00 0x7ff5b80a5410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0005ab0 tx=0x7ff5c002a920 comp rx=0 tx=0).stop 2024-05-20T18:02:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.219+0000 7ff5d5605640 1 -- 172.21.15.129:0/76388503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5d0106c50 msgr2=0x7ff5d019fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.219+0000 7ff5d5605640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5d0106c50 0x7ff5d019fa30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff5c400c970 tx=0x7ff5c400ce30 comp rx=0 tx=0).stop 2024-05-20T18:02:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.221+0000 7ff5d5605640 1 -- 172.21.15.129:0/76388503 shutdown_connections 2024-05-20T18:02:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.221+0000 7ff5d5605640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff5b80a2f00 0x7ff5b80a5410 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.221+0000 7ff5d5605640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5d0106220 0x7ff5d01a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.221+0000 7ff5d5605640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff5d0106c50 0x7ff5d019fa30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.221+0000 7ff5d5605640 1 --2- 172.21.15.129:0/76388503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5d01017d0 0x7ff5d01a5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.221+0000 7ff5d5605640 1 -- 172.21.15.129:0/76388503 >> 172.21.15.129:0/76388503 conn(0x7ff5d00fcec0 msgr2=0x7ff5d00ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.224+0000 7ff5d5605640 1 -- 172.21.15.129:0/76388503 shutdown_connections 2024-05-20T18:02:47.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.225+0000 7ff5d5605640 1 -- 172.21.15.129:0/76388503 wait complete. 2024-05-20T18:02:47.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:02:47.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:47 smithi154 ceph-mon[36482]: pgmap v130: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 945 MiB/s wr, 2.10k op/s 2024-05-20T18:02:47.417 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:47 smithi129 ceph-mon[32029]: pgmap v130: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 945 MiB/s wr, 2.10k op/s 2024-05-20T18:02:47.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:47 smithi204 ceph-mon[37076]: pgmap v130: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 945 MiB/s wr, 2.10k op/s 2024-05-20T18:02:47.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.813+0000 7f7e2759e640 1 -- 172.21.15.129:0/4163574154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e28105fd0 msgr2=0x7f7e28106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:47.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.813+0000 7f7e2759e640 1 --2- 172.21.15.129:0/4163574154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e28105fd0 0x7f7e28106400 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7e18013150 tx=0x7f7e180327a0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.820+0000 7f7e2759e640 1 -- 172.21.15.129:0/4163574154 shutdown_connections 2024-05-20T18:02:47.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.820+0000 7f7e2759e640 1 --2- 172.21.15.129:0/4163574154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e28106a90 0x7f7e28068e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.820+0000 7f7e2759e640 1 --2- 172.21.15.129:0/4163574154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e28105fd0 0x7f7e28106400 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.820+0000 7f7e2759e640 1 --2- 172.21.15.129:0/4163574154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e28101a40 0x7f7e28101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.820+0000 7f7e2759e640 1 -- 172.21.15.129:0/4163574154 >> 172.21.15.129:0/4163574154 conn(0x7f7e280fcec0 msgr2=0x7f7e280ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:47.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.832+0000 7f7e2759e640 1 -- 172.21.15.129:0/4163574154 shutdown_connections 2024-05-20T18:02:47.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.832+0000 7f7e2759e640 1 -- 172.21.15.129:0/4163574154 wait complete. 2024-05-20T18:02:47.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.834+0000 7f7e2759e640 1 Processor -- start 2024-05-20T18:02:47.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.834+0000 7f7e2759e640 1 -- start start 2024-05-20T18:02:47.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.835+0000 7f7e2759e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e28101a40 0x7f7e281a8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:47.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.835+0000 7f7e2759e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e28105fd0 0x7f7e281a89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:47.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.835+0000 7f7e2759e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e28106a90 0x7f7e281accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:47.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.835+0000 7f7e2759e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28110ac0 con 0x7f7e28101a40 2024-05-20T18:02:47.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.835+0000 7f7e2759e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28110940 con 0x7f7e28106a90 2024-05-20T18:02:47.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.835+0000 7f7e2759e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28110c40 con 0x7f7e28105fd0 2024-05-20T18:02:47.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.836+0000 7f7e2659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e28101a40 0x7f7e281a8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:47.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.836+0000 7f7e1ffff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e28105fd0 0x7f7e281a89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:47.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.836+0000 7f7e2659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e28101a40 0x7f7e281a8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40920/0 (socket says 172.21.15.129:40920) 2024-05-20T18:02:47.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.836+0000 7f7e2659c640 1 -- 172.21.15.129:0/405848438 learned_addr learned my addr 172.21.15.129:0/405848438 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:47.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.836+0000 7f7e26d9d640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e28106a90 0x7f7e281accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:47.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.836+0000 7f7e2659c640 1 -- 172.21.15.129:0/405848438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e28105fd0 msgr2=0x7f7e281a89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:47.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.836+0000 7f7e2659c640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e28105fd0 0x7f7e281a89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.836+0000 7f7e2659c640 1 -- 172.21.15.129:0/405848438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e28106a90 msgr2=0x7f7e281accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:47.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.836+0000 7f7e2659c640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e28106a90 0x7f7e281accb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.837+0000 7f7e2659c640 1 -- 172.21.15.129:0/405848438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e281ad2f0 con 0x7f7e28101a40 2024-05-20T18:02:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.838+0000 7f7e2659c640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e28101a40 0x7f7e281a8490 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000b450 tx=0x7f7e1000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.838+0000 7f7e1f7fe640 1 -- 172.21.15.129:0/405848438 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10015020 con 0x7f7e28101a40 2024-05-20T18:02:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.838+0000 7f7e2759e640 1 -- 172.21.15.129:0/405848438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e281ad5e0 con 0x7f7e28101a40 2024-05-20T18:02:47.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.839+0000 7f7e2759e640 1 -- 172.21.15.129:0/405848438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e2806b570 con 0x7f7e28101a40 2024-05-20T18:02:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.840+0000 7f7e1f7fe640 1 -- 172.21.15.129:0/405848438 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e10011d70 con 0x7f7e28101a40 2024-05-20T18:02:47.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.840+0000 7f7e1f7fe640 1 -- 172.21.15.129:0/405848438 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e100103e0 con 0x7f7e28101a40 2024-05-20T18:02:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.845+0000 7f7e1f7fe640 1 -- 172.21.15.129:0/405848438 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7e10018020 con 0x7f7e28101a40 2024-05-20T18:02:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.847+0000 7f7e1f7fe640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e140a2d30 0x7f7e140a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:47.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.850+0000 7f7e33b98740 1 Processor -- start 2024-05-20T18:02:47.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.851+0000 7f7e1f7fe640 1 -- 172.21.15.129:0/405848438 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f7e1009a640 con 0x7f7e28101a40 2024-05-20T18:02:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.851+0000 7f7e33b98740 1 -- start start 2024-05-20T18:02:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.852+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413db390 0x55a141779f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.852+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413b24d0 0x55a14177a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.852+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14177aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.852+0000 7f7e33b98740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a1416dfc40 con 0x55a1413db390 2024-05-20T18:02:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.852+0000 7f7e33b98740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a141687580 con 0x55a1413b24d0 2024-05-20T18:02:47.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.852+0000 7f7e33b98740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a1416dbc00 con 0x55a1413ba6b0 2024-05-20T18:02:47.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.853+0000 7f7e26d9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14177aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:47.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.853+0000 7f7e26d9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14177aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:51562/0 (socket says 172.21.15.129:51562) 2024-05-20T18:02:47.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.853+0000 7f7e26d9d640 1 -- 172.21.15.129:0/1877759974 learned_addr learned my addr 172.21.15.129:0/1877759974 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:47.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.853+0000 7f7e1ffff640 1 --2- 172.21.15.129:0/1877759974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413b24d0 0x55a14177a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:47.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.858+0000 7f7e2659c640 1 --2- 172.21.15.129:0/1877759974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413db390 0x55a141779f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:47.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.858+0000 7f7e1ffff640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e140a2d30 0x7f7e140a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:47.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.859+0000 7f7e26d9d640 1 -- 172.21.15.129:0/1877759974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413b24d0 msgr2=0x55a14177a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:47.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.859+0000 7f7e26d9d640 1 --2- 172.21.15.129:0/1877759974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413b24d0 0x55a14177a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.859+0000 7f7e26d9d640 1 -- 172.21.15.129:0/1877759974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413db390 msgr2=0x55a141779f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:47.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.859+0000 7f7e26d9d640 1 --2- 172.21.15.129:0/1877759974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413db390 0x55a141779f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.859+0000 7f7e26d9d640 1 -- 172.21.15.129:0/1877759974 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1413f33f0 con 0x55a1413ba6b0 2024-05-20T18:02:47.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.859+0000 7f7e1ffff640 1 --2- 172.21.15.129:0/1877759974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413b24d0 0x55a14177a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:02:47.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.862+0000 7f7e26d9d640 1 --2- 172.21.15.129:0/1877759974 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14177aa10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009f20 tx=0x7f7e2000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7b2807511585e26 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:47.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.864+0000 7f7e1cff9640 1 -- 172.21.15.129:0/1877759974 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2001a070 con 0x55a1413ba6b0 2024-05-20T18:02:47.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.864+0000 7f7e1cff9640 1 -- 172.21.15.129:0/1877759974 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e200158f0 con 0x55a1413ba6b0 2024-05-20T18:02:47.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.866+0000 7f7e33b98740 1 -- 172.21.15.129:0/1877759974 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 msgr2=0x55a14177aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:47.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.866+0000 7f7e33b98740 1 --2- 172.21.15.129:0/1877759974 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14177aa10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009f20 tx=0x7f7e2000c8b0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.868+0000 7f7e33b98740 1 -- 172.21.15.129:0/1877759974 shutdown_connections 2024-05-20T18:02:47.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.868+0000 7f7e33b98740 1 --2- 172.21.15.129:0/1877759974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413b24d0 0x55a14177a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.868+0000 7f7e33b98740 1 --2- 172.21.15.129:0/1877759974 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14177aa10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.868+0000 7f7e33b98740 1 --2- 172.21.15.129:0/1877759974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413db390 0x55a141779f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.868+0000 7f7e33b98740 1 -- 172.21.15.129:0/1877759974 >> 172.21.15.129:0/1877759974 conn(0x55a1413cf4e0 msgr2=0x55a141776740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:47.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.877+0000 7f7e33b98740 1 -- 172.21.15.129:0/1877759974 shutdown_connections 2024-05-20T18:02:47.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.877+0000 7f7e33b98740 1 -- 172.21.15.129:0/1877759974 wait complete. 2024-05-20T18:02:47.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.878+0000 7f7e33b98740 1 -- start start 2024-05-20T18:02:47.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.878+0000 7f7e33b98740 1 Processor -- start 2024-05-20T18:02:47.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.879+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413b24d0 0x55a141784470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:47.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.879+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413ba6b0 0x55a1417849b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:47.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.879+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413db390 0x55a141790880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:47.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.879+0000 7f7e33b98740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a141787380 con 0x55a1413ba6b0 2024-05-20T18:02:47.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.879+0000 7f7e33b98740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a14169ac40 con 0x55a1413db390 2024-05-20T18:02:47.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.879+0000 7f7e33b98740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a141787500 con 0x55a1413b24d0 2024-05-20T18:02:47.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.879+0000 7f7e33b98740 10 client.0 _ll_register_callbacks cb 0x55a1413f2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:47.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.880+0000 7f7e1ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413ba6b0 0x55a1417849b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:47.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.880+0000 7f7e1ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413ba6b0 0x55a1417849b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40934/0 (socket says 172.21.15.129:40934) 2024-05-20T18:02:47.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.880+0000 7f7e1ffff640 1 -- 172.21.15.129:0/445153006 learned_addr learned my addr 172.21.15.129:0/445153006 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:47.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.880+0000 7f7e26d9d640 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413db390 0x55a141790880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:47.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.881+0000 7f7e26d9d640 1 -- 172.21.15.129:0/445153006 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413b24d0 msgr2=0x55a141784470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:02:47.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.881+0000 7f7e26d9d640 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413b24d0 0x55a141784470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.881+0000 7f7e26d9d640 1 -- 172.21.15.129:0/445153006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413ba6b0 msgr2=0x55a1417849b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:47.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.881+0000 7f7e26d9d640 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413ba6b0 0x55a1417849b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.881+0000 7f7e26d9d640 1 -- 172.21.15.129:0/445153006 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1414bd7d0 con 0x55a1413db390 2024-05-20T18:02:47.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.883+0000 7f7e26d9d640 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413db390 0x55a141790880 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7e20000f80 tx=0x7f7e2000c2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb720116a586352d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.888+0000 7f7e0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:02:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.888+0000 7f7e0dffb640 1 -- 172.21.15.129:0/445153006 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2001a040 con 0x55a1413db390 2024-05-20T18:02:47.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.889+0000 7f7e33b98740 1 -- 172.21.15.129:0/445153006 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e200158f0 con 0x55a1413db390 2024-05-20T18:02:47.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.889+0000 7f7e0dffb640 1 -- 172.21.15.129:0/445153006 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f7e20012040 con 0x55a1413db390 2024-05-20T18:02:47.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.889+0000 7f7e0dffb640 1 -- 172.21.15.129:0/445153006 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7e2002d050 con 0x55a1413db390 2024-05-20T18:02:47.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.892+0000 7f7e33b98740 10 client.25787 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:47.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.892+0000 7f7e33b98740 1 -- 172.21.15.129:0/445153006 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a141793990 con 0x55a1413db390 2024-05-20T18:02:47.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.898+0000 7f7e0dffb640 1 -- 172.21.15.129:0/445153006 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e200158f0 con 0x55a1413db390 2024-05-20T18:02:47.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.898+0000 7f7e0dffb640 1 -- 172.21.15.129:0/445153006 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f7e20028b10 con 0x55a1413db390 2024-05-20T18:02:47.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.901+0000 7f7e33b98740 10 client.25787 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:47.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.901+0000 7f7e33b98740 10 client.25787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:47.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.901+0000 7f7e33b98740 20 client.25787 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:47.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.901+0000 7f7e33b98740 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a141399f60 0x55a1417972a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:47.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.901+0000 7f7e33b98740 4 client.25787 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:47.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.901+0000 7f7e33b98740 1 -- 172.21.15.129:0/445153006 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1416ed370 con 0x55a141399f60 2024-05-20T18:02:47.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.902+0000 7f7e2659c640 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a141399f60 0x55a1417972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:47.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.903+0000 7f7e1ffff640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e140a2d30 0x7f7e140a5240 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55a141771d80 tx=0x7f7e180327a0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.914+0000 7f7e2659c640 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a141399f60 0x55a1417972a0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ad5deb2b2c855ec server_cookie=9f0554de31fad27a in_seq=0 out_seq=0 2024-05-20T18:02:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.914+0000 7f7e0dffb640 10 client.25787 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:47.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.921+0000 7f7e0dffb640 1 -- 172.21.15.129:0/445153006 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a1416ed370 con 0x55a141399f60 2024-05-20T18:02:47.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.921+0000 7f7e0dffb640 10 client.25787 handle_command_reply: tid=0 2024-05-20T18:02:47.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.921+0000 7f7e33b98740 1 client.25787 shutdown 2024-05-20T18:02:47.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.925+0000 7f7e33b98740 1 -- 172.21.15.129:0/445153006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413db390 msgr2=0x55a141790880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:47.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.925+0000 7f7e33b98740 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413db390 0x55a141790880 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7e20000f80 tx=0x7f7e2000c2e0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.939+0000 7f7e33b98740 1 -- 172.21.15.129:0/445153006 shutdown_connections 2024-05-20T18:02:47.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.939+0000 7f7e33b98740 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a141399f60 0x55a1417972a0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.939+0000 7f7e33b98740 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413db390 0x55a141790880 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.939+0000 7f7e33b98740 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413ba6b0 0x55a1417849b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.939+0000 7f7e33b98740 1 --2- 172.21.15.129:0/445153006 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413b24d0 0x55a141784470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:47.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.939+0000 7f7e33b98740 1 -- 172.21.15.129:0/445153006 >> 172.21.15.129:0/445153006 conn(0x55a1413cf4e0 msgr2=0x55a14177bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:47.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.939+0000 7f7e0dffb640 0 client.25787 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:47.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.943+0000 7f7e33b98740 1 -- 172.21.15.129:0/445153006 shutdown_connections 2024-05-20T18:02:47.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.943+0000 7f7e33b98740 1 -- 172.21.15.129:0/445153006 wait complete. 2024-05-20T18:02:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:47.944+0000 7f7e33b98740 20 client.25787 trim_cache size 0 max 16384 2024-05-20T18:02:48.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.007+0000 7f7e33b98740 1 Processor -- start 2024-05-20T18:02:48.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.008+0000 7f7e33b98740 1 -- start start 2024-05-20T18:02:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.009+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413b24d0 0x55a141790460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.009+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14179caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.009+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413db390 0x55a14179cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.009+0000 7f7e33b98740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0800c8d0 con 0x55a1413b24d0 2024-05-20T18:02:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.009+0000 7f7e33b98740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0800f150 con 0x55a1413db390 2024-05-20T18:02:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.009+0000 7f7e33b98740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0800cf40 con 0x55a1413ba6b0 2024-05-20T18:02:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.010+0000 7f7e1ffff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14179caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.010+0000 7f7e1ffff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14179caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:51564/0 (socket says 172.21.15.129:51564) 2024-05-20T18:02:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.010+0000 7f7e1ffff640 1 -- 172.21.15.129:0/3654691556 learned_addr learned my addr 172.21.15.129:0/3654691556 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.011+0000 7f7e26d9d640 1 --2- 172.21.15.129:0/3654691556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413db390 0x55a14179cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.011+0000 7f7e1ffff640 1 -- 172.21.15.129:0/3654691556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413db390 msgr2=0x55a14179cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.011+0000 7f7e1ffff640 1 --2- 172.21.15.129:0/3654691556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413db390 0x55a14179cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.011+0000 7f7e1ffff640 1 -- 172.21.15.129:0/3654691556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413b24d0 msgr2=0x55a141790460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.011+0000 7f7e1ffff640 1 --2- 172.21.15.129:0/3654691556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413b24d0 0x55a141790460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.011+0000 7f7e1ffff640 1 -- 172.21.15.129:0/3654691556 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a14139a190 con 0x55a1413ba6b0 2024-05-20T18:02:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.014+0000 7f7e1ffff640 1 --2- 172.21.15.129:0/3654691556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14179caa0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7e18032ce0 tx=0x7f7e18005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33bc98ea76c4eba7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.019+0000 7f7e0ffff640 1 -- 172.21.15.129:0/3654691556 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e180043d0 con 0x55a1413ba6b0 2024-05-20T18:02:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.019+0000 7f7e0ffff640 1 -- 172.21.15.129:0/3654691556 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e18004ba0 con 0x55a1413ba6b0 2024-05-20T18:02:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.019+0000 7f7e0ffff640 1 -- 172.21.15.129:0/3654691556 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18003850 con 0x55a1413ba6b0 2024-05-20T18:02:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.021+0000 7f7e33b98740 1 -- 172.21.15.129:0/3654691556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 msgr2=0x55a14179caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.021+0000 7f7e33b98740 1 --2- 172.21.15.129:0/3654691556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14179caa0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7e18032ce0 tx=0x7f7e18005fd0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.021+0000 7f7e33b98740 1 -- 172.21.15.129:0/3654691556 shutdown_connections 2024-05-20T18:02:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.021+0000 7f7e33b98740 1 --2- 172.21.15.129:0/3654691556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413db390 0x55a14179cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.021+0000 7f7e33b98740 1 --2- 172.21.15.129:0/3654691556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413ba6b0 0x55a14179caa0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.021+0000 7f7e33b98740 1 --2- 172.21.15.129:0/3654691556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413b24d0 0x55a141790460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.021+0000 7f7e33b98740 1 -- 172.21.15.129:0/3654691556 >> 172.21.15.129:0/3654691556 conn(0x55a1413cf4e0 msgr2=0x55a141782b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.023+0000 7f7e33b98740 1 -- 172.21.15.129:0/3654691556 shutdown_connections 2024-05-20T18:02:48.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.023+0000 7f7e33b98740 1 -- 172.21.15.129:0/3654691556 wait complete. 2024-05-20T18:02:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.024+0000 7f7e33b98740 1 -- start start 2024-05-20T18:02:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.024+0000 7f7e33b98740 1 Processor -- start 2024-05-20T18:02:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.025+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413b24d0 0x55a14178ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.025+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413ba6b0 0x55a1417877a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.025+0000 7f7e33b98740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413db390 0x55a141787ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.025+0000 7f7e33b98740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0800d610 con 0x55a1413db390 2024-05-20T18:02:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.025+0000 7f7e33b98740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0800ea80 con 0x55a1413ba6b0 2024-05-20T18:02:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.025+0000 7f7e33b98740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0800dce0 con 0x55a1413b24d0 2024-05-20T18:02:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.025+0000 7f7e33b98740 10 client.0 _ll_register_callbacks cb 0x7f7e20018070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.025+0000 7f7e1ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413ba6b0 0x55a1417877a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.025+0000 7f7e2659c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413b24d0 0x55a14178ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.026+0000 7f7e1ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413ba6b0 0x55a1417877a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:54718/0 (socket says 172.21.15.129:54718) 2024-05-20T18:02:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.026+0000 7f7e1ffff640 1 -- 172.21.15.129:0/2445541719 learned_addr learned my addr 172.21.15.129:0/2445541719 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.026+0000 7f7e26d9d640 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413db390 0x55a141787ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.027+0000 7f7e2659c640 1 -- 172.21.15.129:0/2445541719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413ba6b0 msgr2=0x55a1417877a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.027+0000 7f7e2659c640 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413ba6b0 0x55a1417877a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.027+0000 7f7e2659c640 1 -- 172.21.15.129:0/2445541719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413db390 msgr2=0x55a141787ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.027+0000 7f7e2659c640 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413db390 0x55a141787ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.027+0000 7f7e2659c640 1 -- 172.21.15.129:0/2445541719 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a141788220 con 0x55a1413b24d0 2024-05-20T18:02:48.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.027+0000 7f7e2659c640 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413b24d0 0x55a14178ca00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7e100cd7d0 tx=0x7f7e1001a800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6d00fd25040130f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.028+0000 7f7e0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:02:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.028+0000 7f7e0dffb640 1 -- 172.21.15.129:0/2445541719 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e100ba060 con 0x55a1413b24d0 2024-05-20T18:02:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.028+0000 7f7e33b98740 1 -- 172.21.15.129:0/2445541719 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e18004ba0 con 0x55a1413b24d0 2024-05-20T18:02:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.028+0000 7f7e0dffb640 1 -- 172.21.15.129:0/2445541719 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f7e1001a400 con 0x55a1413b24d0 2024-05-20T18:02:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.029+0000 7f7e0dffb640 1 -- 172.21.15.129:0/2445541719 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7e100b5070 con 0x55a1413b24d0 2024-05-20T18:02:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.029+0000 7f7e33b98740 10 client.25813 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.029+0000 7f7e33b98740 1 -- 172.21.15.129:0/2445541719 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a1417fe1c0 con 0x55a1413b24d0 2024-05-20T18:02:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.029+0000 7f7e0dffb640 1 -- 172.21.15.129:0/2445541719 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e18004ba0 con 0x55a1413b24d0 2024-05-20T18:02:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.030+0000 7f7e0dffb640 1 -- 172.21.15.129:0/2445541719 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f7e1009bb90 con 0x55a1413b24d0 2024-05-20T18:02:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.030+0000 7f7e33b98740 10 client.25813 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.030+0000 7f7e33b98740 10 client.25813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.030+0000 7f7e33b98740 20 client.25813 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.030+0000 7f7e33b98740 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a14136f3a0 0x55a1418009b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.030+0000 7f7e33b98740 4 client.25813 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:02:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.030+0000 7f7e33b98740 1 -- 172.21.15.129:0/2445541719 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a1416e76c0 con 0x55a14136f3a0 2024-05-20T18:02:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.032+0000 7f7e1ffff640 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a14136f3a0 0x55a1418009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.037+0000 7f7e1ffff640 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a14136f3a0 0x55a1418009b0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=286909b36e545523 server_cookie=4642fb96161bca07 in_seq=0 out_seq=0 2024-05-20T18:02:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.041+0000 7f7e0dffb640 10 client.25813 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.041+0000 7f7e0dffb640 1 -- 172.21.15.129:0/2445541719 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a1416e76c0 con 0x55a14136f3a0 2024-05-20T18:02:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.041+0000 7f7e0dffb640 10 client.25813 handle_command_reply: tid=0 2024-05-20T18:02:48.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.041+0000 7f7e33b98740 1 client.25813 shutdown 2024-05-20T18:02:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.049+0000 7f7e33b98740 1 -- 172.21.15.129:0/2445541719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413b24d0 msgr2=0x55a14178ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.049+0000 7f7e33b98740 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413b24d0 0x55a14178ca00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7e100cd7d0 tx=0x7f7e1001a800 comp rx=0 tx=0).stop 2024-05-20T18:02:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.052+0000 7f7e33b98740 1 -- 172.21.15.129:0/2445541719 shutdown_connections 2024-05-20T18:02:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.052+0000 7f7e33b98740 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a14136f3a0 0x55a1418009b0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.052+0000 7f7e33b98740 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a1413ba6b0 0x55a1417877a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.052+0000 7f7e33b98740 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a1413db390 0x55a141787ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.052+0000 7f7e33b98740 1 --2- 172.21.15.129:0/2445541719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a1413b24d0 0x55a14178ca00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.052+0000 7f7e0dffb640 0 client.25813 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.052+0000 7f7e33b98740 1 -- 172.21.15.129:0/2445541719 >> 172.21.15.129:0/2445541719 conn(0x55a1413cf4e0 msgr2=0x55a141782af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.062+0000 7f7e33b98740 1 -- 172.21.15.129:0/2445541719 shutdown_connections 2024-05-20T18:02:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.065+0000 7f7e33b98740 1 -- 172.21.15.129:0/2445541719 wait complete. 2024-05-20T18:02:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.065+0000 7f7e33b98740 20 client.25813 trim_cache size 0 max 16384 2024-05-20T18:02:48.067 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:02:48.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.086+0000 7f7e2759e640 1 -- 172.21.15.129:0/405848438 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e140a2d30 msgr2=0x7f7e140a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:48.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.086+0000 7f7e2759e640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e140a2d30 0x7f7e140a5240 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55a141771d80 tx=0x7f7e180327a0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.086+0000 7f7e2759e640 1 -- 172.21.15.129:0/405848438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e28101a40 msgr2=0x7f7e281a8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:48.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.086+0000 7f7e2759e640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e28101a40 0x7f7e281a8490 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000b450 tx=0x7f7e1000b910 comp rx=0 tx=0).stop 2024-05-20T18:02:48.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.095+0000 7f7e2759e640 1 -- 172.21.15.129:0/405848438 shutdown_connections 2024-05-20T18:02:48.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.095+0000 7f7e2759e640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e140a2d30 0x7f7e140a5240 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.095+0000 7f7e2759e640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e28106a90 0x7f7e281accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.095+0000 7f7e2759e640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e28105fd0 0x7f7e281a89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.095+0000 7f7e2759e640 1 --2- 172.21.15.129:0/405848438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e28101a40 0x7f7e281a8490 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:48.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.095+0000 7f7e2759e640 1 -- 172.21.15.129:0/405848438 >> 172.21.15.129:0/405848438 conn(0x7f7e280fcec0 msgr2=0x7f7e281038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:48.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.098+0000 7f7e2759e640 1 -- 172.21.15.129:0/405848438 shutdown_connections 2024-05-20T18:02:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:48.098+0000 7f7e2759e640 1 -- 172.21.15.129:0/405848438 wait complete. 2024-05-20T18:02:48.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:48 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/76388503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:02:48.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:48 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/76388503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:02:48.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:48 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/76388503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:02:49.179 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:02:49.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:02:49.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:02:49.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:49 smithi154 ceph-mon[36482]: pgmap v131: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 742 MiB/s wr, 1.58k op/s 2024-05-20T18:02:49.400 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:49 smithi129 ceph-mon[32029]: pgmap v131: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 742 MiB/s wr, 1.58k op/s 2024-05-20T18:02:49.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:49 smithi204 ceph-mon[37076]: pgmap v131: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 742 MiB/s wr, 1.58k op/s 2024-05-20T18:02:49.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.804+0000 7f2c03475640 1 -- 172.21.15.129:0/2847314339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bfc108f70 msgr2=0x7f2bfc109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:49.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.804+0000 7f2c03475640 1 --2- 172.21.15.129:0/2847314339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bfc108f70 0x7f2bfc109380 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009930 tx=0x7f2bf802a480 comp rx=0 tx=0).stop 2024-05-20T18:02:49.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.807+0000 7f2c03475640 1 -- 172.21.15.129:0/2847314339 shutdown_connections 2024-05-20T18:02:49.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.807+0000 7f2c03475640 1 --2- 172.21.15.129:0/2847314339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bfc11c0d0 0x7f2bfc11e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.807+0000 7f2c03475640 1 --2- 172.21.15.129:0/2847314339 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bfc119770 0x7f2bfc11bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.807+0000 7f2c03475640 1 --2- 172.21.15.129:0/2847314339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bfc108f70 0x7f2bfc109380 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.807+0000 7f2c03475640 1 -- 172.21.15.129:0/2847314339 >> 172.21.15.129:0/2847314339 conn(0x7f2bfc06e5d0 msgr2=0x7f2bfc06ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:49.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.810+0000 7f2c03475640 1 -- 172.21.15.129:0/2847314339 shutdown_connections 2024-05-20T18:02:49.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.810+0000 7f2c03475640 1 -- 172.21.15.129:0/2847314339 wait complete. 2024-05-20T18:02:49.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.811+0000 7f2c03475640 1 Processor -- start 2024-05-20T18:02:49.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.811+0000 7f2c03475640 1 -- start start 2024-05-20T18:02:49.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.812+0000 7f2c03475640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bfc108f70 0x7f2bfc119150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.812+0000 7f2c03475640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bfc119770 0x7f2bfc112270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.812+0000 7f2c03475640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bfc11c0d0 0x7f2bfc1128a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.812+0000 7f2c03475640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2bfc121b20 con 0x7f2bfc11c0d0 2024-05-20T18:02:49.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.812+0000 7f2c03475640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2bfc1219a0 con 0x7f2bfc108f70 2024-05-20T18:02:49.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.812+0000 7f2c03475640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2bfc121ca0 con 0x7f2bfc119770 2024-05-20T18:02:49.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.815+0000 7f2c01a12640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bfc11c0d0 0x7f2bfc1128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:49.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.815+0000 7f2c01a12640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bfc11c0d0 0x7f2bfc1128a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47468/0 (socket says 172.21.15.129:47468) 2024-05-20T18:02:49.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.815+0000 7f2c01a12640 1 -- 172.21.15.129:0/3147761111 learned_addr learned my addr 172.21.15.129:0/3147761111 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:49.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.816+0000 7f2c01a12640 1 -- 172.21.15.129:0/3147761111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bfc119770 msgr2=0x7f2bfc112270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:49.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.816+0000 7f2c01a12640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bfc119770 0x7f2bfc112270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.816+0000 7f2c01a12640 1 -- 172.21.15.129:0/3147761111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bfc108f70 msgr2=0x7f2bfc119150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:49.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.816+0000 7f2c01a12640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bfc108f70 0x7f2bfc119150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.816+0000 7f2c01a12640 1 -- 172.21.15.129:0/3147761111 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bfc116c70 con 0x7f2bfc11c0d0 2024-05-20T18:02:49.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.818+0000 7f2c01a12640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bfc11c0d0 0x7f2bfc1128a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400b9e0 tx=0x7f2bf400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:49.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.819+0000 7f2bf27fc640 1 -- 172.21.15.129:0/3147761111 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4019070 con 0x7f2bfc11c0d0 2024-05-20T18:02:49.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.819+0000 7f2c03475640 1 -- 172.21.15.129:0/3147761111 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bfc116f60 con 0x7f2bfc11c0d0 2024-05-20T18:02:49.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.819+0000 7f2c03475640 1 -- 172.21.15.129:0/3147761111 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bfc1174b0 con 0x7f2bfc11c0d0 2024-05-20T18:02:49.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.819+0000 7f2bf27fc640 1 -- 172.21.15.129:0/3147761111 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bf4014440 con 0x7f2bfc11c0d0 2024-05-20T18:02:49.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.819+0000 7f2bf27fc640 1 -- 172.21.15.129:0/3147761111 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf40125e0 con 0x7f2bfc11c0d0 2024-05-20T18:02:49.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.824+0000 7f2c0a3f1740 1 Processor -- start 2024-05-20T18:02:49.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.825+0000 7f2bf27fc640 1 -- 172.21.15.129:0/3147761111 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f2bf401c070 con 0x7f2bfc11c0d0 2024-05-20T18:02:49.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.827+0000 7f2bf27fc640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2be80a2e80 0x7f2be80a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.827+0000 7f2c0a3f1740 1 -- start start 2024-05-20T18:02:49.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.828+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55925980b410 0x559259baa800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.828+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5592597e2550 0x559259baad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.828+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bab280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.829+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559259b0f990 con 0x5592597e2550 2024-05-20T18:02:49.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.829+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559259ab75b0 con 0x5592597ea730 2024-05-20T18:02:49.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.829+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x559259b0b950 con 0x55925980b410 2024-05-20T18:02:49.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.832+0000 7f2c01211640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55925980b410 0x559259baa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:49.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.832+0000 7f2c01a12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:49.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.832+0000 7f2c01a12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bab280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:50148/0 (socket says 172.21.15.129:50148) 2024-05-20T18:02:49.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.832+0000 7f2c01211640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55925980b410 0x559259baa800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:47024/0 (socket says 172.21.15.129:47024) 2024-05-20T18:02:49.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.832+0000 7f2c01211640 1 -- 172.21.15.129:0/3840680033 learned_addr learned my addr 172.21.15.129:0/3840680033 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:49.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.833+0000 7f2bf27fc640 1 -- 172.21.15.129:0/3147761111 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f2bf4099f80 con 0x7f2bfc11c0d0 2024-05-20T18:02:49.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.836+0000 7f2c01a12640 1 -- 172.21.15.129:0/3840680033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55925980b410 msgr2=0x559259baa800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:49.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.836+0000 7f2c01a12640 1 --2- 172.21.15.129:0/3840680033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55925980b410 0x559259baa800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.836+0000 7f2c01a12640 1 -- 172.21.15.129:0/3840680033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5592597e2550 msgr2=0x559259baad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:49.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.836+0000 7f2c01a12640 1 --2- 172.21.15.129:0/3840680033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5592597e2550 0x559259baad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.836+0000 7f2c01a12640 1 -- 172.21.15.129:0/3840680033 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592598eb7e0 con 0x5592597ea730 2024-05-20T18:02:49.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.842+0000 7f2c01a12640 1 --2- 172.21.15.129:0/3840680033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bab280 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40cd110 tx=0x7f2bf4004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b894dc1b87c2492 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:49.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.844+0000 7f2be77fe640 1 -- 172.21.15.129:0/3840680033 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf40cd280 con 0x5592597ea730 2024-05-20T18:02:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.845+0000 7f2be77fe640 1 -- 172.21.15.129:0/3840680033 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bf40cd420 con 0x5592597ea730 2024-05-20T18:02:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.845+0000 7f2be77fe640 1 -- 172.21.15.129:0/3840680033 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf400cb20 con 0x5592597ea730 2024-05-20T18:02:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.846+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3840680033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 msgr2=0x559259bab280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:49.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.846+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/3840680033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bab280 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40cd110 tx=0x7f2bf4004b50 comp rx=0 tx=0).stop 2024-05-20T18:02:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.847+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3840680033 shutdown_connections 2024-05-20T18:02:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.847+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/3840680033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bab280 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.847+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/3840680033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5592597e2550 0x559259baad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.847+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/3840680033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55925980b410 0x559259baa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.847+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3840680033 >> 172.21.15.129:0/3840680033 conn(0x5592597ff560 msgr2=0x559259ba6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:49.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.852+0000 7f2c00a10640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2be80a2e80 0x7f2be80a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:49.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.852+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3840680033 shutdown_connections 2024-05-20T18:02:49.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.852+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3840680033 wait complete. 2024-05-20T18:02:49.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.853+0000 7f2c0a3f1740 1 -- start start 2024-05-20T18:02:49.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.853+0000 7f2c0a3f1740 1 Processor -- start 2024-05-20T18:02:49.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.855+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bb6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.855+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259bb6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.855+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bb4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.855+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559259bb7b50 con 0x55925980b410 2024-05-20T18:02:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.855+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559259acab40 con 0x5592597ea730 2024-05-20T18:02:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.855+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x559259bb7cd0 con 0x5592597ca1f0 2024-05-20T18:02:49.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.855+0000 7f2c0a3f1740 10 client.0 _ll_register_callbacks cb 0x559259abf0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:49.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.857+0000 7f2c00a10640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259bb6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:49.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.857+0000 7f2c00a10640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259bb6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47476/0 (socket says 172.21.15.129:47476) 2024-05-20T18:02:49.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.857+0000 7f2c00a10640 1 -- 172.21.15.129:0/3684506374 learned_addr learned my addr 172.21.15.129:0/3684506374 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:49.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.858+0000 7f2c00a10640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2be80a2e80 0x7f2be80a5390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x559259ba7f40 tx=0x7f2bec005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:49.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.859+0000 7f2c01211640 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bb6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:49.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.859+0000 7f2c00a10640 1 -- 172.21.15.129:0/3684506374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 msgr2=0x559259bb4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:49.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.859+0000 7f2c00a10640 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bb4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.859+0000 7f2c00a10640 1 -- 172.21.15.129:0/3684506374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 msgr2=0x559259bb6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:49.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.859+0000 7f2c00a10640 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bb6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.860+0000 7f2c00a10640 1 -- 172.21.15.129:0/3684506374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55925994bf70 con 0x55925980b410 2024-05-20T18:02:49.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.860+0000 7f2c00a10640 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259bb6e90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc06b190 tx=0x7f2bec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8806b2443f2aaaba server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:49.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.861+0000 7f2be4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:02:49.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.864+0000 7f2be4ff9640 1 -- 172.21.15.129:0/3684506374 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bec020070 con 0x55925980b410 2024-05-20T18:02:49.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.864+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3684506374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bf40cd420 con 0x55925980b410 2024-05-20T18:02:49.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.864+0000 7f2be4ff9640 1 -- 172.21.15.129:0/3684506374 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f2bec01c8f0 con 0x55925980b410 2024-05-20T18:02:49.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.865+0000 7f2be4ff9640 1 -- 172.21.15.129:0/3684506374 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2bec01caf0 con 0x55925980b410 2024-05-20T18:02:49.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.865+0000 7f2c0a3f1740 10 client.16098 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:49.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.866+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3684506374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5592597e3170 con 0x55925980b410 2024-05-20T18:02:49.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.868+0000 7f2be4ff9640 1 -- 172.21.15.129:0/3684506374 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bf40cd420 con 0x55925980b410 2024-05-20T18:02:49.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.870+0000 7f2be4ff9640 1 -- 172.21.15.129:0/3684506374 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f2bec00f440 con 0x55925980b410 2024-05-20T18:02:49.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.871+0000 7f2c0a3f1740 10 client.16098 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:49.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.871+0000 7f2c0a3f1740 10 client.16098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:49.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.871+0000 7f2c0a3f1740 20 client.16098 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:49.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.871+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55925979f630 0x559259bc78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.871+0000 7f2c0a3f1740 4 client.16098 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:49.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.871+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3684506374 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559259b1d0c0 con 0x55925979f630 2024-05-20T18:02:49.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.873+0000 7f2c01211640 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55925979f630 0x559259bc78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:49.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.885+0000 7f2c01211640 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55925979f630 0x559259bc78f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c1a565739cd3147 server_cookie=de90732840b277fe in_seq=0 out_seq=0 2024-05-20T18:02:49.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.885+0000 7f2be4ff9640 10 client.16098 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:49.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.893+0000 7f2be4ff9640 1 -- 172.21.15.129:0/3684506374 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559259b1d0c0 con 0x55925979f630 2024-05-20T18:02:49.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.894+0000 7f2be4ff9640 10 client.16098 handle_command_reply: tid=0 2024-05-20T18:02:49.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.894+0000 7f2c0a3f1740 1 client.16098 shutdown 2024-05-20T18:02:49.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.894+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3684506374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 msgr2=0x559259bb6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:49.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.894+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259bb6e90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc06b190 tx=0x7f2bec0046c0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.894+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3684506374 shutdown_connections 2024-05-20T18:02:49.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.894+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55925979f630 0x559259bc78f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.894+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bb4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.894+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259bb6e90 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.894+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/3684506374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bb6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:49.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.894+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3684506374 >> 172.21.15.129:0/3684506374 conn(0x5592597ff560 msgr2=0x559259bac520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:49.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.896+0000 7f2be4ff9640 0 client.16098 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:49.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.900+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3684506374 shutdown_connections 2024-05-20T18:02:49.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.900+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/3684506374 wait complete. 2024-05-20T18:02:49.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.903+0000 7f2c0a3f1740 20 client.16098 trim_cache size 0 max 16384 2024-05-20T18:02:49.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.998+0000 7f2c0a3f1740 1 Processor -- start 2024-05-20T18:02:49.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.999+0000 7f2c0a3f1740 1 -- start start 2024-05-20T18:02:49.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.999+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bb3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.999+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:49.999+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259bae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:49.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.000+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2be000c7e0 con 0x55925980b410 2024-05-20T18:02:49.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.000+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2be000efb0 con 0x5592597ea730 2024-05-20T18:02:49.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.000+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2be000cda0 con 0x5592597ca1f0 2024-05-20T18:02:49.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.000+0000 7f2c01211640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bb3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:49.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.000+0000 7f2c01211640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bb3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:47046/0 (socket says 172.21.15.129:47046) 2024-05-20T18:02:49.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.000+0000 7f2c01211640 1 -- 172.21.15.129:0/740265430 learned_addr learned my addr 172.21.15.129:0/740265430 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:50.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.001+0000 7f2c01211640 1 -- 172.21.15.129:0/740265430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 msgr2=0x559259bae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:50.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.001+0000 7f2c01211640 1 --2- 172.21.15.129:0/740265430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.001+0000 7f2c01211640 1 -- 172.21.15.129:0/740265430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 msgr2=0x559259bae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:50.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.001+0000 7f2c01211640 1 --2- 172.21.15.129:0/740265430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259bae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.001+0000 7f2c01211640 1 -- 172.21.15.129:0/740265430 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559259babb70 con 0x5592597ca1f0 2024-05-20T18:02:50.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.002+0000 7f2c01211640 1 --2- 172.21.15.129:0/740265430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bb3dd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009600 tx=0x7f2bf8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=879d73dc45cf683e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:50.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.003+0000 7f2be6ffd640 1 -- 172.21.15.129:0/740265430 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8046530 con 0x5592597ca1f0 2024-05-20T18:02:50.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.003+0000 7f2be6ffd640 1 -- 172.21.15.129:0/740265430 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bf804a030 con 0x5592597ca1f0 2024-05-20T18:02:50.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.005+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/740265430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 msgr2=0x559259bb3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:50.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.005+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/740265430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bb3dd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009600 tx=0x7f2bf8000bc0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.010+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/740265430 shutdown_connections 2024-05-20T18:02:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.010+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/740265430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.010+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/740265430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259bae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.010+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/740265430 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bb3dd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.010+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/740265430 >> 172.21.15.129:0/740265430 conn(0x5592597ff560 msgr2=0x559259bb36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:50.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.015+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/740265430 shutdown_connections 2024-05-20T18:02:50.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.015+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/740265430 wait complete. 2024-05-20T18:02:50.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.016+0000 7f2c0a3f1740 1 -- start start 2024-05-20T18:02:50.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.016+0000 7f2c0a3f1740 1 Processor -- start 2024-05-20T18:02:50.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.018+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bbb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:50.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.018+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bbb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:50.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.018+0000 7f2c0a3f1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259c2ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.018+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2be000d470 con 0x55925980b410 2024-05-20T18:02:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.018+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2be000e8e0 con 0x5592597ea730 2024-05-20T18:02:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.018+0000 7f2c0a3f1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2be000db40 con 0x5592597ca1f0 2024-05-20T18:02:50.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.018+0000 7f2c0a3f1740 10 client.0 _ll_register_callbacks cb 0x7f2bec014050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:50.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.021+0000 7f2c01a12640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259c2ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:50.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.021+0000 7f2c01a12640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259c2ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47486/0 (socket says 172.21.15.129:47486) 2024-05-20T18:02:50.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.021+0000 7f2c01a12640 1 -- 172.21.15.129:0/58619993 learned_addr learned my addr 172.21.15.129:0/58619993 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:50.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.022+0000 7f2c00a10640 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bbb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:50.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.022+0000 7f2c01211640 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bbb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:50.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.022+0000 7f2c01a12640 1 -- 172.21.15.129:0/58619993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 msgr2=0x559259bbb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:50.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.022+0000 7f2c01a12640 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bbb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.027+0000 7f2c01a12640 1 -- 172.21.15.129:0/58619993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 msgr2=0x559259bbb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:50.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.027+0000 7f2c01a12640 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bbb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.027+0000 7f2c01a12640 1 -- 172.21.15.129:0/58619993 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559259c2f1b0 con 0x55925980b410 2024-05-20T18:02:50.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.027+0000 7f2c00a10640 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bbb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:02:50.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.027+0000 7f2c01a12640 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259c2ec70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40099d0 tx=0x7f2bf40cd4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6209906f3bb18f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:50.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.029+0000 7f2be4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:02:50.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.029+0000 7f2be4ff9640 1 -- 172.21.15.129:0/58619993 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf409b330 con 0x55925980b410 2024-05-20T18:02:50.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.029+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/58619993 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bf804a030 con 0x55925980b410 2024-05-20T18:02:50.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.030+0000 7f2be4ff9640 1 -- 172.21.15.129:0/58619993 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f2bf409b4d0 con 0x55925980b410 2024-05-20T18:02:50.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.033+0000 7f2be4ff9640 1 -- 172.21.15.129:0/58619993 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2bf400cb20 con 0x55925980b410 2024-05-20T18:02:50.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.033+0000 7f2c0a3f1740 10 client.16101 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:50.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.033+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/58619993 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x559259c30370 con 0x55925980b410 2024-05-20T18:02:50.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.034+0000 7f2be4ff9640 1 -- 172.21.15.129:0/58619993 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bf40b7030 con 0x55925980b410 2024-05-20T18:02:50.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.036+0000 7f2be4ff9640 1 -- 172.21.15.129:0/58619993 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f2bf4097da0 con 0x55925980b410 2024-05-20T18:02:50.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.036+0000 7f2c0a3f1740 10 client.16101 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:50.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.036+0000 7f2c0a3f1740 10 client.16101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:50.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.036+0000 7f2c0a3f1740 20 client.16101 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:50.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.036+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559259c309e0 0x559259c32e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:50.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.036+0000 7f2c0a3f1740 4 client.16101 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:02:50.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.036+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/58619993 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f2bf804a030 con 0x559259c309e0 2024-05-20T18:02:50.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.045+0000 7f2c01211640 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559259c309e0 0x559259c32e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:50.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.057+0000 7f2c01211640 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559259c309e0 0x559259c32e00 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffd16e399df5634f server_cookie=dff8b3258193b137 in_seq=0 out_seq=0 2024-05-20T18:02:50.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.057+0000 7f2be4ff9640 10 client.16101 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:50.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.614+0000 7f2be4ff9640 1 -- 172.21.15.129:0/58619993 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f2bf804a030 con 0x559259c309e0 2024-05-20T18:02:50.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.614+0000 7f2be4ff9640 10 client.16101 handle_command_reply: tid=0 2024-05-20T18:02:50.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.614+0000 7f2c0a3f1740 1 client.16101 shutdown 2024-05-20T18:02:50.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.615+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/58619993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 msgr2=0x559259c2ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:50.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.615+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259c2ec70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40099d0 tx=0x7f2bf40cd4b0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.615+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/58619993 shutdown_connections 2024-05-20T18:02:50.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.615+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559259c309e0 0x559259c32e00 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.615+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592597ea730 0x559259bbb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.615+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55925980b410 0x559259c2ec70 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.615+0000 7f2c0a3f1740 1 --2- 172.21.15.129:0/58619993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5592597ca1f0 0x559259bbb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.615+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/58619993 >> 172.21.15.129:0/58619993 conn(0x5592597ff560 msgr2=0x559259bacac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:50.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.615+0000 7f2be4ff9640 0 client.16101 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:50.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.616+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/58619993 shutdown_connections 2024-05-20T18:02:50.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.616+0000 7f2c0a3f1740 1 -- 172.21.15.129:0/58619993 wait complete. 2024-05-20T18:02:50.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.616+0000 7f2c0a3f1740 20 client.16101 trim_cache size 0 max 16384 2024-05-20T18:02:50.616 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:02:50.617 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:02:50.617 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "cc33667f-edda-400f-b04b-24ac11175e07", 2024-05-20T18:02:50.617 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:02:50.617 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:02:50.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.659+0000 7f2c03475640 1 -- 172.21.15.129:0/3147761111 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2be80a2e80 msgr2=0x7f2be80a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:50.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.659+0000 7f2c03475640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2be80a2e80 0x7f2be80a5390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x559259ba7f40 tx=0x7f2bec005f90 comp rx=0 tx=0).stop 2024-05-20T18:02:50.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.659+0000 7f2c03475640 1 -- 172.21.15.129:0/3147761111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bfc11c0d0 msgr2=0x7f2bfc1128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:50.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.659+0000 7f2c03475640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bfc11c0d0 0x7f2bfc1128a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400b9e0 tx=0x7f2bf400bea0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.672+0000 7f2c03475640 1 -- 172.21.15.129:0/3147761111 shutdown_connections 2024-05-20T18:02:50.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.672+0000 7f2c03475640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f2be80a2e80 0x7f2be80a5390 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.672+0000 7f2c03475640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bfc11c0d0 0x7f2bfc1128a0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.672+0000 7f2c03475640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bfc119770 0x7f2bfc112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.672+0000 7f2c03475640 1 --2- 172.21.15.129:0/3147761111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bfc108f70 0x7f2bfc119150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:50.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.673+0000 7f2c03475640 1 -- 172.21.15.129:0/3147761111 >> 172.21.15.129:0/3147761111 conn(0x7f2bfc06e5d0 msgr2=0x7f2bfc119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:50.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.675+0000 7f2c03475640 1 -- 172.21.15.129:0/3147761111 shutdown_connections 2024-05-20T18:02:50.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:50.675+0000 7f2c03475640 1 -- 172.21.15.129:0/3147761111 wait complete. 2024-05-20T18:02:50.793 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:02:50.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:02:51.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.238+0000 7f5c9ae56640 1 -- 172.21.15.129:0/3335416411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9410e890 msgr2=0x7f5c9410eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:51.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.238+0000 7f5c9ae56640 1 --2- 172.21.15.129:0/3335416411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9410e890 0x7f5c9410eca0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009900 tx=0x7f5c8402a410 comp rx=0 tx=0).stop 2024-05-20T18:02:51.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.242+0000 7f5c9ae56640 1 -- 172.21.15.129:0/3335416411 shutdown_connections 2024-05-20T18:02:51.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.242+0000 7f5c9ae56640 1 --2- 172.21.15.129:0/3335416411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c94109090 0x7f5c94109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.242+0000 7f5c9ae56640 1 --2- 172.21.15.129:0/3335416411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c9410f270 0x7f5c94108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.242+0000 7f5c9ae56640 1 --2- 172.21.15.129:0/3335416411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9410e890 0x7f5c9410eca0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.242+0000 7f5c9ae56640 1 -- 172.21.15.129:0/3335416411 >> 172.21.15.129:0/3335416411 conn(0x7f5c9406d580 msgr2=0x7f5c9406d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:51.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.250+0000 7f5c9ae56640 1 -- 172.21.15.129:0/3335416411 shutdown_connections 2024-05-20T18:02:51.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.255+0000 7f5c9ae56640 1 -- 172.21.15.129:0/3335416411 wait complete. 2024-05-20T18:02:51.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.256+0000 7f5c9ae56640 1 Processor -- start 2024-05-20T18:02:51.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.256+0000 7f5c9ae56640 1 -- start start 2024-05-20T18:02:51.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.257+0000 7f5c9ae56640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c94109090 0x7f5c941a8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.257+0000 7f5c9ae56640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9410e890 0x7f5c941a8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.257+0000 7f5c9ae56640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c9410f270 0x7f5c941acd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.257+0000 7f5c9ae56640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94121b70 con 0x7f5c94109090 2024-05-20T18:02:51.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.257+0000 7f5c9ae56640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941219f0 con 0x7f5c9410e890 2024-05-20T18:02:51.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.257+0000 7f5c9ae56640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94121cf0 con 0x7f5c9410f270 2024-05-20T18:02:51.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.259+0000 7f5c99653640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9410e890 0x7f5c941a8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.259+0000 7f5c99653640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9410e890 0x7f5c941a8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:50194/0 (socket says 172.21.15.129:50194) 2024-05-20T18:02:51.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.259+0000 7f5c99653640 1 -- 172.21.15.129:0/1286383752 learned_addr learned my addr 172.21.15.129:0/1286383752 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:51.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.264+0000 7f5c9a655640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c9410f270 0x7f5c941acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.265+0000 7f5c99e54640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c94109090 0x7f5c941a8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.271+0000 7f5c99653640 1 -- 172.21.15.129:0/1286383752 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c9410f270 msgr2=0x7f5c941acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:51.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.271+0000 7f5c99653640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c9410f270 0x7f5c941acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.271+0000 7f5c99653640 1 -- 172.21.15.129:0/1286383752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c94109090 msgr2=0x7f5c941a8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:51.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.271+0000 7f5c99653640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c94109090 0x7f5c941a8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.271+0000 7f5c99653640 1 -- 172.21.15.129:0/1286383752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c941ad3b0 con 0x7f5c9410e890 2024-05-20T18:02:51.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.273+0000 7f5c99653640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9410e890 0x7f5c941a8ab0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00b9c0 tx=0x7f5c8c00be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:51.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.276+0000 7f5c8affd640 1 -- 172.21.15.129:0/1286383752 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c019070 con 0x7f5c9410e890 2024-05-20T18:02:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.276+0000 7f5c8affd640 1 -- 172.21.15.129:0/1286383752 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c8c00cba0 con 0x7f5c9410e890 2024-05-20T18:02:51.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.276+0000 7f5c8affd640 1 -- 172.21.15.129:0/1286383752 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c012640 con 0x7f5c9410e890 2024-05-20T18:02:51.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.278+0000 7f5c9ae56640 1 -- 172.21.15.129:0/1286383752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c941ad6a0 con 0x7f5c9410e890 2024-05-20T18:02:51.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.278+0000 7f5c9ae56640 1 -- 172.21.15.129:0/1286383752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c941adba0 con 0x7f5c9410e890 2024-05-20T18:02:51.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.284+0000 7f5ca3033740 1 Processor -- start 2024-05-20T18:02:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.290+0000 7f5ca3033740 1 -- start start 2024-05-20T18:02:51.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.290+0000 7f5c8affd640 1 -- 172.21.15.129:0/1286383752 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5c8c014440 con 0x7f5c9410e890 2024-05-20T18:02:51.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.291+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c1638390 0x55a2c19d6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.291+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c160f4d0 0x55a2c19d7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.291+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19d79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.291+0000 7f5ca3033740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a2c193cd00 con 0x55a2c1638390 2024-05-20T18:02:51.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.291+0000 7f5ca3033740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a2c18e44c0 con 0x55a2c160f4d0 2024-05-20T18:02:51.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.291+0000 7f5ca3033740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a2c1938cc0 con 0x55a2c16176b0 2024-05-20T18:02:51.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.295+0000 7f5c99e54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c160f4d0 0x55a2c19d7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.296+0000 7f5c9a655640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c1638390 0x55a2c19d6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.296+0000 7f5c9a655640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c1638390 0x55a2c19d6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47518/0 (socket says 172.21.15.129:47518) 2024-05-20T18:02:51.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.296+0000 7f5c9a655640 1 -- 172.21.15.129:0/4025359874 learned_addr learned my addr 172.21.15.129:0/4025359874 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:51.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.297+0000 7f5c99653640 1 --2- 172.21.15.129:0/4025359874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19d79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.298+0000 7f5c8affd640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5c800a2f00 0x7f5c800a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.298+0000 7f5c8affd640 1 -- 172.21.15.129:0/1286383752 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5c8c0ce970 con 0x7f5c9410e890 2024-05-20T18:02:51.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.300+0000 7f5c9a655640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5c800a2f00 0x7f5c800a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.301+0000 7f5c9a655640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5c800a2f00 0x7f5c800a5410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5c90016600 tx=0x7f5c9000b830 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:51.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.303+0000 7f5c99e54640 1 -- 172.21.15.129:0/4025359874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 msgr2=0x55a2c19d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:51.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.303+0000 7f5c99e54640 1 --2- 172.21.15.129:0/4025359874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19d79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.303+0000 7f5c99e54640 1 -- 172.21.15.129:0/4025359874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c1638390 msgr2=0x55a2c19d6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:51.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.303+0000 7f5c99e54640 1 --2- 172.21.15.129:0/4025359874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c1638390 0x55a2c19d6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.303+0000 7f5c99e54640 1 -- 172.21.15.129:0/4025359874 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2c171a7d0 con 0x55a2c160f4d0 2024-05-20T18:02:51.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.309+0000 7f5c99e54640 1 --2- 172.21.15.129:0/4025359874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c160f4d0 0x55a2c19d7470 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5c8402aa00 tx=0x7f5c84004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf4d2eefcb6fd11 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:51.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.315+0000 7f5c7ffff640 1 -- 172.21.15.129:0/4025359874 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8403f070 con 0x55a2c160f4d0 2024-05-20T18:02:51.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.315+0000 7f5c7ffff640 1 -- 172.21.15.129:0/4025359874 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c84004470 con 0x55a2c160f4d0 2024-05-20T18:02:51.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.316+0000 7f5ca3033740 1 -- 172.21.15.129:0/4025359874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c160f4d0 msgr2=0x55a2c19d7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:51.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.317+0000 7f5ca3033740 1 --2- 172.21.15.129:0/4025359874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c160f4d0 0x55a2c19d7470 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5c8402aa00 tx=0x7f5c84004040 comp rx=0 tx=0).stop 2024-05-20T18:02:51.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.317+0000 7f5ca3033740 1 -- 172.21.15.129:0/4025359874 shutdown_connections 2024-05-20T18:02:51.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.317+0000 7f5ca3033740 1 --2- 172.21.15.129:0/4025359874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c160f4d0 0x55a2c19d7470 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5c8402aa00 tx=0x7f5c84004040 comp rx=0 tx=0).stop 2024-05-20T18:02:51.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.317+0000 7f5ca3033740 1 --2- 172.21.15.129:0/4025359874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19d79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.317+0000 7f5ca3033740 1 --2- 172.21.15.129:0/4025359874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c1638390 0x55a2c19d6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.317+0000 7f5ca3033740 1 -- 172.21.15.129:0/4025359874 >> 172.21.15.129:0/4025359874 conn(0x55a2c162c4e0 msgr2=0x55a2c19d36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:51.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.320+0000 7f5ca3033740 1 -- 172.21.15.129:0/4025359874 shutdown_connections 2024-05-20T18:02:51.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.321+0000 7f5ca3033740 1 -- 172.21.15.129:0/4025359874 wait complete. 2024-05-20T18:02:51.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.322+0000 7f5ca3033740 1 -- start start 2024-05-20T18:02:51.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.322+0000 7f5ca3033740 1 Processor -- start 2024-05-20T18:02:51.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.323+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c16176b0 0x55a2c19e2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.323+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 0x55a2c19e3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.323+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c15f6f60 0x55a2c19e1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.323+0000 7f5ca3033740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a2c19e4270 con 0x55a2c16176b0 2024-05-20T18:02:51.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.324+0000 7f5ca3033740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a2c18f7b90 con 0x55a2c1638390 2024-05-20T18:02:51.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.324+0000 7f5ca3033740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a2c19e43f0 con 0x55a2c15f6f60 2024-05-20T18:02:51.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.324+0000 7f5ca3033740 10 client.0 _ll_register_callbacks cb 0x55a2c1933da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:51.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.324+0000 7f5c9a655640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c15f6f60 0x55a2c19e1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.325+0000 7f5c99e54640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c16176b0 0x55a2c19e2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.325+0000 7f5c99e54640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c16176b0 0x55a2c19e2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47526/0 (socket says 172.21.15.129:47526) 2024-05-20T18:02:51.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.325+0000 7f5c99e54640 1 -- 172.21.15.129:0/500513613 learned_addr learned my addr 172.21.15.129:0/500513613 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:51.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.325+0000 7f5c99653640 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 0x55a2c19e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.328+0000 7f5c99653640 1 -- 172.21.15.129:0/500513613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c15f6f60 msgr2=0x55a2c19e1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:51.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.328+0000 7f5c99653640 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c15f6f60 0x55a2c19e1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.328+0000 7f5c99653640 1 -- 172.21.15.129:0/500513613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c16176b0 msgr2=0x55a2c19e2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:51.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.328+0000 7f5c99653640 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c16176b0 0x55a2c19e2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.328+0000 7f5c99653640 1 -- 172.21.15.129:0/500513613 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2c16503f0 con 0x55a2c1638390 2024-05-20T18:02:51.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.330+0000 7f5c99653640 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 0x55a2c19e3530 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c0cc160 tx=0x7f5c8c09ad00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f235c9328984caae server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:51.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.330+0000 7f5c7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:02:51.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.334+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/500513613 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c09b630 con 0x55a2c1638390 2024-05-20T18:02:51.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.335+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/500513613 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5c8c0975a0 con 0x55a2c1638390 2024-05-20T18:02:51.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.335+0000 7f5ca3033740 1 -- 172.21.15.129:0/500513613 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c84004470 con 0x55a2c1638390 2024-05-20T18:02:51.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.338+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/500513613 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5c8c09af30 con 0x55a2c1638390 2024-05-20T18:02:51.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.340+0000 7f5ca3033740 10 client.25814 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:51.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.340+0000 7f5ca3033740 1 -- 172.21.15.129:0/500513613 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a2c16100f0 con 0x55a2c1638390 2024-05-20T18:02:51.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.346+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/500513613 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c84004470 con 0x55a2c1638390 2024-05-20T18:02:51.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.348+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/500513613 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f5c8c0b9b70 con 0x55a2c1638390 2024-05-20T18:02:51.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.351+0000 7f5ca3033740 10 client.25814 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:51.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.351+0000 7f5ca3033740 10 client.25814 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:51.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.351+0000 7f5ca3033740 20 client.25814 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:51.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.351+0000 7f5ca3033740 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a2c15cc3a0 0x55a2c19f3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.351+0000 7f5ca3033740 4 client.25814 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:51.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.351+0000 7f5ca3033740 1 -- 172.21.15.129:0/500513613 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2c194a430 con 0x55a2c15cc3a0 2024-05-20T18:02:51.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.353+0000 7f5c99e54640 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a2c15cc3a0 0x55a2c19f3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.363+0000 7f5c99e54640 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a2c15cc3a0 0x55a2c19f3ee0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4632c9ce47f40e26 server_cookie=f29add636011310 in_seq=0 out_seq=0 2024-05-20T18:02:51.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.365+0000 7f5c7d7fa640 10 client.25814 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:51.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.365+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/500513613 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a2c194a430 con 0x55a2c15cc3a0 2024-05-20T18:02:51.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.365+0000 7f5c7d7fa640 10 client.25814 handle_command_reply: tid=0 2024-05-20T18:02:51.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.365+0000 7f5ca3033740 1 client.25814 shutdown 2024-05-20T18:02:51.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.373+0000 7f5ca3033740 1 -- 172.21.15.129:0/500513613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 msgr2=0x55a2c19e3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:51.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.373+0000 7f5ca3033740 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 0x55a2c19e3530 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c0cc160 tx=0x7f5c8c09ad00 comp rx=0 tx=0).stop 2024-05-20T18:02:51.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.379+0000 7f5ca3033740 1 -- 172.21.15.129:0/500513613 shutdown_connections 2024-05-20T18:02:51.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.379+0000 7f5ca3033740 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a2c15cc3a0 0x55a2c19f3ee0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.379+0000 7f5ca3033740 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 0x55a2c19e3530 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.379+0000 7f5ca3033740 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c15f6f60 0x55a2c19e1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.379+0000 7f5ca3033740 1 --2- 172.21.15.129:0/500513613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c16176b0 0x55a2c19e2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.379+0000 7f5ca3033740 1 -- 172.21.15.129:0/500513613 >> 172.21.15.129:0/500513613 conn(0x55a2c162c4e0 msgr2=0x55a2c19d6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:51.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.379+0000 7f5c7d7fa640 0 client.25814 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:51.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.383+0000 7f5ca3033740 1 -- 172.21.15.129:0/500513613 shutdown_connections 2024-05-20T18:02:51.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.386+0000 7f5ca3033740 1 -- 172.21.15.129:0/500513613 wait complete. 2024-05-20T18:02:51.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.390+0000 7f5ca3033740 20 client.25814 trim_cache size 0 max 16384 2024-05-20T18:02:51.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:51 smithi154 ceph-mon[36482]: pgmap v132: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 931 MiB/s wr, 1.87k op/s 2024-05-20T18:02:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:51 smithi129 ceph-mon[32029]: pgmap v132: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 931 MiB/s wr, 1.87k op/s 2024-05-20T18:02:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:51 smithi204 ceph-mon[37076]: pgmap v132: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 931 MiB/s wr, 1.87k op/s 2024-05-20T18:02:51.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.459+0000 7f5ca3033740 1 Processor -- start 2024-05-20T18:02:51.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.459+0000 7f5ca3033740 1 -- start start 2024-05-20T18:02:51.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.459+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c162c4e0 0x55a2c19e0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.459+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c15f6f60 0x55a2c19dc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.460+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19d73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.460+0000 7f5ca3033740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9410a220 con 0x55a2c15f6f60 2024-05-20T18:02:51.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.460+0000 7f5ca3033740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941c66b0 con 0x55a2c162c4e0 2024-05-20T18:02:51.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.460+0000 7f5ca3033740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94118df0 con 0x55a2c16176b0 2024-05-20T18:02:51.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.460+0000 7f5c99e54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c162c4e0 0x55a2c19e0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.460+0000 7f5c99e54640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c162c4e0 0x55a2c19e0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:50208/0 (socket says 172.21.15.129:50208) 2024-05-20T18:02:51.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.460+0000 7f5c99e54640 1 -- 172.21.15.129:0/601796306 learned_addr learned my addr 172.21.15.129:0/601796306 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:51.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.463+0000 7f5c99653640 1 --2- 172.21.15.129:0/601796306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c15f6f60 0x55a2c19dc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.463+0000 7f5c99e54640 1 -- 172.21.15.129:0/601796306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 msgr2=0x55a2c19d73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:51.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.463+0000 7f5c99e54640 1 --2- 172.21.15.129:0/601796306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19d73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.463+0000 7f5c99e54640 1 -- 172.21.15.129:0/601796306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c15f6f60 msgr2=0x55a2c19dc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:51.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.463+0000 7f5c99e54640 1 --2- 172.21.15.129:0/601796306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c15f6f60 0x55a2c19dc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.463+0000 7f5c99e54640 1 -- 172.21.15.129:0/601796306 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2c19d4830 con 0x55a2c162c4e0 2024-05-20T18:02:51.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.468+0000 7f5c99e54640 1 --2- 172.21.15.129:0/601796306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c162c4e0 0x55a2c19e0440 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5c84004fd0 tx=0x7f5c8402a910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=634b1e9b2461d7e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:51.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.486+0000 7f5c7f7fe640 1 -- 172.21.15.129:0/601796306 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c84054070 con 0x55a2c162c4e0 2024-05-20T18:02:51.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.486+0000 7f5c7f7fe640 1 -- 172.21.15.129:0/601796306 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c84049040 con 0x55a2c162c4e0 2024-05-20T18:02:51.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.486+0000 7f5c7f7fe640 1 -- 172.21.15.129:0/601796306 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8403f040 con 0x55a2c162c4e0 2024-05-20T18:02:51.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.488+0000 7f5ca3033740 1 -- 172.21.15.129:0/601796306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c162c4e0 msgr2=0x55a2c19e0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:51.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.488+0000 7f5ca3033740 1 --2- 172.21.15.129:0/601796306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c162c4e0 0x55a2c19e0440 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5c84004fd0 tx=0x7f5c8402a910 comp rx=0 tx=0).stop 2024-05-20T18:02:51.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.489+0000 7f5ca3033740 1 -- 172.21.15.129:0/601796306 shutdown_connections 2024-05-20T18:02:51.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.489+0000 7f5ca3033740 1 --2- 172.21.15.129:0/601796306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19d73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.489+0000 7f5ca3033740 1 --2- 172.21.15.129:0/601796306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c15f6f60 0x55a2c19dc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.489+0000 7f5ca3033740 1 --2- 172.21.15.129:0/601796306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c162c4e0 0x55a2c19e0440 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5c84004fd0 tx=0x7f5c8402a910 comp rx=0 tx=0).stop 2024-05-20T18:02:51.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.489+0000 7f5ca3033740 1 -- 172.21.15.129:0/601796306 >> 172.21.15.129:0/601796306 conn(0x55a2c19dce80 msgr2=0x55a2c19d6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:51.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.498+0000 7f5ca3033740 1 -- 172.21.15.129:0/601796306 shutdown_connections 2024-05-20T18:02:51.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.503+0000 7f5ca3033740 1 -- 172.21.15.129:0/601796306 wait complete. 2024-05-20T18:02:51.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.504+0000 7f5ca3033740 1 -- start start 2024-05-20T18:02:51.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.504+0000 7f5ca3033740 1 Processor -- start 2024-05-20T18:02:51.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.505+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c15f6f60 0x55a2c19e9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.505+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.505+0000 7f5ca3033740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 0x55a2c19e7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.505+0000 7f5ca3033740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94119430 con 0x55a2c15f6f60 2024-05-20T18:02:51.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.505+0000 7f5ca3033740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941c6030 con 0x55a2c1638390 2024-05-20T18:02:51.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.505+0000 7f5ca3033740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94119b00 con 0x55a2c16176b0 2024-05-20T18:02:51.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.506+0000 7f5ca3033740 10 client.0 _ll_register_callbacks cb 0x7f5c8c0971d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:51.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.509+0000 7f5c9a655640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 0x55a2c19e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.509+0000 7f5c9a655640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 0x55a2c19e7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:50212/0 (socket says 172.21.15.129:50212) 2024-05-20T18:02:51.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.509+0000 7f5c9a655640 1 -- 172.21.15.129:0/289604463 learned_addr learned my addr 172.21.15.129:0/289604463 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:51.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.510+0000 7f5c99653640 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.510+0000 7f5c9a655640 1 -- 172.21.15.129:0/289604463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 msgr2=0x55a2c19e6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:51.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.510+0000 7f5c9a655640 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19e6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.510+0000 7f5c9a655640 1 -- 172.21.15.129:0/289604463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c15f6f60 msgr2=0x55a2c19e9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:51.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.510+0000 7f5c9a655640 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c15f6f60 0x55a2c19e9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.510+0000 7f5c9a655640 1 -- 172.21.15.129:0/289604463 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2c19e7b70 con 0x55a2c1638390 2024-05-20T18:02:51.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.510+0000 7f5c99653640 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19e6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:02:51.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.514+0000 7f5c9a655640 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 0x55a2c19e7480 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5c90016e90 tx=0x7f5c90008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98f40316141f8ecb server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:51.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.514+0000 7f5c7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:02:51.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.514+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/289604463 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c90021420 con 0x55a2c1638390 2024-05-20T18:02:51.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.514+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/289604463 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7f5c900215c0 con 0x55a2c1638390 2024-05-20T18:02:51.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.515+0000 7f5ca3033740 1 -- 172.21.15.129:0/289604463 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c84049040 con 0x55a2c1638390 2024-05-20T18:02:51.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.517+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/289604463 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5c90005990 con 0x55a2c1638390 2024-05-20T18:02:51.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.517+0000 7f5ca3033740 10 client.25820 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:51.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.517+0000 7f5ca3033740 1 -- 172.21.15.129:0/289604463 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55a2c1a5c0d0 con 0x55a2c1638390 2024-05-20T18:02:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.520+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/289604463 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c84049040 con 0x55a2c1638390 2024-05-20T18:02:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.520+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/289604463 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7f5c90006020 con 0x55a2c1638390 2024-05-20T18:02:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.520+0000 7f5ca3033740 10 client.25820 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.520+0000 7f5ca3033740 10 client.25820 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.520+0000 7f5ca3033740 20 client.25820 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.520+0000 7f5ca3033740 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a2c1a5c860 0x55a2c1a5ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.520+0000 7f5ca3033740 4 client.25820 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:02:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.520+0000 7f5ca3033740 1 -- 172.21.15.129:0/289604463 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a2c1944780 con 0x55a2c1a5c860 2024-05-20T18:02:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.523+0000 7f5c99e54640 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a2c1a5c860 0x55a2c1a5ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:51.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.525+0000 7f5c99e54640 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a2c1a5c860 0x55a2c1a5ec80 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46141bd88c112490 server_cookie=daeafd3d0786dbad in_seq=0 out_seq=0 2024-05-20T18:02:51.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.525+0000 7f5c7d7fa640 10 client.25820 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:51.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.527+0000 7f5c7d7fa640 1 -- 172.21.15.129:0/289604463 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55a2c1944780 con 0x55a2c1a5c860 2024-05-20T18:02:51.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.527+0000 7f5c7d7fa640 10 client.25820 handle_command_reply: tid=0 2024-05-20T18:02:51.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.527+0000 7f5ca3033740 1 client.25820 shutdown 2024-05-20T18:02:51.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.530+0000 7f5ca3033740 1 -- 172.21.15.129:0/289604463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 msgr2=0x55a2c19e7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:51.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.530+0000 7f5ca3033740 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 0x55a2c19e7480 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5c90016e90 tx=0x7f5c90008040 comp rx=0 tx=0).stop 2024-05-20T18:02:51.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.533+0000 7f5ca3033740 1 -- 172.21.15.129:0/289604463 shutdown_connections 2024-05-20T18:02:51.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.533+0000 7f5ca3033740 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a2c1a5c860 0x55a2c1a5ec80 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.533+0000 7f5ca3033740 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2c1638390 0x55a2c19e7480 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.533+0000 7f5ca3033740 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a2c16176b0 0x55a2c19e6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.533+0000 7f5ca3033740 1 --2- 172.21.15.129:0/289604463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a2c15f6f60 0x55a2c19e9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.533+0000 7f5c7d7fa640 0 client.25820 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:51.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.533+0000 7f5ca3033740 1 -- 172.21.15.129:0/289604463 >> 172.21.15.129:0/289604463 conn(0x55a2c19dce80 msgr2=0x55a2c19daa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:51.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.533+0000 7f5ca3033740 1 -- 172.21.15.129:0/289604463 shutdown_connections 2024-05-20T18:02:51.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.533+0000 7f5ca3033740 1 -- 172.21.15.129:0/289604463 wait complete. 2024-05-20T18:02:51.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.537+0000 7f5ca3033740 20 client.25820 trim_cache size 0 max 16384 2024-05-20T18:02:51.537 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:02:51.537 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "scrub active (0 inodes in the stack)", 2024-05-20T18:02:51.537 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:02:51.537 INFO:teuthology.orchestra.run.smithi129.stdout: "cc33667f-edda-400f-b04b-24ac11175e07": { 2024-05-20T18:02:51.537 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:02:51.537 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "cc33667f-edda-400f-b04b-24ac11175e07", 2024-05-20T18:02:51.537 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:02:51.537 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:02:51.537 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:02:51.537 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:02:51.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.542+0000 7f5c9ae56640 1 -- 172.21.15.129:0/1286383752 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5c800a2f00 msgr2=0x7f5c800a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:51.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.542+0000 7f5c9ae56640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5c800a2f00 0x7f5c800a5410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5c90016600 tx=0x7f5c9000b830 comp rx=0 tx=0).stop 2024-05-20T18:02:51.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.542+0000 7f5c9ae56640 1 -- 172.21.15.129:0/1286383752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9410e890 msgr2=0x7f5c941a8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:51.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.542+0000 7f5c9ae56640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9410e890 0x7f5c941a8ab0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00b9c0 tx=0x7f5c8c00be80 comp rx=0 tx=0).stop 2024-05-20T18:02:51.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.545+0000 7f5c9ae56640 1 -- 172.21.15.129:0/1286383752 shutdown_connections 2024-05-20T18:02:51.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.545+0000 7f5c9ae56640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5c800a2f00 0x7f5c800a5410 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.545+0000 7f5c9ae56640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c9410e890 0x7f5c941a8ab0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.545+0000 7f5c9ae56640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5c9410f270 0x7f5c941acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.545+0000 7f5c9ae56640 1 --2- 172.21.15.129:0/1286383752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c94109090 0x7f5c941a8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:51.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.545+0000 7f5c9ae56640 1 -- 172.21.15.129:0/1286383752 >> 172.21.15.129:0/1286383752 conn(0x7f5c9406d580 msgr2=0x7f5c9410a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:51.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.548+0000 7f5c9ae56640 1 -- 172.21.15.129:0/1286383752 shutdown_connections 2024-05-20T18:02:51.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:02:51.555+0000 7f5c9ae56640 1 -- 172.21.15.129:0/1286383752 wait complete. 2024-05-20T18:02:51.645 INFO:tasks.cephfs.filesystem:scrub status for tag:cc33667f-edda-400f-b04b-24ac11175e07 - {'path': '/', 'tag': 'cc33667f-edda-400f-b04b-24ac11175e07', 'options': 'recursive,force'} 2024-05-20T18:02:52.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:52 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:02:52.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:52 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:52.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:52 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:02:52.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:52 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:02:52.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:52 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:52.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:52 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:02:52.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:52 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:02:52.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:52 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:52.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:52 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:02:53.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:53 smithi129 ceph-mon[32029]: pgmap v133: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 714 MiB/s wr, 1.19k op/s 2024-05-20T18:02:53.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:53 smithi204 ceph-mon[37076]: pgmap v133: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 714 MiB/s wr, 1.19k op/s 2024-05-20T18:02:53.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:53 smithi154 ceph-mon[36482]: pgmap v133: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 714 MiB/s wr, 1.19k op/s 2024-05-20T18:02:54.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:54 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:54.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:54 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:54.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:54 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:02:55.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:55 smithi204 ceph-mon[37076]: pgmap v134: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 760 MiB/s wr, 1.23k op/s 2024-05-20T18:02:55.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:55 smithi154 ceph-mon[36482]: pgmap v134: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 760 MiB/s wr, 1.23k op/s 2024-05-20T18:02:55.668 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:55 smithi129 ceph-mon[32029]: pgmap v134: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 760 MiB/s wr, 1.23k op/s 2024-05-20T18:02:56.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:56 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:02:56.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:56 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:02:56.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:56 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:02:57.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:57 smithi154 ceph-mon[36482]: pgmap v135: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 960 MiB/s wr, 1.62k op/s 2024-05-20T18:02:57.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:57 smithi129 ceph-mon[32029]: pgmap v135: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 960 MiB/s wr, 1.62k op/s 2024-05-20T18:02:57.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:57 smithi204 ceph-mon[37076]: pgmap v135: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 960 MiB/s wr, 1.62k op/s 2024-05-20T18:02:58.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:58 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:02:58.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:58 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:02:58.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:58 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:02:59.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.274+0000 7efc750fe640 1 -- 172.21.15.129:0/1420212879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc70109080 msgr2=0x7efc70109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:59.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.274+0000 7efc750fe640 1 --2- 172.21.15.129:0/1420212879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc70109080 0x7efc70109490 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7efc60009900 tx=0x7efc6002a480 comp rx=0 tx=0).stop 2024-05-20T18:02:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.283+0000 7efc750fe640 1 -- 172.21.15.129:0/1420212879 shutdown_connections 2024-05-20T18:02:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.283+0000 7efc750fe640 1 --2- 172.21.15.129:0/1420212879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc7011c0d0 0x7efc7011e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.283+0000 7efc750fe640 1 --2- 172.21.15.129:0/1420212879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc70119770 0x7efc7011bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.283+0000 7efc750fe640 1 --2- 172.21.15.129:0/1420212879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc70109080 0x7efc70109490 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.283+0000 7efc750fe640 1 -- 172.21.15.129:0/1420212879 >> 172.21.15.129:0/1420212879 conn(0x7efc7006d780 msgr2=0x7efc7006dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.290+0000 7efc750fe640 1 -- 172.21.15.129:0/1420212879 shutdown_connections 2024-05-20T18:02:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.291+0000 7efc750fe640 1 -- 172.21.15.129:0/1420212879 wait complete. 2024-05-20T18:02:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.293+0000 7efc750fe640 1 Processor -- start 2024-05-20T18:02:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.294+0000 7efc750fe640 1 -- start start 2024-05-20T18:02:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.295+0000 7efc750fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc70109080 0x7efc701a8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.295+0000 7efc750fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc70119770 0x7efc701a8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.295+0000 7efc750fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7011c0d0 0x7efc701ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.295+0000 7efc750fe640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efc70121c50 con 0x7efc70119770 2024-05-20T18:02:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.295+0000 7efc750fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc70121ad0 con 0x7efc70109080 2024-05-20T18:02:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.295+0000 7efc750fe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efc70121dd0 con 0x7efc7011c0d0 2024-05-20T18:02:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.296+0000 7efc6e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc70119770 0x7efc701a8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.296+0000 7efc6e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc70119770 0x7efc701a8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47546/0 (socket says 172.21.15.129:47546) 2024-05-20T18:02:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.296+0000 7efc6e59c640 1 -- 172.21.15.129:0/3245105695 learned_addr learned my addr 172.21.15.129:0/3245105695 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.296+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc70109080 0x7efc701a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.296+0000 7efc6f59e640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7011c0d0 0x7efc701ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.297+0000 7efc6ed9d640 1 -- 172.21.15.129:0/3245105695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7011c0d0 msgr2=0x7efc701ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.297+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7011c0d0 0x7efc701ace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.297+0000 7efc6ed9d640 1 -- 172.21.15.129:0/3245105695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc70119770 msgr2=0x7efc701a8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.297+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc70119770 0x7efc701a8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.297+0000 7efc6ed9d640 1 -- 172.21.15.129:0/3245105695 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc701ad470 con 0x7efc70109080 2024-05-20T18:02:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.297+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc70109080 0x7efc701a8610 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efc6002aa70 tx=0x7efc60003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.301+0000 7efc57fff640 1 -- 172.21.15.129:0/3245105695 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6003f070 con 0x7efc70109080 2024-05-20T18:02:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.301+0000 7efc57fff640 1 -- 172.21.15.129:0/3245105695 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc60031c90 con 0x7efc70109080 2024-05-20T18:02:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.301+0000 7efc57fff640 1 -- 172.21.15.129:0/3245105695 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc60035030 con 0x7efc70109080 2024-05-20T18:02:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.301+0000 7efc750fe640 1 -- 172.21.15.129:0/3245105695 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc701ad700 con 0x7efc70109080 2024-05-20T18:02:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.301+0000 7efc750fe640 1 -- 172.21.15.129:0/3245105695 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc701adb70 con 0x7efc70109080 2024-05-20T18:02:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.310+0000 7efc57fff640 1 -- 172.21.15.129:0/3245105695 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7efc60042070 con 0x7efc70109080 2024-05-20T18:02:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.311+0000 7efc57fff640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7efc580a2f00 0x7efc580a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.312+0000 7efc6e59c640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7efc580a2f00 0x7efc580a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.315+0000 7efc6e59c640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7efc580a2f00 0x7efc580a5410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efc7011b530 tx=0x7efc64005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.315+0000 7efc57fff640 1 -- 172.21.15.129:0/3245105695 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7efc600f0a50 con 0x7efc70109080 2024-05-20T18:02:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.327+0000 7efc7bfa3740 1 Processor -- start 2024-05-20T18:02:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.327+0000 7efc7bfa3740 1 -- start start 2024-05-20T18:02:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.328+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8cb31b0 0x5570e9052080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.328+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 0x5570e90525c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.328+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e9052b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.328+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5570e8fb7340 con 0x5570e8c8a2f0 2024-05-20T18:02:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.328+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5570e8f5f290 con 0x5570e8c924d0 2024-05-20T18:02:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.328+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5570e8fb3300 con 0x5570e8cb31b0 2024-05-20T18:02:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.329+0000 7efc6e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 0x5570e90525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.329+0000 7efc6e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 0x5570e90525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47562/0 (socket says 172.21.15.129:47562) 2024-05-20T18:02:59.327 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.329+0000 7efc6e59c640 1 -- 172.21.15.129:0/3487222959 learned_addr learned my addr 172.21.15.129:0/3487222959 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:59.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.330+0000 7efc6e59c640 1 -- 172.21.15.129:0/3487222959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8cb31b0 msgr2=0x5570e9052080 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:02:59.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.330+0000 7efc6e59c640 1 --2- 172.21.15.129:0/3487222959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8cb31b0 0x5570e9052080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.330+0000 7efc6e59c640 1 -- 172.21.15.129:0/3487222959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 msgr2=0x5570e9052b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:02:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.330+0000 7efc6e59c640 1 --2- 172.21.15.129:0/3487222959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e9052b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.330+0000 7efc6e59c640 1 -- 172.21.15.129:0/3487222959 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570e8f838e0 con 0x5570e8c8a2f0 2024-05-20T18:02:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.333+0000 7efc6e59c640 1 --2- 172.21.15.129:0/3487222959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 0x5570e90525c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7efc64009f90 tx=0x7efc64014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287ee44d23821218 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.334+0000 7efc557fa640 1 -- 172.21.15.129:0/3487222959 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6402b070 con 0x5570e8c8a2f0 2024-05-20T18:02:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.334+0000 7efc557fa640 1 -- 172.21.15.129:0/3487222959 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc64026b00 con 0x5570e8c8a2f0 2024-05-20T18:02:59.334 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.336+0000 7efc7bfa3740 1 -- 172.21.15.129:0/3487222959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 msgr2=0x5570e90525c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:59.334 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.336+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/3487222959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 0x5570e90525c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7efc64009f90 tx=0x7efc64014000 comp rx=0 tx=0).stop 2024-05-20T18:02:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.341+0000 7efc7bfa3740 1 -- 172.21.15.129:0/3487222959 shutdown_connections 2024-05-20T18:02:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.341+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/3487222959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e9052b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.341+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/3487222959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 0x5570e90525c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.341+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/3487222959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8cb31b0 0x5570e9052080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.341+0000 7efc7bfa3740 1 -- 172.21.15.129:0/3487222959 >> 172.21.15.129:0/3487222959 conn(0x5570e8ca7300 msgr2=0x5570e904e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:59.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.343+0000 7efc7bfa3740 1 -- 172.21.15.129:0/3487222959 shutdown_connections 2024-05-20T18:02:59.343 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.344+0000 7efc7bfa3740 1 -- 172.21.15.129:0/3487222959 wait complete. 2024-05-20T18:02:59.343 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.345+0000 7efc7bfa3740 1 -- start start 2024-05-20T18:02:59.343 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.345+0000 7efc7bfa3740 1 Processor -- start 2024-05-20T18:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.346+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 0x5570e905e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.346+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e905e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.346+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8cb31b0 0x5570e905c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.346+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5570e9070210 con 0x5570e8c8a2f0 2024-05-20T18:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.346+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5570e8f72790 con 0x5570e8c924d0 2024-05-20T18:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.346+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5570e9070390 con 0x5570e8cb31b0 2024-05-20T18:02:59.345 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.346+0000 7efc7bfa3740 10 client.0 _ll_register_callbacks cb 0x5570e8fada20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:59.346 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.347+0000 7efc6e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e905e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.346 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.348+0000 7efc6e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e905e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:50260/0 (socket says 172.21.15.129:50260) 2024-05-20T18:02:59.346 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.348+0000 7efc6e59c640 1 -- 172.21.15.129:0/2554332161 learned_addr learned my addr 172.21.15.129:0/2554332161 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:59.346 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.348+0000 7efc6f59e640 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8cb31b0 0x5570e905c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.346 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.348+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 0x5570e905e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.349+0000 7efc6f59e640 1 -- 172.21.15.129:0/2554332161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 msgr2=0x5570e905e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.349+0000 7efc6f59e640 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e905e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.349+0000 7efc6f59e640 1 -- 172.21.15.129:0/2554332161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 msgr2=0x5570e905e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.349+0000 7efc6f59e640 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 0x5570e905e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.349+0000 7efc6f59e640 1 -- 172.21.15.129:0/2554332161 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570e8ccb210 con 0x5570e8cb31b0 2024-05-20T18:02:59.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.350+0000 7efc6f59e640 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8cb31b0 0x5570e905c6b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efc68010790 tx=0x7efc68010c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eb7ec065ef3eb51 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.371+0000 7efc52ff5640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:02:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.371+0000 7efc52ff5640 1 -- 172.21.15.129:0/2554332161 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc68017070 con 0x5570e8cb31b0 2024-05-20T18:02:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.371+0000 7efc7bfa3740 1 -- 172.21.15.129:0/2554332161 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5570e8fc4a70 con 0x5570e8cb31b0 2024-05-20T18:02:59.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.372+0000 7efc52ff5640 1 -- 172.21.15.129:0/2554332161 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7efc68004d20 con 0x5570e8cb31b0 2024-05-20T18:02:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.375+0000 7efc52ff5640 1 -- 172.21.15.129:0/2554332161 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7efc6800ea60 con 0x5570e8cb31b0 2024-05-20T18:02:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.375+0000 7efc7bfa3740 10 client.25840 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.375+0000 7efc7bfa3740 1 -- 172.21.15.129:0/2554332161 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5570e8d97620 con 0x5570e8cb31b0 2024-05-20T18:02:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.376+0000 7efc52ff5640 1 -- 172.21.15.129:0/2554332161 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5570e8fc4a70 con 0x5570e8cb31b0 2024-05-20T18:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.378+0000 7efc52ff5640 1 -- 172.21.15.129:0/2554332161 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7efc6801b410 con 0x5570e8cb31b0 2024-05-20T18:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.378+0000 7efc7bfa3740 10 client.25840 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.378+0000 7efc7bfa3740 10 client.25840 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.378+0000 7efc7bfa3740 20 client.25840 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.378+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570e8c71d80 0x5570e9080140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.378+0000 7efc7bfa3740 4 client.25840 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:02:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.378+0000 7efc7bfa3740 1 -- 172.21.15.129:0/2554332161 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570e8fbedc0 con 0x5570e8c71d80 2024-05-20T18:02:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.380+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570e8c71d80 0x5570e9080140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.383+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570e8c71d80 0x5570e9080140 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82394a6bf5a6d6b1 server_cookie=22f16678a137ef79 in_seq=0 out_seq=0 2024-05-20T18:02:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.383+0000 7efc52ff5640 10 client.25840 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.388+0000 7efc52ff5640 1 -- 172.21.15.129:0/2554332161 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5570e8fbedc0 con 0x5570e8c71d80 2024-05-20T18:02:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.388+0000 7efc52ff5640 10 client.25840 handle_command_reply: tid=0 2024-05-20T18:02:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.388+0000 7efc7bfa3740 1 client.25840 shutdown 2024-05-20T18:02:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.389+0000 7efc7bfa3740 1 -- 172.21.15.129:0/2554332161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8cb31b0 msgr2=0x5570e905c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.389+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8cb31b0 0x5570e905c6b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efc68010790 tx=0x7efc68010c50 comp rx=0 tx=0).stop 2024-05-20T18:02:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.389+0000 7efc7bfa3740 1 -- 172.21.15.129:0/2554332161 shutdown_connections 2024-05-20T18:02:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.389+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570e8c71d80 0x5570e9080140 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.389+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e905e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.389+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8cb31b0 0x5570e905c6b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.389+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/2554332161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c8a2f0 0x5570e905e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.389+0000 7efc52ff5640 0 client.25840 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.389+0000 7efc7bfa3740 1 -- 172.21.15.129:0/2554332161 >> 172.21.15.129:0/2554332161 conn(0x5570e8ca7300 msgr2=0x5570e9052020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.389+0000 7efc7bfa3740 1 -- 172.21.15.129:0/2554332161 shutdown_connections 2024-05-20T18:02:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.389+0000 7efc7bfa3740 1 -- 172.21.15.129:0/2554332161 wait complete. 2024-05-20T18:02:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.390+0000 7efc7bfa3740 20 client.25840 trim_cache size 0 max 16384 2024-05-20T18:02:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.482+0000 7efc7bfa3740 1 Processor -- start 2024-05-20T18:02:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.482+0000 7efc7bfa3740 1 -- start start 2024-05-20T18:02:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.483+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 0x5570e9052730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.483+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e9085940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.484+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8cb31b0 0x5570e9085e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.484+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c00c7a0 con 0x5570e8cb31b0 2024-05-20T18:02:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.484+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c00ef70 con 0x5570e8c924d0 2024-05-20T18:02:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.484+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c00cd60 con 0x5570e8c8a2f0 2024-05-20T18:02:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.485+0000 7efc6e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e9085940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.485+0000 7efc6ed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 0x5570e9052730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.485+0000 7efc6e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e9085940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:50264/0 (socket says 172.21.15.129:50264) 2024-05-20T18:02:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.485+0000 7efc6e59c640 1 -- 172.21.15.129:0/512657005 learned_addr learned my addr 172.21.15.129:0/512657005 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.486+0000 7efc6ed9d640 1 -- 172.21.15.129:0/512657005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 msgr2=0x5570e9085940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.486+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/512657005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e9085940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.486+0000 7efc6ed9d640 1 -- 172.21.15.129:0/512657005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8cb31b0 msgr2=0x5570e9085e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.486+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/512657005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8cb31b0 0x5570e9085e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.486+0000 7efc6ed9d640 1 -- 172.21.15.129:0/512657005 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570e904f420 con 0x5570e8c8a2f0 2024-05-20T18:02:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.487+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/512657005 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 0x5570e9052730 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc600ee130 tx=0x7efc600dd050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45422e42e2ceb45c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.489+0000 7efc54ff9640 1 -- 172.21.15.129:0/512657005 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc600bd600 con 0x5570e8c8a2f0 2024-05-20T18:02:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.489+0000 7efc54ff9640 1 -- 172.21.15.129:0/512657005 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc600bcf00 con 0x5570e8c8a2f0 2024-05-20T18:02:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.489+0000 7efc54ff9640 1 -- 172.21.15.129:0/512657005 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc600bd160 con 0x5570e8c8a2f0 2024-05-20T18:02:59.489 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.490+0000 7efc7bfa3740 1 -- 172.21.15.129:0/512657005 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 msgr2=0x5570e9052730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:02:59.489 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.491+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/512657005 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 0x5570e9052730 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efc600ee130 tx=0x7efc600dd050 comp rx=0 tx=0).stop 2024-05-20T18:02:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.499+0000 7efc7bfa3740 1 -- 172.21.15.129:0/512657005 shutdown_connections 2024-05-20T18:02:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.499+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/512657005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8c924d0 0x5570e9085940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.499+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/512657005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8cb31b0 0x5570e9085e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.499+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/512657005 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 0x5570e9052730 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.499+0000 7efc7bfa3740 1 -- 172.21.15.129:0/512657005 >> 172.21.15.129:0/512657005 conn(0x5570e8ca7300 msgr2=0x5570e9052020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:02:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.502+0000 7efc7bfa3740 1 -- 172.21.15.129:0/512657005 shutdown_connections 2024-05-20T18:02:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.502+0000 7efc7bfa3740 1 -- 172.21.15.129:0/512657005 wait complete. 2024-05-20T18:02:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.503+0000 7efc7bfa3740 1 -- start start 2024-05-20T18:02:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.503+0000 7efc7bfa3740 1 Processor -- start 2024-05-20T18:02:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.503+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 0x5570e90729e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.504+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c924d0 0x5570e9072f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.504+0000 7efc7bfa3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8cb31b0 0x5570e90e64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.504+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c00d430 con 0x5570e8c924d0 2024-05-20T18:02:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.504+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c00e8a0 con 0x5570e8cb31b0 2024-05-20T18:02:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.504+0000 7efc7bfa3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c00db00 con 0x5570e8c8a2f0 2024-05-20T18:02:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.504+0000 7efc7bfa3740 10 client.0 _ll_register_callbacks cb 0x7efc68009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:02:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.504+0000 7efc6ed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 0x5570e90729e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.504+0000 7efc6ed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 0x5570e90729e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:47150/0 (socket says 172.21.15.129:47150) 2024-05-20T18:02:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.504+0000 7efc6ed9d640 1 -- 172.21.15.129:0/1991479956 learned_addr learned my addr 172.21.15.129:0/1991479956 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:02:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.505+0000 7efc6f59e640 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8cb31b0 0x5570e90e64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.505+0000 7efc6ed9d640 1 -- 172.21.15.129:0/1991479956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8cb31b0 msgr2=0x5570e90e64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.505+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8cb31b0 0x5570e90e64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.505+0000 7efc6ed9d640 1 -- 172.21.15.129:0/1991479956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c924d0 msgr2=0x5570e9072f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:02:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.505+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c924d0 0x5570e9072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:02:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.505+0000 7efc6ed9d640 1 -- 172.21.15.129:0/1991479956 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570e90e6a30 con 0x5570e8c8a2f0 2024-05-20T18:02:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.506+0000 7efc6ed9d640 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 0x5570e90729e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efc600ee160 tx=0x7efc600ee0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9dfaa176eb4a144 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:02:59.510 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.512+0000 7efc52ff5640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:02:59.510 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.512+0000 7efc52ff5640 1 -- 172.21.15.129:0/1991479956 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc600ee5b0 con 0x5570e8c8a2f0 2024-05-20T18:02:59.511 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.512+0000 7efc7bfa3740 1 -- 172.21.15.129:0/1991479956 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc600bcf00 con 0x5570e8c8a2f0 2024-05-20T18:02:59.511 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.512+0000 7efc52ff5640 1 -- 172.21.15.129:0/1991479956 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7efc600bd160 con 0x5570e8c8a2f0 2024-05-20T18:02:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.517+0000 7efc52ff5640 1 -- 172.21.15.129:0/1991479956 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7efc600c12b0 con 0x5570e8c8a2f0 2024-05-20T18:02:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.517+0000 7efc7bfa3740 10 client.25846 fetch_fsmap learned FSMap version 22 2024-05-20T18:02:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.517+0000 7efc7bfa3740 1 -- 172.21.15.129:0/1991479956 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5570e90e7bd0 con 0x5570e8c8a2f0 2024-05-20T18:02:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.518+0000 7efc52ff5640 1 -- 172.21.15.129:0/1991479956 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc600bcf00 con 0x5570e8c8a2f0 2024-05-20T18:02:59.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.520+0000 7efc52ff5640 1 -- 172.21.15.129:0/1991479956 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7efc600a9c60 con 0x5570e8c8a2f0 2024-05-20T18:02:59.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.520+0000 7efc7bfa3740 10 client.25846 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:02:59.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.520+0000 7efc7bfa3740 10 client.25846 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:02:59.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.520+0000 7efc7bfa3740 20 client.25846 populate_metadata read hostname 'smithi129' 2024-05-20T18:02:59.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.520+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570e8c471c0 0x5570e90ea280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:02:59.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.520+0000 7efc7bfa3740 4 client.25846 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:02:59.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.520+0000 7efc7bfa3740 1 -- 172.21.15.129:0/1991479956 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5570e8f3c8e0 con 0x5570e8c471c0 2024-05-20T18:02:59.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.528+0000 7efc6e59c640 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570e8c471c0 0x5570e90ea280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:02:59.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.532+0000 7efc52ff5640 10 client.25846 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:02:59.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:02:59.532+0000 7efc6e59c640 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570e8c471c0 0x5570e90ea280 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c3eb60a41485d29 server_cookie=babb302343c0e17e in_seq=0 out_seq=0 2024-05-20T18:02:59.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:02:59 smithi154 ceph-mon[36482]: pgmap v136: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 721 MiB/s wr, 1.29k op/s 2024-05-20T18:02:59.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:02:59 smithi129 ceph-mon[32029]: pgmap v136: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 721 MiB/s wr, 1.29k op/s 2024-05-20T18:02:59.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:02:59 smithi204 ceph-mon[37076]: pgmap v136: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 721 MiB/s wr, 1.29k op/s 2024-05-20T18:03:00.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:00 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:03:00.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:00 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:03:00.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:00 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:03:00.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:00 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:03:00.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:00 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:03:00.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:00 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:03:00.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:00 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:03:00.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:00 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:03:00.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:00 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:03:01.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:01 smithi154 ceph-mon[36482]: pgmap v137: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 941 MiB/s wr, 1.69k op/s 2024-05-20T18:03:01.676 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:01 smithi129 ceph-mon[32029]: pgmap v137: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 941 MiB/s wr, 1.69k op/s 2024-05-20T18:03:01.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:01 smithi204 ceph-mon[37076]: pgmap v137: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 941 MiB/s wr, 1.69k op/s 2024-05-20T18:03:03.263 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.263+0000 7efc52ff5640 1 -- 172.21.15.129:0/1991479956 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5570e8f3c8e0 con 0x5570e8c471c0 2024-05-20T18:03:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.263+0000 7efc52ff5640 10 client.25846 handle_command_reply: tid=0 2024-05-20T18:03:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.263+0000 7efc7bfa3740 1 client.25846 shutdown 2024-05-20T18:03:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.264+0000 7efc7bfa3740 1 -- 172.21.15.129:0/1991479956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 msgr2=0x5570e90729e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.264+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 0x5570e90729e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efc600ee160 tx=0x7efc600ee0a0 comp rx=0 tx=0).stop 2024-05-20T18:03:03.264 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.265+0000 7efc7bfa3740 1 -- 172.21.15.129:0/1991479956 shutdown_connections 2024-05-20T18:03:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.265+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570e8c471c0 0x5570e90ea280 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.265+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570e8cb31b0 0x5570e90e64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.265+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570e8c924d0 0x5570e9072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.265+0000 7efc7bfa3740 1 --2- 172.21.15.129:0/1991479956 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570e8c8a2f0 0x5570e90729e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:03.265 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.265+0000 7efc7bfa3740 1 -- 172.21.15.129:0/1991479956 >> 172.21.15.129:0/1991479956 conn(0x5570e8ca7300 msgr2=0x5570e9055d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:03.266 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.265+0000 7efc52ff5640 0 client.25846 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.266+0000 7efc7bfa3740 1 -- 172.21.15.129:0/1991479956 shutdown_connections 2024-05-20T18:03:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.266+0000 7efc7bfa3740 1 -- 172.21.15.129:0/1991479956 wait complete. 2024-05-20T18:03:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.266+0000 7efc7bfa3740 20 client.25846 trim_cache size 0 max 16384 2024-05-20T18:03:03.267 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:03:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:03:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:03:03.268 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:03:03.270 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.269+0000 7efc750fe640 1 -- 172.21.15.129:0/3245105695 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7efc580a2f00 msgr2=0x7efc580a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:03.270 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.270+0000 7efc750fe640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7efc580a2f00 0x7efc580a5410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efc7011b530 tx=0x7efc64005f90 comp rx=0 tx=0).stop 2024-05-20T18:03:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.270+0000 7efc750fe640 1 -- 172.21.15.129:0/3245105695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc70109080 msgr2=0x7efc701a8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.270+0000 7efc750fe640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc70109080 0x7efc701a8610 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efc6002aa70 tx=0x7efc60003b90 comp rx=0 tx=0).stop 2024-05-20T18:03:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.271+0000 7efc750fe640 1 -- 172.21.15.129:0/3245105695 shutdown_connections 2024-05-20T18:03:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.271+0000 7efc750fe640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7efc580a2f00 0x7efc580a5410 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.271+0000 7efc750fe640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efc7011c0d0 0x7efc701ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.271+0000 7efc750fe640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc70119770 0x7efc701a8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.271+0000 7efc750fe640 1 --2- 172.21.15.129:0/3245105695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc70109080 0x7efc701a8610 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.271+0000 7efc750fe640 1 -- 172.21.15.129:0/3245105695 >> 172.21.15.129:0/3245105695 conn(0x7efc7006d780 msgr2=0x7efc70119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:03.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.272+0000 7efc750fe640 1 -- 172.21.15.129:0/3245105695 shutdown_connections 2024-05-20T18:03:03.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:03.272+0000 7efc750fe640 1 -- 172.21.15.129:0/3245105695 wait complete. 2024-05-20T18:03:03.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:03 smithi154 ceph-mon[36482]: pgmap v138: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 712 MiB/s wr, 1.29k op/s 2024-05-20T18:03:03.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:03 smithi129 ceph-mon[32029]: pgmap v138: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 712 MiB/s wr, 1.29k op/s 2024-05-20T18:03:03.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:03 smithi204 ceph-mon[37076]: pgmap v138: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 712 MiB/s wr, 1.29k op/s 2024-05-20T18:03:05.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:05 smithi154 ceph-mon[36482]: pgmap v139: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 750 MiB/s wr, 1.43k op/s 2024-05-20T18:03:05.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:05 smithi129 ceph-mon[32029]: pgmap v139: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 750 MiB/s wr, 1.43k op/s 2024-05-20T18:03:05.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:05 smithi204 ceph-mon[37076]: pgmap v139: 97 pgs: 97 active+clean; 30 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 750 MiB/s wr, 1.43k op/s 2024-05-20T18:03:07.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:07 smithi154 ceph-mon[36482]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 945 MiB/s wr, 2.06k op/s 2024-05-20T18:03:07.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:07 smithi129 ceph-mon[32029]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 945 MiB/s wr, 2.06k op/s 2024-05-20T18:03:07.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:07 smithi204 ceph-mon[37076]: pgmap v140: 97 pgs: 97 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 945 MiB/s wr, 2.06k op/s 2024-05-20T18:03:09.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:09 smithi154 ceph-mon[36482]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 708 MiB/s wr, 1.70k op/s 2024-05-20T18:03:09.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:09 smithi129 ceph-mon[32029]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 708 MiB/s wr, 1.70k op/s 2024-05-20T18:03:09.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:09 smithi204 ceph-mon[37076]: pgmap v141: 97 pgs: 97 active+clean; 30 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 708 MiB/s wr, 1.70k op/s 2024-05-20T18:03:11.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:11 smithi154 ceph-mon[36482]: pgmap v142: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 944 MiB/s wr, 3.56k op/s 2024-05-20T18:03:11.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:11 smithi129 ceph-mon[32029]: pgmap v142: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 944 MiB/s wr, 3.56k op/s 2024-05-20T18:03:11.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:11 smithi204 ceph-mon[37076]: pgmap v142: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 944 MiB/s wr, 3.56k op/s 2024-05-20T18:03:13.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:13 smithi154 ceph-mon[36482]: pgmap v143: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 675 MiB/s wr, 3.21k op/s 2024-05-20T18:03:13.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:13 smithi129 ceph-mon[32029]: pgmap v143: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 675 MiB/s wr, 3.21k op/s 2024-05-20T18:03:13.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:13 smithi204 ceph-mon[37076]: pgmap v143: 97 pgs: 97 active+clean; 32 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 675 MiB/s wr, 3.21k op/s 2024-05-20T18:03:15.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:15 smithi154 ceph-mon[36482]: pgmap v144: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 MiB/s wr, 3.74k op/s 2024-05-20T18:03:15.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:15 smithi129 ceph-mon[32029]: pgmap v144: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 MiB/s wr, 3.74k op/s 2024-05-20T18:03:15.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:15 smithi204 ceph-mon[37076]: pgmap v144: 97 pgs: 97 active+clean; 33 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 MiB/s wr, 3.74k op/s 2024-05-20T18:03:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.605+0000 7fa539ffb640 1 -- 172.21.15.129:0/278196930 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534030e30 con 0x7fa53c106460 2024-05-20T18:03:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.607+0000 7fa54144a640 1 -- 172.21.15.129:0/278196930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa53c106460 msgr2=0x7fa53c106870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.607+0000 7fa54144a640 1 --2- 172.21.15.129:0/278196930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa53c106460 0x7fa53c106870 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa534009be0 tx=0x7fa53402a6d0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.607+0000 7fa54144a640 1 -- 172.21.15.129:0/278196930 shutdown_connections 2024-05-20T18:03:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.607+0000 7fa54144a640 1 --2- 172.21.15.129:0/278196930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa53c106e90 0x7fa53c107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.607+0000 7fa54144a640 1 --2- 172.21.15.129:0/278196930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa53c106460 0x7fa53c106870 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.607+0000 7fa54144a640 1 --2- 172.21.15.129:0/278196930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa53c101a10 0x7fa53c101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.607+0000 7fa54144a640 1 -- 172.21.15.129:0/278196930 >> 172.21.15.129:0/278196930 conn(0x7fa53c077160 msgr2=0x7fa53c0775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.608+0000 7fa54144a640 1 -- 172.21.15.129:0/278196930 shutdown_connections 2024-05-20T18:03:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.608+0000 7fa54144a640 1 -- 172.21.15.129:0/278196930 wait complete. 2024-05-20T18:03:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.609+0000 7fa54144a640 1 Processor -- start 2024-05-20T18:03:16.607 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.609+0000 7fa54144a640 1 -- start start 2024-05-20T18:03:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.609+0000 7fa54144a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa53c101a10 0x7fa53c115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.609+0000 7fa54144a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa53c106460 0x7fa53c10ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.609+0000 7fa54144a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa53c106e90 0x7fa53c10f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.610+0000 7fa54144a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c1193f0 con 0x7fa53c106e90 2024-05-20T18:03:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.610+0000 7fa54144a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c119270 con 0x7fa53c106460 2024-05-20T18:03:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.610+0000 7fa54144a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c119570 con 0x7fa53c101a10 2024-05-20T18:03:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.610+0000 7fa53a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa53c106460 0x7fa53c10ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.610+0000 7fa53a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa53c106460 0x7fa53c10ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59100/0 (socket says 172.21.15.129:59100) 2024-05-20T18:03:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.610+0000 7fa53a7fc640 1 -- 172.21.15.129:0/222157013 learned_addr learned my addr 172.21.15.129:0/222157013 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.610+0000 7fa53affd640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa53c101a10 0x7fa53c115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.610+0000 7fa53b7fe640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa53c106e90 0x7fa53c10f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.611+0000 7fa53a7fc640 1 -- 172.21.15.129:0/222157013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa53c101a10 msgr2=0x7fa53c115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.611+0000 7fa53a7fc640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa53c101a10 0x7fa53c115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.611+0000 7fa53a7fc640 1 -- 172.21.15.129:0/222157013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa53c106e90 msgr2=0x7fa53c10f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.611+0000 7fa53a7fc640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa53c106e90 0x7fa53c10f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.611+0000 7fa53a7fc640 1 -- 172.21.15.129:0/222157013 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53c113850 con 0x7fa53c106460 2024-05-20T18:03:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.616+0000 7fa53a7fc640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa53c106460 0x7fa53c10ee70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa53402acc0 tx=0x7fa534030710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.622+0000 7fa523fff640 1 -- 172.21.15.129:0/222157013 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534036040 con 0x7fa53c106460 2024-05-20T18:03:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.622+0000 7fa523fff640 1 -- 172.21.15.129:0/222157013 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa534042070 con 0x7fa53c106460 2024-05-20T18:03:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.622+0000 7fa523fff640 1 -- 172.21.15.129:0/222157013 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534004330 con 0x7fa53c106460 2024-05-20T18:03:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.622+0000 7fa54144a640 1 -- 172.21.15.129:0/222157013 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa53c113ae0 con 0x7fa53c106460 2024-05-20T18:03:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.623+0000 7fa54144a640 1 -- 172.21.15.129:0/222157013 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa53c113f80 con 0x7fa53c106460 2024-05-20T18:03:16.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.627+0000 7fa5483c5740 1 Processor -- start 2024-05-20T18:03:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.630+0000 7fa5483c5740 1 -- start start 2024-05-20T18:03:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.630+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 0x55ce23a94080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.630+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23a945c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.630+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 0x55ce23a94b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.630+0000 7fa5483c5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ce239f9340 con 0x55ce236d44d0 2024-05-20T18:03:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.630+0000 7fa5483c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ce239a1290 con 0x55ce236cc2f0 2024-05-20T18:03:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.630+0000 7fa5483c5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ce239f5300 con 0x55ce236f51b0 2024-05-20T18:03:16.629 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.631+0000 7fa523fff640 1 -- 172.21.15.129:0/222157013 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa5340044d0 con 0x7fa53c106460 2024-05-20T18:03:16.629 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.632+0000 7fa523fff640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5240a2f00 0x7fa5240a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.632+0000 7fa523fff640 1 -- 172.21.15.129:0/222157013 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa5340efd20 con 0x7fa53c106460 2024-05-20T18:03:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.635+0000 7fa53b7fe640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5240a2f00 0x7fa5240a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.635+0000 7fa53b7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 0x55ce23a94080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.635+0000 7fa53b7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 0x55ce23a94080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50510/0 (socket says 172.21.15.129:50510) 2024-05-20T18:03:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.635+0000 7fa53b7fe640 1 -- 172.21.15.129:0/2296089323 learned_addr learned my addr 172.21.15.129:0/2296089323 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.636+0000 7fa53affd640 1 --2- 172.21.15.129:0/2296089323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23a945c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.636+0000 7fa53a7fc640 1 --2- 172.21.15.129:0/2296089323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 0x55ce23a94b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.636+0000 7fa53b7fe640 1 -- 172.21.15.129:0/2296089323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 msgr2=0x55ce23a945c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.636+0000 7fa53b7fe640 1 --2- 172.21.15.129:0/2296089323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23a945c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.636+0000 7fa53b7fe640 1 -- 172.21.15.129:0/2296089323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 msgr2=0x55ce23a94b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.636+0000 7fa53b7fe640 1 --2- 172.21.15.129:0/2296089323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 0x55ce23a94b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.636+0000 7fa53b7fe640 1 -- 172.21.15.129:0/2296089323 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce239c58e0 con 0x55ce236f51b0 2024-05-20T18:03:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.636+0000 7fa53b7fe640 1 --2- 172.21.15.129:0/2296089323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 0x55ce23a94080 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa5300127c0 tx=0x7fa530012c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dccda3890e2b01c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.637+0000 7fa53b7fe640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5240a2f00 0x7fa5240a5410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa530015000 tx=0x7fa53001d000 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.638+0000 7fa5217fa640 1 -- 172.21.15.129:0/2296089323 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530010ea0 con 0x55ce236f51b0 2024-05-20T18:03:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.638+0000 7fa5217fa640 1 -- 172.21.15.129:0/2296089323 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa530005510 con 0x55ce236f51b0 2024-05-20T18:03:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.639+0000 7fa5483c5740 1 -- 172.21.15.129:0/2296089323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 msgr2=0x55ce23a94080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.639+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2296089323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 0x55ce23a94080 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa5300127c0 tx=0x7fa530012c80 comp rx=0 tx=0).stop 2024-05-20T18:03:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.641+0000 7fa5483c5740 1 -- 172.21.15.129:0/2296089323 shutdown_connections 2024-05-20T18:03:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.641+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2296089323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23a945c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.641+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2296089323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 0x55ce23a94b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.641+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2296089323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 0x55ce23a94080 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.641+0000 7fa5483c5740 1 -- 172.21.15.129:0/2296089323 >> 172.21.15.129:0/2296089323 conn(0x55ce236e9300 msgr2=0x55ce23a90830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.642+0000 7fa5483c5740 1 -- 172.21.15.129:0/2296089323 shutdown_connections 2024-05-20T18:03:16.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.642+0000 7fa5483c5740 1 -- 172.21.15.129:0/2296089323 wait complete. 2024-05-20T18:03:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.643+0000 7fa5483c5740 1 -- start start 2024-05-20T18:03:16.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.643+0000 7fa5483c5740 1 Processor -- start 2024-05-20T18:03:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.643+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23a9e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.643+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 0x55ce23a9ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.643+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 0x55ce23aa4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.643+0000 7fa5483c5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ce23aa11a0 con 0x55ce236d44d0 2024-05-20T18:03:16.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.643+0000 7fa5483c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ce239b4790 con 0x55ce236cc2f0 2024-05-20T18:03:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.643+0000 7fa5483c5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ce23aa1320 con 0x55ce236f51b0 2024-05-20T18:03:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.643+0000 7fa5483c5740 10 client.0 _ll_register_callbacks cb 0x55ce239efa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.644+0000 7fa53affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23a9e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.644+0000 7fa53affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23a9e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59126/0 (socket says 172.21.15.129:59126) 2024-05-20T18:03:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.644+0000 7fa53affd640 1 -- 172.21.15.129:0/2395552825 learned_addr learned my addr 172.21.15.129:0/2395552825 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:16.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.644+0000 7fa53b7fe640 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 0x55ce23aa4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.644+0000 7fa53affd640 1 -- 172.21.15.129:0/2395552825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 msgr2=0x55ce23aa4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.644+0000 7fa53affd640 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 0x55ce23aa4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.644+0000 7fa53affd640 1 -- 172.21.15.129:0/2395552825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 msgr2=0x55ce23a9ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.644+0000 7fa53affd640 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 0x55ce23a9ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.644+0000 7fa53affd640 1 -- 172.21.15.129:0/2395552825 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce2370d210 con 0x55ce236cc2f0 2024-05-20T18:03:16.643 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.645+0000 7fa53affd640 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23a9e720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0136e0 tx=0x7fa52c013ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c551e8bb73c89397 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.652+0000 7fa51eff5640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:03:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.652+0000 7fa51eff5640 1 -- 172.21.15.129:0/2395552825 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c011e90 con 0x55ce236cc2f0 2024-05-20T18:03:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.652+0000 7fa51eff5640 1 -- 172.21.15.129:0/2395552825 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa52c004030 con 0x55ce236cc2f0 2024-05-20T18:03:16.649 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.652+0000 7fa5483c5740 1 -- 172.21.15.129:0/2395552825 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa530005510 con 0x55ce236cc2f0 2024-05-20T18:03:16.651 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.654+0000 7fa51eff5640 1 -- 172.21.15.129:0/2395552825 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa52c004230 con 0x55ce236cc2f0 2024-05-20T18:03:16.651 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.654+0000 7fa5483c5740 10 client.25844 fetch_fsmap learned FSMap version 22 2024-05-20T18:03:16.651 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.654+0000 7fa5483c5740 1 -- 172.21.15.129:0/2395552825 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ce237d9620 con 0x55ce236cc2f0 2024-05-20T18:03:16.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.657+0000 7fa51eff5640 1 -- 172.21.15.129:0/2395552825 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa530005510 con 0x55ce236cc2f0 2024-05-20T18:03:16.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.657+0000 7fa51eff5640 1 -- 172.21.15.129:0/2395552825 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fa52c01b5c0 con 0x55ce236cc2f0 2024-05-20T18:03:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.660+0000 7fa5483c5740 10 client.25844 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:03:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.660+0000 7fa5483c5740 10 client.25844 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.660+0000 7fa5483c5740 20 client.25844 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.660+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ce236b3d80 0x55ce23ab0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.660+0000 7fa5483c5740 4 client.25844 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:03:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.660+0000 7fa5483c5740 1 -- 172.21.15.129:0/2395552825 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce23a06a70 con 0x55ce236b3d80 2024-05-20T18:03:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.660+0000 7fa53a7fc640 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ce236b3d80 0x55ce23ab0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.661+0000 7fa51eff5640 10 client.25844 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.661+0000 7fa53a7fc640 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ce236b3d80 0x55ce23ab0a30 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea1e7f3bc7effbfb server_cookie=165b3360a1a793fa in_seq=0 out_seq=0 2024-05-20T18:03:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.664+0000 7fa51eff5640 1 -- 172.21.15.129:0/2395552825 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ce23a06a70 con 0x55ce236b3d80 2024-05-20T18:03:16.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.664+0000 7fa51eff5640 10 client.25844 handle_command_reply: tid=0 2024-05-20T18:03:16.665 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.668+0000 7fa5483c5740 1 client.25844 shutdown 2024-05-20T18:03:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.669+0000 7fa5483c5740 1 -- 172.21.15.129:0/2395552825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 msgr2=0x55ce23a9e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.669+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23a9e720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0136e0 tx=0x7fa52c013ba0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.672+0000 7fa5483c5740 1 -- 172.21.15.129:0/2395552825 shutdown_connections 2024-05-20T18:03:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.672+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ce236b3d80 0x55ce23ab0a30 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.672+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236f51b0 0x55ce23aa4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.672+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 0x55ce23a9ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.672+0000 7fa51eff5640 0 client.25844 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.672+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2395552825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23a9e720 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.672+0000 7fa5483c5740 1 -- 172.21.15.129:0/2395552825 >> 172.21.15.129:0/2395552825 conn(0x55ce236e9300 msgr2=0x55ce23a93e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.672+0000 7fa5483c5740 1 -- 172.21.15.129:0/2395552825 shutdown_connections 2024-05-20T18:03:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.672+0000 7fa5483c5740 1 -- 172.21.15.129:0/2395552825 wait complete. 2024-05-20T18:03:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.673+0000 7fa5483c5740 20 client.25844 trim_cache size 0 max 16384 2024-05-20T18:03:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.709+0000 7fa5483c5740 1 Processor -- start 2024-05-20T18:03:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.710+0000 7fa5483c5740 1 -- start start 2024-05-20T18:03:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.711+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236cc2f0 0x55ce23a8f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.711+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 0x55ce23aa4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.711+0000 7fa53affd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236cc2f0 0x55ce23a8f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.711+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236f51b0 0x55ce23aa4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.711+0000 7fa53affd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236cc2f0 0x55ce23a8f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50526/0 (socket says 172.21.15.129:50526) 2024-05-20T18:03:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.711+0000 7fa5483c5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa52800c920 con 0x55ce236d44d0 2024-05-20T18:03:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.711+0000 7fa5483c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa52800f1a0 con 0x55ce236f51b0 2024-05-20T18:03:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.711+0000 7fa5483c5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa52800cf90 con 0x55ce236cc2f0 2024-05-20T18:03:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.712+0000 7fa53affd640 1 -- 172.21.15.129:0/3941206341 learned_addr learned my addr 172.21.15.129:0/3941206341 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:16.709 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.712+0000 7fa53b7fe640 1 --2- 172.21.15.129:0/3941206341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236f51b0 0x55ce23aa4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.712+0000 7fa53a7fc640 1 --2- 172.21.15.129:0/3941206341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 0x55ce23aa4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.712+0000 7fa53affd640 1 -- 172.21.15.129:0/3941206341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236f51b0 msgr2=0x55ce23aa4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.712+0000 7fa53affd640 1 --2- 172.21.15.129:0/3941206341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236f51b0 0x55ce23aa4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.712+0000 7fa53affd640 1 -- 172.21.15.129:0/3941206341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 msgr2=0x55ce23aa4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.712+0000 7fa53affd640 1 --2- 172.21.15.129:0/3941206341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 0x55ce23aa4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.712+0000 7fa53affd640 1 -- 172.21.15.129:0/3941206341 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce23a93a10 con 0x55ce236cc2f0 2024-05-20T18:03:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.713+0000 7fa53affd640 1 --2- 172.21.15.129:0/3941206341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236cc2f0 0x55ce23a8f4f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa52c01b9e0 tx=0x7fa52c009dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55e93f6630df25da server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:16.710 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.713+0000 7fa520ff9640 1 -- 172.21.15.129:0/3941206341 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c032020 con 0x55ce236cc2f0 2024-05-20T18:03:16.711 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.713+0000 7fa520ff9640 1 -- 172.21.15.129:0/3941206341 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa52c011d90 con 0x55ce236cc2f0 2024-05-20T18:03:16.711 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.713+0000 7fa520ff9640 1 -- 172.21.15.129:0/3941206341 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c025750 con 0x55ce236cc2f0 2024-05-20T18:03:16.712 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.715+0000 7fa5483c5740 1 -- 172.21.15.129:0/3941206341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236cc2f0 msgr2=0x55ce23a8f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:16.712 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.715+0000 7fa5483c5740 1 --2- 172.21.15.129:0/3941206341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236cc2f0 0x55ce23a8f4f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa52c01b9e0 tx=0x7fa52c009dd0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.712 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.715+0000 7fa5483c5740 1 -- 172.21.15.129:0/3941206341 shutdown_connections 2024-05-20T18:03:16.712 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.715+0000 7fa5483c5740 1 --2- 172.21.15.129:0/3941206341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236f51b0 0x55ce23aa4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.712 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.715+0000 7fa5483c5740 1 --2- 172.21.15.129:0/3941206341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236d44d0 0x55ce23aa4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.712 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.715+0000 7fa5483c5740 1 --2- 172.21.15.129:0/3941206341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236cc2f0 0x55ce23a8f4f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.712 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.715+0000 7fa5483c5740 1 -- 172.21.15.129:0/3941206341 >> 172.21.15.129:0/3941206341 conn(0x55ce236e9300 msgr2=0x55ce23a8ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.719+0000 7fa5483c5740 1 -- 172.21.15.129:0/3941206341 shutdown_connections 2024-05-20T18:03:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.720+0000 7fa5483c5740 1 -- 172.21.15.129:0/3941206341 wait complete. 2024-05-20T18:03:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.721+0000 7fa5483c5740 1 -- start start 2024-05-20T18:03:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.721+0000 7fa5483c5740 1 Processor -- start 2024-05-20T18:03:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.722+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23aa5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.722+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236d44d0 0x55ce23aa5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.722+0000 7fa5483c5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236f51b0 0x55ce23b10da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.722+0000 7fa5483c5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa52800d660 con 0x55ce236f51b0 2024-05-20T18:03:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.722+0000 7fa5483c5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa52800ead0 con 0x55ce236cc2f0 2024-05-20T18:03:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.722+0000 7fa5483c5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa52800dd30 con 0x55ce236d44d0 2024-05-20T18:03:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.722+0000 7fa5483c5740 10 client.0 _ll_register_callbacks cb 0x7fa52c009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.722+0000 7fa53affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23aa5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.722+0000 7fa53affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23aa5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59144/0 (socket says 172.21.15.129:59144) 2024-05-20T18:03:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.722+0000 7fa53affd640 1 -- 172.21.15.129:0/2326949763 learned_addr learned my addr 172.21.15.129:0/2326949763 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:16.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.723+0000 7fa53affd640 1 -- 172.21.15.129:0/2326949763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236d44d0 msgr2=0x55ce23aa5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:16.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.723+0000 7fa53affd640 1 --2- 172.21.15.129:0/2326949763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236d44d0 0x55ce23aa5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.723+0000 7fa53affd640 1 -- 172.21.15.129:0/2326949763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236f51b0 msgr2=0x55ce23b10da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:16.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.723+0000 7fa53affd640 1 --2- 172.21.15.129:0/2326949763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236f51b0 0x55ce23b10da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:16.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.723+0000 7fa53affd640 1 -- 172.21.15.129:0/2326949763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce23b112e0 con 0x55ce236cc2f0 2024-05-20T18:03:16.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.723+0000 7fa53affd640 1 --2- 172.21.15.129:0/2326949763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23aa5840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa52c000c00 tx=0x7fa52c0055b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb7b2641b630a92c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:16.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.724+0000 7fa51eff5640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:03:16.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.728+0000 7fa51eff5640 1 -- 172.21.15.129:0/2326949763 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c005620 con 0x55ce236cc2f0 2024-05-20T18:03:16.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.728+0000 7fa5483c5740 1 -- 172.21.15.129:0/2326949763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa52c011d90 con 0x55ce236cc2f0 2024-05-20T18:03:16.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.728+0000 7fa51eff5640 1 -- 172.21.15.129:0/2326949763 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8543+0+0 (secure 0 0 0) 0x7fa52c02a040 con 0x55ce236cc2f0 2024-05-20T18:03:16.728 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.731+0000 7fa51eff5640 1 -- 172.21.15.129:0/2326949763 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa52c01b630 con 0x55ce236cc2f0 2024-05-20T18:03:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.738+0000 7fa5483c5740 10 client.25850 fetch_fsmap learned FSMap version 22 2024-05-20T18:03:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.738+0000 7fa5483c5740 1 -- 172.21.15.129:0/2326949763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55ce23b185e0 con 0x55ce236cc2f0 2024-05-20T18:03:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.739+0000 7fa51eff5640 1 -- 172.21.15.129:0/2326949763 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa52c011d90 con 0x55ce236cc2f0 2024-05-20T18:03:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.739+0000 7fa51eff5640 1 -- 172.21.15.129:0/2326949763 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 22) ==== 8322+0+0 (secure 0 0 0) 0x7fa52c038670 con 0x55ce236cc2f0 2024-05-20T18:03:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.741+0000 7fa5483c5740 10 client.25850 fetch_fsmap finished waiting for FSMap version 22 2024-05-20T18:03:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.741+0000 7fa5483c5740 10 client.25850 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.741+0000 7fa5483c5740 20 client.25850 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.741+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2326949763 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ce236891c0 0x55ce23b1ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.741+0000 7fa5483c5740 4 client.25850 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:03:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.741+0000 7fa5483c5740 1 -- 172.21.15.129:0/2326949763 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ce23a00dc0 con 0x55ce236891c0 2024-05-20T18:03:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.742+0000 7fa53a7fc640 1 --2- 172.21.15.129:0/2326949763 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ce236891c0 0x55ce23b1ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.746+0000 7fa53a7fc640 1 --2- 172.21.15.129:0/2326949763 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ce236891c0 0x55ce23b1ac70 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9b75d4583be5247 server_cookie=31849083264584f5 in_seq=0 out_seq=0 2024-05-20T18:03:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:16.747+0000 7fa51eff5640 10 client.25850 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:17.085 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.086+0000 7fa51eff5640 1 -- 172.21.15.129:0/2326949763 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ce23a00dc0 con 0x55ce236891c0 2024-05-20T18:03:17.085 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.086+0000 7fa51eff5640 10 client.25850 handle_command_reply: tid=0 2024-05-20T18:03:17.085 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.086+0000 7fa5483c5740 1 client.25850 shutdown 2024-05-20T18:03:17.085 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa5483c5740 1 -- 172.21.15.129:0/2326949763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 msgr2=0x55ce23aa5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:17.085 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2326949763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23aa5840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa52c000c00 tx=0x7fa52c0055b0 comp rx=0 tx=0).stop 2024-05-20T18:03:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa5483c5740 1 -- 172.21.15.129:0/2326949763 shutdown_connections 2024-05-20T18:03:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2326949763 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ce236891c0 0x55ce23b1ac70 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2326949763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce236f51b0 0x55ce23b10da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2326949763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ce236d44d0 0x55ce23aa5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa5483c5740 1 --2- 172.21.15.129:0/2326949763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ce236cc2f0 0x55ce23aa5840 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa5483c5740 1 -- 172.21.15.129:0/2326949763 >> 172.21.15.129:0/2326949763 conn(0x55ce236e9300 msgr2=0x55ce23a907c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa51eff5640 0 client.25850 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa5483c5740 1 -- 172.21.15.129:0/2326949763 shutdown_connections 2024-05-20T18:03:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa5483c5740 1 -- 172.21.15.129:0/2326949763 wait complete. 2024-05-20T18:03:17.087 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.087+0000 7fa5483c5740 20 client.25850 trim_cache size 0 max 16384 2024-05-20T18:03:17.087 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:03:17.087 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:03:17.087 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:03:17.087 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:03:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.092+0000 7fa54144a640 1 -- 172.21.15.129:0/222157013 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5240a2f00 msgr2=0x7fa5240a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.092+0000 7fa54144a640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5240a2f00 0x7fa5240a5410 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa530015000 tx=0x7fa53001d000 comp rx=0 tx=0).stop 2024-05-20T18:03:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.092+0000 7fa54144a640 1 -- 172.21.15.129:0/222157013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa53c106460 msgr2=0x7fa53c10ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.092+0000 7fa54144a640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa53c106460 0x7fa53c10ee70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa53402acc0 tx=0x7fa534030710 comp rx=0 tx=0).stop 2024-05-20T18:03:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.092+0000 7fa54144a640 1 -- 172.21.15.129:0/222157013 shutdown_connections 2024-05-20T18:03:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.092+0000 7fa54144a640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5240a2f00 0x7fa5240a5410 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.093+0000 7fa54144a640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa53c106460 0x7fa53c10ee70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.093+0000 7fa54144a640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa53c106e90 0x7fa53c10f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.093+0000 7fa54144a640 1 --2- 172.21.15.129:0/222157013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa53c101a10 0x7fa53c115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:17.094 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.093+0000 7fa54144a640 1 -- 172.21.15.129:0/222157013 >> 172.21.15.129:0/222157013 conn(0x7fa53c077160 msgr2=0x7fa53c103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:17.094 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.094+0000 7fa54144a640 1 -- 172.21.15.129:0/222157013 shutdown_connections 2024-05-20T18:03:17.094 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:17.094+0000 7fa54144a640 1 -- 172.21.15.129:0/222157013 wait complete. 2024-05-20T18:03:17.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:17 smithi154 ceph-mon[36482]: pgmap v145: 97 pgs: 97 active+clean; 33 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 903 MiB/s wr, 5.33k op/s 2024-05-20T18:03:17.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:03:17 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: ignoring --setuser ceph since I am not root 2024-05-20T18:03:17.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:03:17 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: ignoring --setgroup ceph since I am not root 2024-05-20T18:03:17.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:03:17 smithi154 ceph-mds[69727]: -- 172.21.15.154:0/2513972466 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559c01af3a40 con 0x559c01b6ba80 2024-05-20T18:03:17.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:03:17 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: starting mds.h at 2024-05-20T18:03:17.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:17 smithi129 ceph-mon[32029]: pgmap v145: 97 pgs: 97 active+clean; 33 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 903 MiB/s wr, 5.33k op/s 2024-05-20T18:03:17.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:17 smithi204 ceph-mon[37076]: pgmap v145: 97 pgs: 97 active+clean; 33 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 903 MiB/s wr, 5.33k op/s 2024-05-20T18:03:18.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:18 smithi154 ceph-mon[36482]: daemon mds.h restarted 2024-05-20T18:03:18.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:18 smithi129 ceph-mon[32029]: daemon mds.h restarted 2024-05-20T18:03:18.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:18 smithi204 ceph-mon[37076]: daemon mds.h restarted 2024-05-20T18:03:19.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:19 smithi154 ceph-mon[36482]: pgmap v146: 97 pgs: 97 active+clean; 33 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 687 MiB/s wr, 4.96k op/s 2024-05-20T18:03:19.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:19 smithi154 ceph-mon[36482]: osdmap e96: 12 total, 12 up, 12 in 2024-05-20T18:03:19.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:19 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.154:6836/4068528573,v1:172.21.15.154:6837/4068528573] up:boot 2024-05-20T18:03:19.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:19 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T18:03:19.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:19 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T18:03:19.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:19 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T18:03:19.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:19 smithi129 ceph-mon[32029]: pgmap v146: 97 pgs: 97 active+clean; 33 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 687 MiB/s wr, 4.96k op/s 2024-05-20T18:03:19.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:19 smithi129 ceph-mon[32029]: osdmap e96: 12 total, 12 up, 12 in 2024-05-20T18:03:19.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:19 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.154:6836/4068528573,v1:172.21.15.154:6837/4068528573] up:boot 2024-05-20T18:03:19.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:19 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T18:03:19.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:19 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T18:03:19.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:19 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T18:03:19.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:19 smithi204 ceph-mon[37076]: pgmap v146: 97 pgs: 97 active+clean; 33 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 687 MiB/s wr, 4.96k op/s 2024-05-20T18:03:19.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:19 smithi204 ceph-mon[37076]: osdmap e96: 12 total, 12 up, 12 in 2024-05-20T18:03:19.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:19 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.154:6836/4068528573,v1:172.21.15.154:6837/4068528573] up:boot 2024-05-20T18:03:19.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:19 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T18:03:19.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:19 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T18:03:19.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:19 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T18:03:21.646 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:03:21.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:03:21.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:21 smithi204 ceph-mon[37076]: pgmap v148: 97 pgs: 97 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 696 MiB/s wr, 5.18k op/s 2024-05-20T18:03:21.757 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:21 smithi129 ceph-mon[32029]: pgmap v148: 97 pgs: 97 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 696 MiB/s wr, 5.18k op/s 2024-05-20T18:03:21.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:21 smithi154 ceph-mon[36482]: pgmap v148: 97 pgs: 97 active+clean; 35 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 696 MiB/s wr, 5.18k op/s 2024-05-20T18:03:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.206+0000 7ff7d880c640 1 -- 172.21.15.129:0/2213892428 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d410e210 msgr2=0x7ff7d41090b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.206+0000 7ff7d880c640 1 --2- 172.21.15.129:0/2213892428 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d410e210 0x7ff7d41090b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8007380 tx=0x7ff7c802aff0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.207+0000 7ff7d880c640 1 -- 172.21.15.129:0/2213892428 shutdown_connections 2024-05-20T18:03:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.207+0000 7ff7d880c640 1 --2- 172.21.15.129:0/2213892428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7d4109740 0x7ff7d4079e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.207+0000 7ff7d880c640 1 --2- 172.21.15.129:0/2213892428 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d410e210 0x7ff7d41090b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.207+0000 7ff7d880c640 1 --2- 172.21.15.129:0/2213892428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7d410de00 0x7ff7d4108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.207+0000 7ff7d880c640 1 -- 172.21.15.129:0/2213892428 >> 172.21.15.129:0/2213892428 conn(0x7ff7d406d580 msgr2=0x7ff7d406d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:22.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.207+0000 7ff7d880c640 1 -- 172.21.15.129:0/2213892428 shutdown_connections 2024-05-20T18:03:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.207+0000 7ff7d880c640 1 -- 172.21.15.129:0/2213892428 wait complete. 2024-05-20T18:03:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.211+0000 7ff7d880c640 1 Processor -- start 2024-05-20T18:03:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.212+0000 7ff7d880c640 1 -- start start 2024-05-20T18:03:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.213+0000 7ff7d880c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7d410de00 0x7ff7d407ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.213+0000 7ff7d880c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d410e210 0x7ff7d407f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.213+0000 7ff7d880c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7d407f800 0x7ff7d41bc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.214+0000 7ff7d880c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d407d680 con 0x7ff7d407f800 2024-05-20T18:03:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.214+0000 7ff7d880c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d407d500 con 0x7ff7d410de00 2024-05-20T18:03:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.214+0000 7ff7d259c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d410e210 0x7ff7d407f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.214+0000 7ff7d880c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d407d800 con 0x7ff7d410e210 2024-05-20T18:03:22.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.214+0000 7ff7d259c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d410e210 0x7ff7d407f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50164/0 (socket says 172.21.15.129:50164) 2024-05-20T18:03:22.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.214+0000 7ff7d259c640 1 -- 172.21.15.129:0/208882382 learned_addr learned my addr 172.21.15.129:0/208882382 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:22.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.217+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7d410de00 0x7ff7d407ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.217+0000 7ff7d359e640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7d407f800 0x7ff7d41bc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.218+0000 7ff7d2d9d640 1 -- 172.21.15.129:0/208882382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d410e210 msgr2=0x7ff7d407f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:22.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.218+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d410e210 0x7ff7d407f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.218+0000 7ff7d2d9d640 1 -- 172.21.15.129:0/208882382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7d407f800 msgr2=0x7ff7d41bc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:22.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.218+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7d407f800 0x7ff7d41bc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.218+0000 7ff7d2d9d640 1 -- 172.21.15.129:0/208882382 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d41bcd70 con 0x7ff7d410de00 2024-05-20T18:03:22.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.219+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7d410de00 0x7ff7d407ed80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400d6d0 tx=0x7ff7c400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:22.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.225+0000 7ff7bbfff640 1 -- 172.21.15.129:0/208882382 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4017070 con 0x7ff7d410de00 2024-05-20T18:03:22.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.225+0000 7ff7bbfff640 1 -- 172.21.15.129:0/208882382 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c4009820 con 0x7ff7d410de00 2024-05-20T18:03:22.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.227+0000 7ff7bbfff640 1 -- 172.21.15.129:0/208882382 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4004ce0 con 0x7ff7d410de00 2024-05-20T18:03:22.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.227+0000 7ff7d880c640 1 -- 172.21.15.129:0/208882382 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d41bd060 con 0x7ff7d410de00 2024-05-20T18:03:22.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.227+0000 7ff7d880c640 1 -- 172.21.15.129:0/208882382 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d41bd530 con 0x7ff7d410de00 2024-05-20T18:03:22.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.242+0000 7ff7dff8d740 1 Processor -- start 2024-05-20T18:03:22.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.242+0000 7ff7bbfff640 1 -- 172.21.15.129:0/208882382 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ff7c4005220 con 0x7ff7d410de00 2024-05-20T18:03:22.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.243+0000 7ff7dff8d740 1 -- start start 2024-05-20T18:03:22.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.244+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816ddc390 0x55e81717af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.244+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 0x55e81717b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.244+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816dbb6b0 0x55e81717b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.244+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e8170e0d00 con 0x55e816ddc390 2024-05-20T18:03:22.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.244+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e8170884c0 con 0x55e816db34d0 2024-05-20T18:03:22.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.244+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e8170dccc0 con 0x55e816dbb6b0 2024-05-20T18:03:22.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.246+0000 7ff7bbfff640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff7bc0a2d10 0x7ff7bc0a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.246+0000 7ff7d2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 0x55e81717b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.247+0000 7ff7d359e640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff7bc0a2d10 0x7ff7bc0a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.247+0000 7ff7bbfff640 1 -- 172.21.15.129:0/208882382 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7ff7c40cec20 con 0x7ff7d410de00 2024-05-20T18:03:22.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.248+0000 7ff7d2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 0x55e81717b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:33932/0 (socket says 172.21.15.129:33932) 2024-05-20T18:03:22.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.249+0000 7ff7d2d9d640 1 -- 172.21.15.129:0/2176684334 learned_addr learned my addr 172.21.15.129:0/2176684334 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:22.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.253+0000 7ff7d259c640 1 --2- 172.21.15.129:0/2176684334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816dbb6b0 0x55e81717b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.254+0000 7ff7d2d9d640 1 -- 172.21.15.129:0/2176684334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816dbb6b0 msgr2=0x55e81717b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:22.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.254+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/2176684334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816dbb6b0 0x55e81717b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.254+0000 7ff7d2d9d640 1 -- 172.21.15.129:0/2176684334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816ddc390 msgr2=0x55e81717af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:22.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.254+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/2176684334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816ddc390 0x55e81717af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.254+0000 7ff7d2d9d640 1 -- 172.21.15.129:0/2176684334 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e816ebe7d0 con 0x55e816db34d0 2024-05-20T18:03:22.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.259+0000 7ff7d359e640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff7bc0a2d10 0x7ff7bc0a5220 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc005c40 tx=0x7ff7cc009570 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:22.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.260+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/2176684334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 0x55e81717b470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009610 tx=0x7ff7c409ac90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9645c9f62687a8cb server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:22.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.271+0000 7ff7b97fa640 1 -- 172.21.15.129:0/2176684334 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c409b150 con 0x55e816db34d0 2024-05-20T18:03:22.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.271+0000 7ff7b97fa640 1 -- 172.21.15.129:0/2176684334 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c409f500 con 0x55e816db34d0 2024-05-20T18:03:22.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.273+0000 7ff7dff8d740 1 -- 172.21.15.129:0/2176684334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 msgr2=0x55e81717b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:22.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.273+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/2176684334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 0x55e81717b470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009610 tx=0x7ff7c409ac90 comp rx=0 tx=0).stop 2024-05-20T18:03:22.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.286+0000 7ff7dff8d740 1 -- 172.21.15.129:0/2176684334 shutdown_connections 2024-05-20T18:03:22.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.286+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/2176684334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 0x55e81717b470 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.286+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/2176684334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816dbb6b0 0x55e81717b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.286+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/2176684334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816ddc390 0x55e81717af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.286+0000 7ff7dff8d740 1 -- 172.21.15.129:0/2176684334 >> 172.21.15.129:0/2176684334 conn(0x55e816dd04e0 msgr2=0x55e8171776e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:22.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.293+0000 7ff7dff8d740 1 -- 172.21.15.129:0/2176684334 shutdown_connections 2024-05-20T18:03:22.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.300+0000 7ff7dff8d740 1 -- 172.21.15.129:0/2176684334 wait complete. 2024-05-20T18:03:22.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.300+0000 7ff7dff8d740 1 -- start start 2024-05-20T18:03:22.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.301+0000 7ff7dff8d740 1 Processor -- start 2024-05-20T18:03:22.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.301+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 0x55e817185540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.301+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816dbb6b0 0x55e817185a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.301+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816ddc390 0x55e817191820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.301+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e817188320 con 0x55e816dbb6b0 2024-05-20T18:03:22.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.301+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e81709bb90 con 0x55e816db34d0 2024-05-20T18:03:22.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.301+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e8171884a0 con 0x55e816ddc390 2024-05-20T18:03:22.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.302+0000 7ff7dff8d740 10 client.0 _ll_register_callbacks cb 0x55e8170d7da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:22.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.303+0000 7ff7d359e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816ddc390 0x55e817191820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.303+0000 7ff7d2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 0x55e817185540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.303+0000 7ff7d2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 0x55e817185540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:33942/0 (socket says 172.21.15.129:33942) 2024-05-20T18:03:22.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.303+0000 7ff7d2d9d640 1 -- 172.21.15.129:0/375533665 learned_addr learned my addr 172.21.15.129:0/375533665 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:22.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.303+0000 7ff7d359e640 1 -- 172.21.15.129:0/375533665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 msgr2=0x55e817185540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:22.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.303+0000 7ff7d359e640 1 --2- 172.21.15.129:0/375533665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 0x55e817185540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.303+0000 7ff7d359e640 1 -- 172.21.15.129:0/375533665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816dbb6b0 msgr2=0x55e817185a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:03:22.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.304+0000 7ff7d359e640 1 --2- 172.21.15.129:0/375533665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816dbb6b0 0x55e817185a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.304+0000 7ff7d359e640 1 -- 172.21.15.129:0/375533665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e816df43f0 con 0x55e816ddc390 2024-05-20T18:03:22.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.304+0000 7ff7d359e640 1 --2- 172.21.15.129:0/375533665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816ddc390 0x55e817191820 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc005fd0 tx=0x7ff7cc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d0473ceca8cd6aa server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:22.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.318+0000 7ff7b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:03:22.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.318+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/375533665 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc01ad90 con 0x55e816ddc390 2024-05-20T18:03:22.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.322+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/375533665 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7ff7cc016050 con 0x55e816ddc390 2024-05-20T18:03:22.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.323+0000 7ff7dff8d740 1 -- 172.21.15.129:0/375533665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c409f500 con 0x55e816ddc390 2024-05-20T18:03:22.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.332+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/375533665 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff7cc00cd80 con 0x55e816ddc390 2024-05-20T18:03:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.339+0000 7ff7dff8d740 10 client.25876 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.339+0000 7ff7dff8d740 1 -- 172.21.15.129:0/375533665 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e816db40f0 con 0x55e816ddc390 2024-05-20T18:03:22.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.346+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/375533665 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c409f500 con 0x55e816ddc390 2024-05-20T18:03:22.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.346+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/375533665 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7ff7cc018070 con 0x55e816ddc390 2024-05-20T18:03:22.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.347+0000 7ff7dff8d740 10 client.25876 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:22.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.347+0000 7ff7dff8d740 10 client.25876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:22.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.347+0000 7ff7dff8d740 20 client.25876 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:22.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.347+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/375533665 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e816d9af60 0x55e8171b9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.347+0000 7ff7dff8d740 4 client.25876 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:03:22.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.347+0000 7ff7dff8d740 1 -- 172.21.15.129:0/375533665 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8170ee430 con 0x55e816d9af60 2024-05-20T18:03:22.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.349+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/375533665 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e816d9af60 0x55e8171b9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.357+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/375533665 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e816d9af60 0x55e8171b9d70 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb44923238a5edf9 server_cookie=3c456789f00c4d03 in_seq=0 out_seq=0 2024-05-20T18:03:22.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.362+0000 7ff7b6ff5640 10 client.25876 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:22.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.362+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/375533665 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e8170ee430 con 0x55e816d9af60 2024-05-20T18:03:22.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.362+0000 7ff7b6ff5640 10 client.25876 handle_command_reply: tid=0 2024-05-20T18:03:22.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.362+0000 7ff7dff8d740 1 client.25876 shutdown 2024-05-20T18:03:22.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.362+0000 7ff7dff8d740 1 -- 172.21.15.129:0/375533665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816ddc390 msgr2=0x55e817191820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:22.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.362+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/375533665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816ddc390 0x55e817191820 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc005fd0 tx=0x7ff7cc004ab0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.363+0000 7ff7dff8d740 1 -- 172.21.15.129:0/375533665 shutdown_connections 2024-05-20T18:03:22.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.363+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/375533665 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e816d9af60 0x55e8171b9d70 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.363+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/375533665 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816ddc390 0x55e817191820 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.363+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/375533665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816dbb6b0 0x55e817185a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.363+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/375533665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816db34d0 0x55e817185540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.363+0000 7ff7dff8d740 1 -- 172.21.15.129:0/375533665 >> 172.21.15.129:0/375533665 conn(0x55e816dd04e0 msgr2=0x55e81717cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.367+0000 7ff7b6ff5640 0 client.25876 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:22.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.371+0000 7ff7dff8d740 1 -- 172.21.15.129:0/375533665 shutdown_connections 2024-05-20T18:03:22.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.371+0000 7ff7dff8d740 1 -- 172.21.15.129:0/375533665 wait complete. 2024-05-20T18:03:22.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.372+0000 7ff7dff8d740 20 client.25876 trim_cache size 0 max 16384 2024-05-20T18:03:22.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.444+0000 7ff7dff8d740 1 Processor -- start 2024-05-20T18:03:22.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.444+0000 7ff7dff8d740 1 -- start start 2024-05-20T18:03:22.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.445+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816db34d0 0x55e817184270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.445+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816dbb6b0 0x55e817180510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.445+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816ddc390 0x55e81717d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.445+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c3d80 con 0x55e816dbb6b0 2024-05-20T18:03:22.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.445+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c6600 con 0x55e816ddc390 2024-05-20T18:03:22.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.445+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c43f0 con 0x55e816db34d0 2024-05-20T18:03:22.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.446+0000 7ff7d259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816dbb6b0 0x55e817180510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.446+0000 7ff7d259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816dbb6b0 0x55e817180510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55666/0 (socket says 172.21.15.129:55666) 2024-05-20T18:03:22.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.446+0000 7ff7d259c640 1 -- 172.21.15.129:0/818950313 learned_addr learned my addr 172.21.15.129:0/818950313 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:22.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.446+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/818950313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816db34d0 0x55e817184270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.447+0000 7ff7d2d9d640 1 -- 172.21.15.129:0/818950313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816ddc390 msgr2=0x55e81717d0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:03:22.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.447+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/818950313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816ddc390 0x55e81717d0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.447+0000 7ff7d2d9d640 1 -- 172.21.15.129:0/818950313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816dbb6b0 msgr2=0x55e817180510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:22.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.447+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/818950313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816dbb6b0 0x55e817180510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.447+0000 7ff7d2d9d640 1 -- 172.21.15.129:0/818950313 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e816d9b190 con 0x55e816db34d0 2024-05-20T18:03:22.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.448+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/818950313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816db34d0 0x55e817184270 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40975b0 tx=0x7ff7c4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dd8ba1847e11752 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:22.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.450+0000 7ff7b8ff9640 1 -- 172.21.15.129:0/818950313 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c409f500 con 0x55e816db34d0 2024-05-20T18:03:22.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.451+0000 7ff7b8ff9640 1 -- 172.21.15.129:0/818950313 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c409b150 con 0x55e816db34d0 2024-05-20T18:03:22.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.451+0000 7ff7b8ff9640 1 -- 172.21.15.129:0/818950313 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c40a3930 con 0x55e816db34d0 2024-05-20T18:03:22.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.453+0000 7ff7dff8d740 1 -- 172.21.15.129:0/818950313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816db34d0 msgr2=0x55e817184270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:22.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.453+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/818950313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816db34d0 0x55e817184270 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40975b0 tx=0x7ff7c4000bc0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.454+0000 7ff7dff8d740 1 -- 172.21.15.129:0/818950313 shutdown_connections 2024-05-20T18:03:22.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.454+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/818950313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816ddc390 0x55e81717d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.454+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/818950313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816dbb6b0 0x55e817180510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.454+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/818950313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816db34d0 0x55e817184270 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.454+0000 7ff7dff8d740 1 -- 172.21.15.129:0/818950313 >> 172.21.15.129:0/818950313 conn(0x55e816dd04e0 msgr2=0x55e81717c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:22.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.461+0000 7ff7dff8d740 1 -- 172.21.15.129:0/818950313 shutdown_connections 2024-05-20T18:03:22.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.463+0000 7ff7dff8d740 1 -- 172.21.15.129:0/818950313 wait complete. 2024-05-20T18:03:22.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.464+0000 7ff7dff8d740 1 -- start start 2024-05-20T18:03:22.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.464+0000 7ff7dff8d740 1 Processor -- start 2024-05-20T18:03:22.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.467+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816db34d0 0x55e81718dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.467+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816dbb6b0 0x55e817189040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.467+0000 7ff7dff8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816ddc390 0x55e817189580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.467+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c4ac0 con 0x55e816db34d0 2024-05-20T18:03:22.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.467+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c5f30 con 0x55e816ddc390 2024-05-20T18:03:22.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.467+0000 7ff7dff8d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c5190 con 0x55e816dbb6b0 2024-05-20T18:03:22.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.467+0000 7ff7dff8d740 10 client.0 _ll_register_callbacks cb 0x7ff7cc01a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:22.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.468+0000 7ff7d359e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816ddc390 0x55e817189580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.468+0000 7ff7d359e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816ddc390 0x55e817189580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:33954/0 (socket says 172.21.15.129:33954) 2024-05-20T18:03:22.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.468+0000 7ff7d359e640 1 -- 172.21.15.129:0/1154516845 learned_addr learned my addr 172.21.15.129:0/1154516845 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:22.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.469+0000 7ff7d259c640 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816dbb6b0 0x55e817189040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.470+0000 7ff7d259c640 1 -- 172.21.15.129:0/1154516845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816ddc390 msgr2=0x55e817189580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:22.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.470+0000 7ff7d259c640 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816ddc390 0x55e817189580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.470+0000 7ff7d259c640 1 -- 172.21.15.129:0/1154516845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816db34d0 msgr2=0x55e81718dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:22.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.470+0000 7ff7d259c640 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816db34d0 0x55e81718dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.470+0000 7ff7d259c640 1 -- 172.21.15.129:0/1154516845 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e817189cc0 con 0x55e816dbb6b0 2024-05-20T18:03:22.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.471+0000 7ff7d259c640 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816dbb6b0 0x55e817189040 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8007350 tx=0x7ff7c8033a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15ab4dcb6c53f734 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:22.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.471+0000 7ff7d359e640 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816ddc390 0x55e817189580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:03:22.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.471+0000 7ff7b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:03:22.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.471+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/1154516845 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8004240 con 0x55e816dbb6b0 2024-05-20T18:03:22.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.471+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/1154516845 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7ff7c80043e0 con 0x55e816dbb6b0 2024-05-20T18:03:22.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.472+0000 7ff7dff8d740 1 -- 172.21.15.129:0/1154516845 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c409b150 con 0x55e816dbb6b0 2024-05-20T18:03:22.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.472+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/1154516845 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff7c802b930 con 0x55e816dbb6b0 2024-05-20T18:03:22.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.474+0000 7ff7dff8d740 10 client.25882 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:22.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.474+0000 7ff7dff8d740 1 -- 172.21.15.129:0/1154516845 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e817221ff0 con 0x55e816dbb6b0 2024-05-20T18:03:22.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.477+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/1154516845 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c409b150 con 0x55e816dbb6b0 2024-05-20T18:03:22.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.477+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/1154516845 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7ff7c8004790 con 0x55e816dbb6b0 2024-05-20T18:03:22.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.478+0000 7ff7dff8d740 10 client.25882 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:22.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.478+0000 7ff7dff8d740 10 client.25882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:22.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.478+0000 7ff7dff8d740 20 client.25882 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:22.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.478+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e816d703a0 0x55e8172247e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:22.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.478+0000 7ff7dff8d740 4 client.25882 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:03:22.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.478+0000 7ff7dff8d740 1 -- 172.21.15.129:0/1154516845 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e8170e8780 con 0x55e816d703a0 2024-05-20T18:03:22.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.480+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e816d703a0 0x55e8172247e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:22.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.483+0000 7ff7d2d9d640 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e816d703a0 0x55e8172247e0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10b4ac8e48871f7b server_cookie=d5fa398b65ca70cd in_seq=0 out_seq=0 2024-05-20T18:03:22.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.484+0000 7ff7b6ff5640 10 client.25882 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.488+0000 7ff7b6ff5640 1 -- 172.21.15.129:0/1154516845 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e8170e8780 con 0x55e816d703a0 2024-05-20T18:03:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.488+0000 7ff7b6ff5640 10 client.25882 handle_command_reply: tid=0 2024-05-20T18:03:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.488+0000 7ff7dff8d740 1 client.25882 shutdown 2024-05-20T18:03:22.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.490+0000 7ff7dff8d740 1 -- 172.21.15.129:0/1154516845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816dbb6b0 msgr2=0x55e817189040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:22.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.490+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816dbb6b0 0x55e817189040 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8007350 tx=0x7ff7c8033a60 comp rx=0 tx=0).stop 2024-05-20T18:03:22.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.495+0000 7ff7dff8d740 1 -- 172.21.15.129:0/1154516845 shutdown_connections 2024-05-20T18:03:22.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.495+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e816d703a0 0x55e8172247e0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.495+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e816ddc390 0x55e817189580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.495+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e816dbb6b0 0x55e817189040 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.495+0000 7ff7dff8d740 1 --2- 172.21.15.129:0/1154516845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e816db34d0 0x55e81718dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.495+0000 7ff7dff8d740 1 -- 172.21.15.129:0/1154516845 >> 172.21.15.129:0/1154516845 conn(0x55e816dd04e0 msgr2=0x55e81717b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:22.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.496+0000 7ff7b6ff5640 0 client.25882 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:22.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.496+0000 7ff7dff8d740 1 -- 172.21.15.129:0/1154516845 shutdown_connections 2024-05-20T18:03:22.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.496+0000 7ff7dff8d740 1 -- 172.21.15.129:0/1154516845 wait complete. 2024-05-20T18:03:22.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.500+0000 7ff7dff8d740 20 client.25882 trim_cache size 0 max 16384 2024-05-20T18:03:22.507 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:03:22.507 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:03:22.507 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:03:22.507 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:03:22.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.506+0000 7ff7d880c640 1 -- 172.21.15.129:0/208882382 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff7bc0a2d10 msgr2=0x7ff7bc0a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:22.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.506+0000 7ff7d880c640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff7bc0a2d10 0x7ff7bc0a5220 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc005c40 tx=0x7ff7cc009570 comp rx=0 tx=0).stop 2024-05-20T18:03:22.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.506+0000 7ff7d880c640 1 -- 172.21.15.129:0/208882382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7d410de00 msgr2=0x7ff7d407ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:22.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.506+0000 7ff7d880c640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7d410de00 0x7ff7d407ed80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400d6d0 tx=0x7ff7c400db90 comp rx=0 tx=0).stop 2024-05-20T18:03:22.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.506+0000 7ff7d880c640 1 -- 172.21.15.129:0/208882382 shutdown_connections 2024-05-20T18:03:22.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.506+0000 7ff7d880c640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff7bc0a2d10 0x7ff7bc0a5220 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.506+0000 7ff7d880c640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7d407f800 0x7ff7d41bc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.506+0000 7ff7d880c640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d410e210 0x7ff7d407f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.506+0000 7ff7d880c640 1 --2- 172.21.15.129:0/208882382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7d410de00 0x7ff7d407ed80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:22.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.506+0000 7ff7d880c640 1 -- 172.21.15.129:0/208882382 >> 172.21.15.129:0/208882382 conn(0x7ff7d406d580 msgr2=0x7ff7d40728c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:22.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.507+0000 7ff7d880c640 1 -- 172.21.15.129:0/208882382 shutdown_connections 2024-05-20T18:03:22.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:22.507+0000 7ff7d880c640 1 -- 172.21.15.129:0/208882382 wait complete. 2024-05-20T18:03:22.607 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:03:22.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:03:23.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.197+0000 7f20257fb640 1 -- 172.21.15.129:0/2282257464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f202010f270 msgr2=0x7f2020108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:23.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.197+0000 7f20257fb640 1 --2- 172.21.15.129:0/2282257464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f202010f270 0x7f2020108b50 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2010009900 tx=0x7f201002a440 comp rx=0 tx=0).stop 2024-05-20T18:03:23.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.198+0000 7f20257fb640 1 -- 172.21.15.129:0/2282257464 shutdown_connections 2024-05-20T18:03:23.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.198+0000 7f20257fb640 1 --2- 172.21.15.129:0/2282257464 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2020109090 0x7f2020109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:23.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.198+0000 7f20257fb640 1 --2- 172.21.15.129:0/2282257464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f202010f270 0x7f2020108b50 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:23.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.198+0000 7f20257fb640 1 --2- 172.21.15.129:0/2282257464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f202010e890 0x7f202010eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:23.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.198+0000 7f20257fb640 1 -- 172.21.15.129:0/2282257464 >> 172.21.15.129:0/2282257464 conn(0x7f202006d580 msgr2=0x7f202006d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:23.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.198+0000 7f20257fb640 1 -- 172.21.15.129:0/2282257464 shutdown_connections 2024-05-20T18:03:23.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.198+0000 7f20257fb640 1 -- 172.21.15.129:0/2282257464 wait complete. 2024-05-20T18:03:23.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.200+0000 7f20257fb640 1 Processor -- start 2024-05-20T18:03:23.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.201+0000 7f20257fb640 1 -- start start 2024-05-20T18:03:23.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.202+0000 7f20257fb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2020109090 0x7f202011f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:23.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.202+0000 7f20257fb640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f202010e890 0x7f20201182a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:23.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.202+0000 7f20257fb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f202010f270 0x7f20201188d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:23.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.202+0000 7f20257fb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2020121a90 con 0x7f202010f270 2024-05-20T18:03:23.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.202+0000 7f20257fb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2020121910 con 0x7f2020109090 2024-05-20T18:03:23.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.203+0000 7f20257fb640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2020121c10 con 0x7f202010e890 2024-05-20T18:03:23.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.205+0000 7f201f7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f202010e890 0x7f20201182a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:23.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.205+0000 7f201f7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f202010e890 0x7f20201182a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50210/0 (socket says 172.21.15.129:50210) 2024-05-20T18:03:23.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.205+0000 7f201f7fe640 1 -- 172.21.15.129:0/1946755793 learned_addr learned my addr 172.21.15.129:0/1946755793 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:23.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.205+0000 7f2024ffa640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f202010f270 0x7f20201188d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:23.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.206+0000 7f201ffff640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2020109090 0x7f202011f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:23.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.206+0000 7f201f7fe640 1 -- 172.21.15.129:0/1946755793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2020109090 msgr2=0x7f202011f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:23.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.206+0000 7f201f7fe640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2020109090 0x7f202011f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:23.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.206+0000 7f201f7fe640 1 -- 172.21.15.129:0/1946755793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f202010f270 msgr2=0x7f20201188d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:23.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.206+0000 7f201f7fe640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f202010f270 0x7f20201188d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:23.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.206+0000 7f201f7fe640 1 -- 172.21.15.129:0/1946755793 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202011cca0 con 0x7f202010e890 2024-05-20T18:03:23.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.211+0000 7f201f7fe640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f202010e890 0x7f20201182a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2010005ab0 tx=0x7f2010005230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:23.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.213+0000 7f201d7fa640 1 -- 172.21.15.129:0/1946755793 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2010040070 con 0x7f202010e890 2024-05-20T18:03:23.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.213+0000 7f201d7fa640 1 -- 172.21.15.129:0/1946755793 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2010004a30 con 0x7f202010e890 2024-05-20T18:03:23.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.214+0000 7f201d7fa640 1 -- 172.21.15.129:0/1946755793 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201003ca80 con 0x7f202010e890 2024-05-20T18:03:23.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.217+0000 7f20257fb640 1 -- 172.21.15.129:0/1946755793 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202011cf30 con 0x7f202010e890 2024-05-20T18:03:23.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.217+0000 7f20257fb640 1 -- 172.21.15.129:0/1946755793 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f202011d3d0 con 0x7f202010e890 2024-05-20T18:03:23.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.220+0000 7f20257fb640 1 -- 172.21.15.129:0/1946755793 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f202010a1e0 con 0x7f202010e890 2024-05-20T18:03:23.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.229+0000 7f201d7fa640 1 -- 172.21.15.129:0/1946755793 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f201003a040 con 0x7f202010e890 2024-05-20T18:03:23.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.231+0000 7f201d7fa640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20080a2f00 0x7f20080a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:23.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.231+0000 7f201d7fa640 1 -- 172.21.15.129:0/1946755793 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f20100f04f0 con 0x7f202010e890 2024-05-20T18:03:23.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.234+0000 7f201d7fa640 1 -- 172.21.15.129:0/1946755793 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2010049070 con 0x7f202010e890 2024-05-20T18:03:23.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.235+0000 7f201ffff640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20080a2f00 0x7f20080a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:23.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.244+0000 7f201ffff640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20080a2f00 0x7f20080a5410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2020119af0 tx=0x7f2014005c90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:23.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:23 smithi204 ceph-mon[37076]: pgmap v149: 97 pgs: 97 active+clean; 35 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 729 MiB/s wr, 5.33k op/s 2024-05-20T18:03:23.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:23 smithi154 ceph-mon[36482]: pgmap v149: 97 pgs: 97 active+clean; 35 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 729 MiB/s wr, 5.33k op/s 2024-05-20T18:03:23.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:23 smithi129 ceph-mon[32029]: pgmap v149: 97 pgs: 97 active+clean; 35 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 729 MiB/s wr, 5.33k op/s 2024-05-20T18:03:23.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.955+0000 7f20257fb640 1 -- 172.21.15.129:0/1946755793 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20201190c0 con 0x7f202010e890 2024-05-20T18:03:23.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.958+0000 7f201d7fa640 1 -- 172.21.15.129:0/1946755793 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11829 (secure 0 0 0) 0x7f20100bd120 con 0x7f202010e890 2024-05-20T18:03:23.956 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:03:23.956 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:03:18.451223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/4068528573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":4068528573},{"type":"v1","addr":"172.21.15.154:6837","nonce":4068528573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:03:23.956 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 24 2024-05-20T18:03:23.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.966+0000 7f20257fb640 1 -- 172.21.15.129:0/1946755793 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20080a2f00 msgr2=0x7f20080a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:23.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.966+0000 7f20257fb640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20080a2f00 0x7f20080a5410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2020119af0 tx=0x7f2014005c90 comp rx=0 tx=0).stop 2024-05-20T18:03:23.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.966+0000 7f20257fb640 1 -- 172.21.15.129:0/1946755793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f202010e890 msgr2=0x7f20201182a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:23.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.966+0000 7f20257fb640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f202010e890 0x7f20201182a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2010005ab0 tx=0x7f2010005230 comp rx=0 tx=0).stop 2024-05-20T18:03:23.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.966+0000 7f20257fb640 1 -- 172.21.15.129:0/1946755793 shutdown_connections 2024-05-20T18:03:23.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.966+0000 7f20257fb640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20080a2f00 0x7f20080a5410 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2020119af0 tx=0x7f2014005c90 comp rx=0 tx=0).stop 2024-05-20T18:03:23.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.966+0000 7f20257fb640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f202010f270 0x7f20201188d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:23.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.966+0000 7f20257fb640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f202010e890 0x7f20201182a0 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2010005ab0 tx=0x7f2010005230 comp rx=0 tx=0).stop 2024-05-20T18:03:23.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.966+0000 7f20257fb640 1 --2- 172.21.15.129:0/1946755793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2020109090 0x7f202011f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:23.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.966+0000 7f20257fb640 1 -- 172.21.15.129:0/1946755793 >> 172.21.15.129:0/1946755793 conn(0x7f202006d580 msgr2=0x7f20200728c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.968+0000 7f20257fb640 1 -- 172.21.15.129:0/1946755793 shutdown_connections 2024-05-20T18:03:23.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:23.970+0000 7f20257fb640 1 -- 172.21.15.129:0/1946755793 wait complete. 2024-05-20T18:03:24.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:03:24.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.530+0000 7f443214b640 1 -- 172.21.15.129:0/4072535314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f442c109060 msgr2=0x7f442c109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:24.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.530+0000 7f443214b640 1 --2- 172.21.15.129:0/4072535314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f442c109060 0x7f442c109470 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4420009900 tx=0x7f442002a480 comp rx=0 tx=0).stop 2024-05-20T18:03:24.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.534+0000 7f443214b640 1 -- 172.21.15.129:0/4072535314 shutdown_connections 2024-05-20T18:03:24.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.534+0000 7f443214b640 1 --2- 172.21.15.129:0/4072535314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f442c119770 0x7f442c11bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.534+0000 7f443214b640 1 --2- 172.21.15.129:0/4072535314 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f442c11c0d0 0x7f442c11e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.534+0000 7f443214b640 1 --2- 172.21.15.129:0/4072535314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f442c109060 0x7f442c109470 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.534+0000 7f443214b640 1 -- 172.21.15.129:0/4072535314 >> 172.21.15.129:0/4072535314 conn(0x7f442c06d750 msgr2=0x7f442c06dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:24.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.540+0000 7f443214b640 1 -- 172.21.15.129:0/4072535314 shutdown_connections 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.541+0000 7f443214b640 1 -- 172.21.15.129:0/4072535314 wait complete. 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.542+0000 7f443214b640 1 Processor -- start 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.543+0000 7f443214b640 1 -- start start 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.543+0000 7f443214b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f442c109060 0x7f442c1133e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.543+0000 7f443214b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f442c119770 0x7f442c113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.543+0000 7f443214b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f442c11c0d0 0x7f442c113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.543+0000 7f443214b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f442c111970 con 0x7f442c119770 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.543+0000 7f443214b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f442c1117f0 con 0x7f442c11c0d0 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.543+0000 7f443214b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f442c111af0 con 0x7f442c109060 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.543+0000 7f442affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f442c119770 0x7f442c113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.543+0000 7f442affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f442c119770 0x7f442c113920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55712/0 (socket says 172.21.15.129:55712) 2024-05-20T18:03:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.544+0000 7f442affd640 1 -- 172.21.15.129:0/1710220673 learned_addr learned my addr 172.21.15.129:0/1710220673 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:24.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.544+0000 7f442affd640 1 -- 172.21.15.129:0/1710220673 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f442c109060 msgr2=0x7f442c1133e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:03:24.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.544+0000 7f442affd640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f442c109060 0x7f442c1133e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.544+0000 7f442affd640 1 -- 172.21.15.129:0/1710220673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f442c11c0d0 msgr2=0x7f442c113e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:03:24.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.544+0000 7f442affd640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f442c11c0d0 0x7f442c113e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.544+0000 7f442affd640 1 -- 172.21.15.129:0/1710220673 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442c1bcf60 con 0x7f442c119770 2024-05-20T18:03:24.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.545+0000 7f442affd640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f442c119770 0x7f442c113920 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f442400cce0 tx=0x7f442400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:24.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.547+0000 7f4428ff9640 1 -- 172.21.15.129:0/1710220673 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424004b20 con 0x7f442c119770 2024-05-20T18:03:24.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.547+0000 7f443214b640 1 -- 172.21.15.129:0/1710220673 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f442c1bd1f0 con 0x7f442c119770 2024-05-20T18:03:24.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.547+0000 7f443214b640 1 -- 172.21.15.129:0/1710220673 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f442c1bd740 con 0x7f442c119770 2024-05-20T18:03:24.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.548+0000 7f4428ff9640 1 -- 172.21.15.129:0/1710220673 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4424004cc0 con 0x7f442c119770 2024-05-20T18:03:24.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.550+0000 7f4428ff9640 1 -- 172.21.15.129:0/1710220673 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442400f8c0 con 0x7f442c119770 2024-05-20T18:03:24.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.552+0000 7f4438ff1740 1 Processor -- start 2024-05-20T18:03:24.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.553+0000 7f4438ff1740 1 -- start start 2024-05-20T18:03:24.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.553+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a0886ebd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.553+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0883244d0 0x55a0886ec260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.553+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a08832c6b0 0x55a0886ec7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.553+0000 7f4438ff1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a088651c40 con 0x55a0883244d0 2024-05-20T18:03:24.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.553+0000 7f4438ff1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a0885f9580 con 0x55a08832c6b0 2024-05-20T18:03:24.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.553+0000 7f4438ff1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a08864dc00 con 0x55a08834d390 2024-05-20T18:03:24.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.554+0000 7f442bfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a0886ebd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:24.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.554+0000 7f442bfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a0886ebd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50248/0 (socket says 172.21.15.129:50248) 2024-05-20T18:03:24.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.554+0000 7f442bfff640 1 -- 172.21.15.129:0/425513350 learned_addr learned my addr 172.21.15.129:0/425513350 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:24.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.554+0000 7f442bfff640 1 -- 172.21.15.129:0/425513350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a08832c6b0 msgr2=0x55a0886ec7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:03:24.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.554+0000 7f442bfff640 1 --2- 172.21.15.129:0/425513350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a08832c6b0 0x55a0886ec7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.554+0000 7f442bfff640 1 -- 172.21.15.129:0/425513350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0883244d0 msgr2=0x55a0886ec260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:03:24.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.555+0000 7f442bfff640 1 --2- 172.21.15.129:0/425513350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0883244d0 0x55a0886ec260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.555+0000 7f442bfff640 1 -- 172.21.15.129:0/425513350 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0883653f0 con 0x55a08834d390 2024-05-20T18:03:24.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.557+0000 7f4428ff9640 1 -- 172.21.15.129:0/1710220673 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f442400fa60 con 0x7f442c119770 2024-05-20T18:03:24.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.559+0000 7f4428ff9640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44140a2f00 0x7f44140a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.559+0000 7f4428ff9640 1 -- 172.21.15.129:0/1710220673 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f44240cf520 con 0x7f442c119770 2024-05-20T18:03:24.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.563+0000 7f442b7fe640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44140a2f00 0x7f44140a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:24.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.567+0000 7f442bfff640 1 --2- 172.21.15.129:0/425513350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a0886ebd20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f441c00b980 tx=0x7f441c00be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80fc5fbfedf4fbb4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:24.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.568+0000 7f4411ffb640 1 -- 172.21.15.129:0/425513350 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c016070 con 0x55a08834d390 2024-05-20T18:03:24.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.568+0000 7f4411ffb640 1 -- 172.21.15.129:0/425513350 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441c005500 con 0x55a08834d390 2024-05-20T18:03:24.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.570+0000 7f4438ff1740 1 -- 172.21.15.129:0/425513350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 msgr2=0x55a0886ebd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:24.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.570+0000 7f4438ff1740 1 --2- 172.21.15.129:0/425513350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a0886ebd20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f441c00b980 tx=0x7f441c00be40 comp rx=0 tx=0).stop 2024-05-20T18:03:24.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.570+0000 7f4438ff1740 1 -- 172.21.15.129:0/425513350 shutdown_connections 2024-05-20T18:03:24.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.570+0000 7f4438ff1740 1 --2- 172.21.15.129:0/425513350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a08832c6b0 0x55a0886ec7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.570+0000 7f4438ff1740 1 --2- 172.21.15.129:0/425513350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0883244d0 0x55a0886ec260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.570+0000 7f4438ff1740 1 --2- 172.21.15.129:0/425513350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a0886ebd20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.570+0000 7f4438ff1740 1 -- 172.21.15.129:0/425513350 >> 172.21.15.129:0/425513350 conn(0x55a0883414e0 msgr2=0x55a0886e84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:24.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.575+0000 7f4438ff1740 1 -- 172.21.15.129:0/425513350 shutdown_connections 2024-05-20T18:03:24.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.575+0000 7f442b7fe640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44140a2f00 0x7f44140a5410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4420009230 tx=0x7f4420005fd0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:24.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.579+0000 7f4438ff1740 1 -- 172.21.15.129:0/425513350 wait complete. 2024-05-20T18:03:24.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.580+0000 7f4438ff1740 1 -- start start 2024-05-20T18:03:24.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.580+0000 7f4438ff1740 1 Processor -- start 2024-05-20T18:03:24.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.582+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a0883244d0 0x55a0886f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.582+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a08832c6b0 0x55a0886f67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.582+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a088702610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.582+0000 7f4438ff1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a0886f90c0 con 0x55a08832c6b0 2024-05-20T18:03:24.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.582+0000 7f4438ff1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a08860cc40 con 0x55a0883244d0 2024-05-20T18:03:24.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.582+0000 7f4438ff1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a0886f9240 con 0x55a08834d390 2024-05-20T18:03:24.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.582+0000 7f4438ff1740 10 client.0 _ll_register_callbacks cb 0x55a088364be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:24.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.583+0000 7f442bfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a088702610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:24.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.583+0000 7f442bfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a088702610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50264/0 (socket says 172.21.15.129:50264) 2024-05-20T18:03:24.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.583+0000 7f442bfff640 1 -- 172.21.15.129:0/2223769504 learned_addr learned my addr 172.21.15.129:0/2223769504 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:24.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.583+0000 7f442b7fe640 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a0883244d0 0x55a0886f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:24.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.587+0000 7f442affd640 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a08832c6b0 0x55a0886f67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:24.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.587+0000 7f442bfff640 1 -- 172.21.15.129:0/2223769504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a0883244d0 msgr2=0x55a0886f62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:24.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.587+0000 7f442bfff640 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a0883244d0 0x55a0886f62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.587+0000 7f442bfff640 1 -- 172.21.15.129:0/2223769504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a08832c6b0 msgr2=0x55a0886f67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:24.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.587+0000 7f442bfff640 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a08832c6b0 0x55a0886f67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.587+0000 7f442bfff640 1 -- 172.21.15.129:0/2223769504 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a08842f7d0 con 0x55a08834d390 2024-05-20T18:03:24.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.588+0000 7f442bfff640 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a088702610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f441c0049e0 tx=0x7f441c00dbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d86fb939bc2db51 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:24.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.588+0000 7f440affd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:03:24.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.588+0000 7f440affd640 1 -- 172.21.15.129:0/2223769504 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c016070 con 0x55a08834d390 2024-05-20T18:03:24.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.589+0000 7f440affd640 1 -- 172.21.15.129:0/2223769504 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f441c004040 con 0x55a08834d390 2024-05-20T18:03:24.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.590+0000 7f4438ff1740 1 -- 172.21.15.129:0/2223769504 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f441c005500 con 0x55a08834d390 2024-05-20T18:03:24.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.594+0000 7f440affd640 1 -- 172.21.15.129:0/2223769504 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f441c02a050 con 0x55a08834d390 2024-05-20T18:03:24.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.594+0000 7f4438ff1740 10 client.25897 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:24.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.594+0000 7f4438ff1740 1 -- 172.21.15.129:0/2223769504 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a0887057b0 con 0x55a08834d390 2024-05-20T18:03:24.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.599+0000 7f440affd640 1 -- 172.21.15.129:0/2223769504 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441c005500 con 0x55a08834d390 2024-05-20T18:03:24.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.601+0000 7f440affd640 1 -- 172.21.15.129:0/2223769504 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f441c010460 con 0x55a08834d390 2024-05-20T18:03:24.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.603+0000 7f4438ff1740 10 client.25897 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:24.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.603+0000 7f4438ff1740 10 client.25897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:24.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.603+0000 7f4438ff1740 20 client.25897 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:24.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.603+0000 7f4438ff1740 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a08830bf60 0x55a088708c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.603+0000 7f4438ff1740 4 client.25897 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:03:24.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.603+0000 7f4438ff1740 1 -- 172.21.15.129:0/2223769504 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a08865f370 con 0x55a08830bf60 2024-05-20T18:03:24.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.604+0000 7f442b7fe640 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a08830bf60 0x55a088708c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:24.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.610+0000 7f442b7fe640 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a08830bf60 0x55a088708c20 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d2449c29e3d91eb server_cookie=b7c21955f1151e8a in_seq=0 out_seq=0 2024-05-20T18:03:24.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.610+0000 7f440affd640 10 client.25897 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.614+0000 7f440affd640 1 -- 172.21.15.129:0/2223769504 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a08865f370 con 0x55a08830bf60 2024-05-20T18:03:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.614+0000 7f440affd640 10 client.25897 handle_command_reply: tid=0 2024-05-20T18:03:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.615+0000 7f4438ff1740 1 client.25897 shutdown 2024-05-20T18:03:24.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.641+0000 7f4438ff1740 1 -- 172.21.15.129:0/2223769504 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 msgr2=0x55a088702610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:24.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.641+0000 7f4438ff1740 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a088702610 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f441c0049e0 tx=0x7f441c00dbb0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.641+0000 7f4438ff1740 1 -- 172.21.15.129:0/2223769504 shutdown_connections 2024-05-20T18:03:24.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.641+0000 7f4438ff1740 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a08830bf60 0x55a088708c20 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.641+0000 7f4438ff1740 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08834d390 0x55a088702610 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.641+0000 7f4438ff1740 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a08832c6b0 0x55a0886f67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.641+0000 7f4438ff1740 1 --2- 172.21.15.129:0/2223769504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a0883244d0 0x55a0886f62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.641+0000 7f4438ff1740 1 -- 172.21.15.129:0/2223769504 >> 172.21.15.129:0/2223769504 conn(0x55a0883414e0 msgr2=0x55a0886ebb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:24.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.641+0000 7f440affd640 0 client.25897 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:24.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.641+0000 7f4438ff1740 1 -- 172.21.15.129:0/2223769504 shutdown_connections 2024-05-20T18:03:24.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.641+0000 7f4438ff1740 1 -- 172.21.15.129:0/2223769504 wait complete. 2024-05-20T18:03:24.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.642+0000 7f4438ff1740 20 client.25897 trim_cache size 0 max 16384 2024-05-20T18:03:24.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.682+0000 7f4438ff1740 1 Processor -- start 2024-05-20T18:03:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.685+0000 7f4438ff1740 1 -- start start 2024-05-20T18:03:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.685+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a0883244d0 0x55a0886f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.685+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0887021f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.686+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a08834d390 0x55a0886e7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.686+0000 7f4438ff1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f441800c840 con 0x55a08834d390 2024-05-20T18:03:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.686+0000 7f4438ff1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f441800f010 con 0x55a0883244d0 2024-05-20T18:03:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.686+0000 7f4438ff1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f441800ce00 con 0x55a08832c6b0 2024-05-20T18:03:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.686+0000 7f442affd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0887021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.686+0000 7f442affd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0887021f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50272/0 (socket says 172.21.15.129:50272) 2024-05-20T18:03:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.686+0000 7f442affd640 1 -- 172.21.15.129:0/2246694362 learned_addr learned my addr 172.21.15.129:0/2246694362 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:24.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.687+0000 7f442affd640 1 -- 172.21.15.129:0/2246694362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a0883244d0 msgr2=0x55a0886f65e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:03:24.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.687+0000 7f442affd640 1 --2- 172.21.15.129:0/2246694362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a0883244d0 0x55a0886f65e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.687+0000 7f442affd640 1 -- 172.21.15.129:0/2246694362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a08834d390 msgr2=0x55a0886e7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:24.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.687+0000 7f442affd640 1 --2- 172.21.15.129:0/2246694362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a08834d390 0x55a0886e7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.687+0000 7f442affd640 1 -- 172.21.15.129:0/2246694362 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0886f6e00 con 0x55a08832c6b0 2024-05-20T18:03:24.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.688+0000 7f442affd640 1 --2- 172.21.15.129:0/2246694362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0887021f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f442409b6d0 tx=0x7f442409b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a90907287b4885c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:24.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.692+0000 7f44117fa640 1 -- 172.21.15.129:0/2246694362 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44240b3020 con 0x55a08832c6b0 2024-05-20T18:03:24.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.693+0000 7f44117fa640 1 -- 172.21.15.129:0/2246694362 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f442409fd80 con 0x55a08832c6b0 2024-05-20T18:03:24.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.693+0000 7f44117fa640 1 -- 172.21.15.129:0/2246694362 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442401d070 con 0x55a08832c6b0 2024-05-20T18:03:24.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.694+0000 7f4438ff1740 1 -- 172.21.15.129:0/2246694362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 msgr2=0x55a0887021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:24.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.694+0000 7f4438ff1740 1 --2- 172.21.15.129:0/2246694362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0887021f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f442409b6d0 tx=0x7f442409b330 comp rx=0 tx=0).stop 2024-05-20T18:03:24.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.694+0000 7f4438ff1740 1 -- 172.21.15.129:0/2246694362 shutdown_connections 2024-05-20T18:03:24.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.694+0000 7f4438ff1740 1 --2- 172.21.15.129:0/2246694362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a08834d390 0x55a0886e7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.694+0000 7f4438ff1740 1 --2- 172.21.15.129:0/2246694362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0887021f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.694+0000 7f4438ff1740 1 --2- 172.21.15.129:0/2246694362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a0883244d0 0x55a0886f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.694+0000 7f4438ff1740 1 -- 172.21.15.129:0/2246694362 >> 172.21.15.129:0/2246694362 conn(0x55a0883414e0 msgr2=0x55a0886e6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:24.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.694+0000 7f4438ff1740 1 -- 172.21.15.129:0/2246694362 shutdown_connections 2024-05-20T18:03:24.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.695+0000 7f4438ff1740 1 -- 172.21.15.129:0/2246694362 wait complete. 2024-05-20T18:03:24.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.695+0000 7f4438ff1740 1 -- start start 2024-05-20T18:03:24.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.696+0000 7f4438ff1740 1 Processor -- start 2024-05-20T18:03:24.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.697+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0883244d0 0x55a0886f9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.697+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0886f9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.697+0000 7f4438ff1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a08834d390 0x55a08876ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.697+0000 7f4438ff1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f441800d4d0 con 0x55a0883244d0 2024-05-20T18:03:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.697+0000 7f4438ff1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f441800e940 con 0x55a08834d390 2024-05-20T18:03:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.697+0000 7f4438ff1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f441800dba0 con 0x55a08832c6b0 2024-05-20T18:03:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.697+0000 7f4438ff1740 10 client.0 _ll_register_callbacks cb 0x7f441c00b7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.698+0000 7f442affd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0886f9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.698+0000 7f442affd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0886f9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50280/0 (socket says 172.21.15.129:50280) 2024-05-20T18:03:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.698+0000 7f442affd640 1 -- 172.21.15.129:0/1587296839 learned_addr learned my addr 172.21.15.129:0/1587296839 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:24.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.698+0000 7f442bfff640 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a08834d390 0x55a08876ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:24.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.701+0000 7f442affd640 1 -- 172.21.15.129:0/1587296839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a08834d390 msgr2=0x55a08876ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:24.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.701+0000 7f442affd640 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a08834d390 0x55a08876ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.701+0000 7f442affd640 1 -- 172.21.15.129:0/1587296839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0883244d0 msgr2=0x55a0886f9530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:03:24.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.701+0000 7f442affd640 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0883244d0 0x55a0886f9530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.702+0000 7f442affd640 1 -- 172.21.15.129:0/1587296839 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0886ed310 con 0x55a08832c6b0 2024-05-20T18:03:24.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.704+0000 7f442affd640 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0886f9ad0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f44240a4110 tx=0x7f442409b460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d3786aa7bf7e50f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:24.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.705+0000 7f440affd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:03:24.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.705+0000 7f440affd640 1 -- 172.21.15.129:0/1587296839 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44240b3030 con 0x55a08832c6b0 2024-05-20T18:03:24.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.705+0000 7f4438ff1740 1 -- 172.21.15.129:0/1587296839 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f442409fd80 con 0x55a08832c6b0 2024-05-20T18:03:24.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.705+0000 7f440affd640 1 -- 172.21.15.129:0/1587296839 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f44240ba1f0 con 0x55a08832c6b0 2024-05-20T18:03:24.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.705+0000 7f440affd640 1 -- 172.21.15.129:0/1587296839 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f442401d070 con 0x55a08832c6b0 2024-05-20T18:03:24.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.708+0000 7f4438ff1740 10 client.25903 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:24.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.708+0000 7f4438ff1740 1 -- 172.21.15.129:0/1587296839 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a0887701f0 con 0x55a08832c6b0 2024-05-20T18:03:24.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.714+0000 7f440affd640 1 -- 172.21.15.129:0/1587296839 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f442409fd80 con 0x55a08832c6b0 2024-05-20T18:03:24.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.716+0000 7f440affd640 1 -- 172.21.15.129:0/1587296839 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f442409ae20 con 0x55a08832c6b0 2024-05-20T18:03:24.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.718+0000 7f4438ff1740 10 client.25903 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:24.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.718+0000 7f4438ff1740 10 client.25903 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:24.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.718+0000 7f4438ff1740 20 client.25903 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:24.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.718+0000 7f4438ff1740 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a0882e13a0 0x55a0887728a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:24.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.718+0000 7f4438ff1740 4 client.25903 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:03:24.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.718+0000 7f4438ff1740 1 -- 172.21.15.129:0/1587296839 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a0886596c0 con 0x55a0882e13a0 2024-05-20T18:03:24.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.722+0000 7f442b7fe640 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a0882e13a0 0x55a0887728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:24.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.730+0000 7f442b7fe640 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a0882e13a0 0x55a0887728a0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=139db893e10b6c8e server_cookie=96bf37ee6ce79c9d in_seq=0 out_seq=0 2024-05-20T18:03:24.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.730+0000 7f440affd640 10 client.25903 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:24.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.737+0000 7f440affd640 1 -- 172.21.15.129:0/1587296839 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a0886596c0 con 0x55a0882e13a0 2024-05-20T18:03:24.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.737+0000 7f440affd640 10 client.25903 handle_command_reply: tid=0 2024-05-20T18:03:24.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.737+0000 7f4438ff1740 1 client.25903 shutdown 2024-05-20T18:03:24.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.744+0000 7f4438ff1740 1 -- 172.21.15.129:0/1587296839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 msgr2=0x55a0886f9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:24.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.744+0000 7f4438ff1740 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0886f9ad0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f44240a4110 tx=0x7f442409b460 comp rx=0 tx=0).stop 2024-05-20T18:03:24.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.744+0000 7f4438ff1740 1 -- 172.21.15.129:0/1587296839 shutdown_connections 2024-05-20T18:03:24.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.744+0000 7f4438ff1740 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a0882e13a0 0x55a0887728a0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.744+0000 7f4438ff1740 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a08834d390 0x55a08876ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.744+0000 7f4438ff1740 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a08832c6b0 0x55a0886f9ad0 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f44240a4110 tx=0x7f442409b460 comp rx=0 tx=0).stop 2024-05-20T18:03:24.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.744+0000 7f4438ff1740 1 --2- 172.21.15.129:0/1587296839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a0883244d0 0x55a0886f9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.744+0000 7f4438ff1740 1 -- 172.21.15.129:0/1587296839 >> 172.21.15.129:0/1587296839 conn(0x55a0883414e0 msgr2=0x55a0886e7570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:24.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.744+0000 7f440affd640 0 client.25903 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:24.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.750+0000 7f4438ff1740 1 -- 172.21.15.129:0/1587296839 shutdown_connections 2024-05-20T18:03:24.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.750+0000 7f4438ff1740 1 -- 172.21.15.129:0/1587296839 wait complete. 2024-05-20T18:03:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.753+0000 7f4438ff1740 20 client.25903 trim_cache size 0 max 16384 2024-05-20T18:03:24.755 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:03:24.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.759+0000 7f443214b640 1 -- 172.21.15.129:0/1710220673 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44140a2f00 msgr2=0x7f44140a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:24.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.759+0000 7f443214b640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44140a2f00 0x7f44140a5410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4420009230 tx=0x7f4420005fd0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.759+0000 7f443214b640 1 -- 172.21.15.129:0/1710220673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f442c119770 msgr2=0x7f442c113920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:24.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.759+0000 7f443214b640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f442c119770 0x7f442c113920 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f442400cce0 tx=0x7f442400a620 comp rx=0 tx=0).stop 2024-05-20T18:03:24.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.759+0000 7f443214b640 1 -- 172.21.15.129:0/1710220673 shutdown_connections 2024-05-20T18:03:24.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.759+0000 7f443214b640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44140a2f00 0x7f44140a5410 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.759+0000 7f443214b640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f442c11c0d0 0x7f442c113e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.759+0000 7f443214b640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f442c119770 0x7f442c113920 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.759+0000 7f443214b640 1 --2- 172.21.15.129:0/1710220673 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f442c109060 0x7f442c1133e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:24.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.760+0000 7f443214b640 1 -- 172.21.15.129:0/1710220673 >> 172.21.15.129:0/1710220673 conn(0x7f442c06d750 msgr2=0x7f442c119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:24.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.760+0000 7f443214b640 1 -- 172.21.15.129:0/1710220673 shutdown_connections 2024-05-20T18:03:24.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:24.760+0000 7f443214b640 1 -- 172.21.15.129:0/1710220673 wait complete. 2024-05-20T18:03:24.851 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:24 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1946755793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:03:24.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:24 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1946755793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:03:24.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:24 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1946755793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:03:25.852 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:03:25.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:03:25.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:03:25.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:25 smithi154 ceph-mon[36482]: pgmap v150: 97 pgs: 97 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 715 MiB/s wr, 4.79k op/s 2024-05-20T18:03:25.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:25 smithi129 ceph-mon[32029]: pgmap v150: 97 pgs: 97 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 715 MiB/s wr, 4.79k op/s 2024-05-20T18:03:25.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:25 smithi204 ceph-mon[37076]: pgmap v150: 97 pgs: 97 active+clean; 35 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 715 MiB/s wr, 4.79k op/s 2024-05-20T18:03:26.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.276+0000 7fd84a4ca640 1 -- 172.21.15.129:0/16484523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd83c0b8780 msgr2=0x7fd83c0baba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:26.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.276+0000 7fd84a4ca640 1 --2- 172.21.15.129:0/16484523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd83c0b8780 0x7fd83c0baba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd83801a910 tx=0x7fd83803b500 comp rx=0 tx=0).stop 2024-05-20T18:03:26.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.279+0000 7fd84a4ca640 1 -- 172.21.15.129:0/16484523 shutdown_connections 2024-05-20T18:03:26.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.279+0000 7fd84a4ca640 1 --2- 172.21.15.129:0/16484523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd83c0b5e20 0x7fd83c0b8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.279+0000 7fd84a4ca640 1 --2- 172.21.15.129:0/16484523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd83c0b8780 0x7fd83c0baba0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.279+0000 7fd84a4ca640 1 --2- 172.21.15.129:0/16484523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd83c0a57b0 0x7fd83c0a5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.279+0000 7fd84a4ca640 1 -- 172.21.15.129:0/16484523 >> 172.21.15.129:0/16484523 conn(0x7fd83c01af00 msgr2=0x7fd83c01b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:26.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.279+0000 7fd84a4ca640 1 -- 172.21.15.129:0/16484523 shutdown_connections 2024-05-20T18:03:26.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.279+0000 7fd84a4ca640 1 -- 172.21.15.129:0/16484523 wait complete. 2024-05-20T18:03:26.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.281+0000 7fd84a4ca640 1 Processor -- start 2024-05-20T18:03:26.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.281+0000 7fd84a4ca640 1 -- start start 2024-05-20T18:03:26.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.281+0000 7fd84a4ca640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd83c0b5e20 0x7fd83c0b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.281+0000 7fd84a4ca640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd83c0b8780 0x7fd83c0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.282+0000 7fd84a4ca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd83c0b2580 0x7fd83c16a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.282+0000 7fd84a4ca640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c0b00d0 con 0x7fd83c0b2580 2024-05-20T18:03:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.282+0000 7fd84a4ca640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c0aff50 con 0x7fd83c0b5e20 2024-05-20T18:03:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.282+0000 7fd84a4ca640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c0b0250 con 0x7fd83c0b8780 2024-05-20T18:03:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.282+0000 7fd8437fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd83c0b8780 0x7fd83c0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.282+0000 7fd8437fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd83c0b8780 0x7fd83c0b2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50296/0 (socket says 172.21.15.129:50296) 2024-05-20T18:03:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.282+0000 7fd8437fe640 1 -- 172.21.15.129:0/1721070261 learned_addr learned my addr 172.21.15.129:0/1721070261 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.283+0000 7fd848a67640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd83c0b2580 0x7fd83c16a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.283+0000 7fd8437fe640 1 -- 172.21.15.129:0/1721070261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd83c0b5e20 msgr2=0x7fd83c0b1b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T18:03:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.283+0000 7fd8437fe640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd83c0b5e20 0x7fd83c0b1b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.283+0000 7fd8437fe640 1 -- 172.21.15.129:0/1721070261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd83c0b2580 msgr2=0x7fd83c16a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:26.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.283+0000 7fd8437fe640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd83c0b2580 0x7fd83c16a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.283+0000 7fd8437fe640 1 -- 172.21.15.129:0/1721070261 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83c16ac60 con 0x7fd83c0b8780 2024-05-20T18:03:26.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.283+0000 7fd8437fe640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd83c0b8780 0x7fd83c0b2040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd84404f9f0 tx=0x7fd84406ead0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:26.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.287+0000 7fd8417fa640 1 -- 172.21.15.129:0/1721070261 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84407d070 con 0x7fd83c0b8780 2024-05-20T18:03:26.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.288+0000 7fd8417fa640 1 -- 172.21.15.129:0/1721070261 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd84406bbf0 con 0x7fd83c0b8780 2024-05-20T18:03:26.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.288+0000 7fd8417fa640 1 -- 172.21.15.129:0/1721070261 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd844078db0 con 0x7fd83c0b8780 2024-05-20T18:03:26.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.288+0000 7fd84a4ca640 1 -- 172.21.15.129:0/1721070261 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd83c16ae30 con 0x7fd83c0b8780 2024-05-20T18:03:26.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.288+0000 7fd84a4ca640 1 -- 172.21.15.129:0/1721070261 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd83c16b240 con 0x7fd83c0b8780 2024-05-20T18:03:26.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.296+0000 7fd851369740 1 Processor -- start 2024-05-20T18:03:26.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.296+0000 7fd851369740 1 -- start start 2024-05-20T18:03:26.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.297+0000 7fd851369740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aac6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.297+0000 7fd851369740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aac6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.297+0000 7fd851369740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aac7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.297+0000 7fd851369740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55907aa2b990 con 0x55907a727410 2024-05-20T18:03:26.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.297+0000 7fd851369740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55907a9d35b0 con 0x55907a706730 2024-05-20T18:03:26.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.297+0000 7fd851369740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55907aa27950 con 0x55907a6fe550 2024-05-20T18:03:26.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.297+0000 7fd848a67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aac6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.298+0000 7fd848a67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aac6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55766/0 (socket says 172.21.15.129:55766) 2024-05-20T18:03:26.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.298+0000 7fd848a67640 1 -- 172.21.15.129:0/2885664276 learned_addr learned my addr 172.21.15.129:0/2885664276 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:26.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.298+0000 7fd8437fe640 1 --2- 172.21.15.129:0/2885664276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aac7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.298+0000 7fd848a67640 1 -- 172.21.15.129:0/2885664276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 msgr2=0x55907aac6d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:03:26.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.298+0000 7fd848a67640 1 --2- 172.21.15.129:0/2885664276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aac6d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.298+0000 7fd848a67640 1 -- 172.21.15.129:0/2885664276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 msgr2=0x55907aac7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:26.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.298+0000 7fd848a67640 1 --2- 172.21.15.129:0/2885664276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aac7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.298+0000 7fd848a67640 1 -- 172.21.15.129:0/2885664276 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55907a8077e0 con 0x55907a727410 2024-05-20T18:03:26.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.298+0000 7fd8417fa640 1 -- 172.21.15.129:0/1721070261 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fd844080070 con 0x7fd83c0b8780 2024-05-20T18:03:26.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.299+0000 7fd848a67640 1 --2- 172.21.15.129:0/2885664276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aac6800 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd8380409f0 tx=0x7fd8380405c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de6a6f04c1b0bff server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:26.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.299+0000 7fd82e7fc640 1 -- 172.21.15.129:0/2885664276 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd838013020 con 0x55907a727410 2024-05-20T18:03:26.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.300+0000 7fd8417fa640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd8340a2f00 0x7fd8340a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.300+0000 7fd82e7fc640 1 -- 172.21.15.129:0/2885664276 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd838004050 con 0x55907a727410 2024-05-20T18:03:26.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.302+0000 7fd851369740 1 -- 172.21.15.129:0/2885664276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 msgr2=0x55907aac6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:26.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.302+0000 7fd851369740 1 --2- 172.21.15.129:0/2885664276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aac6800 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd8380409f0 tx=0x7fd8380405c0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.303+0000 7fd8417fa640 1 -- 172.21.15.129:0/1721070261 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fd8441323d0 con 0x7fd83c0b8780 2024-05-20T18:03:26.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.304+0000 7fd851369740 1 -- 172.21.15.129:0/2885664276 shutdown_connections 2024-05-20T18:03:26.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.305+0000 7fd851369740 1 --2- 172.21.15.129:0/2885664276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aac7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.305+0000 7fd851369740 1 --2- 172.21.15.129:0/2885664276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aac6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.305+0000 7fd851369740 1 --2- 172.21.15.129:0/2885664276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aac6800 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.305+0000 7fd851369740 1 -- 172.21.15.129:0/2885664276 >> 172.21.15.129:0/2885664276 conn(0x55907a71b560 msgr2=0x55907aac2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:26.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.306+0000 7fd843fff640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd8340a2f00 0x7fd8340a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.310+0000 7fd851369740 1 -- 172.21.15.129:0/2885664276 shutdown_connections 2024-05-20T18:03:26.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.318+0000 7fd851369740 1 -- 172.21.15.129:0/2885664276 wait complete. 2024-05-20T18:03:26.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.319+0000 7fd851369740 1 -- start start 2024-05-20T18:03:26.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.319+0000 7fd851369740 1 Processor -- start 2024-05-20T18:03:26.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.320+0000 7fd851369740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aad28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.320+0000 7fd851369740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aad2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.320+0000 7fd851369740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aad0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.320+0000 7fd851369740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55907aad3b90 con 0x55907a727410 2024-05-20T18:03:26.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.320+0000 7fd851369740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55907a9e6b40 con 0x55907a706730 2024-05-20T18:03:26.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.320+0000 7fd851369740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55907aad3d10 con 0x55907a6fe550 2024-05-20T18:03:26.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.320+0000 7fd851369740 10 client.0 _ll_register_callbacks cb 0x55907a9db0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:26.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.322+0000 7fd848a67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aad0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.322+0000 7fd848a67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aad0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55782/0 (socket says 172.21.15.129:55782) 2024-05-20T18:03:26.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.322+0000 7fd848a67640 1 -- 172.21.15.129:0/1030889560 learned_addr learned my addr 172.21.15.129:0/1030889560 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:26.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.322+0000 7fd8437fe640 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aad2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.323+0000 7fd843fff640 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aad28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.324+0000 7fd8437fe640 1 -- 172.21.15.129:0/1030889560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 msgr2=0x55907aad28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:26.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.324+0000 7fd8437fe640 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aad28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.324+0000 7fd8437fe640 1 -- 172.21.15.129:0/1030889560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 msgr2=0x55907aad0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:26.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.324+0000 7fd8437fe640 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aad0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.324+0000 7fd8437fe640 1 -- 172.21.15.129:0/1030889560 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55907a867f70 con 0x55907a706730 2024-05-20T18:03:26.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.325+0000 7fd8437fe640 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aad2e00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd84412fab0 tx=0x7fd8440fab20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=befe5ba85e70894f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:26.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.332+0000 7fd8237fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:03:26.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.332+0000 7fd8237fe640 1 -- 172.21.15.129:0/1030889560 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8440fe3c0 con 0x55907a706730 2024-05-20T18:03:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.332+0000 7fd851369740 1 -- 172.21.15.129:0/1030889560 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55907aa390c0 con 0x55907a706730 2024-05-20T18:03:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.333+0000 7fd8237fe640 1 -- 172.21.15.129:0/1030889560 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fd844078410 con 0x55907a706730 2024-05-20T18:03:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.333+0000 7fd8237fe640 1 -- 172.21.15.129:0/1030889560 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd8440fe1e0 con 0x55907a706730 2024-05-20T18:03:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.334+0000 7fd851369740 10 client.25886 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.334+0000 7fd851369740 1 -- 172.21.15.129:0/1030889560 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55907a6ff170 con 0x55907a706730 2024-05-20T18:03:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.334+0000 7fd8237fe640 1 -- 172.21.15.129:0/1030889560 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55907aa390c0 con 0x55907a706730 2024-05-20T18:03:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.336+0000 7fd843fff640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd8340a2f00 0x7fd8340a5410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55907aac7dc0 tx=0x7fd830006c00 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.336+0000 7fd8237fe640 1 -- 172.21.15.129:0/1030889560 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fd84412fd30 con 0x55907a706730 2024-05-20T18:03:26.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.337+0000 7fd851369740 10 client.25886 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:26.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.337+0000 7fd851369740 10 client.25886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.337+0000 7fd851369740 20 client.25886 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.337+0000 7fd851369740 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55907a6e61f0 0x55907aae3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.337+0000 7fd851369740 4 client.25886 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:03:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.337+0000 7fd851369740 1 -- 172.21.15.129:0/1030889560 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55907aa33410 con 0x55907a6e61f0 2024-05-20T18:03:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.338+0000 7fd843fff640 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55907a6e61f0 0x55907aae3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.349+0000 7fd8237fe640 10 client.25886 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:26.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.349+0000 7fd843fff640 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55907a6e61f0 0x55907aae3620 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=830355fc13f920c6 server_cookie=4bba89ec58813824 in_seq=0 out_seq=0 2024-05-20T18:03:26.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.352+0000 7fd8237fe640 1 -- 172.21.15.129:0/1030889560 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55907aa33410 con 0x55907a6e61f0 2024-05-20T18:03:26.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.352+0000 7fd8237fe640 10 client.25886 handle_command_reply: tid=0 2024-05-20T18:03:26.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.355+0000 7fd851369740 1 client.25886 shutdown 2024-05-20T18:03:26.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.360+0000 7fd851369740 1 -- 172.21.15.129:0/1030889560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 msgr2=0x55907aad2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:26.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.360+0000 7fd851369740 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aad2e00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd84412fab0 tx=0x7fd8440fab20 comp rx=0 tx=0).stop 2024-05-20T18:03:26.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.362+0000 7fd851369740 1 -- 172.21.15.129:0/1030889560 shutdown_connections 2024-05-20T18:03:26.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.362+0000 7fd851369740 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55907a6e61f0 0x55907aae3620 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.362+0000 7fd851369740 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aad2e00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.362+0000 7fd851369740 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aad0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.362+0000 7fd851369740 1 --2- 172.21.15.129:0/1030889560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aad28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.362+0000 7fd851369740 1 -- 172.21.15.129:0/1030889560 >> 172.21.15.129:0/1030889560 conn(0x55907a71b560 msgr2=0x55907aac6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.362+0000 7fd8237fe640 0 client.25886 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:26.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.364+0000 7fd851369740 1 -- 172.21.15.129:0/1030889560 shutdown_connections 2024-05-20T18:03:26.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.365+0000 7fd851369740 1 -- 172.21.15.129:0/1030889560 wait complete. 2024-05-20T18:03:26.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.365+0000 7fd851369740 20 client.25886 trim_cache size 0 max 16384 2024-05-20T18:03:26.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.411+0000 7fd851369740 1 Processor -- start 2024-05-20T18:03:26.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.411+0000 7fd851369740 1 -- start start 2024-05-20T18:03:26.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.412+0000 7fd851369740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a6fe550 0x55907aacac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.412+0000 7fd851369740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a706730 0x55907aae8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.412+0000 7fd851369740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aae9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.413+0000 7fd843fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a6fe550 0x55907aacac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.413+0000 7fd843fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a6fe550 0x55907aacac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:34054/0 (socket says 172.21.15.129:34054) 2024-05-20T18:03:26.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.413+0000 7fd8437fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a706730 0x55907aae8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.413+0000 7fd8437fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a706730 0x55907aae8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50320/0 (socket says 172.21.15.129:50320) 2024-05-20T18:03:26.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.413+0000 7fd848a67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aae9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.413+0000 7fd848a67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aae9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55788/0 (socket says 172.21.15.129:55788) 2024-05-20T18:03:26.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.414+0000 7fd843fff640 1 -- 172.21.15.129:0/1235324547 learned_addr learned my addr 172.21.15.129:0/1235324547 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:26.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.414+0000 7fd851369740 1 -- 172.21.15.129:0/1235324547 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd82800c830 con 0x55907a727410 2024-05-20T18:03:26.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.414+0000 7fd851369740 1 -- 172.21.15.129:0/1235324547 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd82800f0b0 con 0x55907a6fe550 2024-05-20T18:03:26.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.414+0000 7fd851369740 1 -- 172.21.15.129:0/1235324547 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd82800cea0 con 0x55907a706730 2024-05-20T18:03:26.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.414+0000 7fd843fff640 1 -- 172.21.15.129:0/1235324547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a706730 msgr2=0x55907aae8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:26.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.414+0000 7fd843fff640 1 --2- 172.21.15.129:0/1235324547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a706730 0x55907aae8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.414+0000 7fd843fff640 1 -- 172.21.15.129:0/1235324547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 msgr2=0x55907aae9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:26.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.414+0000 7fd843fff640 1 --2- 172.21.15.129:0/1235324547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aae9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.414+0000 7fd843fff640 1 -- 172.21.15.129:0/1235324547 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55907aac7df0 con 0x55907a6fe550 2024-05-20T18:03:26.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.415+0000 7fd843fff640 1 --2- 172.21.15.129:0/1235324547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a6fe550 0x55907aacac00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0a52c0 tx=0x7fd83000e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=483818219493bdc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:26.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.418+0000 7fd82dffb640 1 -- 172.21.15.129:0/1235324547 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd830013ae0 con 0x55907a6fe550 2024-05-20T18:03:26.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.418+0000 7fd82dffb640 1 -- 172.21.15.129:0/1235324547 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd830013c80 con 0x55907a6fe550 2024-05-20T18:03:26.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.419+0000 7fd851369740 1 -- 172.21.15.129:0/1235324547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a6fe550 msgr2=0x55907aacac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:26.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.419+0000 7fd851369740 1 --2- 172.21.15.129:0/1235324547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a6fe550 0x55907aacac00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0a52c0 tx=0x7fd83000e000 comp rx=0 tx=0).stop 2024-05-20T18:03:26.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.419+0000 7fd851369740 1 -- 172.21.15.129:0/1235324547 shutdown_connections 2024-05-20T18:03:26.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.419+0000 7fd851369740 1 --2- 172.21.15.129:0/1235324547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aae9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.419+0000 7fd851369740 1 --2- 172.21.15.129:0/1235324547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a706730 0x55907aae8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.419+0000 7fd851369740 1 --2- 172.21.15.129:0/1235324547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a6fe550 0x55907aacac00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.419+0000 7fd851369740 1 -- 172.21.15.129:0/1235324547 >> 172.21.15.129:0/1235324547 conn(0x55907a71b560 msgr2=0x55907aaca4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:26.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.419+0000 7fd851369740 1 -- 172.21.15.129:0/1235324547 shutdown_connections 2024-05-20T18:03:26.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.420+0000 7fd851369740 1 -- 172.21.15.129:0/1235324547 wait complete. 2024-05-20T18:03:26.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.420+0000 7fd851369740 1 -- start start 2024-05-20T18:03:26.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.420+0000 7fd851369740 1 Processor -- start 2024-05-20T18:03:26.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.421+0000 7fd851369740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aad94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.421+0000 7fd851369740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aad6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.421+0000 7fd851369740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aad6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.421+0000 7fd851369740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd82800d570 con 0x55907a727410 2024-05-20T18:03:26.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.421+0000 7fd843fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aad94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.421+0000 7fd851369740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd82800e9e0 con 0x55907a706730 2024-05-20T18:03:26.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.421+0000 7fd851369740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd82800dc40 con 0x55907a6fe550 2024-05-20T18:03:26.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.421+0000 7fd843fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aad94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50322/0 (socket says 172.21.15.129:50322) 2024-05-20T18:03:26.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.421+0000 7fd843fff640 1 -- 172.21.15.129:0/2280909418 learned_addr learned my addr 172.21.15.129:0/2280909418 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:26.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.421+0000 7fd8437fe640 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aad6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.422+0000 7fd848a67640 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aad6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.422+0000 7fd851369740 10 client.0 _ll_register_callbacks cb 0x7fd844135bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:26.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.422+0000 7fd843fff640 1 -- 172.21.15.129:0/2280909418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 msgr2=0x55907aad6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:26.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.422+0000 7fd843fff640 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aad6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.422+0000 7fd843fff640 1 -- 172.21.15.129:0/2280909418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 msgr2=0x55907aad6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:26.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.422+0000 7fd843fff640 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aad6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.422+0000 7fd843fff640 1 -- 172.21.15.129:0/2280909418 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55907aae3b80 con 0x55907a6fe550 2024-05-20T18:03:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.422+0000 7fd848a67640 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aad6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:03:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.422+0000 7fd843fff640 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aad94d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd830024c40 tx=0x7fd830024b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dd5049c00f07f87 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.424+0000 7fd8237fe640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:03:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.424+0000 7fd8237fe640 1 -- 172.21.15.129:0/2280909418 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8300059c0 con 0x55907a6fe550 2024-05-20T18:03:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.424+0000 7fd851369740 1 -- 172.21.15.129:0/2280909418 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd830013c80 con 0x55907a6fe550 2024-05-20T18:03:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.424+0000 7fd8237fe640 1 -- 172.21.15.129:0/2280909418 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fd830005b60 con 0x55907a6fe550 2024-05-20T18:03:26.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.434+0000 7fd8237fe640 1 -- 172.21.15.129:0/2280909418 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd830005d60 con 0x55907a6fe550 2024-05-20T18:03:26.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.434+0000 7fd851369740 10 client.25918 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:26.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.434+0000 7fd851369740 1 -- 172.21.15.129:0/2280909418 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55907ab4b5a0 con 0x55907a6fe550 2024-05-20T18:03:26.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.440+0000 7fd8237fe640 1 -- 172.21.15.129:0/2280909418 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd830013c80 con 0x55907a6fe550 2024-05-20T18:03:26.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.443+0000 7fd8237fe640 1 -- 172.21.15.129:0/2280909418 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fd83002b020 con 0x55907a6fe550 2024-05-20T18:03:26.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.443+0000 7fd851369740 10 client.25918 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:26.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.443+0000 7fd851369740 10 client.25918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:26.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.443+0000 7fd851369740 20 client.25918 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:26.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.443+0000 7fd851369740 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55907a6bb630 0x55907ab4db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:26.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.443+0000 7fd851369740 4 client.25918 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:03:26.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.443+0000 7fd851369740 1 -- 172.21.15.129:0/2280909418 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55907a9b0da0 con 0x55907a6bb630 2024-05-20T18:03:26.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.444+0000 7fd8437fe640 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55907a6bb630 0x55907ab4db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:26.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.446+0000 7fd8437fe640 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55907a6bb630 0x55907ab4db80 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ea5622e3d0df718 server_cookie=48813b8f96d6f6f3 in_seq=0 out_seq=0 2024-05-20T18:03:26.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.446+0000 7fd8237fe640 10 client.25918 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:26.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.938+0000 7fd8237fe640 1 -- 172.21.15.129:0/2280909418 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55907a9b0da0 con 0x55907a6bb630 2024-05-20T18:03:26.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.938+0000 7fd8237fe640 10 client.25918 handle_command_reply: tid=0 2024-05-20T18:03:26.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.939+0000 7fd851369740 1 client.25918 shutdown 2024-05-20T18:03:26.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.939+0000 7fd851369740 1 -- 172.21.15.129:0/2280909418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 msgr2=0x55907aad94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:26.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.939+0000 7fd851369740 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aad94d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd830024c40 tx=0x7fd830024b80 comp rx=0 tx=0).stop 2024-05-20T18:03:26.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.939+0000 7fd851369740 1 -- 172.21.15.129:0/2280909418 shutdown_connections 2024-05-20T18:03:26.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.939+0000 7fd851369740 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55907a6bb630 0x55907ab4db80 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.939+0000 7fd851369740 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55907a706730 0x55907aad6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.939+0000 7fd851369740 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55907a727410 0x55907aad6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.939+0000 7fd851369740 1 --2- 172.21.15.129:0/2280909418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55907a6fe550 0x55907aad94d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.939+0000 7fd851369740 1 -- 172.21.15.129:0/2280909418 >> 172.21.15.129:0/2280909418 conn(0x55907a71b560 msgr2=0x55907aad0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:26.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.939+0000 7fd8237fe640 0 client.25918 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:26.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.940+0000 7fd851369740 1 -- 172.21.15.129:0/2280909418 shutdown_connections 2024-05-20T18:03:26.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.940+0000 7fd851369740 1 -- 172.21.15.129:0/2280909418 wait complete. 2024-05-20T18:03:26.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.940+0000 7fd851369740 20 client.25918 trim_cache size 0 max 16384 2024-05-20T18:03:26.939 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:03:26.939 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:03:26.939 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "688b3ed9-6c87-4b94-9365-b915a6acbf36", 2024-05-20T18:03:26.940 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:03:26.940 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:03:26.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.951+0000 7fd84a4ca640 1 -- 172.21.15.129:0/1721070261 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd8340a2f00 msgr2=0x7fd8340a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:26.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.951+0000 7fd84a4ca640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd8340a2f00 0x7fd8340a5410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55907aac7dc0 tx=0x7fd830006c00 comp rx=0 tx=0).stop 2024-05-20T18:03:26.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.952+0000 7fd84a4ca640 1 -- 172.21.15.129:0/1721070261 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd83c0b8780 msgr2=0x7fd83c0b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:26.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.952+0000 7fd84a4ca640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd83c0b8780 0x7fd83c0b2040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd84404f9f0 tx=0x7fd84406ead0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.952+0000 7fd84a4ca640 1 -- 172.21.15.129:0/1721070261 shutdown_connections 2024-05-20T18:03:26.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.952+0000 7fd84a4ca640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd8340a2f00 0x7fd8340a5410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.952+0000 7fd84a4ca640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd83c0b2580 0x7fd83c16a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.952+0000 7fd84a4ca640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd83c0b8780 0x7fd83c0b2040 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.952+0000 7fd84a4ca640 1 --2- 172.21.15.129:0/1721070261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd83c0b5e20 0x7fd83c0b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:26.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.952+0000 7fd84a4ca640 1 -- 172.21.15.129:0/1721070261 >> 172.21.15.129:0/1721070261 conn(0x7fd83c01af00 msgr2=0x7fd83c0b6230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:26.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.953+0000 7fd84a4ca640 1 -- 172.21.15.129:0/1721070261 shutdown_connections 2024-05-20T18:03:26.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:26.953+0000 7fd84a4ca640 1 -- 172.21.15.129:0/1721070261 wait complete. 2024-05-20T18:03:27.008 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:03:27.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:03:27.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.504+0000 7f1417c44640 1 -- 172.21.15.129:0/4040029775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1410109060 msgr2=0x7f1410109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:27.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.504+0000 7f1417c44640 1 --2- 172.21.15.129:0/4040029775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1410109060 0x7f1410109470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1400009930 tx=0x7f140002a510 comp rx=0 tx=0).stop 2024-05-20T18:03:27.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.505+0000 7f1417c44640 1 -- 172.21.15.129:0/4040029775 shutdown_connections 2024-05-20T18:03:27.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.505+0000 7f1417c44640 1 --2- 172.21.15.129:0/4040029775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f141011c0d0 0x7f141011e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.505+0000 7f1417c44640 1 --2- 172.21.15.129:0/4040029775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1410119770 0x7f141011bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.505+0000 7f1417c44640 1 --2- 172.21.15.129:0/4040029775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1410109060 0x7f1410109470 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1400009930 tx=0x7f140002a510 comp rx=0 tx=0).stop 2024-05-20T18:03:27.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.505+0000 7f1417c44640 1 -- 172.21.15.129:0/4040029775 >> 172.21.15.129:0/4040029775 conn(0x7f141006d750 msgr2=0x7f141006dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:27.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.511+0000 7f1417c44640 1 -- 172.21.15.129:0/4040029775 shutdown_connections 2024-05-20T18:03:27.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.515+0000 7f1417c44640 1 -- 172.21.15.129:0/4040029775 wait complete. 2024-05-20T18:03:27.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.516+0000 7f1417c44640 1 Processor -- start 2024-05-20T18:03:27.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.518+0000 7f1417c44640 1 -- start start 2024-05-20T18:03:27.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.519+0000 7f1417c44640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1410119770 0x7f14101122f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.519+0000 7f1417c44640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f141011c0d0 0x7f1410112830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.519+0000 7f1417c44640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1410112d70 0x7f1410116cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.519+0000 7f1417c44640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1410121c20 con 0x7f1410119770 2024-05-20T18:03:27.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.519+0000 7f1417c44640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1410121aa0 con 0x7f1410112d70 2024-05-20T18:03:27.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.519+0000 7f1417c44640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1410121da0 con 0x7f141011c0d0 2024-05-20T18:03:27.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.520+0000 7f14151df640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f141011c0d0 0x7f1410112830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.520+0000 7f14151df640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f141011c0d0 0x7f1410112830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50342/0 (socket says 172.21.15.129:50342) 2024-05-20T18:03:27.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.520+0000 7f14151df640 1 -- 172.21.15.129:0/3183263902 learned_addr learned my addr 172.21.15.129:0/3183263902 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:27.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.521+0000 7f14151df640 1 -- 172.21.15.129:0/3183263902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1410112d70 msgr2=0x7f1410116cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:27.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.521+0000 7f14151df640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1410112d70 0x7f1410116cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.521+0000 7f14151df640 1 -- 172.21.15.129:0/3183263902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1410119770 msgr2=0x7f14101122f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:27.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.521+0000 7f14151df640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1410119770 0x7f14101122f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.521+0000 7f14151df640 1 -- 172.21.15.129:0/3183263902 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1410117230 con 0x7f141011c0d0 2024-05-20T18:03:27.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.522+0000 7f14151df640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f141011c0d0 0x7f1410112830 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f140800eab0 tx=0x7f140800ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:27.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.523+0000 7f1406ffd640 1 -- 172.21.15.129:0/3183263902 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140800ce40 con 0x7f141011c0d0 2024-05-20T18:03:27.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.523+0000 7f1417c44640 1 -- 172.21.15.129:0/3183263902 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1410117520 con 0x7f141011c0d0 2024-05-20T18:03:27.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.523+0000 7f1417c44640 1 -- 172.21.15.129:0/3183263902 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1410117a70 con 0x7f141011c0d0 2024-05-20T18:03:27.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.523+0000 7f1406ffd640 1 -- 172.21.15.129:0/3183263902 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1408011950 con 0x7f141011c0d0 2024-05-20T18:03:27.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.526+0000 7f1406ffd640 1 -- 172.21.15.129:0/3183263902 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140801ab70 con 0x7f141011c0d0 2024-05-20T18:03:27.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.531+0000 7f1406ffd640 1 -- 172.21.15.129:0/3183263902 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f14080233f0 con 0x7f141011c0d0 2024-05-20T18:03:27.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.533+0000 7f1406ffd640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13fc0a2db0 0x7f13fc0a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.534+0000 7f1406ffd640 1 -- 172.21.15.129:0/3183263902 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f14080cf970 con 0x7f141011c0d0 2024-05-20T18:03:27.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.540+0000 7f14159e0640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13fc0a2db0 0x7f13fc0a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.554+0000 7f141ebc8740 1 Processor -- start 2024-05-20T18:03:27.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.555+0000 7f141ebc8740 1 -- start start 2024-05-20T18:03:27.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.556+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704abd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.556+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700e44d0 0x5633704ac270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.556+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700ec6b0 0x5633704ac7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.558+0000 7f14161e1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700ec6b0 0x5633704ac7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.558+0000 7f14161e1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700ec6b0 0x5633704ac7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55828/0 (socket says 172.21.15.129:55828) 2024-05-20T18:03:27.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.558+0000 7f14161e1640 1 -- 172.21.15.129:0/3308464610 learned_addr learned my addr 172.21.15.129:0/3308464610 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:27.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.558+0000 7f141ebc8740 1 -- 172.21.15.129:0/3308464610 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563370411d00 con 0x5633700ec6b0 2024-05-20T18:03:27.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.558+0000 7f141ebc8740 1 -- 172.21.15.129:0/3308464610 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5633703b94c0 con 0x56337010d390 2024-05-20T18:03:27.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.558+0000 7f141ebc8740 1 -- 172.21.15.129:0/3308464610 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56337040dcc0 con 0x5633700e44d0 2024-05-20T18:03:27.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.558+0000 7f14151df640 1 --2- 172.21.15.129:0/3308464610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700e44d0 0x5633704ac270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.559+0000 7f14159e0640 1 --2- 172.21.15.129:0/3308464610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.563+0000 7f14151df640 1 -- 172.21.15.129:0/3308464610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 msgr2=0x5633704abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:27.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.563+0000 7f14151df640 1 --2- 172.21.15.129:0/3308464610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704abd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.563+0000 7f14151df640 1 -- 172.21.15.129:0/3308464610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700ec6b0 msgr2=0x5633704ac7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:27.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.563+0000 7f14151df640 1 --2- 172.21.15.129:0/3308464610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700ec6b0 0x5633704ac7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.563+0000 7f14151df640 1 -- 172.21.15.129:0/3308464610 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633701ef7d0 con 0x5633700e44d0 2024-05-20T18:03:27.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.570+0000 7f14151df640 1 --2- 172.21.15.129:0/3308464610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700e44d0 0x5633704ac270 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14080cd050 tx=0x7f140809b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff8a1c5a8a729b90 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:27.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.571+0000 7f13fbfff640 1 -- 172.21.15.129:0/3308464610 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14080b9070 con 0x5633700e44d0 2024-05-20T18:03:27.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.571+0000 7f13fbfff640 1 -- 172.21.15.129:0/3308464610 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14080a01d0 con 0x5633700e44d0 2024-05-20T18:03:27.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.573+0000 7f141ebc8740 1 -- 172.21.15.129:0/3308464610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700e44d0 msgr2=0x5633704ac270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:27.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.573+0000 7f141ebc8740 1 --2- 172.21.15.129:0/3308464610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700e44d0 0x5633704ac270 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14080cd050 tx=0x7f140809b840 comp rx=0 tx=0).stop 2024-05-20T18:03:27.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.574+0000 7f141ebc8740 1 -- 172.21.15.129:0/3308464610 shutdown_connections 2024-05-20T18:03:27.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.575+0000 7f14159e0640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13fc0a2db0 0x7f13fc0a52c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1410113b40 tx=0x7f1400005e90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:27.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.575+0000 7f141ebc8740 1 --2- 172.21.15.129:0/3308464610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700ec6b0 0x5633704ac7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.575+0000 7f141ebc8740 1 --2- 172.21.15.129:0/3308464610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700e44d0 0x5633704ac270 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.575+0000 7f141ebc8740 1 --2- 172.21.15.129:0/3308464610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.575+0000 7f141ebc8740 1 -- 172.21.15.129:0/3308464610 >> 172.21.15.129:0/3308464610 conn(0x5633701014e0 msgr2=0x5633704a84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:27.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.586+0000 7f141ebc8740 1 -- 172.21.15.129:0/3308464610 shutdown_connections 2024-05-20T18:03:27.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.587+0000 7f141ebc8740 1 -- 172.21.15.129:0/3308464610 wait complete. 2024-05-20T18:03:27.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.587+0000 7f141ebc8740 1 -- start start 2024-05-20T18:03:27.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.588+0000 7f141ebc8740 1 Processor -- start 2024-05-20T18:03:27.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.589+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700e44d0 0x5633704b7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.589+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704b82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.589+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704b6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.589+0000 7f141ebc8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5633704b9070 con 0x5633700e44d0 2024-05-20T18:03:27.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.589+0000 7f141ebc8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5633703ccb90 con 0x56337010d390 2024-05-20T18:03:27.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.589+0000 7f141ebc8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5633704b91f0 con 0x5633700ec6b0 2024-05-20T18:03:27.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.590+0000 7f141ebc8740 10 client.0 _ll_register_callbacks cb 0x563370408da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:27.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.591+0000 7f14161e1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.591+0000 7f14151df640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704b82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.592+0000 7f14151df640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704b82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50350/0 (socket says 172.21.15.129:50350) 2024-05-20T18:03:27.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.592+0000 7f14151df640 1 -- 172.21.15.129:0/1165334342 learned_addr learned my addr 172.21.15.129:0/1165334342 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:27.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.596+0000 7f14151df640 1 -- 172.21.15.129:0/1165334342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 msgr2=0x5633704b6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:27.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.596+0000 7f14151df640 1 --2- 172.21.15.129:0/1165334342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704b6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.596+0000 7f14151df640 1 -- 172.21.15.129:0/1165334342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700e44d0 msgr2=0x5633704b7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.596+0000 7f14151df640 1 --2- 172.21.15.129:0/1165334342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700e44d0 0x5633704b7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.596+0000 7f14151df640 1 -- 172.21.15.129:0/1165334342 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633701253f0 con 0x5633700ec6b0 2024-05-20T18:03:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.597+0000 7f14151df640 1 --2- 172.21.15.129:0/1165334342 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704b82b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f14080a4560 tx=0x7f1408098150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60d4185785c57b5c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:27.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.601+0000 7f13f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:03:27.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.601+0000 7f13f97fa640 1 -- 172.21.15.129:0/1165334342 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14080b9030 con 0x5633700ec6b0 2024-05-20T18:03:27.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.601+0000 7f141ebc8740 1 -- 172.21.15.129:0/1165334342 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14080a01d0 con 0x5633700ec6b0 2024-05-20T18:03:27.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.601+0000 7f13f97fa640 1 -- 172.21.15.129:0/1165334342 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f140809f070 con 0x5633700ec6b0 2024-05-20T18:03:27.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.610+0000 7f13f97fa640 1 -- 172.21.15.129:0/1165334342 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f14080a4a90 con 0x5633700ec6b0 2024-05-20T18:03:27.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.618+0000 7f141ebc8740 10 client.25930 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:27.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.618+0000 7f141ebc8740 1 -- 172.21.15.129:0/1165334342 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5633700e50f0 con 0x5633700ec6b0 2024-05-20T18:03:27.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.627+0000 7f13f97fa640 1 -- 172.21.15.129:0/1165334342 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14080a01d0 con 0x5633700ec6b0 2024-05-20T18:03:27.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.627+0000 7f13f97fa640 1 -- 172.21.15.129:0/1165334342 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f14080a4ce0 con 0x5633700ec6b0 2024-05-20T18:03:27.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.630+0000 7f141ebc8740 10 client.25930 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:27.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.630+0000 7f141ebc8740 10 client.25930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:27.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.630+0000 7f141ebc8740 20 client.25930 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:27.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.630+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1165334342 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5633700cbf60 0x5633704c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.630+0000 7f141ebc8740 4 client.25930 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:03:27.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.630+0000 7f141ebc8740 1 -- 172.21.15.129:0/1165334342 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56337041f430 con 0x5633700cbf60 2024-05-20T18:03:27.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.633+0000 7f14159e0640 1 --2- 172.21.15.129:0/1165334342 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5633700cbf60 0x5633704c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.635+0000 7f14159e0640 1 --2- 172.21.15.129:0/1165334342 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5633700cbf60 0x5633704c8a80 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e22d50f2918f5f5 server_cookie=e6af936c8734e1a0 in_seq=0 out_seq=0 2024-05-20T18:03:27.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.638+0000 7f13f97fa640 10 client.25930 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:27.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.638+0000 7f13f97fa640 1 -- 172.21.15.129:0/1165334342 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56337041f430 con 0x5633700cbf60 2024-05-20T18:03:27.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.638+0000 7f13f97fa640 10 client.25930 handle_command_reply: tid=0 2024-05-20T18:03:27.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.638+0000 7f141ebc8740 1 client.25930 shutdown 2024-05-20T18:03:27.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.648+0000 7f141ebc8740 1 -- 172.21.15.129:0/1165334342 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 msgr2=0x5633704b82b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:27.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.648+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1165334342 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704b82b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f14080a4560 tx=0x7f1408098150 comp rx=0 tx=0).stop 2024-05-20T18:03:27.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.649+0000 7f141ebc8740 1 -- 172.21.15.129:0/1165334342 shutdown_connections 2024-05-20T18:03:27.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.649+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1165334342 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5633700cbf60 0x5633704c8a80 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.649+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1165334342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704b6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.649+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1165334342 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704b82b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.649+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1165334342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700e44d0 0x5633704b7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.649+0000 7f141ebc8740 1 -- 172.21.15.129:0/1165334342 >> 172.21.15.129:0/1165334342 conn(0x5633701014e0 msgr2=0x5633704ada50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:27.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.650+0000 7f13f97fa640 0 client.25930 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:27.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.653+0000 7f141ebc8740 1 -- 172.21.15.129:0/1165334342 shutdown_connections 2024-05-20T18:03:27.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.657+0000 7f141ebc8740 1 -- 172.21.15.129:0/1165334342 wait complete. 2024-05-20T18:03:27.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.657+0000 7f141ebc8740 20 client.25930 trim_cache size 0 max 16384 2024-05-20T18:03:27.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.723+0000 7f141ebc8740 1 Processor -- start 2024-05-20T18:03:27.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.730+0000 7f141ebc8740 1 -- start start 2024-05-20T18:03:27.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.730+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700e44d0 0x5633704cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.730+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.730+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.731+0000 7f14159e0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700e44d0 0x5633704cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.734+0000 7f141ebc8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f13f400c840 con 0x5633700e44d0 2024-05-20T18:03:27.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.734+0000 7f141ebc8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13f400f0c0 con 0x56337010d390 2024-05-20T18:03:27.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.734+0000 7f141ebc8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13f400ceb0 con 0x5633700ec6b0 2024-05-20T18:03:27.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.734+0000 7f14159e0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700e44d0 0x5633704cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55840/0 (socket says 172.21.15.129:55840) 2024-05-20T18:03:27.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.734+0000 7f14159e0640 1 -- 172.21.15.129:0/2949295066 learned_addr learned my addr 172.21.15.129:0/2949295066 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:27.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.735+0000 7f14161e1640 1 --2- 172.21.15.129:0/2949295066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.736+0000 7f14161e1640 1 -- 172.21.15.129:0/2949295066 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 msgr2=0x5633704cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:27.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.736+0000 7f14161e1640 1 --2- 172.21.15.129:0/2949295066 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.736+0000 7f14161e1640 1 -- 172.21.15.129:0/2949295066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700e44d0 msgr2=0x5633704cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:27.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.736+0000 7f14161e1640 1 --2- 172.21.15.129:0/2949295066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700e44d0 0x5633704cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.736+0000 7f14161e1640 1 -- 172.21.15.129:0/2949295066 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633700cc190 con 0x56337010d390 2024-05-20T18:03:27.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.738+0000 7f14161e1640 1 --2- 172.21.15.129:0/2949295066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704af7e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f140c009bc0 tx=0x7f140c012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc831503b1a83fc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:27.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.740+0000 7f13fb7fe640 1 -- 172.21.15.129:0/2949295066 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c009d70 con 0x56337010d390 2024-05-20T18:03:27.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.740+0000 7f13fb7fe640 1 -- 172.21.15.129:0/2949295066 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f140c0120d0 con 0x56337010d390 2024-05-20T18:03:27.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.741+0000 7f141ebc8740 1 -- 172.21.15.129:0/2949295066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 msgr2=0x5633704af7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:27.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.741+0000 7f141ebc8740 1 --2- 172.21.15.129:0/2949295066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704af7e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f140c009bc0 tx=0x7f140c012820 comp rx=0 tx=0).stop 2024-05-20T18:03:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.744+0000 7f141ebc8740 1 -- 172.21.15.129:0/2949295066 shutdown_connections 2024-05-20T18:03:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.744+0000 7f141ebc8740 1 --2- 172.21.15.129:0/2949295066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56337010d390 0x5633704af7e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.744+0000 7f141ebc8740 1 --2- 172.21.15.129:0/2949295066 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.744+0000 7f141ebc8740 1 --2- 172.21.15.129:0/2949295066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633700e44d0 0x5633704cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.744+0000 7f141ebc8740 1 -- 172.21.15.129:0/2949295066 >> 172.21.15.129:0/2949295066 conn(0x5633701014e0 msgr2=0x5633704b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:27.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.746+0000 7f141ebc8740 1 -- 172.21.15.129:0/2949295066 shutdown_connections 2024-05-20T18:03:27.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.750+0000 7f141ebc8740 1 -- 172.21.15.129:0/2949295066 wait complete. 2024-05-20T18:03:27.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.753+0000 7f141ebc8740 1 -- start start 2024-05-20T18:03:27.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.753+0000 7f141ebc8740 1 Processor -- start 2024-05-20T18:03:27.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.755+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633700e44d0 0x5633704ca870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.755+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.755+0000 7f141ebc8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56337010d390 0x56337052ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.755+0000 7f141ebc8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f13f400d580 con 0x56337010d390 2024-05-20T18:03:27.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.755+0000 7f141ebc8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13f400e9f0 con 0x5633700e44d0 2024-05-20T18:03:27.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.755+0000 7f141ebc8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13f400dc50 con 0x5633700ec6b0 2024-05-20T18:03:27.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.755+0000 7f141ebc8740 10 client.0 _ll_register_callbacks cb 0x7f14080b7070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:27.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.757+0000 7f14151df640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.757+0000 7f14151df640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:50360/0 (socket says 172.21.15.129:50360) 2024-05-20T18:03:27.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.757+0000 7f14151df640 1 -- 172.21.15.129:0/1362906360 learned_addr learned my addr 172.21.15.129:0/1362906360 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:27.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.757+0000 7f14151df640 1 -- 172.21.15.129:0/1362906360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633700e44d0 msgr2=0x5633704ca870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:03:27.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.757+0000 7f14151df640 1 --2- 172.21.15.129:0/1362906360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633700e44d0 0x5633704ca870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.757+0000 7f14151df640 1 -- 172.21.15.129:0/1362906360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56337010d390 msgr2=0x56337052ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:27.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.757+0000 7f14151df640 1 --2- 172.21.15.129:0/1362906360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56337010d390 0x56337052ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.757+0000 7f14151df640 1 -- 172.21.15.129:0/1362906360 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633704cb6a0 con 0x5633700ec6b0 2024-05-20T18:03:27.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.758+0000 7f14151df640 1 --2- 172.21.15.129:0/1362906360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704cadb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f140809bdb0 tx=0x7f140809b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd8916a5103a5e03 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:27.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.761+0000 7f13f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:03:27.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.761+0000 7f13f97fa640 1 -- 172.21.15.129:0/1362906360 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14080b9070 con 0x5633700ec6b0 2024-05-20T18:03:27.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.761+0000 7f141ebc8740 1 -- 172.21.15.129:0/1362906360 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f140c0120d0 con 0x5633700ec6b0 2024-05-20T18:03:27.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.761+0000 7f13f97fa640 1 -- 172.21.15.129:0/1362906360 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f14080a1640 con 0x5633700ec6b0 2024-05-20T18:03:27.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.762+0000 7f13f97fa640 1 -- 172.21.15.129:0/1362906360 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f14080980c0 con 0x5633700ec6b0 2024-05-20T18:03:27.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.765+0000 7f141ebc8740 10 client.25933 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:27.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.765+0000 7f141ebc8740 1 -- 172.21.15.129:0/1362906360 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5633705301b0 con 0x5633700ec6b0 2024-05-20T18:03:27.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.767+0000 7f13f97fa640 1 -- 172.21.15.129:0/1362906360 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f140c0120d0 con 0x5633700ec6b0 2024-05-20T18:03:27.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.770+0000 7f13f97fa640 1 -- 172.21.15.129:0/1362906360 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f1408022030 con 0x5633700ec6b0 2024-05-20T18:03:27.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.770+0000 7f141ebc8740 10 client.25933 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:27.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.770+0000 7f141ebc8740 10 client.25933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:27.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.770+0000 7f141ebc8740 20 client.25933 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:27.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.770+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1362906360 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5633700a13a0 0x563370532710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:27.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.770+0000 7f141ebc8740 4 client.25933 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:03:27.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.770+0000 7f141ebc8740 1 -- 172.21.15.129:0/1362906360 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563370419780 con 0x5633700a13a0 2024-05-20T18:03:27.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.774+0000 7f14159e0640 1 --2- 172.21.15.129:0/1362906360 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5633700a13a0 0x563370532710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:27.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.777+0000 7f14159e0640 1 --2- 172.21.15.129:0/1362906360 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5633700a13a0 0x563370532710 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74a30ec7a7b41c0d server_cookie=d6b148b9c779a07a in_seq=0 out_seq=0 2024-05-20T18:03:27.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.777+0000 7f13f97fa640 10 client.25933 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:27.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.785+0000 7f13f97fa640 1 -- 172.21.15.129:0/1362906360 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x563370419780 con 0x5633700a13a0 2024-05-20T18:03:27.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.785+0000 7f13f97fa640 10 client.25933 handle_command_reply: tid=0 2024-05-20T18:03:27.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.785+0000 7f141ebc8740 1 client.25933 shutdown 2024-05-20T18:03:27.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.787+0000 7f141ebc8740 1 -- 172.21.15.129:0/1362906360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 msgr2=0x5633704cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:27.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.787+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1362906360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704cadb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f140809bdb0 tx=0x7f140809b6b0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.789+0000 7f141ebc8740 1 -- 172.21.15.129:0/1362906360 shutdown_connections 2024-05-20T18:03:27.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.789+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1362906360 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5633700a13a0 0x563370532710 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.789+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1362906360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56337010d390 0x56337052ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.789+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1362906360 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5633700ec6b0 0x5633704cadb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.789+0000 7f141ebc8740 1 --2- 172.21.15.129:0/1362906360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633700e44d0 0x5633704ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.789+0000 7f141ebc8740 1 -- 172.21.15.129:0/1362906360 >> 172.21.15.129:0/1362906360 conn(0x5633701014e0 msgr2=0x5633704b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:27.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.789+0000 7f13f97fa640 0 client.25933 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:27.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.794+0000 7f141ebc8740 1 -- 172.21.15.129:0/1362906360 shutdown_connections 2024-05-20T18:03:27.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.796+0000 7f141ebc8740 1 -- 172.21.15.129:0/1362906360 wait complete. 2024-05-20T18:03:27.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.798+0000 7f141ebc8740 20 client.25933 trim_cache size 0 max 16384 2024-05-20T18:03:27.806 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:03:27.806 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "scrub active (4 inodes in the stack)", 2024-05-20T18:03:27.806 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:03:27.806 INFO:teuthology.orchestra.run.smithi129.stdout: "688b3ed9-6c87-4b94-9365-b915a6acbf36": { 2024-05-20T18:03:27.806 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:03:27.806 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "688b3ed9-6c87-4b94-9365-b915a6acbf36", 2024-05-20T18:03:27.806 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:03:27.807 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:03:27.807 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:03:27.807 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:03:27.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.812+0000 7f1417c44640 1 -- 172.21.15.129:0/3183263902 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13fc0a2db0 msgr2=0x7f13fc0a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:27.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.812+0000 7f1417c44640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13fc0a2db0 0x7f13fc0a52c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1410113b40 tx=0x7f1400005e90 comp rx=0 tx=0).stop 2024-05-20T18:03:27.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.822+0000 7f1417c44640 1 -- 172.21.15.129:0/3183263902 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f141011c0d0 msgr2=0x7f1410112830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:27.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.822+0000 7f1417c44640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f141011c0d0 0x7f1410112830 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f140800eab0 tx=0x7f140800ef70 comp rx=0 tx=0).stop 2024-05-20T18:03:27.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.828+0000 7f1417c44640 1 -- 172.21.15.129:0/3183263902 shutdown_connections 2024-05-20T18:03:27.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.828+0000 7f1417c44640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13fc0a2db0 0x7f13fc0a52c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.828+0000 7f1417c44640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1410112d70 0x7f1410116cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.828+0000 7f1417c44640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f141011c0d0 0x7f1410112830 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.828+0000 7f1417c44640 1 --2- 172.21.15.129:0/3183263902 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1410119770 0x7f14101122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:27.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.828+0000 7f1417c44640 1 -- 172.21.15.129:0/3183263902 >> 172.21.15.129:0/3183263902 conn(0x7f141006d750 msgr2=0x7f1410119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:27.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.835+0000 7f1417c44640 1 -- 172.21.15.129:0/3183263902 shutdown_connections 2024-05-20T18:03:27.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:27.837+0000 7f1417c44640 1 -- 172.21.15.129:0/3183263902 wait complete. 2024-05-20T18:03:27.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:27 smithi154 ceph-mon[36482]: pgmap v151: 97 pgs: 97 active+clean; 36 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 716 MiB/s wr, 3.19k op/s 2024-05-20T18:03:27.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:27 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:03:27.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:27 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:03:27.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:27 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:03:27.917 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:27 smithi129 ceph-mon[32029]: pgmap v151: 97 pgs: 97 active+clean; 36 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 716 MiB/s wr, 3.19k op/s 2024-05-20T18:03:27.918 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:27 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:03:27.918 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:27 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:03:27.918 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:27 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:03:27.922 INFO:tasks.cephfs.filesystem:scrub status for tag:688b3ed9-6c87-4b94-9365-b915a6acbf36 - {'path': '/', 'tag': '688b3ed9-6c87-4b94-9365-b915a6acbf36', 'options': 'recursive,force'} 2024-05-20T18:03:27.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:27 smithi204 ceph-mon[37076]: pgmap v151: 97 pgs: 97 active+clean; 36 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 716 MiB/s wr, 3.19k op/s 2024-05-20T18:03:27.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:27 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:03:27.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:27 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:03:27.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:27 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:03:29.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:29 smithi154 ceph-mon[36482]: pgmap v152: 97 pgs: 97 active+clean; 36 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 729 MiB/s wr, 2.89k op/s 2024-05-20T18:03:29.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:29 smithi129 ceph-mon[32029]: pgmap v152: 97 pgs: 97 active+clean; 36 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 729 MiB/s wr, 2.89k op/s 2024-05-20T18:03:29.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:29 smithi204 ceph-mon[37076]: pgmap v152: 97 pgs: 97 active+clean; 36 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 729 MiB/s wr, 2.89k op/s 2024-05-20T18:03:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.527+0000 7f3a22173640 1 -- 172.21.15.129:0/3369180574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c109090 msgr2=0x7f3a1c109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.527+0000 7f3a22173640 1 --2- 172.21.15.129:0/3369180574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c109090 0x7f3a1c109520 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009ca0 tx=0x7f3a1402a8f0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.528+0000 7f3a22173640 1 -- 172.21.15.129:0/3369180574 shutdown_connections 2024-05-20T18:03:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.528+0000 7f3a22173640 1 --2- 172.21.15.129:0/3369180574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c109090 0x7f3a1c109520 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.528+0000 7f3a22173640 1 --2- 172.21.15.129:0/3369180574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a1c10f270 0x7f3a1c108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.528+0000 7f3a22173640 1 --2- 172.21.15.129:0/3369180574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a1c10e890 0x7f3a1c10eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.528+0000 7f3a22173640 1 -- 172.21.15.129:0/3369180574 >> 172.21.15.129:0/3369180574 conn(0x7f3a1c06e690 msgr2=0x7f3a1c06eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:30.537 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.539+0000 7f3a22173640 1 -- 172.21.15.129:0/3369180574 shutdown_connections 2024-05-20T18:03:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.544+0000 7f3a22173640 1 -- 172.21.15.129:0/3369180574 wait complete. 2024-05-20T18:03:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.546+0000 7f3a22173640 1 Processor -- start 2024-05-20T18:03:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.548+0000 7f3a22173640 1 -- start start 2024-05-20T18:03:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.549+0000 7f3a22173640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c109090 0x7f3a1c1a81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.549+0000 7f3a22173640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a1c10e890 0x7f3a1c1a8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.549+0000 7f3a22173640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a1c10f270 0x7f3a1c1ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.549+0000 7f3a22173640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c121aa0 con 0x7f3a1c10f270 2024-05-20T18:03:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.549+0000 7f3a22173640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c121920 con 0x7f3a1c109090 2024-05-20T18:03:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.549+0000 7f3a22173640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c121c20 con 0x7f3a1c10e890 2024-05-20T18:03:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.550+0000 7f3a21972640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a1c10f270 0x7f3a1c1ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.550+0000 7f3a21972640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a1c10f270 0x7f3a1c1ac9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33290/0 (socket says 172.21.15.129:33290) 2024-05-20T18:03:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.550+0000 7f3a21972640 1 -- 172.21.15.129:0/4063655312 learned_addr learned my addr 172.21.15.129:0/4063655312 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.551+0000 7f3a21972640 1 -- 172.21.15.129:0/4063655312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a1c10e890 msgr2=0x7f3a1c1a8730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:03:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.551+0000 7f3a21972640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a1c10e890 0x7f3a1c1a8730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.551+0000 7f3a21972640 1 -- 172.21.15.129:0/4063655312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c109090 msgr2=0x7f3a1c1a81d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T18:03:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.551+0000 7f3a21972640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c109090 0x7f3a1c1a81d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.551+0000 7f3a21972640 1 -- 172.21.15.129:0/4063655312 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c1acfb0 con 0x7f3a1c10f270 2024-05-20T18:03:30.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.552+0000 7f3a21972640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a1c10f270 0x7f3a1c1ac9f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3a1402ae60 tx=0x7f3a1402f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:30.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.559+0000 7f3a127fc640 1 -- 172.21.15.129:0/4063655312 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1402fbc0 con 0x7f3a1c10f270 2024-05-20T18:03:30.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.559+0000 7f3a22173640 1 -- 172.21.15.129:0/4063655312 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1c1ad240 con 0x7f3a1c10f270 2024-05-20T18:03:30.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.559+0000 7f3a22173640 1 -- 172.21.15.129:0/4063655312 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a1c1ad790 con 0x7f3a1c10f270 2024-05-20T18:03:30.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.559+0000 7f3a127fc640 1 -- 172.21.15.129:0/4063655312 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a1402d070 con 0x7f3a1c10f270 2024-05-20T18:03:30.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.559+0000 7f3a127fc640 1 -- 172.21.15.129:0/4063655312 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14045850 con 0x7f3a1c10f270 2024-05-20T18:03:30.566 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.568+0000 7f3a2a280740 1 Processor -- start 2024-05-20T18:03:30.567 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.569+0000 7f3a2a280740 1 -- start start 2024-05-20T18:03:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.572+0000 7f3a127fc640 1 -- 172.21.15.129:0/4063655312 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f3a1402fd60 con 0x7f3a1c10f270 2024-05-20T18:03:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.573+0000 7f3a127fc640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a080a2f00 0x7f3a080a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.573+0000 7f3a127fc640 1 -- 172.21.15.129:0/4063655312 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f3a140f06a0 con 0x7f3a1c10f270 2024-05-20T18:03:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.579+0000 7f3a20970640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a080a2f00 0x7f3a080a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.579+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa9b61b0 0x5563aad55280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.580+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa98d2f0 0x5563aad557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.580+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad55d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.580+0000 7f3a2a280740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5563aacba340 con 0x5563aa9b61b0 2024-05-20T18:03:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.580+0000 7f3a2a280740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5563aac62290 con 0x5563aa9954d0 2024-05-20T18:03:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.581+0000 7f3a21972640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad55d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.581+0000 7f3a21972640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad55d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:57922/0 (socket says 172.21.15.129:57922) 2024-05-20T18:03:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.581+0000 7f3a21972640 1 -- 172.21.15.129:0/928326159 learned_addr learned my addr 172.21.15.129:0/928326159 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.581+0000 7f3a20970640 1 --2- 172.21.15.129:0/928326159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa98d2f0 0x5563aad557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.581+0000 7f3a2a280740 1 -- 172.21.15.129:0/928326159 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5563aacb6300 con 0x5563aa98d2f0 2024-05-20T18:03:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.587+0000 7f3a20970640 1 -- 172.21.15.129:0/928326159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 msgr2=0x5563aad55d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.587+0000 7f3a20970640 1 --2- 172.21.15.129:0/928326159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad55d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.587+0000 7f3a20970640 1 -- 172.21.15.129:0/928326159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa9b61b0 msgr2=0x5563aad55280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.587+0000 7f3a20970640 1 --2- 172.21.15.129:0/928326159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa9b61b0 0x5563aad55280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.587+0000 7f3a20970640 1 -- 172.21.15.129:0/928326159 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563aac868e0 con 0x5563aa98d2f0 2024-05-20T18:03:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.588+0000 7f3a20970640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a080a2f00 0x7f3a080a5410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3a18005fd0 tx=0x7f3a1800aa10 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.588+0000 7f3a20970640 1 --2- 172.21.15.129:0/928326159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa98d2f0 0x5563aad557c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c06b570 tx=0x7f3a18018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77a5fef88776a04e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.588+0000 7f3a077fe640 1 -- 172.21.15.129:0/928326159 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a18011e20 con 0x5563aa98d2f0 2024-05-20T18:03:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.588+0000 7f3a077fe640 1 -- 172.21.15.129:0/928326159 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a18005020 con 0x5563aa98d2f0 2024-05-20T18:03:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.588+0000 7f3a077fe640 1 -- 172.21.15.129:0/928326159 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a18004890 con 0x5563aa98d2f0 2024-05-20T18:03:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.591+0000 7f3a2a280740 1 -- 172.21.15.129:0/928326159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa98d2f0 msgr2=0x5563aad557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.591+0000 7f3a2a280740 1 --2- 172.21.15.129:0/928326159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa98d2f0 0x5563aad557c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c06b570 tx=0x7f3a18018000 comp rx=0 tx=0).stop 2024-05-20T18:03:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.591+0000 7f3a2a280740 1 -- 172.21.15.129:0/928326159 shutdown_connections 2024-05-20T18:03:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.591+0000 7f3a2a280740 1 --2- 172.21.15.129:0/928326159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad55d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.591+0000 7f3a2a280740 1 --2- 172.21.15.129:0/928326159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa98d2f0 0x5563aad557c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.591+0000 7f3a2a280740 1 --2- 172.21.15.129:0/928326159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa9b61b0 0x5563aad55280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.592+0000 7f3a2a280740 1 -- 172.21.15.129:0/928326159 >> 172.21.15.129:0/928326159 conn(0x5563aa9aa300 msgr2=0x5563aad51a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.592+0000 7f3a2a280740 1 -- 172.21.15.129:0/928326159 shutdown_connections 2024-05-20T18:03:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.592+0000 7f3a2a280740 1 -- 172.21.15.129:0/928326159 wait complete. 2024-05-20T18:03:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.593+0000 7f3a2a280740 1 -- start start 2024-05-20T18:03:30.592 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.593+0000 7f3a2a280740 1 Processor -- start 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.595+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa98d2f0 0x5563aad612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.595+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad61820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.595+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa974d80 0x5563aad5f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.595+0000 7f3a2a280740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5563aad62350 con 0x5563aa98d2f0 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.595+0000 7f3a2a280740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5563aac75790 con 0x5563aa9954d0 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.595+0000 7f3a2a280740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5563aad624d0 con 0x5563aa974d80 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.595+0000 7f3a2a280740 10 client.0 _ll_register_callbacks cb 0x5563aacb0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.596+0000 7f3a21972640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa974d80 0x5563aad5f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.596+0000 7f3a21972640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa974d80 0x5563aad5f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:47634/0 (socket says 172.21.15.129:47634) 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.596+0000 7f3a21972640 1 -- 172.21.15.129:0/4069048014 learned_addr learned my addr 172.21.15.129:0/4069048014 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.596+0000 7f3a20970640 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad61820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.596+0000 7f3a21972640 1 -- 172.21.15.129:0/4069048014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 msgr2=0x5563aad61820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.596+0000 7f3a21972640 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad61820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.596+0000 7f3a21972640 1 -- 172.21.15.129:0/4069048014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa98d2f0 msgr2=0x5563aad612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.596+0000 7f3a21972640 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa98d2f0 0x5563aad612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.596+0000 7f3a21972640 1 -- 172.21.15.129:0/4069048014 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563aa9ce210 con 0x5563aa974d80 2024-05-20T18:03:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.597+0000 7f3a20970640 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad61820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:03:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.597+0000 7f3a21972640 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa974d80 0x5563aad5f7d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a140094e0 tx=0x7f3a140bcd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79f213259fc7c051 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.600+0000 7f3a04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:03:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.603+0000 7f3a04ff9640 1 -- 172.21.15.129:0/4069048014 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a140da060 con 0x5563aa974d80 2024-05-20T18:03:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.603+0000 7f3a04ff9640 1 -- 172.21.15.129:0/4069048014 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f3a14004b40 con 0x5563aa974d80 2024-05-20T18:03:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.603+0000 7f3a2a280740 1 -- 172.21.15.129:0/4069048014 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a18005020 con 0x5563aa974d80 2024-05-20T18:03:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.604+0000 7f3a04ff9640 1 -- 172.21.15.129:0/4069048014 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3a140d5e60 con 0x5563aa974d80 2024-05-20T18:03:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.604+0000 7f3a2a280740 10 client.25939 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.605+0000 7f3a2a280740 1 -- 172.21.15.129:0/4069048014 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5563aaa9a620 con 0x5563aa974d80 2024-05-20T18:03:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.610+0000 7f3a04ff9640 1 -- 172.21.15.129:0/4069048014 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a140d2070 con 0x5563aa974d80 2024-05-20T18:03:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.610+0000 7f3a04ff9640 1 -- 172.21.15.129:0/4069048014 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f3a140b89b0 con 0x5563aa974d80 2024-05-20T18:03:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.616+0000 7f3a2a280740 10 client.25939 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.616+0000 7f3a2a280740 10 client.25939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.616+0000 7f3a2a280740 20 client.25939 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.616+0000 7f3a2a280740 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5563aa94a1c0 0x5563aad6fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.616+0000 7f3a2a280740 4 client.25939 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:03:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.616+0000 7f3a2a280740 1 -- 172.21.15.129:0/4069048014 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3a18005020 con 0x5563aa94a1c0 2024-05-20T18:03:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.616+0000 7f3a21171640 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5563aa94a1c0 0x5563aad6fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.621+0000 7f3a21171640 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5563aa94a1c0 0x5563aad6fd40 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b917dfd2b872770b server_cookie=9be907d2d05fa776 in_seq=0 out_seq=0 2024-05-20T18:03:30.619 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.622+0000 7f3a04ff9640 10 client.25939 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.625+0000 7f3a04ff9640 1 -- 172.21.15.129:0/4069048014 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f3a18005020 con 0x5563aa94a1c0 2024-05-20T18:03:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.625+0000 7f3a04ff9640 10 client.25939 handle_command_reply: tid=0 2024-05-20T18:03:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.625+0000 7f3a2a280740 1 client.25939 shutdown 2024-05-20T18:03:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.626+0000 7f3a2a280740 1 -- 172.21.15.129:0/4069048014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa974d80 msgr2=0x5563aad5f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.626+0000 7f3a2a280740 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa974d80 0x5563aad5f7d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a140094e0 tx=0x7f3a140bcd20 comp rx=0 tx=0).stop 2024-05-20T18:03:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.626+0000 7f3a2a280740 1 -- 172.21.15.129:0/4069048014 shutdown_connections 2024-05-20T18:03:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.626+0000 7f3a2a280740 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5563aa94a1c0 0x5563aad6fd40 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.626+0000 7f3a2a280740 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad61820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.626+0000 7f3a2a280740 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa974d80 0x5563aad5f7d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.626+0000 7f3a2a280740 1 --2- 172.21.15.129:0/4069048014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa98d2f0 0x5563aad612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.626+0000 7f3a2a280740 1 -- 172.21.15.129:0/4069048014 >> 172.21.15.129:0/4069048014 conn(0x5563aa9aa300 msgr2=0x5563aad551c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:30.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.627+0000 7f3a04ff9640 0 client.25939 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:30.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.630+0000 7f3a2a280740 1 -- 172.21.15.129:0/4069048014 shutdown_connections 2024-05-20T18:03:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.632+0000 7f3a2a280740 1 -- 172.21.15.129:0/4069048014 wait complete. 2024-05-20T18:03:30.633 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.634+0000 7f3a2a280740 20 client.25939 trim_cache size 0 max 16384 2024-05-20T18:03:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.765+0000 7f3a2a280740 1 Processor -- start 2024-05-20T18:03:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.769+0000 7f3a2a280740 1 -- start start 2024-05-20T18:03:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.771+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa974d80 0x5563aad558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.771+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa98d2f0 0x5563aad75500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.771+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa9954d0 0x5563aad75a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.770 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.771+0000 7f3a2a280740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0000c8c0 con 0x5563aa98d2f0 2024-05-20T18:03:30.770 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.771+0000 7f3a2a280740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0000f140 con 0x5563aa974d80 2024-05-20T18:03:30.770 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.771+0000 7f3a2a280740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0000cf30 con 0x5563aa9954d0 2024-05-20T18:03:30.770 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.772+0000 7f3a21171640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa974d80 0x5563aad558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.770 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.772+0000 7f3a21171640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa974d80 0x5563aad558d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:57934/0 (socket says 172.21.15.129:57934) 2024-05-20T18:03:30.770 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.772+0000 7f3a21171640 1 -- 172.21.15.129:0/765133004 learned_addr learned my addr 172.21.15.129:0/765133004 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.774+0000 7f3a21972640 1 --2- 172.21.15.129:0/765133004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa9954d0 0x5563aad75a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.776+0000 7f3a20970640 1 --2- 172.21.15.129:0/765133004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa98d2f0 0x5563aad75500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.777+0000 7f3a21171640 1 -- 172.21.15.129:0/765133004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa9954d0 msgr2=0x5563aad75a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.777+0000 7f3a21171640 1 --2- 172.21.15.129:0/765133004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa9954d0 0x5563aad75a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.777+0000 7f3a21171640 1 -- 172.21.15.129:0/765133004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa98d2f0 msgr2=0x5563aad75500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.777+0000 7f3a21171640 1 --2- 172.21.15.129:0/765133004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa98d2f0 0x5563aad75500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.777+0000 7f3a21171640 1 -- 172.21.15.129:0/765133004 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563aad70270 con 0x5563aa974d80 2024-05-20T18:03:30.778 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.780+0000 7f3a21171640 1 --2- 172.21.15.129:0/765133004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa974d80 0x5563aad558d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0047b0 tx=0x7f3a0c009d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad51866cd182d23 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.796+0000 7f3a06ffd640 1 -- 172.21.15.129:0/765133004 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c009e90 con 0x5563aa974d80 2024-05-20T18:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.796+0000 7f3a06ffd640 1 -- 172.21.15.129:0/765133004 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a0c01a030 con 0x5563aa974d80 2024-05-20T18:03:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.796+0000 7f3a06ffd640 1 -- 172.21.15.129:0/765133004 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c015d20 con 0x5563aa974d80 2024-05-20T18:03:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.799+0000 7f3a2a280740 1 -- 172.21.15.129:0/765133004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa974d80 msgr2=0x5563aad558d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:30.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.799+0000 7f3a2a280740 1 --2- 172.21.15.129:0/765133004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa974d80 0x5563aad558d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0047b0 tx=0x7f3a0c009d60 comp rx=0 tx=0).stop 2024-05-20T18:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.803+0000 7f3a2a280740 1 -- 172.21.15.129:0/765133004 shutdown_connections 2024-05-20T18:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.803+0000 7f3a2a280740 1 --2- 172.21.15.129:0/765133004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa9954d0 0x5563aad75a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.803+0000 7f3a2a280740 1 --2- 172.21.15.129:0/765133004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa98d2f0 0x5563aad75500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.803+0000 7f3a2a280740 1 --2- 172.21.15.129:0/765133004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa974d80 0x5563aad558d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.803+0000 7f3a2a280740 1 -- 172.21.15.129:0/765133004 >> 172.21.15.129:0/765133004 conn(0x5563aa9aa300 msgr2=0x5563aad551c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.805+0000 7f3a2a280740 1 -- 172.21.15.129:0/765133004 shutdown_connections 2024-05-20T18:03:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.805+0000 7f3a2a280740 1 -- 172.21.15.129:0/765133004 wait complete. 2024-05-20T18:03:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.806+0000 7f3a2a280740 1 -- start start 2024-05-20T18:03:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.806+0000 7f3a2a280740 1 Processor -- start 2024-05-20T18:03:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.808+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa974d80 0x5563aad67970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.808+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa98d2f0 0x5563aad67eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.808+0000 7f3a2a280740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad62ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.809+0000 7f3a2a280740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0000d600 con 0x5563aa974d80 2024-05-20T18:03:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.809+0000 7f3a2a280740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0000ea70 con 0x5563aa9954d0 2024-05-20T18:03:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.809+0000 7f3a2a280740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0000dcd0 con 0x5563aa98d2f0 2024-05-20T18:03:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.809+0000 7f3a2a280740 10 client.0 _ll_register_callbacks cb 0x7f3a140bc800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.809+0000 7f3a21972640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad62ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.809+0000 7f3a21972640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad62ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:57938/0 (socket says 172.21.15.129:57938) 2024-05-20T18:03:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.809+0000 7f3a21972640 1 -- 172.21.15.129:0/1594845230 learned_addr learned my addr 172.21.15.129:0/1594845230 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.809+0000 7f3a20970640 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa98d2f0 0x5563aad67eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.810+0000 7f3a21972640 1 -- 172.21.15.129:0/1594845230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa98d2f0 msgr2=0x5563aad67eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.810+0000 7f3a21972640 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa98d2f0 0x5563aad67eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.810+0000 7f3a21972640 1 -- 172.21.15.129:0/1594845230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa974d80 msgr2=0x5563aad67970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.810+0000 7f3a21972640 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa974d80 0x5563aad67970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.810+0000 7f3a21972640 1 -- 172.21.15.129:0/1594845230 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563aad63670 con 0x5563aa9954d0 2024-05-20T18:03:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.812+0000 7f3a21972640 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad62ec0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3a1402f570 tx=0x7f3a14005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7927468c9e765047 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.819+0000 7f3a04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:03:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.819+0000 7f3a04ff9640 1 -- 172.21.15.129:0/1594845230 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a140bd360 con 0x5563aa9954d0 2024-05-20T18:03:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.819+0000 7f3a04ff9640 1 -- 172.21.15.129:0/1594845230 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f3a14004b40 con 0x5563aa9954d0 2024-05-20T18:03:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.819+0000 7f3a2a280740 1 -- 172.21.15.129:0/1594845230 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a0c01a030 con 0x5563aa9954d0 2024-05-20T18:03:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.823+0000 7f3a04ff9640 1 -- 172.21.15.129:0/1594845230 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3a140c1010 con 0x5563aa9954d0 2024-05-20T18:03:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.823+0000 7f3a2a280740 10 client.25916 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.823+0000 7f3a2a280740 1 -- 172.21.15.129:0/1594845230 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5563aad63670 con 0x5563aa9954d0 2024-05-20T18:03:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.828+0000 7f3a04ff9640 1 -- 172.21.15.129:0/1594845230 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a140d2040 con 0x5563aa9954d0 2024-05-20T18:03:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.828+0000 7f3a04ff9640 1 -- 172.21.15.129:0/1594845230 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f3a1409bcc0 con 0x5563aa9954d0 2024-05-20T18:03:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.828+0000 7f3a2a280740 10 client.25916 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.828+0000 7f3a2a280740 10 client.25916 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.828+0000 7f3a2a280740 20 client.25916 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.828+0000 7f3a2a280740 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5563aadd99b0 0x5563aaddbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.828+0000 7f3a2a280740 4 client.25916 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:03:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.828+0000 7f3a2a280740 1 -- 172.21.15.129:0/1594845230 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f3a0c01a030 con 0x5563aadd99b0 2024-05-20T18:03:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.830+0000 7f3a21171640 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5563aadd99b0 0x5563aaddbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.831+0000 7f3a21171640 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5563aadd99b0 0x5563aaddbe20 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c428c04e8e331be4 server_cookie=b9efc2616fc8e099 in_seq=0 out_seq=0 2024-05-20T18:03:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:30.835+0000 7f3a04ff9640 10 client.25916 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:31.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:31 smithi154 ceph-mon[36482]: pgmap v153: 97 pgs: 97 active+clean; 38 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 861 MiB/s wr, 2.84k op/s 2024-05-20T18:03:31.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:31 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:03:31.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:31 smithi129 ceph-mon[32029]: pgmap v153: 97 pgs: 97 active+clean; 38 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 861 MiB/s wr, 2.84k op/s 2024-05-20T18:03:31.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:31 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:03:31.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:31 smithi204 ceph-mon[37076]: pgmap v153: 97 pgs: 97 active+clean; 38 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 861 MiB/s wr, 2.84k op/s 2024-05-20T18:03:31.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:31 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:03:33.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:33 smithi154 ceph-mon[36482]: pgmap v154: 97 pgs: 97 active+clean; 38 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 715 MiB/s wr, 1.69k op/s 2024-05-20T18:03:33.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:33 smithi129 ceph-mon[32029]: pgmap v154: 97 pgs: 97 active+clean; 38 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 715 MiB/s wr, 1.69k op/s 2024-05-20T18:03:33.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:33 smithi204 ceph-mon[37076]: pgmap v154: 97 pgs: 97 active+clean; 38 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 715 MiB/s wr, 1.69k op/s 2024-05-20T18:03:35.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:35 smithi154 ceph-mon[36482]: pgmap v155: 97 pgs: 97 active+clean; 39 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 729 MiB/s wr, 1.58k op/s 2024-05-20T18:03:35.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:35 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:03:35.921 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:35 smithi129 ceph-mon[32029]: pgmap v155: 97 pgs: 97 active+clean; 39 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 729 MiB/s wr, 1.58k op/s 2024-05-20T18:03:35.921 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:35 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:03:35.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:35 smithi204 ceph-mon[37076]: pgmap v155: 97 pgs: 97 active+clean; 39 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 729 MiB/s wr, 1.58k op/s 2024-05-20T18:03:35.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:35 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:03:37.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:37 smithi154 ceph-mon[36482]: pgmap v156: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 903 MiB/s wr, 2.07k op/s 2024-05-20T18:03:37.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:37 smithi129 ceph-mon[32029]: pgmap v156: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 903 MiB/s wr, 2.07k op/s 2024-05-20T18:03:37.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:37 smithi204 ceph-mon[37076]: pgmap v156: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 903 MiB/s wr, 2.07k op/s 2024-05-20T18:03:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.715+0000 7f3a04ff9640 1 -- 172.21.15.129:0/1594845230 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f3a0c01a030 con 0x5563aadd99b0 2024-05-20T18:03:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.715+0000 7f3a04ff9640 10 client.25916 handle_command_reply: tid=0 2024-05-20T18:03:39.714 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.715+0000 7f3a2a280740 1 client.25916 shutdown 2024-05-20T18:03:39.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.717+0000 7f3a2a280740 1 -- 172.21.15.129:0/1594845230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 msgr2=0x5563aad62ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:39.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.717+0000 7f3a2a280740 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad62ec0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3a1402f570 tx=0x7f3a14005fd0 comp rx=0 tx=0).stop 2024-05-20T18:03:39.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.717+0000 7f3a2a280740 1 -- 172.21.15.129:0/1594845230 shutdown_connections 2024-05-20T18:03:39.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.717+0000 7f3a2a280740 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5563aadd99b0 0x5563aaddbe20 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:39.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.717+0000 7f3a2a280740 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563aa9954d0 0x5563aad62ec0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:39.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.717+0000 7f3a2a280740 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5563aa98d2f0 0x5563aad67eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:39.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.717+0000 7f3a2a280740 1 --2- 172.21.15.129:0/1594845230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5563aa974d80 0x5563aad67970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:39.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.717+0000 7f3a2a280740 1 -- 172.21.15.129:0/1594845230 >> 172.21.15.129:0/1594845230 conn(0x5563aa9aa300 msgr2=0x5563aad513f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:39.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.717+0000 7f3a04ff9640 0 client.25916 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:39.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.718+0000 7f3a2a280740 1 -- 172.21.15.129:0/1594845230 shutdown_connections 2024-05-20T18:03:39.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.718+0000 7f3a2a280740 1 -- 172.21.15.129:0/1594845230 wait complete. 2024-05-20T18:03:39.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.718+0000 7f3a2a280740 20 client.25916 trim_cache size 0 max 16384 2024-05-20T18:03:39.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:03:39.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:03:39.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:03:39.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:03:39.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.722+0000 7f3a22173640 1 -- 172.21.15.129:0/4063655312 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a080a2f00 msgr2=0x7f3a080a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:39.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.722+0000 7f3a22173640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a080a2f00 0x7f3a080a5410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3a18005fd0 tx=0x7f3a1800aa10 comp rx=0 tx=0).stop 2024-05-20T18:03:39.722 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.722+0000 7f3a22173640 1 -- 172.21.15.129:0/4063655312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a1c10f270 msgr2=0x7f3a1c1ac9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:39.722 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.722+0000 7f3a22173640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a1c10f270 0x7f3a1c1ac9f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3a1402ae60 tx=0x7f3a1402f640 comp rx=0 tx=0).stop 2024-05-20T18:03:39.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.722+0000 7f3a22173640 1 -- 172.21.15.129:0/4063655312 shutdown_connections 2024-05-20T18:03:39.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.722+0000 7f3a22173640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3a080a2f00 0x7f3a080a5410 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:39.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.722+0000 7f3a22173640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a1c10f270 0x7f3a1c1ac9f0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:39.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.722+0000 7f3a22173640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3a1c10e890 0x7f3a1c1a8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:39.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.722+0000 7f3a22173640 1 --2- 172.21.15.129:0/4063655312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c109090 0x7f3a1c1a81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:39.723 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.722+0000 7f3a22173640 1 -- 172.21.15.129:0/4063655312 >> 172.21.15.129:0/4063655312 conn(0x7f3a1c06e690 msgr2=0x7f3a1c072920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.738+0000 7f3a22173640 1 -- 172.21.15.129:0/4063655312 shutdown_connections 2024-05-20T18:03:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:39.743+0000 7f3a22173640 1 -- 172.21.15.129:0/4063655312 wait complete. 2024-05-20T18:03:39.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:39 smithi154 ceph-mon[36482]: pgmap v157: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 683 MiB/s wr, 1.68k op/s 2024-05-20T18:03:39.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:39 smithi129 ceph-mon[32029]: pgmap v157: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 683 MiB/s wr, 1.68k op/s 2024-05-20T18:03:39.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:39 smithi204 ceph-mon[37076]: pgmap v157: 97 pgs: 97 active+clean; 40 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 683 MiB/s wr, 1.68k op/s 2024-05-20T18:03:41.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:41 smithi154 ceph-mon[36482]: pgmap v158: 97 pgs: 97 active+clean; 40 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 910 MiB/s wr, 2.30k op/s 2024-05-20T18:03:41.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:41 smithi129 ceph-mon[32029]: pgmap v158: 97 pgs: 97 active+clean; 40 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 910 MiB/s wr, 2.30k op/s 2024-05-20T18:03:41.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:41 smithi204 ceph-mon[37076]: pgmap v158: 97 pgs: 97 active+clean; 40 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 910 MiB/s wr, 2.30k op/s 2024-05-20T18:03:43.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:43 smithi154 ceph-mon[36482]: pgmap v159: 97 pgs: 97 active+clean; 40 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 735 MiB/s wr, 1.95k op/s 2024-05-20T18:03:43.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:43 smithi129 ceph-mon[32029]: pgmap v159: 97 pgs: 97 active+clean; 40 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 735 MiB/s wr, 1.95k op/s 2024-05-20T18:03:43.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:43 smithi204 ceph-mon[37076]: pgmap v159: 97 pgs: 97 active+clean; 40 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 735 MiB/s wr, 1.95k op/s 2024-05-20T18:03:45.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:45 smithi154 ceph-mon[36482]: pgmap v160: 97 pgs: 97 active+clean; 40 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 744 MiB/s wr, 1.93k op/s 2024-05-20T18:03:45.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:45 smithi129 ceph-mon[32029]: pgmap v160: 97 pgs: 97 active+clean; 40 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 744 MiB/s wr, 1.93k op/s 2024-05-20T18:03:45.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:45 smithi204 ceph-mon[37076]: pgmap v160: 97 pgs: 97 active+clean; 40 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 744 MiB/s wr, 1.93k op/s 2024-05-20T18:03:47.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:47 smithi154 ceph-mon[36482]: pgmap v161: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 948 MiB/s wr, 2.32k op/s 2024-05-20T18:03:47.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:47 smithi129 ceph-mon[32029]: pgmap v161: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 948 MiB/s wr, 2.32k op/s 2024-05-20T18:03:47.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:47 smithi204 ceph-mon[37076]: pgmap v161: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 948 MiB/s wr, 2.32k op/s 2024-05-20T18:03:49.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:48 smithi154 ceph-mon[36482]: pgmap v162: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 733 MiB/s wr, 1.74k op/s 2024-05-20T18:03:49.169 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:48 smithi129 ceph-mon[32029]: pgmap v162: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 733 MiB/s wr, 1.74k op/s 2024-05-20T18:03:49.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:48 smithi204 ceph-mon[37076]: pgmap v162: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 733 MiB/s wr, 1.74k op/s 2024-05-20T18:03:51.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:50 smithi154 ceph-mon[36482]: pgmap v163: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 979 MiB/s wr, 2.55k op/s 2024-05-20T18:03:51.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:50 smithi129 ceph-mon[32029]: pgmap v163: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 979 MiB/s wr, 2.55k op/s 2024-05-20T18:03:51.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:50 smithi204 ceph-mon[37076]: pgmap v163: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 979 MiB/s wr, 2.55k op/s 2024-05-20T18:03:53.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.158+0000 7f13df6f7640 1 -- 172.21.15.129:0/1593792664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8108f70 msgr2=0x7f13d8109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:53.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.158+0000 7f13df6f7640 1 --2- 172.21.15.129:0/1593792664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8108f70 0x7f13d8109380 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009900 tx=0x7f13d002a4d0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.160+0000 7f13df6f7640 1 -- 172.21.15.129:0/1593792664 shutdown_connections 2024-05-20T18:03:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.160+0000 7f13df6f7640 1 --2- 172.21.15.129:0/1593792664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d811c0d0 0x7f13d811e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.160+0000 7f13df6f7640 1 --2- 172.21.15.129:0/1593792664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13d8119770 0x7f13d811bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.160+0000 7f13df6f7640 1 --2- 172.21.15.129:0/1593792664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8108f70 0x7f13d8109380 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.160+0000 7f13df6f7640 1 -- 172.21.15.129:0/1593792664 >> 172.21.15.129:0/1593792664 conn(0x7f13d806e600 msgr2=0x7f13d806ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:53.167 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.165+0000 7f13df6f7640 1 -- 172.21.15.129:0/1593792664 shutdown_connections 2024-05-20T18:03:53.168 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.166+0000 7f13df6f7640 1 -- 172.21.15.129:0/1593792664 wait complete. 2024-05-20T18:03:53.168 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.169+0000 7f13df6f7640 1 Processor -- start 2024-05-20T18:03:53.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.169+0000 7f13df6f7640 1 -- start start 2024-05-20T18:03:53.170 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.170+0000 7f13df6f7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d8108f70 0x7f13d81a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.170+0000 7f13df6f7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8119770 0x7f13d81a8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.170+0000 7f13df6f7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13d811c0d0 0x7f13d81accc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.171+0000 7f13df6f7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8121b80 con 0x7f13d811c0d0 2024-05-20T18:03:53.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.171+0000 7f13df6f7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8121a00 con 0x7f13d8108f70 2024-05-20T18:03:53.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.171+0000 7f13df6f7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8121d00 con 0x7f13d8119770 2024-05-20T18:03:53.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.171+0000 7f13ddc94640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13d811c0d0 0x7f13d81accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.171+0000 7f13ddc94640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13d811c0d0 0x7f13d81accc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50968/0 (socket says 172.21.15.129:50968) 2024-05-20T18:03:53.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.171+0000 7f13ddc94640 1 -- 172.21.15.129:0/1274214654 learned_addr learned my addr 172.21.15.129:0/1274214654 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:53.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.173+0000 7f13dcc92640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8119770 0x7f13d81a8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.172 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.174+0000 7f13dd493640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d8108f70 0x7f13d81a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.175+0000 7f13dcc92640 1 -- 172.21.15.129:0/1274214654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d8108f70 msgr2=0x7f13d81a84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:53.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.175+0000 7f13dcc92640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d8108f70 0x7f13d81a84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.175+0000 7f13dcc92640 1 -- 172.21.15.129:0/1274214654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13d811c0d0 msgr2=0x7f13d81accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:53.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.175+0000 7f13dcc92640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13d811c0d0 0x7f13d81accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.175+0000 7f13dcc92640 1 -- 172.21.15.129:0/1274214654 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d81ad300 con 0x7f13d8119770 2024-05-20T18:03:53.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.176+0000 7f13dcc92640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8119770 0x7f13d81a8a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00bd70 tx=0x7f13cc00be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:53.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.176+0000 7f13d67fc640 1 -- 172.21.15.129:0/1274214654 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc019070 con 0x7f13d8119770 2024-05-20T18:03:53.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.177+0000 7f13df6f7640 1 -- 172.21.15.129:0/1274214654 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d81ad5f0 con 0x7f13d8119770 2024-05-20T18:03:53.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.177+0000 7f13df6f7640 1 -- 172.21.15.129:0/1274214654 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d8110f60 con 0x7f13d8119770 2024-05-20T18:03:53.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.177+0000 7f13d67fc640 1 -- 172.21.15.129:0/1274214654 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13cc014920 con 0x7f13d8119770 2024-05-20T18:03:53.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.178+0000 7f13d67fc640 1 -- 172.21.15.129:0/1274214654 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc012940 con 0x7f13d8119770 2024-05-20T18:03:53.183 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.185+0000 7f13d67fc640 1 -- 172.21.15.129:0/1274214654 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f13cc01c070 con 0x7f13d8119770 2024-05-20T18:03:53.187 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.189+0000 7f13d67fc640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13c80a2db0 0x7f13c80a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.188 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.190+0000 7f13d67fc640 1 -- 172.21.15.129:0/1274214654 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f13cc0cfb80 con 0x7f13d8119770 2024-05-20T18:03:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.190+0000 7f13dd493640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13c80a2db0 0x7f13c80a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.190+0000 7f13e659d740 1 Processor -- start 2024-05-20T18:03:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.190+0000 7f13e659d740 1 -- start start 2024-05-20T18:03:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.191+0000 7f13e659d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76ca0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.191+0000 7f13e659d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76ca07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.191+0000 7f13e659d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76ca0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.196+0000 7f13e659d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562e76c05340 con 0x562e768d82f0 2024-05-20T18:03:53.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.196+0000 7f13e659d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562e76bad290 con 0x562e768e04d0 2024-05-20T18:03:53.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.196+0000 7f13e659d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x562e76c01300 con 0x562e769011b0 2024-05-20T18:03:53.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.197+0000 7f13dcc92640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76ca07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.197+0000 7f13dcc92640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76ca07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50976/0 (socket says 172.21.15.129:50976) 2024-05-20T18:03:53.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.197+0000 7f13dcc92640 1 -- 172.21.15.129:0/2467284768 learned_addr learned my addr 172.21.15.129:0/2467284768 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:53.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.197+0000 7f13dcc92640 1 -- 172.21.15.129:0/2467284768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 msgr2=0x562e76ca0270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:03:53.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.197+0000 7f13dcc92640 1 --2- 172.21.15.129:0/2467284768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76ca0270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.197+0000 7f13dcc92640 1 -- 172.21.15.129:0/2467284768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 msgr2=0x562e76ca0cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:03:53.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.197+0000 7f13dcc92640 1 --2- 172.21.15.129:0/2467284768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76ca0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.197+0000 7f13dcc92640 1 -- 172.21.15.129:0/2467284768 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e76bd18e0 con 0x562e768d82f0 2024-05-20T18:03:53.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.198+0000 7f13dcc92640 1 --2- 172.21.15.129:0/2467284768 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76ca07b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f13cc098c20 tx=0x7f13cc09be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb6693284919bfa server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:53.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.199+0000 7f13c37fe640 1 -- 172.21.15.129:0/2467284768 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc0b5070 con 0x562e768d82f0 2024-05-20T18:03:53.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.204+0000 7f13c37fe640 1 -- 172.21.15.129:0/2467284768 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13cc0a0550 con 0x562e768d82f0 2024-05-20T18:03:53.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.205+0000 7f13c37fe640 1 -- 172.21.15.129:0/2467284768 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc09c4b0 con 0x562e768d82f0 2024-05-20T18:03:53.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.206+0000 7f13e659d740 1 -- 172.21.15.129:0/2467284768 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 msgr2=0x562e76ca07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:53.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.206+0000 7f13e659d740 1 --2- 172.21.15.129:0/2467284768 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76ca07b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f13cc098c20 tx=0x7f13cc09be50 comp rx=0 tx=0).stop 2024-05-20T18:03:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.210+0000 7f13e659d740 1 -- 172.21.15.129:0/2467284768 shutdown_connections 2024-05-20T18:03:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.210+0000 7f13e659d740 1 --2- 172.21.15.129:0/2467284768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76ca0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.210+0000 7f13e659d740 1 --2- 172.21.15.129:0/2467284768 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76ca07b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.210+0000 7f13e659d740 1 --2- 172.21.15.129:0/2467284768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76ca0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.212 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.210+0000 7f13e659d740 1 -- 172.21.15.129:0/2467284768 >> 172.21.15.129:0/2467284768 conn(0x562e768f5300 msgr2=0x562e76c9ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:53.218 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.220+0000 7f13e659d740 1 -- 172.21.15.129:0/2467284768 shutdown_connections 2024-05-20T18:03:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.225+0000 7f13e659d740 1 -- 172.21.15.129:0/2467284768 wait complete. 2024-05-20T18:03:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.226+0000 7f13e659d740 1 -- start start 2024-05-20T18:03:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.226+0000 7f13e659d740 1 Processor -- start 2024-05-20T18:03:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.228+0000 7f13e659d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76caa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.228+0000 7f13e659d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76caaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.228+0000 7f13e659d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76cb08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.228+0000 7f13e659d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562e76cad3a0 con 0x562e768d82f0 2024-05-20T18:03:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.228+0000 7f13e659d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562e76bc0790 con 0x562e768e04d0 2024-05-20T18:03:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.228+0000 7f13e659d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x562e76cad520 con 0x562e769011b0 2024-05-20T18:03:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.228+0000 7f13e659d740 10 client.0 _ll_register_callbacks cb 0x562e76bfba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.229+0000 7f13ddc94640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76cb08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.230+0000 7f13dcc92640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76caaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.230+0000 7f13dcc92640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76caaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38214/0 (socket says 172.21.15.129:38214) 2024-05-20T18:03:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.230+0000 7f13dcc92640 1 -- 172.21.15.129:0/3779311941 learned_addr learned my addr 172.21.15.129:0/3779311941 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.231+0000 7f13dd493640 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76caa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.232+0000 7f13ddc94640 1 -- 172.21.15.129:0/3779311941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 msgr2=0x562e76caaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.232+0000 7f13ddc94640 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76caaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.232+0000 7f13ddc94640 1 -- 172.21.15.129:0/3779311941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 msgr2=0x562e76caa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.232+0000 7f13ddc94640 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76caa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.232+0000 7f13ddc94640 1 -- 172.21.15.129:0/3779311941 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e76919210 con 0x562e769011b0 2024-05-20T18:03:53.243 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.241+0000 7f13ddc94640 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76cb08f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f13c400e4b0 tx=0x7f13c400e970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4549af42bbfcfe64 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.247+0000 7f13c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:03:53.245 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.247+0000 7f13c0ff9640 1 -- 172.21.15.129:0/3779311941 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4017070 con 0x562e769011b0 2024-05-20T18:03:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.247+0000 7f13e659d740 1 -- 172.21.15.129:0/3779311941 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13cc0a0550 con 0x562e769011b0 2024-05-20T18:03:53.246 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.248+0000 7f13c0ff9640 1 -- 172.21.15.129:0/3779311941 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f13c4013680 con 0x562e769011b0 2024-05-20T18:03:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.249+0000 7f13c0ff9640 1 -- 172.21.15.129:0/3779311941 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f13c4023050 con 0x562e769011b0 2024-05-20T18:03:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.249+0000 7f13e659d740 10 client.25954 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:53.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.249+0000 7f13e659d740 1 -- 172.21.15.129:0/3779311941 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562e769e5620 con 0x562e769011b0 2024-05-20T18:03:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.250+0000 7f13c0ff9640 1 -- 172.21.15.129:0/3779311941 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13cc0a0550 con 0x562e769011b0 2024-05-20T18:03:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.251+0000 7f13c0ff9640 1 -- 172.21.15.129:0/3779311941 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f13c4012a20 con 0x562e769011b0 2024-05-20T18:03:53.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.251+0000 7f13e659d740 10 client.25954 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.251+0000 7f13e659d740 10 client.25954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.251+0000 7f13e659d740 20 client.25954 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.251+0000 7f13e659d740 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562e768bfd80 0x562e76cbcee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.251+0000 7f13e659d740 4 client.25954 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:03:53.250 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.251+0000 7f13e659d740 1 -- 172.21.15.129:0/3779311941 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e76c12a70 con 0x562e768bfd80 2024-05-20T18:03:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.255+0000 7f13dd493640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13c80a2db0 0x7f13c80a52c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x562e76c9d3e0 tx=0x7f13d0000bc0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.256+0000 7f13dd493640 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562e768bfd80 0x562e76cbcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.258+0000 7f13dd493640 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562e768bfd80 0x562e76cbcee0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e1c4e23502b735c server_cookie=45e51bb17167241a in_seq=0 out_seq=0 2024-05-20T18:03:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.258+0000 7f13c0ff9640 10 client.25954 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.282+0000 7f13c0ff9640 1 -- 172.21.15.129:0/3779311941 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562e76c12a70 con 0x562e768bfd80 2024-05-20T18:03:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.282+0000 7f13c0ff9640 10 client.25954 handle_command_reply: tid=0 2024-05-20T18:03:53.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.282+0000 7f13e659d740 1 client.25954 shutdown 2024-05-20T18:03:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.282+0000 7f13e659d740 1 -- 172.21.15.129:0/3779311941 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 msgr2=0x562e76cb08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.282+0000 7f13e659d740 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76cb08f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f13c400e4b0 tx=0x7f13c400e970 comp rx=0 tx=0).stop 2024-05-20T18:03:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.287+0000 7f13e659d740 1 -- 172.21.15.129:0/3779311941 shutdown_connections 2024-05-20T18:03:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.287+0000 7f13e659d740 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562e768bfd80 0x562e76cbcee0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.287+0000 7f13e659d740 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76caaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.287+0000 7f13e659d740 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76cb08f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.287+0000 7f13e659d740 1 --2- 172.21.15.129:0/3779311941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76caa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.287+0000 7f13e659d740 1 -- 172.21.15.129:0/3779311941 >> 172.21.15.129:0/3779311941 conn(0x562e768f5300 msgr2=0x562e76ca0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.290+0000 7f13c0ff9640 0 client.25954 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.294+0000 7f13e659d740 1 -- 172.21.15.129:0/3779311941 shutdown_connections 2024-05-20T18:03:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.294+0000 7f13e659d740 1 -- 172.21.15.129:0/3779311941 wait complete. 2024-05-20T18:03:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.298+0000 7f13e659d740 20 client.25954 trim_cache size 0 max 16384 2024-05-20T18:03:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.374+0000 7f13e659d740 1 Processor -- start 2024-05-20T18:03:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.380+0000 7f13e659d740 1 -- start start 2024-05-20T18:03:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.381+0000 7f13e659d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e768d82f0 0x562e76cc26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.381+0000 7f13e659d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76cc2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.381+0000 7f13e659d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e769011b0 0x562e76cb07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.381+0000 7f13e659d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc00c860 con 0x562e769011b0 2024-05-20T18:03:53.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.381+0000 7f13e659d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc00f0e0 con 0x562e768e04d0 2024-05-20T18:03:53.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.381+0000 7f13e659d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc00ced0 con 0x562e768d82f0 2024-05-20T18:03:53.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.382+0000 7f13dcc92640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76cc2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.382+0000 7f13dcc92640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76cc2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38226/0 (socket says 172.21.15.129:38226) 2024-05-20T18:03:53.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.382+0000 7f13dcc92640 1 -- 172.21.15.129:0/3935092249 learned_addr learned my addr 172.21.15.129:0/3935092249 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.382+0000 7f13dd493640 1 --2- 172.21.15.129:0/3935092249 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e768d82f0 0x562e76cc26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.383+0000 7f13dcc92640 1 -- 172.21.15.129:0/3935092249 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e768d82f0 msgr2=0x562e76cc26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.383+0000 7f13dcc92640 1 --2- 172.21.15.129:0/3935092249 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e768d82f0 0x562e76cc26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.383+0000 7f13dcc92640 1 -- 172.21.15.129:0/3935092249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e769011b0 msgr2=0x562e76cb07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.383+0000 7f13dcc92640 1 --2- 172.21.15.129:0/3935092249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e769011b0 0x562e76cb07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.383+0000 7f13dcc92640 1 -- 172.21.15.129:0/3935092249 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e76cab650 con 0x562e768e04d0 2024-05-20T18:03:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.383+0000 7f13dcc92640 1 --2- 172.21.15.129:0/3935092249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76cc2be0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f13cc098c20 tx=0x7f13cc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2e6580c15a762c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:53.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:53 smithi154 ceph-mon[36482]: pgmap v164: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 753 MiB/s wr, 1.97k op/s 2024-05-20T18:03:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.408+0000 7f13c2ffd640 1 -- 172.21.15.129:0/3935092249 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc0ba020 con 0x562e768e04d0 2024-05-20T18:03:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.408+0000 7f13c2ffd640 1 -- 172.21.15.129:0/3935092249 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13cc0a0550 con 0x562e768e04d0 2024-05-20T18:03:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.411+0000 7f13e659d740 1 -- 172.21.15.129:0/3935092249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 msgr2=0x562e76cc2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.411+0000 7f13e659d740 1 --2- 172.21.15.129:0/3935092249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76cc2be0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f13cc098c20 tx=0x7f13cc004a40 comp rx=0 tx=0).stop 2024-05-20T18:03:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.413+0000 7f13e659d740 1 -- 172.21.15.129:0/3935092249 shutdown_connections 2024-05-20T18:03:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.413+0000 7f13e659d740 1 --2- 172.21.15.129:0/3935092249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76cc2be0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.413+0000 7f13e659d740 1 --2- 172.21.15.129:0/3935092249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e769011b0 0x562e76cb07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.413+0000 7f13e659d740 1 --2- 172.21.15.129:0/3935092249 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e768d82f0 0x562e76cc26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.413+0000 7f13e659d740 1 -- 172.21.15.129:0/3935092249 >> 172.21.15.129:0/3935092249 conn(0x562e768f5300 msgr2=0x562e76ca9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:53.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:53 smithi129 ceph-mon[32029]: pgmap v164: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 753 MiB/s wr, 1.97k op/s 2024-05-20T18:03:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.414+0000 7f13e659d740 1 -- 172.21.15.129:0/3935092249 shutdown_connections 2024-05-20T18:03:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.414+0000 7f13e659d740 1 -- 172.21.15.129:0/3935092249 wait complete. 2024-05-20T18:03:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.415+0000 7f13e659d740 1 -- start start 2024-05-20T18:03:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.415+0000 7f13e659d740 1 Processor -- start 2024-05-20T18:03:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.417+0000 7f13e659d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76cb13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.417+0000 7f13e659d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76cb1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.417+0000 7f13e659d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76cb1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.417+0000 7f13e659d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc00d5a0 con 0x562e768d82f0 2024-05-20T18:03:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.417+0000 7f13e659d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc00ea10 con 0x562e768e04d0 2024-05-20T18:03:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.417+0000 7f13e659d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13bc00dc70 con 0x562e769011b0 2024-05-20T18:03:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.417+0000 7f13e659d740 10 client.0 _ll_register_callbacks cb 0x7f13c40093e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.418+0000 7f13dd493640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76cb13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.418+0000 7f13dd493640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76cb13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50996/0 (socket says 172.21.15.129:50996) 2024-05-20T18:03:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.418+0000 7f13dd493640 1 -- 172.21.15.129:0/147238080 learned_addr learned my addr 172.21.15.129:0/147238080 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.420+0000 7f13ddc94640 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76cb1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.418 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.420+0000 7f13dcc92640 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76cb1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.423 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.425+0000 7f13dcc92640 1 -- 172.21.15.129:0/147238080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 msgr2=0x562e76cb1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:53.423 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.425+0000 7f13dcc92640 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76cb1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.423 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.425+0000 7f13dcc92640 1 -- 172.21.15.129:0/147238080 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 msgr2=0x562e76cb13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:53.423 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.425+0000 7f13dcc92640 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76cb13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:53.423 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.425+0000 7f13dcc92640 1 -- 172.21.15.129:0/147238080 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e76d233c0 con 0x562e768e04d0 2024-05-20T18:03:53.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:53 smithi204 ceph-mon[37076]: pgmap v164: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 753 MiB/s wr, 1.97k op/s 2024-05-20T18:03:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.431+0000 7f13dcc92640 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76cb1930 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f13cc004b90 tx=0x7f13cc0a0380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a8c19e75dc1504d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.431+0000 7f13c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:03:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.433+0000 7f13c0ff9640 1 -- 172.21.15.129:0/147238080 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc0ba070 con 0x562e768e04d0 2024-05-20T18:03:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.433+0000 7f13c0ff9640 1 -- 172.21.15.129:0/147238080 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f13cc012bc0 con 0x562e768e04d0 2024-05-20T18:03:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.435+0000 7f13e659d740 1 -- 172.21.15.129:0/147238080 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13cc0a0550 con 0x562e768e04d0 2024-05-20T18:03:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.436+0000 7f13c0ff9640 1 -- 172.21.15.129:0/147238080 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f13cc012dc0 con 0x562e768e04d0 2024-05-20T18:03:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.437+0000 7f13e659d740 10 client.25928 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.437+0000 7f13e659d740 1 -- 172.21.15.129:0/147238080 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562e76d24610 con 0x562e768e04d0 2024-05-20T18:03:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.438+0000 7f13c0ff9640 1 -- 172.21.15.129:0/147238080 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13cc0a0550 con 0x562e768e04d0 2024-05-20T18:03:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.438+0000 7f13c0ff9640 1 -- 172.21.15.129:0/147238080 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f13cc081020 con 0x562e768e04d0 2024-05-20T18:03:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.440+0000 7f13e659d740 10 client.25928 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.440+0000 7f13e659d740 10 client.25928 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:53.438 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.440+0000 7f13e659d740 20 client.25928 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.440+0000 7f13e659d740 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562e768951c0 0x562e76d267f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.440+0000 7f13e659d740 4 client.25928 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:03:53.439 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.440+0000 7f13e659d740 1 -- 172.21.15.129:0/147238080 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e76c0cdc0 con 0x562e768951c0 2024-05-20T18:03:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.442+0000 7f13dd493640 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562e768951c0 0x562e76d267f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.445+0000 7f13dd493640 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562e768951c0 0x562e76d267f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd9bcaba87c8d917 server_cookie=bff1b5d8abef6dce in_seq=0 out_seq=0 2024-05-20T18:03:53.443 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:53.445+0000 7f13c0ff9640 10 client.25928 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:55.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:55 smithi154 ceph-mon[36482]: pgmap v165: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 764 MiB/s wr, 1.95k op/s 2024-05-20T18:03:55.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:55 smithi129 ceph-mon[32029]: pgmap v165: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 764 MiB/s wr, 1.95k op/s 2024-05-20T18:03:55.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:55 smithi204 ceph-mon[37076]: pgmap v165: 97 pgs: 97 active+clean; 40 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 764 MiB/s wr, 1.95k op/s 2024-05-20T18:03:56.497 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.499+0000 7f13c0ff9640 1 -- 172.21.15.129:0/147238080 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e76c0cdc0 con 0x562e768951c0 2024-05-20T18:03:56.497 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.499+0000 7f13c0ff9640 10 client.25928 handle_command_reply: tid=0 2024-05-20T18:03:56.497 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.499+0000 7f13e659d740 1 client.25928 shutdown 2024-05-20T18:03:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.507+0000 7f13e659d740 1 -- 172.21.15.129:0/147238080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 msgr2=0x562e76cb1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.508+0000 7f13e659d740 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76cb1930 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f13cc004b90 tx=0x7f13cc0a0380 comp rx=0 tx=0).stop 2024-05-20T18:03:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.508+0000 7f13e659d740 1 -- 172.21.15.129:0/147238080 shutdown_connections 2024-05-20T18:03:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.508+0000 7f13e659d740 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562e768951c0 0x562e76d267f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.508+0000 7f13e659d740 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562e768e04d0 0x562e76cb1930 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.508+0000 7f13e659d740 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562e769011b0 0x562e76cb1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.508+0000 7f13e659d740 1 --2- 172.21.15.129:0/147238080 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562e768d82f0 0x562e76cb13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.508+0000 7f13e659d740 1 -- 172.21.15.129:0/147238080 >> 172.21.15.129:0/147238080 conn(0x562e768f5300 msgr2=0x562e76caa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.508+0000 7f13c0ff9640 0 client.25928 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:56.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.519+0000 7f13e659d740 1 -- 172.21.15.129:0/147238080 shutdown_connections 2024-05-20T18:03:56.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.519+0000 7f13e659d740 1 -- 172.21.15.129:0/147238080 wait complete. 2024-05-20T18:03:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.519+0000 7f13e659d740 20 client.25928 trim_cache size 0 max 16384 2024-05-20T18:03:56.519 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:03:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:03:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:03:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:03:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.537+0000 7f13df6f7640 1 -- 172.21.15.129:0/1274214654 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13c80a2db0 msgr2=0x7f13c80a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.537+0000 7f13df6f7640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13c80a2db0 0x7f13c80a52c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x562e76c9d3e0 tx=0x7f13d0000bc0 comp rx=0 tx=0).stop 2024-05-20T18:03:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.537+0000 7f13df6f7640 1 -- 172.21.15.129:0/1274214654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8119770 msgr2=0x7f13d81a8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.537+0000 7f13df6f7640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8119770 0x7f13d81a8a00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00bd70 tx=0x7f13cc00be60 comp rx=0 tx=0).stop 2024-05-20T18:03:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.542+0000 7f13df6f7640 1 -- 172.21.15.129:0/1274214654 shutdown_connections 2024-05-20T18:03:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.542+0000 7f13df6f7640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13c80a2db0 0x7f13c80a52c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.542+0000 7f13df6f7640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13d811c0d0 0x7f13d81accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:56.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.542+0000 7f13df6f7640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13d8119770 0x7f13d81a8a00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.542+0000 7f13df6f7640 1 --2- 172.21.15.129:0/1274214654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d8108f70 0x7f13d81a84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.542+0000 7f13df6f7640 1 -- 172.21.15.129:0/1274214654 >> 172.21.15.129:0/1274214654 conn(0x7f13d806e600 msgr2=0x7f13d8119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.542+0000 7f13df6f7640 1 -- 172.21.15.129:0/1274214654 shutdown_connections 2024-05-20T18:03:56.541 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:03:56.542+0000 7f13df6f7640 1 -- 172.21.15.129:0/1274214654 wait complete. 2024-05-20T18:03:57.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:57 smithi154 ceph-mon[36482]: pgmap v166: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 948 MiB/s wr, 2.23k op/s 2024-05-20T18:03:57.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:57 smithi129 ceph-mon[32029]: pgmap v166: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 948 MiB/s wr, 2.23k op/s 2024-05-20T18:03:57.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:57 smithi204 ceph-mon[37076]: pgmap v166: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 948 MiB/s wr, 2.23k op/s 2024-05-20T18:03:57.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:03:57.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:03:58.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.415+0000 7f5bb72ae640 1 -- 172.21.15.129:0/2846815380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bb0109740 msgr2=0x7f5bb0079e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:58.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.415+0000 7f5bb72ae640 1 --2- 172.21.15.129:0/2846815380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bb0109740 0x7f5bb0079e70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800afd0 tx=0x7f5ba802c470 comp rx=0 tx=0).stop 2024-05-20T18:03:58.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.415+0000 7f5bb72ae640 1 -- 172.21.15.129:0/2846815380 shutdown_connections 2024-05-20T18:03:58.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.415+0000 7f5bb72ae640 1 --2- 172.21.15.129:0/2846815380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bb0109740 0x7f5bb0079e70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.415+0000 7f5bb72ae640 1 --2- 172.21.15.129:0/2846815380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb010e210 0x7f5bb01090b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.415+0000 7f5bb72ae640 1 --2- 172.21.15.129:0/2846815380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bb010de00 0x7f5bb0108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.415+0000 7f5bb72ae640 1 -- 172.21.15.129:0/2846815380 >> 172.21.15.129:0/2846815380 conn(0x7f5bb006d580 msgr2=0x7f5bb006d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:58.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.439+0000 7f5bb72ae640 1 -- 172.21.15.129:0/2846815380 shutdown_connections 2024-05-20T18:03:58.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.445+0000 7f5bb72ae640 1 -- 172.21.15.129:0/2846815380 wait complete. 2024-05-20T18:03:58.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.446+0000 7f5bb72ae640 1 Processor -- start 2024-05-20T18:03:58.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.447+0000 7f5bb72ae640 1 -- start start 2024-05-20T18:03:58.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.447+0000 7f5bb72ae640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bb010de00 0x7f5bb007ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.447+0000 7f5bb72ae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bb010e210 0x7f5bb007f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.447+0000 7f5bb72ae640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb007f7e0 0x7f5bb01c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.447+0000 7f5bb72ae640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb007d640 con 0x7f5bb007f7e0 2024-05-20T18:03:58.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.447+0000 7f5bb72ae640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb007d4c0 con 0x7f5bb010e210 2024-05-20T18:03:58.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.447+0000 7f5bb72ae640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb007d7c0 con 0x7f5bb010de00 2024-05-20T18:03:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.448+0000 7f5bb6aad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb007f7e0 0x7f5bb01c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.448+0000 7f5bb6aad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb007f7e0 0x7f5bb01c4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51016/0 (socket says 172.21.15.129:51016) 2024-05-20T18:03:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.448+0000 7f5bb6aad640 1 -- 172.21.15.129:0/1934977296 learned_addr learned my addr 172.21.15.129:0/1934977296 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.449+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bb010de00 0x7f5bb007ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.452+0000 7f5bb62ac640 1 -- 172.21.15.129:0/1934977296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bb010e210 msgr2=0x7f5bb007f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.452+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bb010e210 0x7f5bb007f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.452+0000 7f5bb62ac640 1 -- 172.21.15.129:0/1934977296 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb007f7e0 msgr2=0x7f5bb01c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.452+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb007f7e0 0x7f5bb01c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.452+0000 7f5bb62ac640 1 -- 172.21.15.129:0/1934977296 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb01b6740 con 0x7f5bb010de00 2024-05-20T18:03:58.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.459+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bb010de00 0x7f5bb007ed60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00e9f0 tx=0x7f5bac00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:58.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.466+0000 7f5ba77fe640 1 -- 172.21.15.129:0/1934977296 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac00cc40 con 0x7f5bb010de00 2024-05-20T18:03:58.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.466+0000 7f5ba77fe640 1 -- 172.21.15.129:0/1934977296 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bac00cde0 con 0x7f5bb010de00 2024-05-20T18:03:58.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.466+0000 7f5bb72ae640 1 -- 172.21.15.129:0/1934977296 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb01b6a30 con 0x7f5bb010de00 2024-05-20T18:03:58.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.466+0000 7f5bb72ae640 1 -- 172.21.15.129:0/1934977296 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb01b6f80 con 0x7f5bb010de00 2024-05-20T18:03:58.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.473+0000 7f5ba77fe640 1 -- 172.21.15.129:0/1934977296 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac018920 con 0x7f5bb010de00 2024-05-20T18:03:58.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.474+0000 7f5bbf3b8740 1 Processor -- start 2024-05-20T18:03:58.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.480+0000 7f5ba77fe640 1 -- 172.21.15.129:0/1934977296 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5bac018b80 con 0x7f5bb010de00 2024-05-20T18:03:58.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.489+0000 7f5bbf3b8740 1 -- start start 2024-05-20T18:03:58.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.490+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde704390 0x560bdeaa2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.490+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 0x560bdeaa3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.490+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde6e36b0 0x560bdeaa39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.490+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560bdea08d00 con 0x560bde6e36b0 2024-05-20T18:03:58.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.490+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560bde9b04c0 con 0x560bde6db4d0 2024-05-20T18:03:58.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.490+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560bdea04cc0 con 0x560bde704390 2024-05-20T18:03:58.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.491+0000 7f5bb62ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 0x560bdeaa3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.491+0000 7f5bb62ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 0x560bdeaa3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38258/0 (socket says 172.21.15.129:38258) 2024-05-20T18:03:58.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.491+0000 7f5bb62ac640 1 -- 172.21.15.129:0/1625161187 learned_addr learned my addr 172.21.15.129:0/1625161187 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:58.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.492+0000 7f5ba77fe640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5b9c0a2f00 0x7f5b9c0a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.492+0000 7f5bb62ac640 1 -- 172.21.15.129:0/1625161187 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde704390 msgr2=0x560bdeaa2f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:03:58.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.492+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/1625161187 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde704390 0x560bdeaa2f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.492+0000 7f5bb62ac640 1 -- 172.21.15.129:0/1625161187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde6e36b0 msgr2=0x560bdeaa39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:58.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.492+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/1625161187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde6e36b0 0x560bdeaa39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.492+0000 7f5bb62ac640 1 -- 172.21.15.129:0/1625161187 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bde7e67d0 con 0x560bde6db4d0 2024-05-20T18:03:58.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.505+0000 7f5ba77fe640 1 -- 172.21.15.129:0/1934977296 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f5bac0cf580 con 0x7f5bb010de00 2024-05-20T18:03:58.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.510+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/1625161187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 0x560bdeaa3470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5bac0ccce0 tx=0x7f5bac0afca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc1975526667ef7b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:58.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.512+0000 7f5bb6aad640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5b9c0a2f00 0x7f5b9c0a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.524+0000 7f5bb6aad640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5b9c0a2f00 0x7f5b9c0a5410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800f2a0 tx=0x7f5ba800e040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:58.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.525+0000 7f5ba4bfe640 1 -- 172.21.15.129:0/1625161187 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac09b910 con 0x560bde6db4d0 2024-05-20T18:03:58.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.525+0000 7f5ba4bfe640 1 -- 172.21.15.129:0/1625161187 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bac0c4040 con 0x560bde6db4d0 2024-05-20T18:03:58.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.525+0000 7f5ba4bfe640 1 -- 172.21.15.129:0/1625161187 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac09b4a0 con 0x560bde6db4d0 2024-05-20T18:03:58.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.526+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/1625161187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 msgr2=0x560bdeaa3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:58.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.526+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/1625161187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 0x560bdeaa3470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5bac0ccce0 tx=0x7f5bac0afca0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.527+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/1625161187 shutdown_connections 2024-05-20T18:03:58.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.527+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/1625161187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 0x560bdeaa3470 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.527+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/1625161187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde6e36b0 0x560bdeaa39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.527+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/1625161187 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde704390 0x560bdeaa2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.527+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/1625161187 >> 172.21.15.129:0/1625161187 conn(0x560bde6f84e0 msgr2=0x560bdea9f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:58.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.529+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/1625161187 shutdown_connections 2024-05-20T18:03:58.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.535+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/1625161187 wait complete. 2024-05-20T18:03:58.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.536+0000 7f5bbf3b8740 1 -- start start 2024-05-20T18:03:58.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.536+0000 7f5bbf3b8740 1 Processor -- start 2024-05-20T18:03:58.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.537+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 0x560bdeaad590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.537+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde6e36b0 0x560bdeaadad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.537+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde704390 0x560bdeab3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.537+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560bdeab0280 con 0x560bde6e36b0 2024-05-20T18:03:58.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.537+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560bde9c3b90 con 0x560bde6db4d0 2024-05-20T18:03:58.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.537+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560bdeab0400 con 0x560bde704390 2024-05-20T18:03:58.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.537+0000 7f5bbf3b8740 10 client.0 _ll_register_callbacks cb 0x560bde9ffda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:58.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.538+0000 7f5bb62ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 0x560bdeaad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.538+0000 7f5bb62ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 0x560bdeaad590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38272/0 (socket says 172.21.15.129:38272) 2024-05-20T18:03:58.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.538+0000 7f5bb62ac640 1 -- 172.21.15.129:0/3267503200 learned_addr learned my addr 172.21.15.129:0/3267503200 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:58.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.538+0000 7f5bb6aad640 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde704390 0x560bdeab3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.539+0000 7f5bb6aad640 1 -- 172.21.15.129:0/3267503200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 msgr2=0x560bdeaad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:58.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.539+0000 7f5bb6aad640 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 0x560bdeaad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.539+0000 7f5bb6aad640 1 -- 172.21.15.129:0/3267503200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde6e36b0 msgr2=0x560bdeaadad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:58.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.539+0000 7f5bb6aad640 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde6e36b0 0x560bdeaadad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.539+0000 7f5bb6aad640 1 -- 172.21.15.129:0/3267503200 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bde71c3f0 con 0x560bde704390 2024-05-20T18:03:58.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.550+0000 7f5bb6aad640 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde704390 0x560bdeab3790 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x560bdeaa00a0 tx=0x7f5ba80074e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb6b4c7d352b7447 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:58.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.550+0000 7f5b99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:03:58.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.554+0000 7f5b99ffb640 1 -- 172.21.15.129:0/3267503200 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8044040 con 0x560bde704390 2024-05-20T18:03:58.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.554+0000 7f5b99ffb640 1 -- 172.21.15.129:0/3267503200 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f5ba8004180 con 0x560bde704390 2024-05-20T18:03:58.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.557+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/3267503200 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bac0c4040 con 0x560bde704390 2024-05-20T18:03:58.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.557+0000 7f5b99ffb640 1 -- 172.21.15.129:0/3267503200 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5ba8038040 con 0x560bde704390 2024-05-20T18:03:58.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.559+0000 7f5bbf3b8740 10 client.25969 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:58.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.559+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/3267503200 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560bde6dc0f0 con 0x560bde704390 2024-05-20T18:03:58.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.560+0000 7f5b99ffb640 1 -- 172.21.15.129:0/3267503200 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bac0c4040 con 0x560bde704390 2024-05-20T18:03:58.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.560+0000 7f5b99ffb640 1 -- 172.21.15.129:0/3267503200 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f5ba803b030 con 0x560bde704390 2024-05-20T18:03:58.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.560+0000 7f5bbf3b8740 10 client.25969 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:58.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.560+0000 7f5bbf3b8740 10 client.25969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:58.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.560+0000 7f5bbf3b8740 20 client.25969 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:58.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.560+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560bde6c2f60 0x560bdeac0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.560+0000 7f5bbf3b8740 4 client.25969 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:03:58.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.560+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/3267503200 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bdea16430 con 0x560bde6c2f60 2024-05-20T18:03:58.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.564+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560bde6c2f60 0x560bdeac0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.570+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560bde6c2f60 0x560bdeac0300 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb6116b438b26f9b server_cookie=8c175b7b7292ff71 in_seq=0 out_seq=0 2024-05-20T18:03:58.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.571+0000 7f5b99ffb640 10 client.25969 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:58.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.579+0000 7f5b99ffb640 1 -- 172.21.15.129:0/3267503200 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560bdea16430 con 0x560bde6c2f60 2024-05-20T18:03:58.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.579+0000 7f5b99ffb640 10 client.25969 handle_command_reply: tid=0 2024-05-20T18:03:58.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.584+0000 7f5bbf3b8740 1 client.25969 shutdown 2024-05-20T18:03:58.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.585+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/3267503200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde704390 msgr2=0x560bdeab3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:58.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.585+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde704390 0x560bdeab3790 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x560bdeaa00a0 tx=0x7f5ba80074e0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.590+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/3267503200 shutdown_connections 2024-05-20T18:03:58.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.590+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560bde6c2f60 0x560bdeac0300 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.590+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde704390 0x560bdeab3790 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.590+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde6e36b0 0x560bdeaadad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.590+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/3267503200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6db4d0 0x560bdeaad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.590+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/3267503200 >> 172.21.15.129:0/3267503200 conn(0x560bde6f84e0 msgr2=0x560bdeaa2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:58.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.591+0000 7f5b99ffb640 0 client.25969 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:58.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.591+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/3267503200 shutdown_connections 2024-05-20T18:03:58.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.593+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/3267503200 wait complete. 2024-05-20T18:03:58.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.595+0000 7f5bbf3b8740 20 client.25969 trim_cache size 0 max 16384 2024-05-20T18:03:58.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.663+0000 7f5bbf3b8740 1 Processor -- start 2024-05-20T18:03:58.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.665+0000 7f5bbf3b8740 1 -- start start 2024-05-20T18:03:58.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.666+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde6db4d0 0x560bdeaacfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.666+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 0x560bdeac5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.666+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 0x560bdeac6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.666+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9400c870 con 0x560bde704390 2024-05-20T18:03:58.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.666+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9400f0f0 con 0x560bde6e36b0 2024-05-20T18:03:58.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.666+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9400cee0 con 0x560bde6db4d0 2024-05-20T18:03:58.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.666+0000 7f5bb6aad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 0x560bdeac6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.666+0000 7f5bb6aad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 0x560bdeac6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51046/0 (socket says 172.21.15.129:51046) 2024-05-20T18:03:58.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.667+0000 7f5bb6aad640 1 -- 172.21.15.129:0/2548807130 learned_addr learned my addr 172.21.15.129:0/2548807130 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:58.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.667+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/2548807130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde6db4d0 0x560bdeaacfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.668+0000 7f5bb5aab640 1 --2- 172.21.15.129:0/2548807130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 0x560bdeac5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.672+0000 7f5bb6aad640 1 -- 172.21.15.129:0/2548807130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde6db4d0 msgr2=0x560bdeaacfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:58.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.672+0000 7f5bb6aad640 1 --2- 172.21.15.129:0/2548807130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde6db4d0 0x560bdeaacfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.672+0000 7f5bb6aad640 1 -- 172.21.15.129:0/2548807130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 msgr2=0x560bdeac5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:58.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.672+0000 7f5bb6aad640 1 --2- 172.21.15.129:0/2548807130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 0x560bdeac5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.672+0000 7f5bb6aad640 1 -- 172.21.15.129:0/2548807130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bdeaac680 con 0x560bde704390 2024-05-20T18:03:58.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.675+0000 7f5bb6aad640 1 --2- 172.21.15.129:0/2548807130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 0x560bdeac6040 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8048b40 tx=0x7f5ba8049f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2748b511960ab server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:58.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.682+0000 7f5b9bfff640 1 -- 172.21.15.129:0/2548807130 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8044040 con 0x560bde704390 2024-05-20T18:03:58.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.683+0000 7f5b9bfff640 1 -- 172.21.15.129:0/2548807130 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ba804d070 con 0x560bde704390 2024-05-20T18:03:58.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.683+0000 7f5b9bfff640 1 -- 172.21.15.129:0/2548807130 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8052450 con 0x560bde704390 2024-05-20T18:03:58.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.687+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/2548807130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 msgr2=0x560bdeac6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:58.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.687+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/2548807130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 0x560bdeac6040 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8048b40 tx=0x7f5ba8049f80 comp rx=0 tx=0).stop 2024-05-20T18:03:58.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.687+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/2548807130 shutdown_connections 2024-05-20T18:03:58.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.687+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/2548807130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 0x560bdeac5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.687+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/2548807130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 0x560bdeac6040 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.687+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/2548807130 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde6db4d0 0x560bdeaacfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.687+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/2548807130 >> 172.21.15.129:0/2548807130 conn(0x560bde6f84e0 msgr2=0x560bdeaac8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:58.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.693+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/2548807130 shutdown_connections 2024-05-20T18:03:58.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.693+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/2548807130 wait complete. 2024-05-20T18:03:58.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.694+0000 7f5bbf3b8740 1 -- start start 2024-05-20T18:03:58.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.696+0000 7f5bbf3b8740 1 Processor -- start 2024-05-20T18:03:58.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.697+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde6db4d0 0x560bdeab6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.697+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 0x560bdeab6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.697+0000 7f5bbf3b8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 0x560bdeab71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.698+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9400d5b0 con 0x560bde704390 2024-05-20T18:03:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.698+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9400ea20 con 0x560bde6e36b0 2024-05-20T18:03:58.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.699+0000 7f5bb5aab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 0x560bdeab6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.699+0000 7f5bbf3b8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9400dc80 con 0x560bde6db4d0 2024-05-20T18:03:58.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.699+0000 7f5bbf3b8740 10 client.0 _ll_register_callbacks cb 0x7f5ba8041040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:03:58.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.699+0000 7f5bb5aab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 0x560bdeab6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38278/0 (socket says 172.21.15.129:38278) 2024-05-20T18:03:58.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.699+0000 7f5bb5aab640 1 -- 172.21.15.129:0/258975190 learned_addr learned my addr 172.21.15.129:0/258975190 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:58.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.700+0000 7f5bb6aad640 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 0x560bdeab71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.700+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde6db4d0 0x560bdeab6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.701+0000 7f5bb5aab640 1 -- 172.21.15.129:0/258975190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde6db4d0 msgr2=0x560bdeab6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:58.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.701+0000 7f5bb5aab640 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde6db4d0 0x560bdeab6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.701+0000 7f5bb5aab640 1 -- 172.21.15.129:0/258975190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 msgr2=0x560bdeab71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:03:58.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.701+0000 7f5bb5aab640 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 0x560bdeab71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.701+0000 7f5bb5aab640 1 -- 172.21.15.129:0/258975190 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bdeb266b0 con 0x560bde6e36b0 2024-05-20T18:03:58.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.701+0000 7f5bb5aab640 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 0x560bdeab6cb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0073660 tx=0x7f5ba0003d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7829a5318b93d84f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:58.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.701+0000 7f5b99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:03:58.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.701+0000 7f5b99ffb640 1 -- 172.21.15.129:0/258975190 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba000bde0 con 0x560bde6e36b0 2024-05-20T18:03:58.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.701+0000 7f5b99ffb640 1 -- 172.21.15.129:0/258975190 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f5ba001a470 con 0x560bde6e36b0 2024-05-20T18:03:58.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.703+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/258975190 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ba804d070 con 0x560bde6e36b0 2024-05-20T18:03:58.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.703+0000 7f5b99ffb640 1 -- 172.21.15.129:0/258975190 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5ba0003e30 con 0x560bde6e36b0 2024-05-20T18:03:58.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.705+0000 7f5bbf3b8740 10 client.25943 fetch_fsmap learned FSMap version 24 2024-05-20T18:03:58.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.705+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/258975190 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560bdeb27980 con 0x560bde6e36b0 2024-05-20T18:03:58.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.711+0000 7f5b99ffb640 1 -- 172.21.15.129:0/258975190 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ba0015070 con 0x560bde6e36b0 2024-05-20T18:03:58.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.711+0000 7f5b99ffb640 1 -- 172.21.15.129:0/258975190 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f5ba0011940 con 0x560bde6e36b0 2024-05-20T18:03:58.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.716+0000 7f5bbf3b8740 10 client.25943 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:03:58.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.716+0000 7f5bbf3b8740 10 client.25943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:03:58.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.716+0000 7f5bbf3b8740 20 client.25943 populate_metadata read hostname 'smithi129' 2024-05-20T18:03:58.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.716+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560bde6983a0 0x560bdeb2a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:58.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.717+0000 7f5bbf3b8740 4 client.25943 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:03:58.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.717+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/258975190 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f5ba804d070 con 0x560bde6983a0 2024-05-20T18:03:58.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.717+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560bde6983a0 0x560bdeb2a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:58.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.718+0000 7f5bb62ac640 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560bde6983a0 0x560bdeb2a0c0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=105f8f1da1dcd02 server_cookie=e2e96dd18aafadf3 in_seq=0 out_seq=0 2024-05-20T18:03:58.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.719+0000 7f5b99ffb640 10 client.25943 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:58.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.721+0000 7f5b99ffb640 1 -- 172.21.15.129:0/258975190 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f5ba804d070 con 0x560bde6983a0 2024-05-20T18:03:58.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.721+0000 7f5b99ffb640 10 client.25943 handle_command_reply: tid=0 2024-05-20T18:03:58.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.722+0000 7f5bbf3b8740 1 client.25943 shutdown 2024-05-20T18:03:58.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.722+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/258975190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 msgr2=0x560bdeab6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:58.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.722+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 0x560bdeab6cb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0073660 tx=0x7f5ba0003d00 comp rx=0 tx=0).stop 2024-05-20T18:03:58.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.726+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/258975190 shutdown_connections 2024-05-20T18:03:58.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.726+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560bde6983a0 0x560bdeb2a0c0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.726+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bde6e36b0 0x560bdeab6cb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.726+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bde704390 0x560bdeab71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.727+0000 7f5bbf3b8740 1 --2- 172.21.15.129:0/258975190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560bde6db4d0 0x560bdeab6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.727+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/258975190 >> 172.21.15.129:0/258975190 conn(0x560bde6f84e0 msgr2=0x560bdeaab810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:58.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.727+0000 7f5b99ffb640 0 client.25943 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:03:58.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.727+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/258975190 shutdown_connections 2024-05-20T18:03:58.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.727+0000 7f5bbf3b8740 1 -- 172.21.15.129:0/258975190 wait complete. 2024-05-20T18:03:58.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.728+0000 7f5bbf3b8740 20 client.25943 trim_cache size 0 max 16384 2024-05-20T18:03:58.727 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:03:58.727 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:03:58.727 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:03:58.727 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:03:58.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.731+0000 7f5bb72ae640 1 -- 172.21.15.129:0/1934977296 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5b9c0a2f00 msgr2=0x7f5b9c0a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:58.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.731+0000 7f5bb72ae640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5b9c0a2f00 0x7f5b9c0a5410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800f2a0 tx=0x7f5ba800e040 comp rx=0 tx=0).stop 2024-05-20T18:03:58.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.731+0000 7f5bb72ae640 1 -- 172.21.15.129:0/1934977296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bb010de00 msgr2=0x7f5bb007ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:58.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.731+0000 7f5bb72ae640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bb010de00 0x7f5bb007ed60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00e9f0 tx=0x7f5bac00eeb0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.733+0000 7f5bb72ae640 1 -- 172.21.15.129:0/1934977296 shutdown_connections 2024-05-20T18:03:58.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.733+0000 7f5bb72ae640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5b9c0a2f00 0x7f5b9c0a5410 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.733+0000 7f5bb72ae640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bb010e210 0x7f5bb007f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.733+0000 7f5bb72ae640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb007f7e0 0x7f5bb01c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.733+0000 7f5bb72ae640 1 --2- 172.21.15.129:0/1934977296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bb010de00 0x7f5bb007ed60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:58.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.733+0000 7f5bb72ae640 1 -- 172.21.15.129:0/1934977296 >> 172.21.15.129:0/1934977296 conn(0x7f5bb006d580 msgr2=0x7f5bb0070d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:58.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.735+0000 7f5bb72ae640 1 -- 172.21.15.129:0/1934977296 shutdown_connections 2024-05-20T18:03:58.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:58.738+0000 7f5bb72ae640 1 -- 172.21.15.129:0/1934977296 wait complete. 2024-05-20T18:03:58.788 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:03:58.788 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:03:59.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:03:59 smithi129 ceph-mon[32029]: pgmap v167: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 710 MiB/s wr, 1.73k op/s 2024-05-20T18:03:59.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.326+0000 7f749872f640 1 -- 172.21.15.129:0/1372618951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f749010e350 msgr2=0x7f7490079e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:59.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.326+0000 7f749872f640 1 --2- 172.21.15.129:0/1372618951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f749010e350 0x7f7490079e10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f748800bfd0 tx=0x7f748802ae40 comp rx=0 tx=0).stop 2024-05-20T18:03:59.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.329+0000 7f749872f640 1 -- 172.21.15.129:0/1372618951 shutdown_connections 2024-05-20T18:03:59.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.329+0000 7f749872f640 1 --2- 172.21.15.129:0/1372618951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f749010e350 0x7f7490079e10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:59.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.329+0000 7f749872f640 1 --2- 172.21.15.129:0/1372618951 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749010d890 0x7f749010dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:59.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.329+0000 7f749872f640 1 --2- 172.21.15.129:0/1372618951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7490109300 0x7f7490109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:59.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.329+0000 7f749872f640 1 -- 172.21.15.129:0/1372618951 >> 172.21.15.129:0/1372618951 conn(0x7f749006d760 msgr2=0x7f749006dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:03:59.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.333+0000 7f749872f640 1 -- 172.21.15.129:0/1372618951 shutdown_connections 2024-05-20T18:03:59.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.333+0000 7f749872f640 1 -- 172.21.15.129:0/1372618951 wait complete. 2024-05-20T18:03:59.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.336+0000 7f749872f640 1 Processor -- start 2024-05-20T18:03:59.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.336+0000 7f749872f640 1 -- start start 2024-05-20T18:03:59.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.336+0000 7f749872f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7490109300 0x7f749007f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:59.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.336+0000 7f749872f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f749010d890 0x7f749007f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:59.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.336+0000 7f749872f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749010e350 0x7f74901b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:59.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.336+0000 7f749872f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f749007d750 con 0x7f749010d890 2024-05-20T18:03:59.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.336+0000 7f749872f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f749007d5d0 con 0x7f7490109300 2024-05-20T18:03:59.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.336+0000 7f749872f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f749007d8d0 con 0x7f749010e350 2024-05-20T18:03:59.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.337+0000 7f7495cca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f749010d890 0x7f749007f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:59.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.337+0000 7f7495cca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f749010d890 0x7f749007f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51076/0 (socket says 172.21.15.129:51076) 2024-05-20T18:03:59.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.337+0000 7f7495cca640 1 -- 172.21.15.129:0/4221433600 learned_addr learned my addr 172.21.15.129:0/4221433600 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:03:59.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.338+0000 7f7496ccc640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749010e350 0x7f74901b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:59.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.338+0000 7f74964cb640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7490109300 0x7f749007f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:59.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.339+0000 7f7495cca640 1 -- 172.21.15.129:0/4221433600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749010e350 msgr2=0x7f74901b67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:59.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.339+0000 7f7495cca640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749010e350 0x7f74901b67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:59.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.339+0000 7f7495cca640 1 -- 172.21.15.129:0/4221433600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7490109300 msgr2=0x7f749007f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:03:59.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.339+0000 7f7495cca640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7490109300 0x7f749007f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:03:59.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.339+0000 7f7495cca640 1 -- 172.21.15.129:0/4221433600 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74901b6e60 con 0x7f749010d890 2024-05-20T18:03:59.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.341+0000 7f7495cca640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f749010d890 0x7f749007f950 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f748000e9c0 tx=0x7f748000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:59.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.342+0000 7f74877fe640 1 -- 172.21.15.129:0/4221433600 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748000cd80 con 0x7f749010d890 2024-05-20T18:03:59.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.343+0000 7f749872f640 1 -- 172.21.15.129:0/4221433600 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74901b7150 con 0x7f749010d890 2024-05-20T18:03:59.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.343+0000 7f749872f640 1 -- 172.21.15.129:0/4221433600 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74901bd8c0 con 0x7f749010d890 2024-05-20T18:03:59.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.343+0000 7f74877fe640 1 -- 172.21.15.129:0/4221433600 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7480005500 con 0x7f749010d890 2024-05-20T18:03:59.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.343+0000 7f74877fe640 1 -- 172.21.15.129:0/4221433600 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7480010690 con 0x7f749010d890 2024-05-20T18:03:59.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.350+0000 7f74877fe640 1 -- 172.21.15.129:0/4221433600 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7480005020 con 0x7f749010d890 2024-05-20T18:03:59.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.351+0000 7f749872f640 1 -- 172.21.15.129:0/4221433600 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7474005290 con 0x7f749010d890 2024-05-20T18:03:59.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.352+0000 7f74877fe640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f747c0a2e30 0x7f747c0a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:03:59.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.353+0000 7f74877fe640 1 -- 172.21.15.129:0/4221433600 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f74800cf760 con 0x7f749010d890 2024-05-20T18:03:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.360+0000 7f74964cb640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f747c0a2e30 0x7f747c0a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:03:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.362+0000 7f74877fe640 1 -- 172.21.15.129:0/4221433600 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f748009c370 con 0x7f749010d890 2024-05-20T18:03:59.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:03:59.375+0000 7f74964cb640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f747c0a2e30 0x7f747c0a5340 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f748c00a7d0 tx=0x7f748c008040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:03:59.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:03:59 smithi154 ceph-mon[36482]: pgmap v167: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 710 MiB/s wr, 1.73k op/s 2024-05-20T18:03:59.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:03:59 smithi204 ceph-mon[37076]: pgmap v167: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 710 MiB/s wr, 1.73k op/s 2024-05-20T18:04:00.080 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:04:00.080 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:03:18.451223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/4068528573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":4068528573},{"type":"v1","addr":"172.21.15.154:6837","nonce":4068528573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:04:00.080 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 24 2024-05-20T18:04:00.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.074+0000 7f749872f640 1 -- 172.21.15.129:0/4221433600 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7474005850 con 0x7f749010d890 2024-05-20T18:04:00.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.078+0000 7f74877fe640 1 -- 172.21.15.129:0/4221433600 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11829 (secure 0 0 0) 0x7f748009bc70 con 0x7f749010d890 2024-05-20T18:04:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.086+0000 7f74857fa640 1 -- 172.21.15.129:0/4221433600 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f747c0a2e30 msgr2=0x7f747c0a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:00.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.086+0000 7f74857fa640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f747c0a2e30 0x7f747c0a5340 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f748c00a7d0 tx=0x7f748c008040 comp rx=0 tx=0).stop 2024-05-20T18:04:00.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.086+0000 7f74857fa640 1 -- 172.21.15.129:0/4221433600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f749010d890 msgr2=0x7f749007f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:00.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.086+0000 7f74857fa640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f749010d890 0x7f749007f950 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f748000e9c0 tx=0x7f748000ee80 comp rx=0 tx=0).stop 2024-05-20T18:04:00.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.095+0000 7f74857fa640 1 -- 172.21.15.129:0/4221433600 shutdown_connections 2024-05-20T18:04:00.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.095+0000 7f74857fa640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f747c0a2e30 0x7f747c0a5340 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.095+0000 7f74857fa640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749010e350 0x7f74901b67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.095+0000 7f74857fa640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f749010d890 0x7f749007f950 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.095+0000 7f74857fa640 1 --2- 172.21.15.129:0/4221433600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7490109300 0x7f749007f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.095+0000 7f74857fa640 1 -- 172.21.15.129:0/4221433600 >> 172.21.15.129:0/4221433600 conn(0x7f749006d760 msgr2=0x7f749010cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:00.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.101+0000 7f74857fa640 1 -- 172.21.15.129:0/4221433600 shutdown_connections 2024-05-20T18:04:00.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.101+0000 7f74857fa640 1 -- 172.21.15.129:0/4221433600 wait complete. 2024-05-20T18:04:00.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:04:00.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.743+0000 7f91dfbb5640 1 -- 172.21.15.129:0/3145568716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91d8119770 msgr2=0x7f91d811bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:00.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.743+0000 7f91dfbb5640 1 --2- 172.21.15.129:0/3145568716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91d8119770 0x7f91d811bb90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f91d401a940 tx=0x7f91d403b520 comp rx=0 tx=0).stop 2024-05-20T18:04:00.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.747+0000 7f91dfbb5640 1 -- 172.21.15.129:0/3145568716 shutdown_connections 2024-05-20T18:04:00.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.747+0000 7f91dfbb5640 1 --2- 172.21.15.129:0/3145568716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d811c0d0 0x7f91d811e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.747+0000 7f91dfbb5640 1 --2- 172.21.15.129:0/3145568716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91d8119770 0x7f91d811bb90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.747+0000 7f91dfbb5640 1 --2- 172.21.15.129:0/3145568716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91d8109060 0x7f91d8109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.747+0000 7f91dfbb5640 1 -- 172.21.15.129:0/3145568716 >> 172.21.15.129:0/3145568716 conn(0x7f91d806d750 msgr2=0x7f91d806dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:00.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.755+0000 7f91dfbb5640 1 -- 172.21.15.129:0/3145568716 shutdown_connections 2024-05-20T18:04:00.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.760+0000 7f91dfbb5640 1 -- 172.21.15.129:0/3145568716 wait complete. 2024-05-20T18:04:00.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.762+0000 7f91dfbb5640 1 Processor -- start 2024-05-20T18:04:00.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.763+0000 7f91dfbb5640 1 -- start start 2024-05-20T18:04:00.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.764+0000 7f91dfbb5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91d8109060 0x7f91d81153d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.764+0000 7f91dfbb5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d8119770 0x7f91d8115910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.764+0000 7f91dfbb5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91d811c0d0 0x7f91d8115e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.764+0000 7f91dfbb5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f91d81139a0 con 0x7f91d811c0d0 2024-05-20T18:04:00.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.764+0000 7f91dfbb5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91d8113820 con 0x7f91d8119770 2024-05-20T18:04:00.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.764+0000 7f91dfbb5640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f91d8113b20 con 0x7f91d8109060 2024-05-20T18:04:00.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.765+0000 7f91dd951640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91d8109060 0x7f91d81153d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.765+0000 7f91dd951640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91d8109060 0x7f91d81153d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:49268/0 (socket says 172.21.15.129:49268) 2024-05-20T18:04:00.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.765+0000 7f91dd951640 1 -- 172.21.15.129:0/203913007 learned_addr learned my addr 172.21.15.129:0/203913007 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:00.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.766+0000 7f91dd951640 1 -- 172.21.15.129:0/203913007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d8119770 msgr2=0x7f91d8115910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:00.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.766+0000 7f91dd951640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d8119770 0x7f91d8115910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.766+0000 7f91dd951640 1 -- 172.21.15.129:0/203913007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91d811c0d0 msgr2=0x7f91d8115e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:00.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.766+0000 7f91dd951640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91d811c0d0 0x7f91d8115e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.766+0000 7f91dd951640 1 -- 172.21.15.129:0/203913007 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d81abfa0 con 0x7f91d8109060 2024-05-20T18:04:00.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.767+0000 7f91dd951640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91d8109060 0x7f91d81153d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f91d000eab0 tx=0x7f91d000ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:00.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.781+0000 7f91ceffd640 1 -- 172.21.15.129:0/203913007 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d000ce40 con 0x7f91d8109060 2024-05-20T18:04:00.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.781+0000 7f91dfbb5640 1 -- 172.21.15.129:0/203913007 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91d81ac290 con 0x7f91d8109060 2024-05-20T18:04:00.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.781+0000 7f91ceffd640 1 -- 172.21.15.129:0/203913007 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d0011950 con 0x7f91d8109060 2024-05-20T18:04:00.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.781+0000 7f91dfbb5640 1 -- 172.21.15.129:0/203913007 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91d81ac7b0 con 0x7f91d8109060 2024-05-20T18:04:00.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.781+0000 7f91ceffd640 1 -- 172.21.15.129:0/203913007 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0010ae0 con 0x7f91d8109060 2024-05-20T18:04:00.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.788+0000 7f91ceffd640 1 -- 172.21.15.129:0/203913007 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f91d00223a0 con 0x7f91d8109060 2024-05-20T18:04:00.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.791+0000 7f91ceffd640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f91c40a2e90 0x7f91c40a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.792+0000 7f91ceffd640 1 -- 172.21.15.129:0/203913007 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f91d00d0ab0 con 0x7f91d8109060 2024-05-20T18:04:00.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.798+0000 7f91dd150640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f91c40a2e90 0x7f91c40a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.804+0000 7f91e6a65740 1 Processor -- start 2024-05-20T18:04:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.805+0000 7f91e6a65740 1 -- start start 2024-05-20T18:04:00.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.806+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 0x559f47794d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.808+0000 7f91dd150640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f91c40a2e90 0x7f91c40a53a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f91d401a860 tx=0x7f91d4000bc0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:00.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.808+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f47795260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.808+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477957a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.809+0000 7f91e6a65740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559f476fac40 con 0x559f473f6390 2024-05-20T18:04:00.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.809+0000 7f91e6a65740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559f476a2580 con 0x559f473d56b0 2024-05-20T18:04:00.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.809+0000 7f91e6a65740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x559f476f6c00 con 0x559f473cd4d0 2024-05-20T18:04:00.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.809+0000 7f91de152640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.809+0000 7f91de152640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477957a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52780/0 (socket says 172.21.15.129:52780) 2024-05-20T18:04:00.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.809+0000 7f91de152640 1 -- 172.21.15.129:0/3852320697 learned_addr learned my addr 172.21.15.129:0/3852320697 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:00.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.810+0000 7f91dd150640 1 --2- 172.21.15.129:0/3852320697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f47795260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.810+0000 7f91de152640 1 -- 172.21.15.129:0/3852320697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 msgr2=0x559f47795260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:00.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.810+0000 7f91de152640 1 --2- 172.21.15.129:0/3852320697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f47795260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.810+0000 7f91de152640 1 -- 172.21.15.129:0/3852320697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 msgr2=0x559f47794d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:00.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.810+0000 7f91de152640 1 --2- 172.21.15.129:0/3852320697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 0x559f47794d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.810+0000 7f91de152640 1 -- 172.21.15.129:0/3852320697 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f4740e3f0 con 0x559f473d56b0 2024-05-20T18:04:00.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.811+0000 7f91de152640 1 --2- 172.21.15.129:0/3852320697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477957a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f91c800b9e0 tx=0x7f91c800bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c2fa4c9871a23e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:00.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.815+0000 7f91c3fff640 1 -- 172.21.15.129:0/3852320697 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8016070 con 0x559f473d56b0 2024-05-20T18:04:00.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.815+0000 7f91c3fff640 1 -- 172.21.15.129:0/3852320697 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91c8005500 con 0x559f473d56b0 2024-05-20T18:04:00.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.815+0000 7f91c3fff640 1 -- 172.21.15.129:0/3852320697 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8010460 con 0x559f473d56b0 2024-05-20T18:04:00.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.818+0000 7f91e6a65740 1 -- 172.21.15.129:0/3852320697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 msgr2=0x559f477957a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:00.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.818+0000 7f91e6a65740 1 --2- 172.21.15.129:0/3852320697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477957a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f91c800b9e0 tx=0x7f91c800bea0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.818+0000 7f91e6a65740 1 -- 172.21.15.129:0/3852320697 shutdown_connections 2024-05-20T18:04:00.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.818+0000 7f91e6a65740 1 --2- 172.21.15.129:0/3852320697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477957a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.818+0000 7f91e6a65740 1 --2- 172.21.15.129:0/3852320697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f47795260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.818+0000 7f91e6a65740 1 --2- 172.21.15.129:0/3852320697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 0x559f47794d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.819+0000 7f91e6a65740 1 -- 172.21.15.129:0/3852320697 >> 172.21.15.129:0/3852320697 conn(0x559f473ea4e0 msgr2=0x559f477914d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:00.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.819+0000 7f91e6a65740 1 -- 172.21.15.129:0/3852320697 shutdown_connections 2024-05-20T18:04:00.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.819+0000 7f91e6a65740 1 -- 172.21.15.129:0/3852320697 wait complete. 2024-05-20T18:04:00.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.820+0000 7f91e6a65740 1 -- start start 2024-05-20T18:04:00.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.823+0000 7f91e6a65740 1 Processor -- start 2024-05-20T18:04:00.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.824+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f4779f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.825+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f4779f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.826+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 0x559f477a5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.826+0000 7f91e6a65740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559f477a2070 con 0x559f473f6390 2024-05-20T18:04:00.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.826+0000 7f91e6a65740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559f476b5c40 con 0x559f473d56b0 2024-05-20T18:04:00.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.826+0000 7f91e6a65740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x559f477a21f0 con 0x559f473cd4d0 2024-05-20T18:04:00.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.826+0000 7f91e6a65740 10 client.0 _ll_register_callbacks cb 0x559f4740dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:00.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.830+0000 7f91dd951640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f4779f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.830+0000 7f91dd951640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f4779f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:49296/0 (socket says 172.21.15.129:49296) 2024-05-20T18:04:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.830+0000 7f91dd951640 1 -- 172.21.15.129:0/2552207019 learned_addr learned my addr 172.21.15.129:0/2552207019 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.830+0000 7f91dd150640 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f4779f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.831+0000 7f91dd951640 1 -- 172.21.15.129:0/2552207019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 msgr2=0x559f4779f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.831+0000 7f91dd951640 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f4779f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.831+0000 7f91dd951640 1 -- 172.21.15.129:0/2552207019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 msgr2=0x559f477a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.831+0000 7f91dd951640 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 0x559f477a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.831+0000 7f91dd951640 1 -- 172.21.15.129:0/2552207019 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f474d87d0 con 0x559f473cd4d0 2024-05-20T18:04:00.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.833+0000 7f91dd951640 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f4779f3e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f91d00ce2a0 tx=0x7f91d009d080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ea5f28ac716a4dd server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:00.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.837+0000 7f91c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:00.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.837+0000 7f91c17fa640 1 -- 172.21.15.129:0/2552207019 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d00b5070 con 0x559f473cd4d0 2024-05-20T18:04:00.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.838+0000 7f91e6a65740 1 -- 172.21.15.129:0/2552207019 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91c8005500 con 0x559f473cd4d0 2024-05-20T18:04:00.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.838+0000 7f91c17fa640 1 -- 172.21.15.129:0/2552207019 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f91d0099630 con 0x559f473cd4d0 2024-05-20T18:04:00.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.840+0000 7f91c17fa640 1 -- 172.21.15.129:0/2552207019 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f91d0010c80 con 0x559f473cd4d0 2024-05-20T18:04:00.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.841+0000 7f91e6a65740 10 client.25990 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:00.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.841+0000 7f91e6a65740 1 -- 172.21.15.129:0/2552207019 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559f477ae6f0 con 0x559f473cd4d0 2024-05-20T18:04:00.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.844+0000 7f91c17fa640 1 -- 172.21.15.129:0/2552207019 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91c8005500 con 0x559f473cd4d0 2024-05-20T18:04:00.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.844+0000 7f91c17fa640 1 -- 172.21.15.129:0/2552207019 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f91d00ba030 con 0x559f473cd4d0 2024-05-20T18:04:00.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.845+0000 7f91e6a65740 10 client.25990 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:00.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.845+0000 7f91e6a65740 10 client.25990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:00.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.845+0000 7f91e6a65740 20 client.25990 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:00.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.845+0000 7f91e6a65740 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559f473b4f60 0x559f477b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.845+0000 7f91e6a65740 4 client.25990 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:04:00.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.845+0000 7f91e6a65740 1 -- 172.21.15.129:0/2552207019 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f47708370 con 0x559f473b4f60 2024-05-20T18:04:00.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.846+0000 7f91dd150640 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559f473b4f60 0x559f477b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.852+0000 7f91dd150640 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559f473b4f60 0x559f477b20e0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2cf7b483c133599 server_cookie=bdd03dc16dceae53 in_seq=0 out_seq=0 2024-05-20T18:04:00.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.852+0000 7f91c17fa640 10 client.25990 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:00.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.859+0000 7f91c17fa640 1 -- 172.21.15.129:0/2552207019 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559f47708370 con 0x559f473b4f60 2024-05-20T18:04:00.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.859+0000 7f91c17fa640 10 client.25990 handle_command_reply: tid=0 2024-05-20T18:04:00.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.860+0000 7f91e6a65740 1 client.25990 shutdown 2024-05-20T18:04:00.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.862+0000 7f91e6a65740 1 -- 172.21.15.129:0/2552207019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 msgr2=0x559f4779f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:00.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.862+0000 7f91e6a65740 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f4779f3e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f91d00ce2a0 tx=0x7f91d009d080 comp rx=0 tx=0).stop 2024-05-20T18:04:00.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.862+0000 7f91e6a65740 1 -- 172.21.15.129:0/2552207019 shutdown_connections 2024-05-20T18:04:00.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.863+0000 7f91e6a65740 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559f473b4f60 0x559f477b20e0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.863+0000 7f91e6a65740 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f4779f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.863+0000 7f91e6a65740 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 0x559f477a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.863+0000 7f91e6a65740 1 --2- 172.21.15.129:0/2552207019 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f4779f3e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.863+0000 7f91e6a65740 1 -- 172.21.15.129:0/2552207019 >> 172.21.15.129:0/2552207019 conn(0x559f473ea4e0 msgr2=0x559f47796a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:00.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.863+0000 7f91c17fa640 0 client.25990 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:00.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.863+0000 7f91e6a65740 1 -- 172.21.15.129:0/2552207019 shutdown_connections 2024-05-20T18:04:00.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.863+0000 7f91e6a65740 1 -- 172.21.15.129:0/2552207019 wait complete. 2024-05-20T18:04:00.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.863+0000 7f91e6a65740 20 client.25990 trim_cache size 0 max 16384 2024-05-20T18:04:00.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.904+0000 7f91e6a65740 1 Processor -- start 2024-05-20T18:04:00.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.904+0000 7f91e6a65740 1 -- start start 2024-05-20T18:04:00.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.905+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f477a5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.905+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.905+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 0x559f477b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.905+0000 7f91e6a65740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc00c8b0 con 0x559f473f6390 2024-05-20T18:04:00.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.905+0000 7f91e6a65740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc00f130 con 0x559f473d56b0 2024-05-20T18:04:00.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.905+0000 7f91e6a65740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc00cf20 con 0x559f473cd4d0 2024-05-20T18:04:00.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.905+0000 7f91de152640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 0x559f477b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.905+0000 7f91dd951640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f477a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.905+0000 7f91de152640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 0x559f477b7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34648/0 (socket says 172.21.15.129:34648) 2024-05-20T18:04:00.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.905+0000 7f91de152640 1 -- 172.21.15.129:0/2616227731 learned_addr learned my addr 172.21.15.129:0/2616227731 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:00.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.908+0000 7f91dd150640 1 --2- 172.21.15.129:0/2616227731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.910+0000 7f91dd951640 1 -- 172.21.15.129:0/2616227731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 msgr2=0x559f477b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.910+0000 7f91dd951640 1 --2- 172.21.15.129:0/2616227731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477b78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.910+0000 7f91dd951640 1 -- 172.21.15.129:0/2616227731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 msgr2=0x559f477b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.910+0000 7f91dd951640 1 --2- 172.21.15.129:0/2616227731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 0x559f477b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.910+0000 7f91dd951640 1 -- 172.21.15.129:0/2616227731 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f473b5190 con 0x559f473cd4d0 2024-05-20T18:04:00.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.916+0000 7f91dd951640 1 --2- 172.21.15.129:0/2616227731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f477a5470 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f91d000ea80 tx=0x7f91d00a5db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3238eec3a6a3c5d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:00.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.919+0000 7f91c37fe640 1 -- 172.21.15.129:0/2616227731 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d00ad030 con 0x559f473cd4d0 2024-05-20T18:04:00.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.920+0000 7f91c37fe640 1 -- 172.21.15.129:0/2616227731 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d0010e20 con 0x559f473cd4d0 2024-05-20T18:04:00.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.921+0000 7f91e6a65740 1 -- 172.21.15.129:0/2616227731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 msgr2=0x559f477a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:00.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.921+0000 7f91e6a65740 1 --2- 172.21.15.129:0/2616227731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f477a5470 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f91d000ea80 tx=0x7f91d00a5db0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.923+0000 7f91e6a65740 1 -- 172.21.15.129:0/2616227731 shutdown_connections 2024-05-20T18:04:00.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.923+0000 7f91e6a65740 1 --2- 172.21.15.129:0/2616227731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.923+0000 7f91e6a65740 1 --2- 172.21.15.129:0/2616227731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473f6390 0x559f477b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.923+0000 7f91e6a65740 1 --2- 172.21.15.129:0/2616227731 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473cd4d0 0x559f477a5470 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.923+0000 7f91e6a65740 1 -- 172.21.15.129:0/2616227731 >> 172.21.15.129:0/2616227731 conn(0x559f473ea4e0 msgr2=0x559f4779dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:00.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.929+0000 7f91e6a65740 1 -- 172.21.15.129:0/2616227731 shutdown_connections 2024-05-20T18:04:00.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.930+0000 7f91e6a65740 1 -- 172.21.15.129:0/2616227731 wait complete. 2024-05-20T18:04:00.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.930+0000 7f91e6a65740 1 -- start start 2024-05-20T18:04:00.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.931+0000 7f91e6a65740 1 Processor -- start 2024-05-20T18:04:00.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.932+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473cd4d0 0x559f477a6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.932+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.932+0000 7f91e6a65740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473f6390 0x559f477a6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.932+0000 7f91e6a65740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc00d5f0 con 0x559f473cd4d0 2024-05-20T18:04:00.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.932+0000 7f91e6a65740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc00ea60 con 0x559f473d56b0 2024-05-20T18:04:00.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.932+0000 7f91e6a65740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f91bc00dcc0 con 0x559f473f6390 2024-05-20T18:04:00.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.932+0000 7f91e6a65740 10 client.0 _ll_register_callbacks cb 0x7f91d0098b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:00.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.932+0000 7f91de152640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473f6390 0x559f477a6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.932+0000 7f91de152640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473f6390 0x559f477a6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:49324/0 (socket says 172.21.15.129:49324) 2024-05-20T18:04:00.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.932+0000 7f91de152640 1 -- 172.21.15.129:0/657510487 learned_addr learned my addr 172.21.15.129:0/657510487 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:00.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.933+0000 7f91dd150640 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.933+0000 7f91de152640 1 -- 172.21.15.129:0/657510487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 msgr2=0x559f477a6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:00.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.933+0000 7f91de152640 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477a6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.933+0000 7f91de152640 1 -- 172.21.15.129:0/657510487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473cd4d0 msgr2=0x559f477a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:00.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.933+0000 7f91de152640 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473cd4d0 0x559f477a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.933+0000 7f91de152640 1 -- 172.21.15.129:0/657510487 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f47818480 con 0x559f473f6390 2024-05-20T18:04:00.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.934+0000 7f91de152640 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473f6390 0x559f477a6fc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f91c800b770 tx=0x7f91c800df50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e88324f151d3c14c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:00.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.949+0000 7f91c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:00.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.949+0000 7f91c17fa640 1 -- 172.21.15.129:0/657510487 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8016040 con 0x559f473f6390 2024-05-20T18:04:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.950+0000 7f91c17fa640 1 -- 172.21.15.129:0/657510487 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f91c8004390 con 0x559f473f6390 2024-05-20T18:04:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.952+0000 7f91e6a65740 1 -- 172.21.15.129:0/657510487 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91d0010e20 con 0x559f473f6390 2024-05-20T18:04:00.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.956+0000 7f91c17fa640 1 -- 172.21.15.129:0/657510487 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f91c80035f0 con 0x559f473f6390 2024-05-20T18:04:00.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.959+0000 7f91e6a65740 10 client.25996 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:00.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.959+0000 7f91e6a65740 1 -- 172.21.15.129:0/657510487 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559f478196a0 con 0x559f473f6390 2024-05-20T18:04:00.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.961+0000 7f91c17fa640 1 -- 172.21.15.129:0/657510487 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d0010e20 con 0x559f473f6390 2024-05-20T18:04:00.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.961+0000 7f91c17fa640 1 -- 172.21.15.129:0/657510487 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f91c80518d0 con 0x559f473f6390 2024-05-20T18:04:00.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.961+0000 7f91e6a65740 10 client.25996 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:00.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.961+0000 7f91e6a65740 10 client.25996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:00.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.961+0000 7f91e6a65740 20 client.25996 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:00.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.961+0000 7f91e6a65740 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559f4738a3a0 0x559f4781bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:00.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.961+0000 7f91e6a65740 4 client.25996 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:04:00.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.961+0000 7f91e6a65740 1 -- 172.21.15.129:0/657510487 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f477026c0 con 0x559f4738a3a0 2024-05-20T18:04:00.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.964+0000 7f91dd951640 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559f4738a3a0 0x559f4781bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:00.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.968+0000 7f91dd951640 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559f4738a3a0 0x559f4781bd80 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5718aed84e9cb405 server_cookie=d6a01cc1f90302b8 in_seq=0 out_seq=0 2024-05-20T18:04:00.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.968+0000 7f91c17fa640 10 client.25996 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:00.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.972+0000 7f91c17fa640 1 -- 172.21.15.129:0/657510487 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f477026c0 con 0x559f4738a3a0 2024-05-20T18:04:00.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.972+0000 7f91c17fa640 10 client.25996 handle_command_reply: tid=0 2024-05-20T18:04:00.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.972+0000 7f91e6a65740 1 client.25996 shutdown 2024-05-20T18:04:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.973+0000 7f91e6a65740 1 -- 172.21.15.129:0/657510487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473f6390 msgr2=0x559f477a6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.973+0000 7f91e6a65740 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473f6390 0x559f477a6fc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f91c800b770 tx=0x7f91c800df50 comp rx=0 tx=0).stop 2024-05-20T18:04:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.973+0000 7f91e6a65740 1 -- 172.21.15.129:0/657510487 shutdown_connections 2024-05-20T18:04:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.973+0000 7f91e6a65740 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x559f4738a3a0 0x559f4781bd80 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.973+0000 7f91e6a65740 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559f473d56b0 0x559f477a6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.973+0000 7f91e6a65740 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559f473f6390 0x559f477a6fc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.973+0000 7f91e6a65740 1 --2- 172.21.15.129:0/657510487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559f473cd4d0 0x559f477a6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.973+0000 7f91e6a65740 1 -- 172.21.15.129:0/657510487 >> 172.21.15.129:0/657510487 conn(0x559f473ea4e0 msgr2=0x559f47796810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.973+0000 7f91c17fa640 0 client.25996 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:00.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.974+0000 7f91e6a65740 1 -- 172.21.15.129:0/657510487 shutdown_connections 2024-05-20T18:04:00.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.974+0000 7f91e6a65740 1 -- 172.21.15.129:0/657510487 wait complete. 2024-05-20T18:04:00.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.974+0000 7f91e6a65740 20 client.25996 trim_cache size 0 max 16384 2024-05-20T18:04:00.977 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:04:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.982+0000 7f91dfbb5640 1 -- 172.21.15.129:0/203913007 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f91c40a2e90 msgr2=0x7f91c40a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.982+0000 7f91dfbb5640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f91c40a2e90 0x7f91c40a53a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f91d401a860 tx=0x7f91d4000bc0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.982+0000 7f91dfbb5640 1 -- 172.21.15.129:0/203913007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91d8109060 msgr2=0x7f91d81153d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.982+0000 7f91dfbb5640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91d8109060 0x7f91d81153d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f91d000eab0 tx=0x7f91d000ef70 comp rx=0 tx=0).stop 2024-05-20T18:04:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.984+0000 7f91dfbb5640 1 -- 172.21.15.129:0/203913007 shutdown_connections 2024-05-20T18:04:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.984+0000 7f91dfbb5640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f91c40a2e90 0x7f91c40a53a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.984+0000 7f91dfbb5640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d8119770 0x7f91d8115910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.984+0000 7f91dfbb5640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91d811c0d0 0x7f91d8115e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.984+0000 7f91dfbb5640 1 --2- 172.21.15.129:0/203913007 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f91d8109060 0x7f91d81153d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.984+0000 7f91dfbb5640 1 -- 172.21.15.129:0/203913007 >> 172.21.15.129:0/203913007 conn(0x7f91d806d750 msgr2=0x7f91d8119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:00.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.987+0000 7f91dfbb5640 1 -- 172.21.15.129:0/203913007 shutdown_connections 2024-05-20T18:04:00.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:00.987+0000 7f91dfbb5640 1 -- 172.21.15.129:0/203913007 wait complete. 2024-05-20T18:04:01.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:01 smithi154 ceph-mon[36482]: pgmap v168: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 974 MiB/s wr, 2.43k op/s 2024-05-20T18:04:01.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:01 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/4221433600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:04:01.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:01 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:04:01.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:01 smithi129 ceph-mon[32029]: pgmap v168: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 974 MiB/s wr, 2.43k op/s 2024-05-20T18:04:01.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:01 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4221433600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:04:01.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:01 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:04:01.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:01 smithi204 ceph-mon[37076]: pgmap v168: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 974 MiB/s wr, 2.43k op/s 2024-05-20T18:04:01.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:01 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/4221433600' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:04:01.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:01 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:04:02.057 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:04:02.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:04:02.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:04:02.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.766+0000 7fcbbb59e640 1 -- 172.21.15.129:0/1182584870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbbc119770 msgr2=0x7fcbbc11bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:02.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.766+0000 7fcbbb59e640 1 --2- 172.21.15.129:0/1182584870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbbc119770 0x7fcbbc11bb90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009cd0 tx=0x7fcbb402a930 comp rx=0 tx=0).stop 2024-05-20T18:04:02.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.767+0000 7fcbbb59e640 1 -- 172.21.15.129:0/1182584870 shutdown_connections 2024-05-20T18:04:02.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.767+0000 7fcbbb59e640 1 --2- 172.21.15.129:0/1182584870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbbc119770 0x7fcbbc11bb90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.767+0000 7fcbbb59e640 1 --2- 172.21.15.129:0/1182584870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbbc11c0d0 0x7fcbbc11e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.767+0000 7fcbbb59e640 1 --2- 172.21.15.129:0/1182584870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbbc108f70 0x7fcbbc109380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.767+0000 7fcbbb59e640 1 -- 172.21.15.129:0/1182584870 >> 172.21.15.129:0/1182584870 conn(0x7fcbbc06e600 msgr2=0x7fcbbc06ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:02.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.772+0000 7fcbbb59e640 1 -- 172.21.15.129:0/1182584870 shutdown_connections 2024-05-20T18:04:02.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.772+0000 7fcbbb59e640 1 -- 172.21.15.129:0/1182584870 wait complete. 2024-05-20T18:04:02.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.775+0000 7fcbbb59e640 1 Processor -- start 2024-05-20T18:04:02.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.775+0000 7fcbbb59e640 1 -- start start 2024-05-20T18:04:02.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.775+0000 7fcbbb59e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbbc108f70 0x7fcbbc119150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.775+0000 7fcbbb59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbbc119770 0x7fcbbc112270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.776+0000 7fcbbb59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbbc11c0d0 0x7fcbbc1128a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.776+0000 7fcbbb59e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc121b20 con 0x7fcbbc11c0d0 2024-05-20T18:04:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.776+0000 7fcbbb59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc1219a0 con 0x7fcbbc119770 2024-05-20T18:04:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.776+0000 7fcbbb59e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc121ca0 con 0x7fcbbc108f70 2024-05-20T18:04:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.776+0000 7fcbb9d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbbc119770 0x7fcbbc112270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.776+0000 7fcbb9d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbbc119770 0x7fcbbc112270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52818/0 (socket says 172.21.15.129:52818) 2024-05-20T18:04:02.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.776+0000 7fcbb9d9b640 1 -- 172.21.15.129:0/2069564711 learned_addr learned my addr 172.21.15.129:0/2069564711 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:02.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.777+0000 7fcbba59c640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbbc108f70 0x7fcbbc119150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.777+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbbc11c0d0 0x7fcbbc1128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.777+0000 7fcbba59c640 1 -- 172.21.15.129:0/2069564711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbbc119770 msgr2=0x7fcbbc112270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:02.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.777+0000 7fcbba59c640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbbc119770 0x7fcbbc112270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.777+0000 7fcbba59c640 1 -- 172.21.15.129:0/2069564711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbbc11c0d0 msgr2=0x7fcbbc1128a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:02.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.777+0000 7fcbba59c640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbbc11c0d0 0x7fcbbc1128a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.777+0000 7fcbba59c640 1 -- 172.21.15.129:0/2069564711 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbbc116c70 con 0x7fcbbc108f70 2024-05-20T18:04:02.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.778+0000 7fcbba59c640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbbc108f70 0x7fcbbc119150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009fd0 tx=0x7fcbac00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:02.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.778+0000 7fcbab7fe640 1 -- 172.21.15.129:0/2069564711 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac015020 con 0x7fcbbc108f70 2024-05-20T18:04:02.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.778+0000 7fcbbb59e640 1 -- 172.21.15.129:0/2069564711 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbbc116f60 con 0x7fcbbc108f70 2024-05-20T18:04:02.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.779+0000 7fcbbb59e640 1 -- 172.21.15.129:0/2069564711 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbbc117500 con 0x7fcbbc108f70 2024-05-20T18:04:02.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.784+0000 7fcbab7fe640 1 -- 172.21.15.129:0/2069564711 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbac011d10 con 0x7fcbbc108f70 2024-05-20T18:04:02.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.785+0000 7fcbab7fe640 1 -- 172.21.15.129:0/2069564711 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac00f2e0 con 0x7fcbbc108f70 2024-05-20T18:04:02.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.785+0000 7fcbc79e0740 1 Processor -- start 2024-05-20T18:04:02.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.786+0000 7fcbc79e0740 1 -- start start 2024-05-20T18:04:02.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.787+0000 7fcbab7fe640 1 -- 172.21.15.129:0/2069564711 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fcbac018020 con 0x7fcbbc108f70 2024-05-20T18:04:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.789+0000 7fcbab7fe640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcba40a2dc0 0x7fcba40a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.791+0000 7fcbab7fe640 1 -- 172.21.15.129:0/2069564711 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fcbac0cfa80 con 0x7fcbbc108f70 2024-05-20T18:04:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.792+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8eb8410 0x5629c9257830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.792+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 0x5629c9257d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.792+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 0x5629c92582b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.792+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5629c91bc990 con 0x5629c8e8f550 2024-05-20T18:04:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.792+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5629c91645b0 con 0x5629c8e97730 2024-05-20T18:04:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.792+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5629c91b8950 con 0x5629c8eb8410 2024-05-20T18:04:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.792+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcba40a2dc0 0x7fcba40a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.792+0000 7fcbba59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8eb8410 0x5629c9257830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.793+0000 7fcbb9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 0x5629c9257d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.793+0000 7fcbb9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 0x5629c9257d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34674/0 (socket says 172.21.15.129:34674) 2024-05-20T18:04:02.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.793+0000 7fcbb9d9b640 1 -- 172.21.15.129:0/2545583155 learned_addr learned my addr 172.21.15.129:0/2545583155 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:02.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.793+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/2545583155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 0x5629c92582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.794+0000 7fcbba59c640 1 -- 172.21.15.129:0/2545583155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 msgr2=0x5629c92582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.794+0000 7fcbba59c640 1 --2- 172.21.15.129:0/2545583155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 0x5629c92582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.794+0000 7fcbba59c640 1 -- 172.21.15.129:0/2545583155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 msgr2=0x5629c9257d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.794+0000 7fcbba59c640 1 --2- 172.21.15.129:0/2545583155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 0x5629c9257d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.794+0000 7fcbba59c640 1 -- 172.21.15.129:0/2545583155 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629c8f987e0 con 0x5629c8eb8410 2024-05-20T18:04:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.795+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/2545583155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 0x5629c92582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:04:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.795+0000 7fcbba59c640 1 --2- 172.21.15.129:0/2545583155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8eb8410 0x5629c9257830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcbac0cd2d0 tx=0x7fcbac09c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11e5705414cdc84a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.799+0000 7fcba8ff9640 1 -- 172.21.15.129:0/2545583155 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac0a4d10 con 0x5629c8eb8410 2024-05-20T18:04:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.799+0000 7fcba8ff9640 1 -- 172.21.15.129:0/2545583155 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbac09a3a0 con 0x5629c8eb8410 2024-05-20T18:04:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.799+0000 7fcba8ff9640 1 -- 172.21.15.129:0/2545583155 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac0a4880 con 0x5629c8eb8410 2024-05-20T18:04:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.799+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcba40a2dc0 0x7fcba40a52d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc06b490 tx=0x7fcbb0011810 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:02.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.801+0000 7fcbc79e0740 1 -- 172.21.15.129:0/2545583155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8eb8410 msgr2=0x5629c9257830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:02.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.801+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/2545583155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8eb8410 0x5629c9257830 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcbac0cd2d0 tx=0x7fcbac09c860 comp rx=0 tx=0).stop 2024-05-20T18:04:02.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.801+0000 7fcbc79e0740 1 -- 172.21.15.129:0/2545583155 shutdown_connections 2024-05-20T18:04:02.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.801+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/2545583155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 0x5629c92582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.801+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/2545583155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 0x5629c9257d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.801+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/2545583155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8eb8410 0x5629c9257830 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcbac0cd2d0 tx=0x7fcbac09c860 comp rx=0 tx=0).stop 2024-05-20T18:04:02.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.801+0000 7fcbc79e0740 1 -- 172.21.15.129:0/2545583155 >> 172.21.15.129:0/2545583155 conn(0x5629c8eac560 msgr2=0x5629c9253fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.805+0000 7fcbc79e0740 1 -- 172.21.15.129:0/2545583155 shutdown_connections 2024-05-20T18:04:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.805+0000 7fcbc79e0740 1 -- 172.21.15.129:0/2545583155 wait complete. 2024-05-20T18:04:02.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.806+0000 7fcbc79e0740 1 -- start start 2024-05-20T18:04:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.807+0000 7fcbc79e0740 1 Processor -- start 2024-05-20T18:04:02.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.808+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 0x5629c9263990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.808+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 0x5629c9263ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.808+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e771f0 0x5629c9261e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.808+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5629c9264b20 con 0x5629c8e8f550 2024-05-20T18:04:02.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.808+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5629c9177b40 con 0x5629c8e97730 2024-05-20T18:04:02.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.808+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5629c9264ca0 con 0x5629c8e771f0 2024-05-20T18:04:02.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.808+0000 7fcbc79e0740 10 client.0 _ll_register_callbacks cb 0x5629c916c0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.809+0000 7fcbb9d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 0x5629c9263ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.809+0000 7fcbb9d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 0x5629c9263ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52836/0 (socket says 172.21.15.129:52836) 2024-05-20T18:04:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.809+0000 7fcbb9d9b640 1 -- 172.21.15.129:0/1447284527 learned_addr learned my addr 172.21.15.129:0/1447284527 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.809+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e771f0 0x5629c9261e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.810+0000 7fcbbad9d640 1 -- 172.21.15.129:0/1447284527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 msgr2=0x5629c9263ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:02.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.810+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 0x5629c9263ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.810+0000 7fcbbad9d640 1 -- 172.21.15.129:0/1447284527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 msgr2=0x5629c9263990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:02.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.810+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 0x5629c9263990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.810+0000 7fcbbad9d640 1 -- 172.21.15.129:0/1447284527 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629c8ff8f70 con 0x5629c8e771f0 2024-05-20T18:04:02.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.811+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e771f0 0x5629c9261e10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000f570 tx=0x7fcbb0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c27ca059fe2ed46 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:02.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.819+0000 7fcba1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:02.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.819+0000 7fcba1ffb640 1 -- 172.21.15.129:0/1447284527 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb001a040 con 0x5629c8e771f0 2024-05-20T18:04:02.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.819+0000 7fcbc79e0740 1 -- 172.21.15.129:0/1447284527 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbac09a3a0 con 0x5629c8e771f0 2024-05-20T18:04:02.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.819+0000 7fcba1ffb640 1 -- 172.21.15.129:0/1447284527 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fcbb00209b0 con 0x5629c8e771f0 2024-05-20T18:04:02.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.821+0000 7fcba1ffb640 1 -- 172.21.15.129:0/1447284527 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcbb0020bb0 con 0x5629c8e771f0 2024-05-20T18:04:02.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.828+0000 7fcbc79e0740 10 client.26005 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:02.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.829+0000 7fcbc79e0740 1 -- 172.21.15.129:0/1447284527 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5629c8e90170 con 0x5629c8e771f0 2024-05-20T18:04:02.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.829+0000 7fcba1ffb640 1 -- 172.21.15.129:0/1447284527 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbac09a3a0 con 0x5629c8e771f0 2024-05-20T18:04:02.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.829+0000 7fcba1ffb640 1 -- 172.21.15.129:0/1447284527 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fcbb00313c0 con 0x5629c8e771f0 2024-05-20T18:04:02.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.829+0000 7fcbc79e0740 10 client.26005 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:02.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.829+0000 7fcbc79e0740 10 client.26005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:02.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.830+0000 7fcbc79e0740 20 client.26005 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:02.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.830+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5629c8e4c630 0x5629c92745c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.830+0000 7fcbc79e0740 4 client.26005 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:04:02.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.830+0000 7fcbc79e0740 1 -- 172.21.15.129:0/1447284527 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629c91ca0c0 con 0x5629c8e4c630 2024-05-20T18:04:02.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.853+0000 7fcbba59c640 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5629c8e4c630 0x5629c92745c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.856+0000 7fcbba59c640 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5629c8e4c630 0x5629c92745c0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ce24010202f558d server_cookie=d57b28fabb95b506 in_seq=0 out_seq=0 2024-05-20T18:04:02.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.856+0000 7fcba1ffb640 10 client.26005 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:02.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.861+0000 7fcba1ffb640 1 -- 172.21.15.129:0/1447284527 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5629c91ca0c0 con 0x5629c8e4c630 2024-05-20T18:04:02.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.861+0000 7fcba1ffb640 10 client.26005 handle_command_reply: tid=0 2024-05-20T18:04:02.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.861+0000 7fcbc79e0740 1 client.26005 shutdown 2024-05-20T18:04:02.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.869+0000 7fcbc79e0740 1 -- 172.21.15.129:0/1447284527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e771f0 msgr2=0x5629c9261e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:02.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.869+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e771f0 0x5629c9261e10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000f570 tx=0x7fcbb0004730 comp rx=0 tx=0).stop 2024-05-20T18:04:02.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.869+0000 7fcbc79e0740 1 -- 172.21.15.129:0/1447284527 shutdown_connections 2024-05-20T18:04:02.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.869+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5629c8e4c630 0x5629c92745c0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.869+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e97730 0x5629c9263ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.869+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e771f0 0x5629c9261e10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.869+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/1447284527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 0x5629c9263990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.869+0000 7fcba1ffb640 0 client.26005 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.869+0000 7fcbc79e0740 1 -- 172.21.15.129:0/1447284527 >> 172.21.15.129:0/1447284527 conn(0x5629c8eac560 msgr2=0x5629c92528b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:02.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.871+0000 7fcbc79e0740 1 -- 172.21.15.129:0/1447284527 shutdown_connections 2024-05-20T18:04:02.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.876+0000 7fcbc79e0740 1 -- 172.21.15.129:0/1447284527 wait complete. 2024-05-20T18:04:02.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.876+0000 7fcbc79e0740 20 client.26005 trim_cache size 0 max 16384 2024-05-20T18:04:02.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.951+0000 7fcbc79e0740 1 Processor -- start 2024-05-20T18:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.951+0000 7fcbc79e0740 1 -- start start 2024-05-20T18:04:02.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.952+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e771f0 0x5629c92540b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.952+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 0x5629c9279dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.953+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c927a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.953+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c00c870 con 0x5629c8e8f550 2024-05-20T18:04:02.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.953+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c00f0f0 con 0x5629c8e771f0 2024-05-20T18:04:02.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.953+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c00cee0 con 0x5629c8e97730 2024-05-20T18:04:02.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.954+0000 7fcbbad9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c927a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.954+0000 7fcbbad9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c927a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:49366/0 (socket says 172.21.15.129:49366) 2024-05-20T18:04:02.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.954+0000 7fcbbad9d640 1 -- 172.21.15.129:0/2286866723 learned_addr learned my addr 172.21.15.129:0/2286866723 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:02.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.955+0000 7fcbba59c640 1 --2- 172.21.15.129:0/2286866723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e771f0 0x5629c92540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.955+0000 7fcbbad9d640 1 -- 172.21.15.129:0/2286866723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e771f0 msgr2=0x5629c92540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:02.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.955+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/2286866723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e771f0 0x5629c92540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.955+0000 7fcbbad9d640 1 -- 172.21.15.129:0/2286866723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 msgr2=0x5629c9279dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:02.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.955+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/2286866723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 0x5629c9279dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.955+0000 7fcbbad9d640 1 -- 172.21.15.129:0/2286866723 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629c924dee0 con 0x5629c8e97730 2024-05-20T18:04:02.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.956+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/2286866723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c927a300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00058c0 tx=0x7fcbb0031fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94ff2527a1c6584b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:02.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.959+0000 7fcba3fff640 1 -- 172.21.15.129:0/2286866723 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0036070 con 0x5629c8e97730 2024-05-20T18:04:02.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.959+0000 7fcba3fff640 1 -- 172.21.15.129:0/2286866723 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb0024070 con 0x5629c8e97730 2024-05-20T18:04:02.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.960+0000 7fcbc79e0740 1 -- 172.21.15.129:0/2286866723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 msgr2=0x5629c927a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:02.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.961+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/2286866723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c927a300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00058c0 tx=0x7fcbb0031fb0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.961+0000 7fcbc79e0740 1 -- 172.21.15.129:0/2286866723 shutdown_connections 2024-05-20T18:04:02.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.961+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/2286866723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c927a300 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.961+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/2286866723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e8f550 0x5629c9279dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.961+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/2286866723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e771f0 0x5629c92540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.961+0000 7fcbc79e0740 1 -- 172.21.15.129:0/2286866723 >> 172.21.15.129:0/2286866723 conn(0x5629c8eac560 msgr2=0x5629c92539a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:02.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.963+0000 7fcbc79e0740 1 -- 172.21.15.129:0/2286866723 shutdown_connections 2024-05-20T18:04:02.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.965+0000 7fcbc79e0740 1 -- 172.21.15.129:0/2286866723 wait complete. 2024-05-20T18:04:02.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.966+0000 7fcbc79e0740 1 -- start start 2024-05-20T18:04:02.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.966+0000 7fcbc79e0740 1 Processor -- start 2024-05-20T18:04:02.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.966+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e771f0 0x5629c926c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.967+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e8f550 0x5629c926cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.967+0000 7fcbc79e0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c926d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.967+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c00d5b0 con 0x5629c8e771f0 2024-05-20T18:04:02.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.967+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c00ea20 con 0x5629c8e8f550 2024-05-20T18:04:02.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.967+0000 7fcbc79e0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c00dc80 con 0x5629c8e97730 2024-05-20T18:04:02.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.967+0000 7fcbc79e0740 10 client.0 _ll_register_callbacks cb 0x7fcbb0009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:02.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.967+0000 7fcbbad9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c926d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.967+0000 7fcbbad9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c926d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:49378/0 (socket says 172.21.15.129:49378) 2024-05-20T18:04:02.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.967+0000 7fcbbad9d640 1 -- 172.21.15.129:0/3780167251 learned_addr learned my addr 172.21.15.129:0/3780167251 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:02.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.968+0000 7fcbbad9d640 1 -- 172.21.15.129:0/3780167251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e8f550 msgr2=0x5629c926cb40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:04:02.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.968+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/3780167251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e8f550 0x5629c926cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.968+0000 7fcbbad9d640 1 -- 172.21.15.129:0/3780167251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e771f0 msgr2=0x5629c926c600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:04:02.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.968+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/3780167251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e771f0 0x5629c926c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:02.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.968+0000 7fcbbad9d640 1 -- 172.21.15.129:0/3780167251 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629c92fdb40 con 0x5629c8e97730 2024-05-20T18:04:02.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.975+0000 7fcbbad9d640 1 --2- 172.21.15.129:0/3780167251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c926d080 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00318d0 tx=0x7fcbb0009a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a09d72557f077f28 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:02.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.976+0000 7fcba1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:02.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.976+0000 7fcba1ffb640 1 -- 172.21.15.129:0/3780167251 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0036040 con 0x5629c8e97730 2024-05-20T18:04:02.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.976+0000 7fcbc79e0740 1 -- 172.21.15.129:0/3780167251 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbb0024070 con 0x5629c8e97730 2024-05-20T18:04:02.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.976+0000 7fcba1ffb640 1 -- 172.21.15.129:0/3780167251 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fcbb0004020 con 0x5629c8e97730 2024-05-20T18:04:02.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.977+0000 7fcba1ffb640 1 -- 172.21.15.129:0/3780167251 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fcbb0004220 con 0x5629c8e97730 2024-05-20T18:04:02.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.977+0000 7fcbc79e0740 10 client.26011 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:02.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.978+0000 7fcbc79e0740 1 -- 172.21.15.129:0/3780167251 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5629c92fee00 con 0x5629c8e97730 2024-05-20T18:04:02.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.984+0000 7fcba1ffb640 1 -- 172.21.15.129:0/3780167251 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb0024070 con 0x5629c8e97730 2024-05-20T18:04:02.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.984+0000 7fcba1ffb640 1 -- 172.21.15.129:0/3780167251 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fcbb0029470 con 0x5629c8e97730 2024-05-20T18:04:02.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.984+0000 7fcbc79e0740 10 client.26011 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:02.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.984+0000 7fcbc79e0740 10 client.26011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:02.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.984+0000 7fcbc79e0740 20 client.26011 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:02.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.984+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/3780167251 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5629c92ff450 0x5629c9301870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:02.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.984+0000 7fcbc79e0740 4 client.26011 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:04:02.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.984+0000 7fcbc79e0740 1 -- 172.21.15.129:0/3780167251 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5629c91c4410 con 0x5629c92ff450 2024-05-20T18:04:02.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.987+0000 7fcbba59c640 1 --2- 172.21.15.129:0/3780167251 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5629c92ff450 0x5629c9301870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:02.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.997+0000 7fcbba59c640 1 --2- 172.21.15.129:0/3780167251 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5629c92ff450 0x5629c9301870 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40cddcec138fe94a server_cookie=2540761c4113219a in_seq=0 out_seq=0 2024-05-20T18:04:02.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:02.997+0000 7fcba1ffb640 10 client.26011 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:03.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.278+0000 7fcba1ffb640 1 -- 172.21.15.129:0/3780167251 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5629c91c4410 con 0x5629c92ff450 2024-05-20T18:04:03.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.278+0000 7fcba1ffb640 10 client.26011 handle_command_reply: tid=0 2024-05-20T18:04:03.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.278+0000 7fcbc79e0740 1 client.26011 shutdown 2024-05-20T18:04:03.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.278+0000 7fcbc79e0740 1 -- 172.21.15.129:0/3780167251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 msgr2=0x5629c926d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.278+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/3780167251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c926d080 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00318d0 tx=0x7fcbb0009a00 comp rx=0 tx=0).stop 2024-05-20T18:04:03.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.279+0000 7fcbc79e0740 1 -- 172.21.15.129:0/3780167251 shutdown_connections 2024-05-20T18:04:03.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.279+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/3780167251 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5629c92ff450 0x5629c9301870 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.279+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/3780167251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5629c8e8f550 0x5629c926cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.279+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/3780167251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5629c8e97730 0x5629c926d080 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.279+0000 7fcbc79e0740 1 --2- 172.21.15.129:0/3780167251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5629c8e771f0 0x5629c926c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.279+0000 7fcbc79e0740 1 -- 172.21.15.129:0/3780167251 >> 172.21.15.129:0/3780167251 conn(0x5629c8eac560 msgr2=0x5629c925b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.279+0000 7fcba1ffb640 0 client.26011 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.279+0000 7fcbc79e0740 1 -- 172.21.15.129:0/3780167251 shutdown_connections 2024-05-20T18:04:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.279+0000 7fcbc79e0740 1 -- 172.21.15.129:0/3780167251 wait complete. 2024-05-20T18:04:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.279+0000 7fcbc79e0740 20 client.26011 trim_cache size 0 max 16384 2024-05-20T18:04:03.279 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:04:03.280 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:04:03.280 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "1bf91c3e-45f7-4fda-8303-03f6baf0eadd", 2024-05-20T18:04:03.280 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:04:03.280 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:04:03.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.280+0000 7fcbbb59e640 1 -- 172.21.15.129:0/2069564711 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcba40a2dc0 msgr2=0x7fcba40a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.280+0000 7fcbbb59e640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcba40a2dc0 0x7fcba40a52d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc06b490 tx=0x7fcbb0011810 comp rx=0 tx=0).stop 2024-05-20T18:04:03.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.280+0000 7fcbbb59e640 1 -- 172.21.15.129:0/2069564711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbbc108f70 msgr2=0x7fcbbc119150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.280+0000 7fcbbb59e640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbbc108f70 0x7fcbbc119150 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009fd0 tx=0x7fcbac00b870 comp rx=0 tx=0).stop 2024-05-20T18:04:03.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.280+0000 7fcbbb59e640 1 -- 172.21.15.129:0/2069564711 shutdown_connections 2024-05-20T18:04:03.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.280+0000 7fcbbb59e640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fcba40a2dc0 0x7fcba40a52d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.280+0000 7fcbbb59e640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbbc119770 0x7fcbbc112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.280+0000 7fcbbb59e640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbbc11c0d0 0x7fcbbc1128a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.280+0000 7fcbbb59e640 1 --2- 172.21.15.129:0/2069564711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcbbc108f70 0x7fcbbc119150 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.280+0000 7fcbbb59e640 1 -- 172.21.15.129:0/2069564711 >> 172.21.15.129:0/2069564711 conn(0x7fcbbc06e600 msgr2=0x7fcbbc119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:03.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.280+0000 7fcbbb59e640 1 -- 172.21.15.129:0/2069564711 shutdown_connections 2024-05-20T18:04:03.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.281+0000 7fcbbb59e640 1 -- 172.21.15.129:0/2069564711 wait complete. 2024-05-20T18:04:03.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:04:03.303 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:04:03.342 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:03 smithi129 ceph-mon[32029]: pgmap v169: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 760 MiB/s wr, 1.71k op/s 2024-05-20T18:04:03.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:03 smithi154 ceph-mon[36482]: pgmap v169: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 760 MiB/s wr, 1.71k op/s 2024-05-20T18:04:03.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:03 smithi204 ceph-mon[37076]: pgmap v169: 97 pgs: 97 active+clean; 40 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 760 MiB/s wr, 1.71k op/s 2024-05-20T18:04:03.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.481+0000 7f134e444640 1 -- 172.21.15.129:0/457765847 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1348107e70 msgr2=0x7f1348102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.481+0000 7f134e444640 1 --2- 172.21.15.129:0/457765847 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1348107e70 0x7f1348102110 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f133c009900 tx=0x7f133c02a410 comp rx=0 tx=0).stop 2024-05-20T18:04:03.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.481+0000 7f134e444640 1 -- 172.21.15.129:0/457765847 shutdown_connections 2024-05-20T18:04:03.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.481+0000 7f134e444640 1 --2- 172.21.15.129:0/457765847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1348102780 0x7f134810e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.481+0000 7f134e444640 1 --2- 172.21.15.129:0/457765847 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1348107e70 0x7f1348102110 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.481+0000 7f134e444640 1 --2- 172.21.15.129:0/457765847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13481079d0 0x7f1348101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.481+0000 7f134e444640 1 -- 172.21.15.129:0/457765847 >> 172.21.15.129:0/457765847 conn(0x7f1348076830 msgr2=0x7f1348076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:03.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.482+0000 7f134e444640 1 -- 172.21.15.129:0/457765847 shutdown_connections 2024-05-20T18:04:03.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.482+0000 7f134e444640 1 -- 172.21.15.129:0/457765847 wait complete. 2024-05-20T18:04:03.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.483+0000 7f134e444640 1 Processor -- start 2024-05-20T18:04:03.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.483+0000 7f134e444640 1 -- start start 2024-05-20T18:04:03.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.483+0000 7f134e444640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1348102780 0x7f13481a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.484+0000 7f134e444640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13481079d0 0x7f13481a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.484+0000 7f134e444640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1348107e70 0x7f134819b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.484+0000 7f134e444640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1348115020 con 0x7f13481079d0 2024-05-20T18:04:03.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.484+0000 7f134e444640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1348114ea0 con 0x7f1348107e70 2024-05-20T18:04:03.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.484+0000 7f134e444640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13481151a0 con 0x7f1348102780 2024-05-20T18:04:03.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.485+0000 7f13477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13481079d0 0x7f13481a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.485+0000 7f13477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13481079d0 0x7f13481a1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34700/0 (socket says 172.21.15.129:34700) 2024-05-20T18:04:03.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.485+0000 7f13477fe640 1 -- 172.21.15.129:0/773436340 learned_addr learned my addr 172.21.15.129:0/773436340 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:03.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.485+0000 7f134c9e1640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1348107e70 0x7f134819b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.486+0000 7f1347fff640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1348102780 0x7f13481a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.486+0000 7f13477fe640 1 -- 172.21.15.129:0/773436340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1348102780 msgr2=0x7f13481a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.486+0000 7f13477fe640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1348102780 0x7f13481a1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.486+0000 7f13477fe640 1 -- 172.21.15.129:0/773436340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1348107e70 msgr2=0x7f134819b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.486+0000 7f13477fe640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1348107e70 0x7f134819b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.486+0000 7f13477fe640 1 -- 172.21.15.129:0/773436340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134819bf40 con 0x7f13481079d0 2024-05-20T18:04:03.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.486+0000 7f13477fe640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13481079d0 0x7f13481a1b70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f133c005940 tx=0x7f133c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:03.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.487+0000 7f13457fa640 1 -- 172.21.15.129:0/773436340 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c03f030 con 0x7f13481079d0 2024-05-20T18:04:03.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.487+0000 7f134e444640 1 -- 172.21.15.129:0/773436340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134819c1d0 con 0x7f13481079d0 2024-05-20T18:04:03.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.487+0000 7f134e444640 1 -- 172.21.15.129:0/773436340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13481a8480 con 0x7f13481079d0 2024-05-20T18:04:03.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.487+0000 7f13457fa640 1 -- 172.21.15.129:0/773436340 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c003ad0 con 0x7f13481079d0 2024-05-20T18:04:03.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.487+0000 7f13457fa640 1 -- 172.21.15.129:0/773436340 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c035070 con 0x7f13481079d0 2024-05-20T18:04:03.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.495+0000 7f13457fa640 1 -- 172.21.15.129:0/773436340 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f133c042030 con 0x7f13481079d0 2024-05-20T18:04:03.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.496+0000 7f13553ca740 1 Processor -- start 2024-05-20T18:04:03.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.497+0000 7f13553ca740 1 -- start start 2024-05-20T18:04:03.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.497+0000 7f13457fa640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13300a2df0 0x7f13300a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.497+0000 7f13457fa640 1 -- 172.21.15.129:0/773436340 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f133c0ef990 con 0x7f13481079d0 2024-05-20T18:04:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.499+0000 7f134c9e1640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13300a2df0 0x7f13300a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.499+0000 7f13553ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74e1390 0x55f6b787fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.499+0000 7f13553ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74b84d0 0x55f6b7880270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.499+0000 7f13553ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74c06b0 0x55f6b78807b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.499+0000 7f13553ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f6b77e5d00 con 0x55f6b74b84d0 2024-05-20T18:04:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.499+0000 7f13553ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f6b778d4c0 con 0x55f6b74e1390 2024-05-20T18:04:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.499+0000 7f13553ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f6b77e1cc0 con 0x55f6b74c06b0 2024-05-20T18:04:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.500+0000 7f13477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74b84d0 0x55f6b7880270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.500+0000 7f13477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74b84d0 0x55f6b7880270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34702/0 (socket says 172.21.15.129:34702) 2024-05-20T18:04:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.500+0000 7f13477fe640 1 -- 172.21.15.129:0/1760021660 learned_addr learned my addr 172.21.15.129:0/1760021660 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.500+0000 7f134c9e1640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13300a2df0 0x7f13300a5300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1340005fd0 tx=0x7f1340009500 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.500+0000 7f13477fe640 1 -- 172.21.15.129:0/1760021660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74c06b0 msgr2=0x55f6b78807b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:04:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.500+0000 7f13477fe640 1 --2- 172.21.15.129:0/1760021660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74c06b0 0x55f6b78807b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.500+0000 7f13477fe640 1 -- 172.21.15.129:0/1760021660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74e1390 msgr2=0x55f6b787fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.500+0000 7f13477fe640 1 --2- 172.21.15.129:0/1760021660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74e1390 0x55f6b787fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.500+0000 7f13477fe640 1 -- 172.21.15.129:0/1760021660 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6b75c37d0 con 0x55f6b74b84d0 2024-05-20T18:04:03.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.501+0000 7f13477fe640 1 --2- 172.21.15.129:0/1760021660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74b84d0 0x55f6b7880270 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f133c0c4580 tx=0x7f133c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0fe620155c9b50 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:03.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.503+0000 7f132e7fc640 1 -- 172.21.15.129:0/1760021660 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0be070 con 0x55f6b74b84d0 2024-05-20T18:04:03.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.504+0000 7f132e7fc640 1 -- 172.21.15.129:0/1760021660 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c0c01f0 con 0x55f6b74b84d0 2024-05-20T18:04:03.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.505+0000 7f132e7fc640 1 -- 172.21.15.129:0/1760021660 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0be070 con 0x55f6b74b84d0 2024-05-20T18:04:03.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.506+0000 7f13553ca740 1 -- 172.21.15.129:0/1760021660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74b84d0 msgr2=0x55f6b7880270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.506+0000 7f13553ca740 1 --2- 172.21.15.129:0/1760021660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74b84d0 0x55f6b7880270 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f133c0c4580 tx=0x7f133c02a410 comp rx=0 tx=0).stop 2024-05-20T18:04:03.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.506+0000 7f13553ca740 1 -- 172.21.15.129:0/1760021660 shutdown_connections 2024-05-20T18:04:03.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.506+0000 7f13553ca740 1 --2- 172.21.15.129:0/1760021660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74c06b0 0x55f6b78807b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.506+0000 7f13553ca740 1 --2- 172.21.15.129:0/1760021660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74b84d0 0x55f6b7880270 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.506+0000 7f13553ca740 1 --2- 172.21.15.129:0/1760021660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74e1390 0x55f6b787fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.506+0000 7f13553ca740 1 -- 172.21.15.129:0/1760021660 >> 172.21.15.129:0/1760021660 conn(0x55f6b74d54e0 msgr2=0x55f6b787c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:03.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.506+0000 7f13553ca740 1 -- 172.21.15.129:0/1760021660 shutdown_connections 2024-05-20T18:04:03.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.506+0000 7f13553ca740 1 -- 172.21.15.129:0/1760021660 wait complete. 2024-05-20T18:04:03.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.507+0000 7f13553ca740 1 -- start start 2024-05-20T18:04:03.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.507+0000 7f13553ca740 1 Processor -- start 2024-05-20T18:04:03.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.508+0000 7f13553ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74c06b0 0x55f6b788bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.508+0000 7f13553ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74e1390 0x55f6b788c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.508+0000 7f13553ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b749ff60 0x55f6b788a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.508+0000 7f13553ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f6b788ce40 con 0x55f6b74e1390 2024-05-20T18:04:03.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.508+0000 7f13553ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f6b77a0b90 con 0x55f6b74c06b0 2024-05-20T18:04:03.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.508+0000 7f13553ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f6b788cfc0 con 0x55f6b749ff60 2024-05-20T18:04:03.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.508+0000 7f13553ca740 10 client.0 _ll_register_callbacks cb 0x55f6b77dcda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:03.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.509+0000 7f13477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74e1390 0x55f6b788c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.509+0000 7f13477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74e1390 0x55f6b788c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34706/0 (socket says 172.21.15.129:34706) 2024-05-20T18:04:03.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.509+0000 7f13477fe640 1 -- 172.21.15.129:0/4071763678 learned_addr learned my addr 172.21.15.129:0/4071763678 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:03.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.509+0000 7f13477fe640 1 -- 172.21.15.129:0/4071763678 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b749ff60 msgr2=0x55f6b788a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:03.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.509+0000 7f13477fe640 1 --2- 172.21.15.129:0/4071763678 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b749ff60 0x55f6b788a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.509+0000 7f13477fe640 1 -- 172.21.15.129:0/4071763678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74c06b0 msgr2=0x55f6b788bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:04:03.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.509+0000 7f13477fe640 1 --2- 172.21.15.129:0/4071763678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74c06b0 0x55f6b788bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.510+0000 7f13477fe640 1 -- 172.21.15.129:0/4071763678 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6b74f93f0 con 0x55f6b74e1390 2024-05-20T18:04:03.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.510+0000 7f13277fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:04:03.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.510+0000 7f13477fe640 1 --2- 172.21.15.129:0/4071763678 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74e1390 0x55f6b788c340 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f133c0bbdd0 tx=0x7f133c0c4ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0cba4c5d0350b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:03.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.510+0000 7f13277fe640 1 -- 172.21.15.129:0/4071763678 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c09f060 con 0x55f6b74e1390 2024-05-20T18:04:03.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.511+0000 7f13277fe640 1 -- 172.21.15.129:0/4071763678 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f133c0bc0a0 con 0x55f6b74e1390 2024-05-20T18:04:03.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.511+0000 7f13553ca740 1 -- 172.21.15.129:0/4071763678 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f133c0c01f0 con 0x55f6b74e1390 2024-05-20T18:04:03.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.511+0000 7f13277fe640 1 -- 172.21.15.129:0/4071763678 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f133c0be070 con 0x55f6b74e1390 2024-05-20T18:04:03.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.511+0000 7f13553ca740 10 client.16248 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:03.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.511+0000 7f13553ca740 1 -- 172.21.15.129:0/4071763678 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f6b74b90f0 con 0x55f6b74e1390 2024-05-20T18:04:03.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.512+0000 7f13277fe640 1 -- 172.21.15.129:0/4071763678 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c0c01f0 con 0x55f6b74e1390 2024-05-20T18:04:03.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.512+0000 7f13277fe640 1 -- 172.21.15.129:0/4071763678 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f133c03b9c0 con 0x55f6b74e1390 2024-05-20T18:04:03.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.512+0000 7f13553ca740 10 client.16248 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:03.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.512+0000 7f13553ca740 10 client.16248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:03.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.512+0000 7f13553ca740 20 client.16248 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:03.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.512+0000 7f13553ca740 1 --2- 172.21.15.129:0/4071763678 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f6b74753a0 0x55f6b78a5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.512+0000 7f13553ca740 4 client.16248 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:04:03.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.512+0000 7f13553ca740 1 -- 172.21.15.129:0/4071763678 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6b77f3430 con 0x55f6b74753a0 2024-05-20T18:04:03.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.514+0000 7f1347fff640 1 --2- 172.21.15.129:0/4071763678 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f6b74753a0 0x55f6b78a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.519+0000 7f1347fff640 1 --2- 172.21.15.129:0/4071763678 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f6b74753a0 0x55f6b78a5270 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d28ced58fa4914 server_cookie=22906c8de54c7027 in_seq=0 out_seq=0 2024-05-20T18:04:03.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.519+0000 7f13277fe640 10 client.16248 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.526+0000 7f13277fe640 1 -- 172.21.15.129:0/4071763678 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f6b77f3430 con 0x55f6b74753a0 2024-05-20T18:04:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.526+0000 7f13277fe640 10 client.16248 handle_command_reply: tid=0 2024-05-20T18:04:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.527+0000 7f13553ca740 1 client.16248 shutdown 2024-05-20T18:04:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.527+0000 7f13553ca740 1 -- 172.21.15.129:0/4071763678 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74e1390 msgr2=0x55f6b788c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.527+0000 7f13553ca740 1 --2- 172.21.15.129:0/4071763678 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74e1390 0x55f6b788c340 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f133c0bbdd0 tx=0x7f133c0c4ab0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.527+0000 7f13553ca740 1 -- 172.21.15.129:0/4071763678 shutdown_connections 2024-05-20T18:04:03.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.527+0000 7f13553ca740 1 --2- 172.21.15.129:0/4071763678 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f6b74753a0 0x55f6b78a5270 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.527+0000 7f13553ca740 1 --2- 172.21.15.129:0/4071763678 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b749ff60 0x55f6b788a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.527+0000 7f13553ca740 1 --2- 172.21.15.129:0/4071763678 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74e1390 0x55f6b788c340 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.527+0000 7f13277fe640 0 client.16248 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:03.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.527+0000 7f13553ca740 1 --2- 172.21.15.129:0/4071763678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74c06b0 0x55f6b788bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.527+0000 7f13553ca740 1 -- 172.21.15.129:0/4071763678 >> 172.21.15.129:0/4071763678 conn(0x55f6b74d54e0 msgr2=0x55f6b7881a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:03.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.528+0000 7f13553ca740 1 -- 172.21.15.129:0/4071763678 shutdown_connections 2024-05-20T18:04:03.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.528+0000 7f13553ca740 1 -- 172.21.15.129:0/4071763678 wait complete. 2024-05-20T18:04:03.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.528+0000 7f13553ca740 20 client.16248 trim_cache size 0 max 16384 2024-05-20T18:04:03.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.576+0000 7f13553ca740 1 Processor -- start 2024-05-20T18:04:03.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.581+0000 7f13553ca740 1 -- start start 2024-05-20T18:04:03.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.582+0000 7f13553ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74d54e0 0x55f6b7884090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.582+0000 7f13553ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74753a0 0x55f6b7885410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.583+0000 7f13553ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b749ff60 0x55f6b7889240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.583+0000 7f13553ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f133400c750 con 0x55f6b74753a0 2024-05-20T18:04:03.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.583+0000 7f13553ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f133400f010 con 0x55f6b749ff60 2024-05-20T18:04:03.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.583+0000 7f13553ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f133400ce00 con 0x55f6b74d54e0 2024-05-20T18:04:03.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.583+0000 7f13477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74753a0 0x55f6b7885410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.583+0000 7f13477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74753a0 0x55f6b7885410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34710/0 (socket says 172.21.15.129:34710) 2024-05-20T18:04:03.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.583+0000 7f13477fe640 1 -- 172.21.15.129:0/1133841889 learned_addr learned my addr 172.21.15.129:0/1133841889 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:03.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.586+0000 7f13477fe640 1 -- 172.21.15.129:0/1133841889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74d54e0 msgr2=0x55f6b7884090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:03.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.586+0000 7f13477fe640 1 --2- 172.21.15.129:0/1133841889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74d54e0 0x55f6b7884090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.586+0000 7f13477fe640 1 -- 172.21.15.129:0/1133841889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b749ff60 msgr2=0x55f6b7889240 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:04:03.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.586+0000 7f13477fe640 1 --2- 172.21.15.129:0/1133841889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b749ff60 0x55f6b7889240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.586+0000 7f13477fe640 1 -- 172.21.15.129:0/1133841889 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6b78ab380 con 0x55f6b74753a0 2024-05-20T18:04:03.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.594+0000 7f13477fe640 1 --2- 172.21.15.129:0/1133841889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74753a0 0x55f6b7885410 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f133c0bd0e0 tx=0x7f133c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b0b535d1d1791c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:03.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.598+0000 7f132dffb640 1 -- 172.21.15.129:0/1133841889 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0be070 con 0x55f6b74753a0 2024-05-20T18:04:03.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.598+0000 7f132dffb640 1 -- 172.21.15.129:0/1133841889 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c0ef750 con 0x55f6b74753a0 2024-05-20T18:04:03.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.598+0000 7f132dffb640 1 -- 172.21.15.129:0/1133841889 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0da1b0 con 0x55f6b74753a0 2024-05-20T18:04:03.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.601+0000 7f13553ca740 1 -- 172.21.15.129:0/1133841889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74753a0 msgr2=0x55f6b7885410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.601+0000 7f13553ca740 1 --2- 172.21.15.129:0/1133841889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74753a0 0x55f6b7885410 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f133c0bd0e0 tx=0x7f133c005ef0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.602+0000 7f13553ca740 1 -- 172.21.15.129:0/1133841889 shutdown_connections 2024-05-20T18:04:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.602+0000 7f13553ca740 1 --2- 172.21.15.129:0/1133841889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b749ff60 0x55f6b7889240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.602+0000 7f13553ca740 1 --2- 172.21.15.129:0/1133841889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b74753a0 0x55f6b7885410 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.602+0000 7f13553ca740 1 --2- 172.21.15.129:0/1133841889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74d54e0 0x55f6b7884090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.602+0000 7f13553ca740 1 -- 172.21.15.129:0/1133841889 >> 172.21.15.129:0/1133841889 conn(0x55f6b7885be0 msgr2=0x55f6b7888b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:03.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.608+0000 7f13553ca740 1 -- 172.21.15.129:0/1133841889 shutdown_connections 2024-05-20T18:04:03.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.610+0000 7f13553ca740 1 -- 172.21.15.129:0/1133841889 wait complete. 2024-05-20T18:04:03.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.611+0000 7f13553ca740 1 -- start start 2024-05-20T18:04:03.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.612+0000 7f13553ca740 1 Processor -- start 2024-05-20T18:04:03.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.614+0000 7f13553ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74753a0 0x55f6b78a7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.615+0000 7f13553ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b749ff60 0x55f6b78a7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.615+0000 7f13553ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74d54e0 0x55f6b790b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.615+0000 7f13553ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f133400d4d0 con 0x55f6b749ff60 2024-05-20T18:04:03.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.615+0000 7f13553ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f133400e940 con 0x55f6b74d54e0 2024-05-20T18:04:03.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.615+0000 7f13553ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f133400dba0 con 0x55f6b74753a0 2024-05-20T18:04:03.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.616+0000 7f13553ca740 10 client.0 _ll_register_callbacks cb 0x7f133c0bb980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:03.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.619+0000 7f13477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b749ff60 0x55f6b78a7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.619+0000 7f13477fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b749ff60 0x55f6b78a7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34716/0 (socket says 172.21.15.129:34716) 2024-05-20T18:04:03.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.619+0000 7f13477fe640 1 -- 172.21.15.129:0/692430707 learned_addr learned my addr 172.21.15.129:0/692430707 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:03.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.621+0000 7f134c9e1640 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74d54e0 0x55f6b790b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.621+0000 7f1347fff640 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74753a0 0x55f6b78a7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.622+0000 7f13477fe640 1 -- 172.21.15.129:0/692430707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74753a0 msgr2=0x55f6b78a7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:03.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.622+0000 7f13477fe640 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74753a0 0x55f6b78a7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.622+0000 7f13477fe640 1 -- 172.21.15.129:0/692430707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74d54e0 msgr2=0x55f6b790b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:03.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.622+0000 7f13477fe640 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74d54e0 0x55f6b790b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.622+0000 7f13477fe640 1 -- 172.21.15.129:0/692430707 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6b790ba10 con 0x55f6b749ff60 2024-05-20T18:04:03.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.622+0000 7f134c9e1640 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74d54e0 0x55f6b790b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:04:03.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.626+0000 7f13477fe640 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b749ff60 0x55f6b78a7ab0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f133c0bbb40 tx=0x7f133c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a01130c910f4370 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:03.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.626+0000 7f13277fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:04:03.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.632+0000 7f13277fe640 1 -- 172.21.15.129:0/692430707 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0be070 con 0x55f6b749ff60 2024-05-20T18:04:03.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.633+0000 7f13277fe640 1 -- 172.21.15.129:0/692430707 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f133c0c1380 con 0x55f6b749ff60 2024-05-20T18:04:03.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.634+0000 7f13553ca740 1 -- 172.21.15.129:0/692430707 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f133c0ef750 con 0x55f6b749ff60 2024-05-20T18:04:03.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.643+0000 7f13277fe640 1 -- 172.21.15.129:0/692430707 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f133c0c8050 con 0x55f6b749ff60 2024-05-20T18:04:03.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.651+0000 7f13553ca740 10 client.16254 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:03.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.652+0000 7f13553ca740 1 -- 172.21.15.129:0/692430707 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f6b790ba10 con 0x55f6b749ff60 2024-05-20T18:04:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.660+0000 7f13277fe640 1 -- 172.21.15.129:0/692430707 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c0ef750 con 0x55f6b749ff60 2024-05-20T18:04:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.660+0000 7f13277fe640 1 -- 172.21.15.129:0/692430707 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f133c03b9c0 con 0x55f6b749ff60 2024-05-20T18:04:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.660+0000 7f13553ca740 10 client.16254 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:03.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.660+0000 7f13553ca740 10 client.16254 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:03.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.661+0000 7f13553ca740 20 client.16254 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:03.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.661+0000 7f13553ca740 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f6b74c06b0 0x55f6b790eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:03.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.661+0000 7f13553ca740 4 client.16254 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:04:03.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.661+0000 7f13553ca740 1 -- 172.21.15.129:0/692430707 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f6b77ed780 con 0x55f6b74c06b0 2024-05-20T18:04:03.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.667+0000 7f1347fff640 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f6b74c06b0 0x55f6b790eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:03.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.683+0000 7f1347fff640 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f6b74c06b0 0x55f6b790eed0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55a42e79c53f3bb6 server_cookie=f9572d55af69eec2 in_seq=0 out_seq=0 2024-05-20T18:04:03.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.683+0000 7f13277fe640 10 client.16254 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:03.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.684+0000 7f13277fe640 1 -- 172.21.15.129:0/692430707 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f6b77ed780 con 0x55f6b74c06b0 2024-05-20T18:04:03.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.684+0000 7f13277fe640 10 client.16254 handle_command_reply: tid=0 2024-05-20T18:04:03.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.684+0000 7f13553ca740 1 client.16254 shutdown 2024-05-20T18:04:03.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.687+0000 7f13553ca740 1 -- 172.21.15.129:0/692430707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b749ff60 msgr2=0x55f6b78a7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.687+0000 7f13553ca740 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b749ff60 0x55f6b78a7ab0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f133c0bbb40 tx=0x7f133c02a410 comp rx=0 tx=0).stop 2024-05-20T18:04:03.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.691+0000 7f13553ca740 1 -- 172.21.15.129:0/692430707 shutdown_connections 2024-05-20T18:04:03.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.691+0000 7f13553ca740 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f6b74c06b0 0x55f6b790eed0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.691+0000 7f13553ca740 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f6b74d54e0 0x55f6b790b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.691+0000 7f13553ca740 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f6b749ff60 0x55f6b78a7ab0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.691+0000 7f13553ca740 1 --2- 172.21.15.129:0/692430707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6b74753a0 0x55f6b78a7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.691+0000 7f13553ca740 1 -- 172.21.15.129:0/692430707 >> 172.21.15.129:0/692430707 conn(0x55f6b7885be0 msgr2=0x55f6b788a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:03.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.691+0000 7f13277fe640 0 client.16254 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:03.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.693+0000 7f13553ca740 1 -- 172.21.15.129:0/692430707 shutdown_connections 2024-05-20T18:04:03.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.693+0000 7f13553ca740 1 -- 172.21.15.129:0/692430707 wait complete. 2024-05-20T18:04:03.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.698+0000 7f13553ca740 20 client.16254 trim_cache size 0 max 16384 2024-05-20T18:04:03.699 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:04:03.699 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:04:03.699 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:04:03.699 INFO:teuthology.orchestra.run.smithi129.stdout: "1bf91c3e-45f7-4fda-8303-03f6baf0eadd": { 2024-05-20T18:04:03.699 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:04:03.700 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "1bf91c3e-45f7-4fda-8303-03f6baf0eadd", 2024-05-20T18:04:03.700 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:04:03.700 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:04:03.700 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:04:03.700 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:04:03.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.701+0000 7f134e444640 1 -- 172.21.15.129:0/773436340 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13300a2df0 msgr2=0x7f13300a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.701+0000 7f134e444640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13300a2df0 0x7f13300a5300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1340005fd0 tx=0x7f1340009500 comp rx=0 tx=0).stop 2024-05-20T18:04:03.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.701+0000 7f134e444640 1 -- 172.21.15.129:0/773436340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13481079d0 msgr2=0x7f13481a1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:03.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.701+0000 7f134e444640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13481079d0 0x7f13481a1b70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f133c005940 tx=0x7f133c004270 comp rx=0 tx=0).stop 2024-05-20T18:04:03.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.704+0000 7f134e444640 1 -- 172.21.15.129:0/773436340 shutdown_connections 2024-05-20T18:04:03.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.704+0000 7f134e444640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f13300a2df0 0x7f13300a5300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.704+0000 7f134e444640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1348107e70 0x7f134819b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.704+0000 7f134e444640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13481079d0 0x7f13481a1b70 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.704+0000 7f134e444640 1 --2- 172.21.15.129:0/773436340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1348102780 0x7f13481a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:03.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.704+0000 7f134e444640 1 -- 172.21.15.129:0/773436340 >> 172.21.15.129:0/773436340 conn(0x7f1348076830 msgr2=0x7f13480ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:03.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.705+0000 7f134e444640 1 -- 172.21.15.129:0/773436340 shutdown_connections 2024-05-20T18:04:03.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:03.705+0000 7f134e444640 1 -- 172.21.15.129:0/773436340 wait complete. 2024-05-20T18:04:03.768 INFO:tasks.cephfs.filesystem:scrub status for tag:1bf91c3e-45f7-4fda-8303-03f6baf0eadd - {'path': '/', 'tag': '1bf91c3e-45f7-4fda-8303-03f6baf0eadd', 'options': 'recursive,force'} 2024-05-20T18:04:04.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:04 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:04:04.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:04 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:04:04.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:04 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:04:04.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:04 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:04:04.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:04 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:04:04.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:04 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:04:04.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:04 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:04:04.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:04 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:04:04.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:04 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:04:05.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:05 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:04:05.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:05 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:05.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:05 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:04:05.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:05 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:05.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:05 smithi154 ceph-mon[36482]: pgmap v170: 97 pgs: 97 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 770 MiB/s wr, 1.69k op/s 2024-05-20T18:04:05.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:05 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:04:05.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:05 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:05.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:05 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:04:05.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:05 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:05.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:05 smithi129 ceph-mon[32029]: pgmap v170: 97 pgs: 97 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 770 MiB/s wr, 1.69k op/s 2024-05-20T18:04:05.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:05 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:04:05.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:05 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:05.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:05 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:04:05.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:05 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:05.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:05 smithi204 ceph-mon[37076]: pgmap v170: 97 pgs: 97 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 770 MiB/s wr, 1.69k op/s 2024-05-20T18:04:06.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:06 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:04:06.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:06 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:04:06.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:06 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:04:07.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:07 smithi154 ceph-mon[36482]: pgmap v171: 97 pgs: 97 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 931 MiB/s wr, 2.06k op/s 2024-05-20T18:04:07.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:07 smithi129 ceph-mon[32029]: pgmap v171: 97 pgs: 97 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 931 MiB/s wr, 2.06k op/s 2024-05-20T18:04:07.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:07 smithi204 ceph-mon[37076]: pgmap v171: 97 pgs: 97 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 931 MiB/s wr, 2.06k op/s 2024-05-20T18:04:09.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:09 smithi154 ceph-mon[36482]: pgmap v172: 97 pgs: 97 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 702 MiB/s wr, 1.78k op/s 2024-05-20T18:04:09.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:09 smithi129 ceph-mon[32029]: pgmap v172: 97 pgs: 97 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 702 MiB/s wr, 1.78k op/s 2024-05-20T18:04:09.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:09 smithi204 ceph-mon[37076]: pgmap v172: 97 pgs: 97 active+clean; 40 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 702 MiB/s wr, 1.78k op/s 2024-05-20T18:04:09.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.956+0000 7f54b9145640 1 -- 172.21.15.129:0/768338634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b4109080 msgr2=0x7f54b4109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:09.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.956+0000 7f54b9145640 1 --2- 172.21.15.129:0/768338634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b4109080 0x7f54b4109490 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009860 tx=0x7f54a802a3f0 comp rx=0 tx=0).stop 2024-05-20T18:04:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.957+0000 7f54b9145640 1 -- 172.21.15.129:0/768338634 shutdown_connections 2024-05-20T18:04:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.957+0000 7f54b9145640 1 --2- 172.21.15.129:0/768338634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54b411c0d0 0x7f54b411e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.957+0000 7f54b9145640 1 --2- 172.21.15.129:0/768338634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54b4119770 0x7f54b411bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.957+0000 7f54b9145640 1 --2- 172.21.15.129:0/768338634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b4109080 0x7f54b4109490 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009860 tx=0x7f54a802a3f0 comp rx=0 tx=0).stop 2024-05-20T18:04:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.957+0000 7f54b9145640 1 -- 172.21.15.129:0/768338634 >> 172.21.15.129:0/768338634 conn(0x7f54b406d780 msgr2=0x7f54b406dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:09.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.962+0000 7f54b9145640 1 -- 172.21.15.129:0/768338634 shutdown_connections 2024-05-20T18:04:09.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.966+0000 7f54b9145640 1 -- 172.21.15.129:0/768338634 wait complete. 2024-05-20T18:04:09.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.969+0000 7f54b9145640 1 Processor -- start 2024-05-20T18:04:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.969+0000 7f54b9145640 1 -- start start 2024-05-20T18:04:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.969+0000 7f54b9145640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b4119770 0x7f54b41a8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.969+0000 7f54b9145640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54b411c0d0 0x7f54b41a8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.969+0000 7f54b9145640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54b41acef0 0x7f54b41ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.970+0000 7f54b9145640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54b4121c70 con 0x7f54b41acef0 2024-05-20T18:04:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.970+0000 7f54b9145640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54b4121af0 con 0x7f54b4119770 2024-05-20T18:04:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.970+0000 7f54b9145640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f54b4121df0 con 0x7f54b411c0d0 2024-05-20T18:04:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.970+0000 7f54b259c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54b411c0d0 0x7f54b41a8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.970+0000 7f54b259c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54b411c0d0 0x7f54b41a8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:41390/0 (socket says 172.21.15.129:41390) 2024-05-20T18:04:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.970+0000 7f54b259c640 1 -- 172.21.15.129:0/2092676586 learned_addr learned my addr 172.21.15.129:0/2092676586 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.971+0000 7f54b259c640 1 -- 172.21.15.129:0/2092676586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b4119770 msgr2=0x7f54b41a8790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:04:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.971+0000 7f54b259c640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b4119770 0x7f54b41a8790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.971+0000 7f54b259c640 1 -- 172.21.15.129:0/2092676586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54b41acef0 msgr2=0x7f54b41ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.971+0000 7f54b259c640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54b41acef0 0x7f54b41ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.971+0000 7f54b259c640 1 -- 172.21.15.129:0/2092676586 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b41ada00 con 0x7f54b411c0d0 2024-05-20T18:04:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.972+0000 7f54b259c640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54b411c0d0 0x7f54b41a8cf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f54a400b4a0 tx=0x7f54a400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.972+0000 7f549bfff640 1 -- 172.21.15.129:0/2092676586 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4014070 con 0x7f54b411c0d0 2024-05-20T18:04:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.972+0000 7f549bfff640 1 -- 172.21.15.129:0/2092676586 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a4010af0 con 0x7f54b411c0d0 2024-05-20T18:04:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.972+0000 7f549bfff640 1 -- 172.21.15.129:0/2092676586 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4017d40 con 0x7f54b411c0d0 2024-05-20T18:04:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.972+0000 7f54b9145640 1 -- 172.21.15.129:0/2092676586 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b41adcf0 con 0x7f54b411c0d0 2024-05-20T18:04:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.973+0000 7f54b9145640 1 -- 172.21.15.129:0/2092676586 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b41b5440 con 0x7f54b411c0d0 2024-05-20T18:04:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.982+0000 7f549bfff640 1 -- 172.21.15.129:0/2092676586 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f54a4017070 con 0x7f54b411c0d0 2024-05-20T18:04:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.984+0000 7f549bfff640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f549c0a2e40 0x7f549c0a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.984+0000 7f549bfff640 1 -- 172.21.15.129:0/2092676586 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f54a40ce660 con 0x7f54b411c0d0 2024-05-20T18:04:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.992+0000 7f54bfff3740 1 Processor -- start 2024-05-20T18:04:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.992+0000 7f54bfff3740 1 -- start start 2024-05-20T18:04:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.993+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96fb71b0 0x557a97356080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.993+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973565c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.994+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a97356b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.994+0000 7f54bfff3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557a972bb340 con 0x557a96f8e2f0 2024-05-20T18:04:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.994+0000 7f54bfff3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557a97263290 con 0x557a96fb71b0 2024-05-20T18:04:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.994+0000 7f54bfff3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557a972b7300 con 0x557a96f964d0 2024-05-20T18:04:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.996+0000 7f54b359e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a97356b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.997+0000 7f54b259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.997+0000 7f54b259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973565c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59246/0 (socket says 172.21.15.129:59246) 2024-05-20T18:04:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.997+0000 7f54b259c640 1 -- 172.21.15.129:0/2929728444 learned_addr learned my addr 172.21.15.129:0/2929728444 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.997+0000 7f54b2d9d640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f549c0a2e40 0x7f549c0a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:09.997+0000 7f54b2d9d640 1 --2- 172.21.15.129:0/2929728444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96fb71b0 0x557a97356080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:09.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.001+0000 7f54b259c640 1 -- 172.21.15.129:0/2929728444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 msgr2=0x557a97356b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:09.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.001+0000 7f54b259c640 1 --2- 172.21.15.129:0/2929728444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a97356b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:09.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.001+0000 7f54b259c640 1 -- 172.21.15.129:0/2929728444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96fb71b0 msgr2=0x557a97356080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:09.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.001+0000 7f54b259c640 1 --2- 172.21.15.129:0/2929728444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96fb71b0 0x557a97356080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:09.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.001+0000 7f54b259c640 1 -- 172.21.15.129:0/2929728444 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a972878e0 con 0x557a96f8e2f0 2024-05-20T18:04:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.003+0000 7f54b2d9d640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f549c0a2e40 0x7f549c0a5350 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x557a973531f0 tx=0x7f54a8009410 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.004+0000 7f54b259c640 1 --2- 172.21.15.129:0/2929728444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973565c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f54a409abb0 tx=0x7f54a409b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b6db5fd72108f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.006+0000 7f54997fa640 1 -- 172.21.15.129:0/2929728444 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a401e070 con 0x557a96f8e2f0 2024-05-20T18:04:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.007+0000 7f54997fa640 1 -- 172.21.15.129:0/2929728444 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a40a3890 con 0x557a96f8e2f0 2024-05-20T18:04:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.008+0000 7f54bfff3740 1 -- 172.21.15.129:0/2929728444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 msgr2=0x557a973565c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.008+0000 7f54bfff3740 1 --2- 172.21.15.129:0/2929728444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973565c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f54a409abb0 tx=0x7f54a409b460 comp rx=0 tx=0).stop 2024-05-20T18:04:10.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.018+0000 7f54bfff3740 1 -- 172.21.15.129:0/2929728444 shutdown_connections 2024-05-20T18:04:10.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.018+0000 7f54bfff3740 1 --2- 172.21.15.129:0/2929728444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a97356b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.018+0000 7f54bfff3740 1 --2- 172.21.15.129:0/2929728444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973565c0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.018+0000 7f54bfff3740 1 --2- 172.21.15.129:0/2929728444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96fb71b0 0x557a97356080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.018+0000 7f54bfff3740 1 -- 172.21.15.129:0/2929728444 >> 172.21.15.129:0/2929728444 conn(0x557a96fab300 msgr2=0x557a97352830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:10.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.019+0000 7f54bfff3740 1 -- 172.21.15.129:0/2929728444 shutdown_connections 2024-05-20T18:04:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.022+0000 7f54bfff3740 1 -- 172.21.15.129:0/2929728444 wait complete. 2024-05-20T18:04:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.024+0000 7f54bfff3740 1 -- start start 2024-05-20T18:04:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.024+0000 7f54bfff3740 1 Processor -- start 2024-05-20T18:04:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.025+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96f8e2f0 0x557a97360670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.025+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a97360bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.025+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96fb71b0 0x557a9736c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.025+0000 7f54bfff3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557a973631a0 con 0x557a96fb71b0 2024-05-20T18:04:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.025+0000 7f54bfff3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557a97276790 con 0x557a96f8e2f0 2024-05-20T18:04:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.025+0000 7f54bfff3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557a97363320 con 0x557a96f964d0 2024-05-20T18:04:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.025+0000 7f54bfff3740 10 client.0 _ll_register_callbacks cb 0x557a972b1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.026+0000 7f54b259c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a97360bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.026+0000 7f54b259c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a97360bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:41410/0 (socket says 172.21.15.129:41410) 2024-05-20T18:04:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.026+0000 7f54b259c640 1 -- 172.21.15.129:0/3925021980 learned_addr learned my addr 172.21.15.129:0/3925021980 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.027+0000 7f54b259c640 1 -- 172.21.15.129:0/3925021980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96f8e2f0 msgr2=0x557a97360670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.027+0000 7f54b259c640 1 --2- 172.21.15.129:0/3925021980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96f8e2f0 0x557a97360670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.027+0000 7f54b259c640 1 -- 172.21.15.129:0/3925021980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96fb71b0 msgr2=0x557a9736c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.027+0000 7f54b259c640 1 --2- 172.21.15.129:0/3925021980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96fb71b0 0x557a9736c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.027+0000 7f54b259c640 1 -- 172.21.15.129:0/3925021980 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a96fcf210 con 0x557a96f964d0 2024-05-20T18:04:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.031+0000 7f54b259c640 1 --2- 172.21.15.129:0/3925021980 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a97360bb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f54a409a920 tx=0x7f54a40cbdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e901df34cfc623d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.031+0000 7f5496ff5640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.034+0000 7f5496ff5640 1 -- 172.21.15.129:0/3925021980 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a40b7070 con 0x557a96f964d0 2024-05-20T18:04:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.034+0000 7f5496ff5640 1 -- 172.21.15.129:0/3925021980 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f54a4010610 con 0x557a96f964d0 2024-05-20T18:04:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.035+0000 7f54bfff3740 1 -- 172.21.15.129:0/3925021980 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54a40a3890 con 0x557a96f964d0 2024-05-20T18:04:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.041+0000 7f5496ff5640 1 -- 172.21.15.129:0/3925021980 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f54a401e070 con 0x557a96f964d0 2024-05-20T18:04:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.046+0000 7f54bfff3740 10 client.26026 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.046+0000 7f54bfff3740 1 -- 172.21.15.129:0/3925021980 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557a9709b620 con 0x557a96f964d0 2024-05-20T18:04:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.046+0000 7f5496ff5640 1 -- 172.21.15.129:0/3925021980 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a40a3890 con 0x557a96f964d0 2024-05-20T18:04:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.047+0000 7f5496ff5640 1 -- 172.21.15.129:0/3925021980 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f54a4096970 con 0x557a96f964d0 2024-05-20T18:04:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.052+0000 7f54bfff3740 10 client.26026 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.052+0000 7f54bfff3740 10 client.26026 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.052+0000 7f54bfff3740 20 client.26026 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.052+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3925021980 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a96f75d80 0x557a97372720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.052+0000 7f54bfff3740 4 client.26026 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:04:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.052+0000 7f54bfff3740 1 -- 172.21.15.129:0/3925021980 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a972c8a70 con 0x557a96f75d80 2024-05-20T18:04:10.052 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.055+0000 7f54b2d9d640 1 --2- 172.21.15.129:0/3925021980 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a96f75d80 0x557a97372720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.057+0000 7f54b2d9d640 1 --2- 172.21.15.129:0/3925021980 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a96f75d80 0x557a97372720 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f580191b83e9c6b server_cookie=503b6ffb1b7d2873 in_seq=0 out_seq=0 2024-05-20T18:04:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.057+0000 7f5496ff5640 10 client.26026 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.059+0000 7f5496ff5640 1 -- 172.21.15.129:0/3925021980 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557a972c8a70 con 0x557a96f75d80 2024-05-20T18:04:10.059 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.059+0000 7f5496ff5640 10 client.26026 handle_command_reply: tid=0 2024-05-20T18:04:10.059 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.059+0000 7f54bfff3740 1 client.26026 shutdown 2024-05-20T18:04:10.060 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.062+0000 7f54bfff3740 1 -- 172.21.15.129:0/3925021980 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 msgr2=0x557a97360bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:10.060 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.062+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3925021980 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a97360bb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f54a409a920 tx=0x7f54a40cbdf0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.060 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.062+0000 7f54bfff3740 1 -- 172.21.15.129:0/3925021980 shutdown_connections 2024-05-20T18:04:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.062+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3925021980 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a96f75d80 0x557a97372720 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.062+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3925021980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96fb71b0 0x557a9736c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.062+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3925021980 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a97360bb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.062+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3925021980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96f8e2f0 0x557a97360670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.062+0000 7f54bfff3740 1 -- 172.21.15.129:0/3925021980 >> 172.21.15.129:0/3925021980 conn(0x557a96fab300 msgr2=0x557a97355eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.062+0000 7f5496ff5640 0 client.26026 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.062+0000 7f54bfff3740 1 -- 172.21.15.129:0/3925021980 shutdown_connections 2024-05-20T18:04:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.062+0000 7f54bfff3740 1 -- 172.21.15.129:0/3925021980 wait complete. 2024-05-20T18:04:10.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.067+0000 7f54bfff3740 20 client.26026 trim_cache size 0 max 16384 2024-05-20T18:04:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.128+0000 7f54bfff3740 1 Processor -- start 2024-05-20T18:04:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.128+0000 7f54bfff3740 1 -- start start 2024-05-20T18:04:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.128+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96f8e2f0 0x557a97360010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.128+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a9735ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.128+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96fb71b0 0x557a97356520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.128+0000 7f54bfff3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54a000c8e0 con 0x557a96fb71b0 2024-05-20T18:04:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.128+0000 7f54bfff3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54a000f160 con 0x557a96f8e2f0 2024-05-20T18:04:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.128+0000 7f54bfff3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f54a000cf50 con 0x557a96f964d0 2024-05-20T18:04:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.129+0000 7f54b2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96f8e2f0 0x557a97360010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.129+0000 7f54b2d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96f8e2f0 0x557a97360010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:47366/0 (socket says 172.21.15.129:47366) 2024-05-20T18:04:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.129+0000 7f54b2d9d640 1 -- 172.21.15.129:0/442817362 learned_addr learned my addr 172.21.15.129:0/442817362 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.129+0000 7f54b359e640 1 --2- 172.21.15.129:0/442817362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96fb71b0 0x557a97356520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.130+0000 7f54b259c640 1 --2- 172.21.15.129:0/442817362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a9735ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.130+0000 7f54b259c640 1 -- 172.21.15.129:0/442817362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96f8e2f0 msgr2=0x557a97360010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.130+0000 7f54b259c640 1 --2- 172.21.15.129:0/442817362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96f8e2f0 0x557a97360010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.130+0000 7f54b259c640 1 -- 172.21.15.129:0/442817362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96fb71b0 msgr2=0x557a97356520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.130+0000 7f54b259c640 1 --2- 172.21.15.129:0/442817362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96fb71b0 0x557a97356520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.130+0000 7f54b259c640 1 -- 172.21.15.129:0/442817362 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a9735e9b0 con 0x557a96f964d0 2024-05-20T18:04:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.134+0000 7f54b259c640 1 --2- 172.21.15.129:0/442817362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a9735ace0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f54a4098ed0 tx=0x7f54a40cc300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27eeac7dad42c109 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.134+0000 7f5498ff9640 1 -- 172.21.15.129:0/442817362 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a40a8070 con 0x557a96f964d0 2024-05-20T18:04:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.135+0000 7f5498ff9640 1 -- 172.21.15.129:0/442817362 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a40a3890 con 0x557a96f964d0 2024-05-20T18:04:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.136+0000 7f54bfff3740 1 -- 172.21.15.129:0/442817362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 msgr2=0x557a9735ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.136+0000 7f54bfff3740 1 --2- 172.21.15.129:0/442817362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a9735ace0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f54a4098ed0 tx=0x7f54a40cc300 comp rx=0 tx=0).stop 2024-05-20T18:04:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.139+0000 7f54bfff3740 1 -- 172.21.15.129:0/442817362 shutdown_connections 2024-05-20T18:04:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.139+0000 7f54bfff3740 1 --2- 172.21.15.129:0/442817362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96fb71b0 0x557a97356520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.139+0000 7f54bfff3740 1 --2- 172.21.15.129:0/442817362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a9735ace0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.139+0000 7f54bfff3740 1 --2- 172.21.15.129:0/442817362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96f8e2f0 0x557a97360010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.139+0000 7f54bfff3740 1 -- 172.21.15.129:0/442817362 >> 172.21.15.129:0/442817362 conn(0x557a96fab300 msgr2=0x557a97355e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.143+0000 7f54bfff3740 1 -- 172.21.15.129:0/442817362 shutdown_connections 2024-05-20T18:04:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.144+0000 7f54bfff3740 1 -- 172.21.15.129:0/442817362 wait complete. 2024-05-20T18:04:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.144+0000 7f54bfff3740 1 -- start start 2024-05-20T18:04:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.145+0000 7f54bfff3740 1 Processor -- start 2024-05-20T18:04:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.145+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973d29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.145+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a973d2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.145+0000 7f54bfff3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96fb71b0 0x557a973d3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.145+0000 7f54bfff3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54a000d620 con 0x557a96f8e2f0 2024-05-20T18:04:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.145+0000 7f54bfff3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54a000ea90 con 0x557a96fb71b0 2024-05-20T18:04:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.145+0000 7f54bfff3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f54a000dcf0 con 0x557a96f964d0 2024-05-20T18:04:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.145+0000 7f54bfff3740 10 client.0 _ll_register_callbacks cb 0x7f54a409bf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.146+0000 7f54b2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973d29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.146+0000 7f54b2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973d29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59278/0 (socket says 172.21.15.129:59278) 2024-05-20T18:04:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.146+0000 7f54b2d9d640 1 -- 172.21.15.129:0/3623055668 learned_addr learned my addr 172.21.15.129:0/3623055668 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.147+0000 7f54b259c640 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a973d2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.147+0000 7f54b359e640 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96fb71b0 0x557a973d3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.147+0000 7f54b2d9d640 1 -- 172.21.15.129:0/3623055668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 msgr2=0x557a973d2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.147+0000 7f54b2d9d640 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a973d2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.147+0000 7f54b2d9d640 1 -- 172.21.15.129:0/3623055668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96fb71b0 msgr2=0x557a973d3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.147+0000 7f54b2d9d640 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96fb71b0 0x557a973d3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.147+0000 7f54b2d9d640 1 -- 172.21.15.129:0/3623055668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a973677b0 con 0x557a96f8e2f0 2024-05-20T18:04:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.148+0000 7f54b2d9d640 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973d29d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f54a8030f10 tx=0x7f54a8030b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618ae2714406f3f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.148+0000 7f5496ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:04:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.148+0000 7f5496ff5640 1 -- 172.21.15.129:0/3623055668 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8005310 con 0x557a96f8e2f0 2024-05-20T18:04:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.149+0000 7f54bfff3740 1 -- 172.21.15.129:0/3623055668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54a40a3890 con 0x557a96f8e2f0 2024-05-20T18:04:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.149+0000 7f5496ff5640 1 -- 172.21.15.129:0/3623055668 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f54a80054b0 con 0x557a96f8e2f0 2024-05-20T18:04:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.150+0000 7f5496ff5640 1 -- 172.21.15.129:0/3623055668 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f54a80046c0 con 0x557a96f8e2f0 2024-05-20T18:04:10.149 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.151+0000 7f54bfff3740 10 client.16263 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:10.149 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.151+0000 7f54bfff3740 1 -- 172.21.15.129:0/3623055668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557a97368a50 con 0x557a96f8e2f0 2024-05-20T18:04:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.152+0000 7f5496ff5640 1 -- 172.21.15.129:0/3623055668 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54a40a3890 con 0x557a96f8e2f0 2024-05-20T18:04:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.152+0000 7f5496ff5640 1 -- 172.21.15.129:0/3623055668 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f54a8051630 con 0x557a96f8e2f0 2024-05-20T18:04:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.154+0000 7f54bfff3740 10 client.16263 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.154+0000 7f54bfff3740 10 client.16263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.154+0000 7f54bfff3740 20 client.16263 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.154+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a96f4b1c0 0x557a973ed890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.154+0000 7f54bfff3740 4 client.16263 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:04:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.154+0000 7f54bfff3740 1 -- 172.21.15.129:0/3623055668 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a9735c040 con 0x557a96f4b1c0 2024-05-20T18:04:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.155+0000 7f54b259c640 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a96f4b1c0 0x557a973ed890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.156+0000 7f54b259c640 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a96f4b1c0 0x557a973ed890 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11cf03c014c2a96b server_cookie=496876bdacf69d2c in_seq=0 out_seq=0 2024-05-20T18:04:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:10.158+0000 7f5496ff5640 10 client.16263 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:11.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:11 smithi154 ceph-mon[36482]: pgmap v173: 97 pgs: 97 active+clean; 41 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 915 MiB/s wr, 3.51k op/s 2024-05-20T18:04:11.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:11 smithi129 ceph-mon[32029]: pgmap v173: 97 pgs: 97 active+clean; 41 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 915 MiB/s wr, 3.51k op/s 2024-05-20T18:04:11.680 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:11 smithi204 ceph-mon[37076]: pgmap v173: 97 pgs: 97 active+clean; 41 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 915 MiB/s wr, 3.51k op/s 2024-05-20T18:04:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.804+0000 7f5496ff5640 1 -- 172.21.15.129:0/3623055668 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557a9735c040 con 0x557a96f4b1c0 2024-05-20T18:04:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.804+0000 7f5496ff5640 10 client.16263 handle_command_reply: tid=0 2024-05-20T18:04:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.805+0000 7f54bfff3740 1 client.16263 shutdown 2024-05-20T18:04:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.805+0000 7f54bfff3740 1 -- 172.21.15.129:0/3623055668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 msgr2=0x557a973d29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.805+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973d29d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f54a8030f10 tx=0x7f54a8030b50 comp rx=0 tx=0).stop 2024-05-20T18:04:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.805+0000 7f54bfff3740 1 -- 172.21.15.129:0/3623055668 shutdown_connections 2024-05-20T18:04:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.805+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a96f4b1c0 0x557a973ed890 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.805+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a96fb71b0 0x557a973d3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.805+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a96f964d0 0x557a973d2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.805+0000 7f54bfff3740 1 --2- 172.21.15.129:0/3623055668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a96f8e2f0 0x557a973d29d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.805+0000 7f54bfff3740 1 -- 172.21.15.129:0/3623055668 >> 172.21.15.129:0/3623055668 conn(0x557a96fab300 msgr2=0x557a97356840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.805+0000 7f5496ff5640 0 client.16263 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.806+0000 7f54bfff3740 1 -- 172.21.15.129:0/3623055668 shutdown_connections 2024-05-20T18:04:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.806+0000 7f54bfff3740 1 -- 172.21.15.129:0/3623055668 wait complete. 2024-05-20T18:04:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.806+0000 7f54bfff3740 20 client.16263 trim_cache size 0 max 16384 2024-05-20T18:04:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:04:12.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:04:12.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:04:12.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:04:12.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.807+0000 7f54b9145640 1 -- 172.21.15.129:0/2092676586 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f549c0a2e40 msgr2=0x7f549c0a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:12.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.807+0000 7f54b9145640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f549c0a2e40 0x7f549c0a5350 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x557a973531f0 tx=0x7f54a8009410 comp rx=0 tx=0).stop 2024-05-20T18:04:12.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.807+0000 7f54b9145640 1 -- 172.21.15.129:0/2092676586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54b411c0d0 msgr2=0x7f54b41a8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:12.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.807+0000 7f54b9145640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54b411c0d0 0x7f54b41a8cf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f54a400b4a0 tx=0x7f54a400b960 comp rx=0 tx=0).stop 2024-05-20T18:04:12.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.808+0000 7f54b9145640 1 -- 172.21.15.129:0/2092676586 shutdown_connections 2024-05-20T18:04:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.808+0000 7f54b9145640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f549c0a2e40 0x7f549c0a5350 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.808+0000 7f54b9145640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54b41acef0 0x7f54b41ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.808+0000 7f54b9145640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54b411c0d0 0x7f54b41a8cf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.808+0000 7f54b9145640 1 --2- 172.21.15.129:0/2092676586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54b4119770 0x7f54b41a8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.808+0000 7f54b9145640 1 -- 172.21.15.129:0/2092676586 >> 172.21.15.129:0/2092676586 conn(0x7f54b406d780 msgr2=0x7f54b4119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.809+0000 7f54b9145640 1 -- 172.21.15.129:0/2092676586 shutdown_connections 2024-05-20T18:04:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:12.809+0000 7f54b9145640 1 -- 172.21.15.129:0/2092676586 wait complete. 2024-05-20T18:04:13.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:13 smithi154 ceph-mon[36482]: pgmap v174: 97 pgs: 97 active+clean; 41 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 664 MiB/s wr, 3.00k op/s 2024-05-20T18:04:13.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:13 smithi129 ceph-mon[32029]: pgmap v174: 97 pgs: 97 active+clean; 41 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 664 MiB/s wr, 3.00k op/s 2024-05-20T18:04:13.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:13 smithi204 ceph-mon[37076]: pgmap v174: 97 pgs: 97 active+clean; 41 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 664 MiB/s wr, 3.00k op/s 2024-05-20T18:04:15.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:15 smithi154 ceph-mon[36482]: pgmap v175: 97 pgs: 97 active+clean; 41 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 653 MiB/s wr, 3.15k op/s 2024-05-20T18:04:15.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:15 smithi129 ceph-mon[32029]: pgmap v175: 97 pgs: 97 active+clean; 41 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 653 MiB/s wr, 3.15k op/s 2024-05-20T18:04:15.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:15 smithi204 ceph-mon[37076]: pgmap v175: 97 pgs: 97 active+clean; 41 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 653 MiB/s wr, 3.15k op/s 2024-05-20T18:04:17.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:17 smithi129 ceph-mon[32029]: pgmap v176: 97 pgs: 97 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 793 MiB/s wr, 4.26k op/s 2024-05-20T18:04:17.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:17 smithi204 ceph-mon[37076]: pgmap v176: 97 pgs: 97 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 793 MiB/s wr, 4.26k op/s 2024-05-20T18:04:17.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:17 smithi154 ceph-mon[36482]: pgmap v176: 97 pgs: 97 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 793 MiB/s wr, 4.26k op/s 2024-05-20T18:04:19.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:19 smithi204 ceph-mon[37076]: pgmap v177: 97 pgs: 97 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 610 MiB/s wr, 3.94k op/s 2024-05-20T18:04:19.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:19 smithi154 ceph-mon[36482]: pgmap v177: 97 pgs: 97 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 610 MiB/s wr, 3.94k op/s 2024-05-20T18:04:19.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:19 smithi129 ceph-mon[32029]: pgmap v177: 97 pgs: 97 active+clean; 43 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 610 MiB/s wr, 3.94k op/s 2024-05-20T18:04:21.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:21 smithi204 ceph-mon[37076]: pgmap v178: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 823 MiB/s wr, 4.71k op/s 2024-05-20T18:04:21.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:21 smithi154 ceph-mon[36482]: pgmap v178: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 823 MiB/s wr, 4.71k op/s 2024-05-20T18:04:21.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:21 smithi129 ceph-mon[32029]: pgmap v178: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 823 MiB/s wr, 4.71k op/s 2024-05-20T18:04:23.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:23 smithi129 ceph-mon[32029]: pgmap v179: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 617 MiB/s wr, 3.10k op/s 2024-05-20T18:04:23.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:23 smithi154 ceph-mon[36482]: pgmap v179: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 617 MiB/s wr, 3.10k op/s 2024-05-20T18:04:23.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:23 smithi204 ceph-mon[37076]: pgmap v179: 97 pgs: 97 active+clean; 45 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 617 MiB/s wr, 3.10k op/s 2024-05-20T18:04:25.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:25 smithi154 ceph-mon[36482]: pgmap v180: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 629 MiB/s wr, 3.11k op/s 2024-05-20T18:04:25.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:25 smithi129 ceph-mon[32029]: pgmap v180: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 629 MiB/s wr, 3.11k op/s 2024-05-20T18:04:25.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:25 smithi204 ceph-mon[37076]: pgmap v180: 97 pgs: 97 active+clean; 46 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 629 MiB/s wr, 3.11k op/s 2024-05-20T18:04:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.939+0000 7f415b422640 1 -- 172.21.15.129:0/2337996186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4154109080 msgr2=0x7f4154109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.939+0000 7f415b422640 1 --2- 172.21.15.129:0/2337996186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4154109080 0x7f4154109490 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4150009860 tx=0x7f415002a410 comp rx=0 tx=0).stop 2024-05-20T18:04:26.940 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.943+0000 7f415b422640 1 -- 172.21.15.129:0/2337996186 shutdown_connections 2024-05-20T18:04:26.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.943+0000 7f415b422640 1 --2- 172.21.15.129:0/2337996186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f415411c0d0 0x7f415411e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:26.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.943+0000 7f415b422640 1 --2- 172.21.15.129:0/2337996186 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4154119770 0x7f415411bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:26.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.943+0000 7f415b422640 1 --2- 172.21.15.129:0/2337996186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4154109080 0x7f4154109490 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:26.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.943+0000 7f415b422640 1 -- 172.21.15.129:0/2337996186 >> 172.21.15.129:0/2337996186 conn(0x7f415406d780 msgr2=0x7f415406dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:26.948 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.950+0000 7f415b422640 1 -- 172.21.15.129:0/2337996186 shutdown_connections 2024-05-20T18:04:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.956+0000 7f415b422640 1 -- 172.21.15.129:0/2337996186 wait complete. 2024-05-20T18:04:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.957+0000 7f415b422640 1 Processor -- start 2024-05-20T18:04:26.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.962+0000 7f415b422640 1 -- start start 2024-05-20T18:04:26.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.963+0000 7f415b422640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4154109080 0x7f4154119080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:26.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.963+0000 7f415b422640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4154119770 0x7f41541121a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:26.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.963+0000 7f415b422640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f415411c0d0 0x7f4154112700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:26.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.964+0000 7f415b422640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4154121c50 con 0x7f4154119770 2024-05-20T18:04:26.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.964+0000 7f415b422640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4154121ad0 con 0x7f4154109080 2024-05-20T18:04:26.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.964+0000 7f415b422640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4154121dd0 con 0x7f415411c0d0 2024-05-20T18:04:26.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.965+0000 7f41591be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4154109080 0x7f4154119080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:26.964 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.965+0000 7f41591be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4154109080 0x7f4154119080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:50318/0 (socket says 172.21.15.129:50318) 2024-05-20T18:04:26.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.966+0000 7f41591be640 1 -- 172.21.15.129:0/2413882389 learned_addr learned my addr 172.21.15.129:0/2413882389 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.970+0000 7f41591be640 1 -- 172.21.15.129:0/2413882389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f415411c0d0 msgr2=0x7f4154112700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:04:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.970+0000 7f41589bd640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4154119770 0x7f41541121a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.970+0000 7f41599bf640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f415411c0d0 0x7f4154112700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.970+0000 7f41591be640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f415411c0d0 0x7f4154112700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.970+0000 7f41591be640 1 -- 172.21.15.129:0/2413882389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4154119770 msgr2=0x7f41541121a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.970+0000 7f41591be640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4154119770 0x7f41541121a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.970+0000 7f41591be640 1 -- 172.21.15.129:0/2413882389 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4154112f20 con 0x7f4154109080 2024-05-20T18:04:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.970+0000 7f41589bd640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4154119770 0x7f41541121a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:04:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.973+0000 7f41591be640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4154109080 0x7f4154119080 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f415002a980 tx=0x7f41500309c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.975+0000 7f414a7fc640 1 -- 172.21.15.129:0/2413882389 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4150005460 con 0x7f4154109080 2024-05-20T18:04:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.975+0000 7f414a7fc640 1 -- 172.21.15.129:0/2413882389 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4150005600 con 0x7f4154109080 2024-05-20T18:04:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.975+0000 7f414a7fc640 1 -- 172.21.15.129:0/2413882389 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4150004e10 con 0x7f4154109080 2024-05-20T18:04:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.975+0000 7f415b422640 1 -- 172.21.15.129:0/2413882389 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4154116ba0 con 0x7f4154109080 2024-05-20T18:04:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.978+0000 7f415b422640 1 -- 172.21.15.129:0/2413882389 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4154117040 con 0x7f4154109080 2024-05-20T18:04:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.987+0000 7f414a7fc640 1 -- 172.21.15.129:0/2413882389 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f4150039040 con 0x7f4154109080 2024-05-20T18:04:26.989 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.990+0000 7f414a7fc640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41400a2f00 0x7f41400a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:26.992+0000 7f414a7fc640 1 -- 172.21.15.129:0/2413882389 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f41500f1320 con 0x7f4154109080 2024-05-20T18:04:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.000+0000 7f41589bd640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41400a2f00 0x7f41400a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.011+0000 7f41623a1740 1 Processor -- start 2024-05-20T18:04:27.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.011+0000 7f41589bd640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41400a2f00 0x7f41400a5410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f415411b580 tx=0x7f4144005e50 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:27.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.012+0000 7f41623a1740 1 -- start start 2024-05-20T18:04:27.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.013+0000 7f41623a1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 0x5570dcbc2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.013+0000 7f41623a1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7fa2f0 0x5570dcbc25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.013+0000 7f41623a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbc2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.013+0000 7f41623a1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5570dcb27340 con 0x5570dc8231b0 2024-05-20T18:04:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.013+0000 7f41623a1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5570dcacf290 con 0x5570dc8024d0 2024-05-20T18:04:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.013+0000 7f41623a1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5570dcb23300 con 0x5570dc7fa2f0 2024-05-20T18:04:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.015+0000 7f41589bd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7fa2f0 0x5570dcbc25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.017+0000 7f41599bf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbc2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.017+0000 7f41599bf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbc2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:50334/0 (socket says 172.21.15.129:50334) 2024-05-20T18:04:27.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.017+0000 7f41599bf640 1 -- 172.21.15.129:0/496118390 learned_addr learned my addr 172.21.15.129:0/496118390 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.023+0000 7f41599bf640 1 -- 172.21.15.129:0/496118390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7fa2f0 msgr2=0x5570dcbc25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.023+0000 7f41599bf640 1 --2- 172.21.15.129:0/496118390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7fa2f0 0x5570dcbc25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.023 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.023+0000 7f41599bf640 1 -- 172.21.15.129:0/496118390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 msgr2=0x5570dcbc2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:27.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.023+0000 7f41599bf640 1 --2- 172.21.15.129:0/496118390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 0x5570dcbc2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.023+0000 7f41599bf640 1 -- 172.21.15.129:0/496118390 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570dcaf38e0 con 0x5570dc8024d0 2024-05-20T18:04:27.024 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.026+0000 7f41599bf640 1 --2- 172.21.15.129:0/496118390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbc2b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f414c009fd0 tx=0x7f414c0108a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f31094b83ee9324b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:27.025 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.027+0000 7f413f7fe640 1 -- 172.21.15.129:0/496118390 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c003b80 con 0x5570dc8024d0 2024-05-20T18:04:27.025 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.027+0000 7f413f7fe640 1 -- 172.21.15.129:0/496118390 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f414c003d20 con 0x5570dc8024d0 2024-05-20T18:04:27.025 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.027+0000 7f413f7fe640 1 -- 172.21.15.129:0/496118390 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c00eb50 con 0x5570dc8024d0 2024-05-20T18:04:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.030+0000 7f41623a1740 1 -- 172.21.15.129:0/496118390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 msgr2=0x5570dcbc2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.030+0000 7f41623a1740 1 --2- 172.21.15.129:0/496118390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbc2b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f414c009fd0 tx=0x7f414c0108a0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.031+0000 7f41623a1740 1 -- 172.21.15.129:0/496118390 shutdown_connections 2024-05-20T18:04:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.031+0000 7f41623a1740 1 --2- 172.21.15.129:0/496118390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbc2b00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.031+0000 7f41623a1740 1 --2- 172.21.15.129:0/496118390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7fa2f0 0x5570dcbc25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.031+0000 7f41623a1740 1 --2- 172.21.15.129:0/496118390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 0x5570dcbc2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.031+0000 7f41623a1740 1 -- 172.21.15.129:0/496118390 >> 172.21.15.129:0/496118390 conn(0x5570dc817300 msgr2=0x5570dcbbe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.034+0000 7f41623a1740 1 -- 172.21.15.129:0/496118390 shutdown_connections 2024-05-20T18:04:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.035+0000 7f41623a1740 1 -- 172.21.15.129:0/496118390 wait complete. 2024-05-20T18:04:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.036+0000 7f41623a1740 1 -- start start 2024-05-20T18:04:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.036+0000 7f41623a1740 1 Processor -- start 2024-05-20T18:04:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.040+0000 7f41623a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc7fa2f0 0x5570dcbcc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.040+0000 7f41623a1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8024d0 0x5570dcbccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.040+0000 7f41623a1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc8231b0 0x5570dcbd2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.040+0000 7f41623a1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5570dcbcf1b0 con 0x5570dc8024d0 2024-05-20T18:04:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.040+0000 7f41623a1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5570dcae2790 con 0x5570dc7fa2f0 2024-05-20T18:04:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.040+0000 7f41623a1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5570dcbcf330 con 0x5570dc8231b0 2024-05-20T18:04:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.040+0000 7f41623a1740 10 client.0 _ll_register_callbacks cb 0x5570dcb1da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.041+0000 7f41589bd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8024d0 0x5570dcbccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.041+0000 7f41589bd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8024d0 0x5570dcbccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51332/0 (socket says 172.21.15.129:51332) 2024-05-20T18:04:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.041+0000 7f41589bd640 1 -- 172.21.15.129:0/883948316 learned_addr learned my addr 172.21.15.129:0/883948316 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.042+0000 7f41599bf640 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc8231b0 0x5570dcbd2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.043+0000 7f41599bf640 1 -- 172.21.15.129:0/883948316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc7fa2f0 msgr2=0x5570dcbcc780 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:04:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.043+0000 7f41599bf640 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc7fa2f0 0x5570dcbcc780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.043+0000 7f41599bf640 1 -- 172.21.15.129:0/883948316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8024d0 msgr2=0x5570dcbccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.043+0000 7f41599bf640 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8024d0 0x5570dcbccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.043+0000 7f41599bf640 1 -- 172.21.15.129:0/883948316 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570dc83b210 con 0x5570dc8231b0 2024-05-20T18:04:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.045+0000 7f41599bf640 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc8231b0 0x5570dcbd2730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f414c006010 tx=0x7f414c009d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6a6551209a02975 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.046+0000 7f413cff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.050+0000 7f413cff9640 1 -- 172.21.15.129:0/883948316 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c026070 con 0x5570dc8231b0 2024-05-20T18:04:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.051+0000 7f413cff9640 1 -- 172.21.15.129:0/883948316 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f414c00eac0 con 0x5570dc8231b0 2024-05-20T18:04:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.051+0000 7f41623a1740 1 -- 172.21.15.129:0/883948316 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f414c003d20 con 0x5570dc8231b0 2024-05-20T18:04:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.052+0000 7f413cff9640 1 -- 172.21.15.129:0/883948316 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f414c032050 con 0x5570dc8231b0 2024-05-20T18:04:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.054+0000 7f41623a1740 10 client.26038 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:27.052 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.054+0000 7f41623a1740 1 -- 172.21.15.129:0/883948316 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5570dc907620 con 0x5570dc8231b0 2024-05-20T18:04:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.055+0000 7f413cff9640 1 -- 172.21.15.129:0/883948316 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f414c003d20 con 0x5570dc8231b0 2024-05-20T18:04:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.057+0000 7f413cff9640 1 -- 172.21.15.129:0/883948316 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f414c00b040 con 0x5570dc8231b0 2024-05-20T18:04:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.060+0000 7f41623a1740 10 client.26038 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.060+0000 7f41623a1740 10 client.26038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.060+0000 7f41623a1740 20 client.26038 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.060+0000 7f41623a1740 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570dc7e1d80 0x5570dcbde950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.060+0000 7f41623a1740 4 client.26038 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:04:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.060+0000 7f41623a1740 1 -- 172.21.15.129:0/883948316 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570dcb34a70 con 0x5570dc7e1d80 2024-05-20T18:04:27.062 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.064+0000 7f41591be640 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570dc7e1d80 0x5570dcbde950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.086+0000 7f41591be640 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570dc7e1d80 0x5570dcbde950 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca0b2c8f9f0bafb7 server_cookie=13b181dd86c43861 in_seq=0 out_seq=0 2024-05-20T18:04:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.089+0000 7f413cff9640 10 client.26038 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.094+0000 7f413cff9640 1 -- 172.21.15.129:0/883948316 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5570dcb34a70 con 0x5570dc7e1d80 2024-05-20T18:04:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.094+0000 7f413cff9640 10 client.26038 handle_command_reply: tid=0 2024-05-20T18:04:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.094+0000 7f41623a1740 1 client.26038 shutdown 2024-05-20T18:04:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.094+0000 7f41623a1740 1 -- 172.21.15.129:0/883948316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc8231b0 msgr2=0x5570dcbd2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:27.095 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.094+0000 7f41623a1740 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc8231b0 0x5570dcbd2730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f414c006010 tx=0x7f414c009d80 comp rx=0 tx=0).stop 2024-05-20T18:04:27.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.099+0000 7f41623a1740 1 -- 172.21.15.129:0/883948316 shutdown_connections 2024-05-20T18:04:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.099+0000 7f41623a1740 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570dc7e1d80 0x5570dcbde950 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.099+0000 7f41623a1740 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc8231b0 0x5570dcbd2730 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.099+0000 7f41623a1740 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8024d0 0x5570dcbccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.099+0000 7f41623a1740 1 --2- 172.21.15.129:0/883948316 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc7fa2f0 0x5570dcbcc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.099+0000 7f41623a1740 1 -- 172.21.15.129:0/883948316 >> 172.21.15.129:0/883948316 conn(0x5570dc817300 msgr2=0x5570dcbc3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.099+0000 7f413cff9640 0 client.26038 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.100+0000 7f41623a1740 1 -- 172.21.15.129:0/883948316 shutdown_connections 2024-05-20T18:04:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.105+0000 7f41623a1740 1 -- 172.21.15.129:0/883948316 wait complete. 2024-05-20T18:04:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.105+0000 7f41623a1740 20 client.26038 trim_cache size 0 max 16384 2024-05-20T18:04:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.156+0000 7f41623a1740 1 Processor -- start 2024-05-20T18:04:27.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.157+0000 7f41623a1740 1 -- start start 2024-05-20T18:04:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.157+0000 7f41623a1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7fa2f0 0x5570dcbd2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.158+0000 7f41623a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbe3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.158+0000 7f41623a1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 0x5570dcbe4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.158+0000 7f41623a1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f413800c6f0 con 0x5570dc8231b0 2024-05-20T18:04:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.158+0000 7f41589bd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbe3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.158+0000 7f41589bd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbe3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:50352/0 (socket says 172.21.15.129:50352) 2024-05-20T18:04:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.158+0000 7f41589bd640 1 -- 172.21.15.129:0/2234948931 learned_addr learned my addr 172.21.15.129:0/2234948931 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.160+0000 7f41591be640 1 --2- 172.21.15.129:0/2234948931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7fa2f0 0x5570dcbd2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.160+0000 7f41623a1740 1 -- 172.21.15.129:0/2234948931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f413800eec0 con 0x5570dc8024d0 2024-05-20T18:04:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.160+0000 7f41623a1740 1 -- 172.21.15.129:0/2234948931 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f413800ccb0 con 0x5570dc7fa2f0 2024-05-20T18:04:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.161+0000 7f41599bf640 1 --2- 172.21.15.129:0/2234948931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 0x5570dcbe4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.164+0000 7f41589bd640 1 -- 172.21.15.129:0/2234948931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7fa2f0 msgr2=0x5570dcbd2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.164+0000 7f41589bd640 1 --2- 172.21.15.129:0/2234948931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7fa2f0 0x5570dcbd2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.168 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.164+0000 7f41589bd640 1 -- 172.21.15.129:0/2234948931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 msgr2=0x5570dcbe4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.164+0000 7f41589bd640 1 --2- 172.21.15.129:0/2234948931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 0x5570dcbe4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.164+0000 7f41589bd640 1 -- 172.21.15.129:0/2234948931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570dc7e1fb0 con 0x5570dc8024d0 2024-05-20T18:04:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.167+0000 7f41589bd640 1 --2- 172.21.15.129:0/2234948931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbe3fe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4144011570 tx=0x7f4144011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6895ee18f46a0b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.168+0000 7f413effd640 1 -- 172.21.15.129:0/2234948931 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4144004220 con 0x5570dc8024d0 2024-05-20T18:04:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.168+0000 7f413effd640 1 -- 172.21.15.129:0/2234948931 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4144024070 con 0x5570dc8024d0 2024-05-20T18:04:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.170+0000 7f41623a1740 1 -- 172.21.15.129:0/2234948931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 msgr2=0x5570dcbe3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.170+0000 7f41623a1740 1 --2- 172.21.15.129:0/2234948931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbe3fe0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4144011570 tx=0x7f4144011000 comp rx=0 tx=0).stop 2024-05-20T18:04:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.170+0000 7f41623a1740 1 -- 172.21.15.129:0/2234948931 shutdown_connections 2024-05-20T18:04:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.170+0000 7f41623a1740 1 --2- 172.21.15.129:0/2234948931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc8024d0 0x5570dcbe3fe0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.170+0000 7f41623a1740 1 --2- 172.21.15.129:0/2234948931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 0x5570dcbe4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.170+0000 7f41623a1740 1 --2- 172.21.15.129:0/2234948931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7fa2f0 0x5570dcbd2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.170+0000 7f41623a1740 1 -- 172.21.15.129:0/2234948931 >> 172.21.15.129:0/2234948931 conn(0x5570dc817300 msgr2=0x5570dcbcae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.171+0000 7f41623a1740 1 -- 172.21.15.129:0/2234948931 shutdown_connections 2024-05-20T18:04:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.171+0000 7f41623a1740 1 -- 172.21.15.129:0/2234948931 wait complete. 2024-05-20T18:04:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.172+0000 7f41623a1740 1 -- start start 2024-05-20T18:04:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.172+0000 7f41623a1740 1 Processor -- start 2024-05-20T18:04:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.173+0000 7f41623a1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc7fa2f0 0x5570dcbd3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.173+0000 7f41623a1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 0x5570dcbd4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.173+0000 7f41623a1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7b71c0 0x5570dcc44b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.174+0000 7f41623a1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f413800d380 con 0x5570dc8231b0 2024-05-20T18:04:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.174+0000 7f41623a1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f413800e7f0 con 0x5570dc7fa2f0 2024-05-20T18:04:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.174+0000 7f41623a1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f413800da50 con 0x5570dc7b71c0 2024-05-20T18:04:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.174+0000 7f41591be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc7fa2f0 0x5570dcbd3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.174+0000 7f41591be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc7fa2f0 0x5570dcbd3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:50356/0 (socket says 172.21.15.129:50356) 2024-05-20T18:04:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.174+0000 7f41591be640 1 -- 172.21.15.129:0/1259476485 learned_addr learned my addr 172.21.15.129:0/1259476485 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.174+0000 7f41623a1740 10 client.0 _ll_register_callbacks cb 0x7f4144009750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.174+0000 7f41599bf640 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7b71c0 0x5570dcc44b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.175+0000 7f41599bf640 1 -- 172.21.15.129:0/1259476485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc7fa2f0 msgr2=0x5570dcbd3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.175+0000 7f41599bf640 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc7fa2f0 0x5570dcbd3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.175+0000 7f41599bf640 1 -- 172.21.15.129:0/1259476485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 msgr2=0x5570dcbd4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.175+0000 7f41599bf640 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 0x5570dcbd4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.175+0000 7f41599bf640 1 -- 172.21.15.129:0/1259476485 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570dcbd4b50 con 0x5570dc7b71c0 2024-05-20T18:04:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.176+0000 7f41599bf640 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7b71c0 0x5570dcc44b80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f414c009e10 tx=0x7f414c00e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ceda84d59d3a469 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.177+0000 7f413cff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.177+0000 7f413cff9640 1 -- 172.21.15.129:0/1259476485 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c026070 con 0x5570dc7b71c0 2024-05-20T18:04:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.177+0000 7f41623a1740 1 -- 172.21.15.129:0/1259476485 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5570dcb2edc0 con 0x5570dc7b71c0 2024-05-20T18:04:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.178+0000 7f413cff9640 1 -- 172.21.15.129:0/1259476485 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f414c00eac0 con 0x5570dc7b71c0 2024-05-20T18:04:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.178+0000 7f413cff9640 1 -- 172.21.15.129:0/1259476485 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f414c00ece0 con 0x5570dc7b71c0 2024-05-20T18:04:27.181 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.182+0000 7f41623a1740 10 client.26044 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:27.181 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.182+0000 7f41623a1740 1 -- 172.21.15.129:0/1259476485 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5570dcc461c0 con 0x5570dc7b71c0 2024-05-20T18:04:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.188+0000 7f413cff9640 1 -- 172.21.15.129:0/1259476485 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5570dcb2edc0 con 0x5570dc7b71c0 2024-05-20T18:04:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.188+0000 7f413cff9640 1 -- 172.21.15.129:0/1259476485 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f414c00b040 con 0x5570dc7b71c0 2024-05-20T18:04:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.188+0000 7f41623a1740 10 client.26044 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.188+0000 7f41623a1740 10 client.26044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.188+0000 7f41623a1740 20 client.26044 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.189+0000 7f41623a1740 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570dcc46830 0x5570dcc48cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.189+0000 7f41623a1740 4 client.26044 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:04:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.189+0000 7f41623a1740 1 -- 172.21.15.129:0/1259476485 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5570dcaac8e0 con 0x5570dcc46830 2024-05-20T18:04:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.189+0000 7f41591be640 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570dcc46830 0x5570dcc48cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.194+0000 7f41591be640 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570dcc46830 0x5570dcc48cc0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68a6fd8c3405a20f server_cookie=9211e44ae100c00e in_seq=0 out_seq=0 2024-05-20T18:04:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:27.194+0000 7f413cff9640 10 client.26044 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:27.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:27 smithi154 ceph-mon[36482]: pgmap v181: 97 pgs: 97 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 816 MiB/s wr, 3.52k op/s 2024-05-20T18:04:27.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:27 smithi129 ceph-mon[32029]: pgmap v181: 97 pgs: 97 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 816 MiB/s wr, 3.52k op/s 2024-05-20T18:04:27.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:27 smithi204 ceph-mon[37076]: pgmap v181: 97 pgs: 97 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 816 MiB/s wr, 3.52k op/s 2024-05-20T18:04:29.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:29 smithi154 ceph-mon[36482]: pgmap v182: 97 pgs: 97 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 667 MiB/s wr, 2.48k op/s 2024-05-20T18:04:29.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:29 smithi129 ceph-mon[32029]: pgmap v182: 97 pgs: 97 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 667 MiB/s wr, 2.48k op/s 2024-05-20T18:04:29.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:29 smithi204 ceph-mon[37076]: pgmap v182: 97 pgs: 97 active+clean; 48 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 667 MiB/s wr, 2.48k op/s 2024-05-20T18:04:31.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:31 smithi154 ceph-mon[36482]: pgmap v183: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 890 MiB/s wr, 2.91k op/s 2024-05-20T18:04:31.916 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:31 smithi129 ceph-mon[32029]: pgmap v183: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 890 MiB/s wr, 2.91k op/s 2024-05-20T18:04:31.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:31 smithi204 ceph-mon[37076]: pgmap v183: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 890 MiB/s wr, 2.91k op/s 2024-05-20T18:04:32.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.552+0000 7f413cff9640 1 -- 172.21.15.129:0/1259476485 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5570dcaac8e0 con 0x5570dcc46830 2024-05-20T18:04:32.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.552+0000 7f413cff9640 10 client.26044 handle_command_reply: tid=0 2024-05-20T18:04:32.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.553+0000 7f41623a1740 1 client.26044 shutdown 2024-05-20T18:04:32.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.555+0000 7f41623a1740 1 -- 172.21.15.129:0/1259476485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7b71c0 msgr2=0x5570dcc44b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:32.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.555+0000 7f41623a1740 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7b71c0 0x5570dcc44b80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f414c009e10 tx=0x7f414c00e6c0 comp rx=0 tx=0).stop 2024-05-20T18:04:32.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.556+0000 7f41623a1740 1 -- 172.21.15.129:0/1259476485 shutdown_connections 2024-05-20T18:04:32.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.556+0000 7f41623a1740 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5570dcc46830 0x5570dcc48cc0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:32.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.556+0000 7f41623a1740 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5570dc7b71c0 0x5570dcc44b80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:32.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.556+0000 7f41623a1740 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570dc8231b0 0x5570dcbd4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:32.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.556+0000 7f41623a1740 1 --2- 172.21.15.129:0/1259476485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5570dc7fa2f0 0x5570dcbd3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:32.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.556+0000 7f41623a1740 1 -- 172.21.15.129:0/1259476485 >> 172.21.15.129:0/1259476485 conn(0x5570dc817300 msgr2=0x5570dcbcade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:32.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.556+0000 7f413cff9640 0 client.26044 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:32.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.557+0000 7f41623a1740 1 -- 172.21.15.129:0/1259476485 shutdown_connections 2024-05-20T18:04:32.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.557+0000 7f41623a1740 1 -- 172.21.15.129:0/1259476485 wait complete. 2024-05-20T18:04:32.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.557+0000 7f41623a1740 20 client.26044 trim_cache size 0 max 16384 2024-05-20T18:04:32.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:04:32.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:04:32.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:04:32.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:04:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.560+0000 7f415b422640 1 -- 172.21.15.129:0/2413882389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41400a2f00 msgr2=0x7f41400a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.560+0000 7f415b422640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41400a2f00 0x7f41400a5410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f415411b580 tx=0x7f4144005e50 comp rx=0 tx=0).stop 2024-05-20T18:04:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.560+0000 7f415b422640 1 -- 172.21.15.129:0/2413882389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4154109080 msgr2=0x7f4154119080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.560+0000 7f415b422640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4154109080 0x7f4154119080 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f415002a980 tx=0x7f41500309c0 comp rx=0 tx=0).stop 2024-05-20T18:04:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.561+0000 7f415b422640 1 -- 172.21.15.129:0/2413882389 shutdown_connections 2024-05-20T18:04:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.561+0000 7f415b422640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41400a2f00 0x7f41400a5410 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.561+0000 7f415b422640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f415411c0d0 0x7f4154112700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.561+0000 7f415b422640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4154119770 0x7f41541121a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.561+0000 7f415b422640 1 --2- 172.21.15.129:0/2413882389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4154109080 0x7f4154119080 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.561+0000 7f415b422640 1 -- 172.21.15.129:0/2413882389 >> 172.21.15.129:0/2413882389 conn(0x7f415406d780 msgr2=0x7f4154119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.561+0000 7f415b422640 1 -- 172.21.15.129:0/2413882389 shutdown_connections 2024-05-20T18:04:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:32.561+0000 7f415b422640 1 -- 172.21.15.129:0/2413882389 wait complete. 2024-05-20T18:04:33.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:04:33.770 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:04:33.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:33 smithi154 ceph-mon[36482]: pgmap v184: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 691 MiB/s wr, 2.08k op/s 2024-05-20T18:04:33.916 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:33 smithi129 ceph-mon[32029]: pgmap v184: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 691 MiB/s wr, 2.08k op/s 2024-05-20T18:04:33.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:33 smithi204 ceph-mon[37076]: pgmap v184: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 691 MiB/s wr, 2.08k op/s 2024-05-20T18:04:34.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.476+0000 7fbd1b902640 1 -- 172.21.15.129:0/651531518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd14119770 msgr2=0x7fbd1411bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:34.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.476+0000 7fbd1b902640 1 --2- 172.21.15.129:0/651531518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd14119770 0x7fbd1411bb90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c009ca0 tx=0x7fbd0c02a980 comp rx=0 tx=0).stop 2024-05-20T18:04:34.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.477+0000 7fbd1b902640 1 -- 172.21.15.129:0/651531518 shutdown_connections 2024-05-20T18:04:34.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.477+0000 7fbd1b902640 1 --2- 172.21.15.129:0/651531518 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd1411c0d0 0x7fbd1411e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.477+0000 7fbd1b902640 1 --2- 172.21.15.129:0/651531518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd14119770 0x7fbd1411bb90 secure :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c009ca0 tx=0x7fbd0c02a980 comp rx=0 tx=0).stop 2024-05-20T18:04:34.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.477+0000 7fbd1b902640 1 --2- 172.21.15.129:0/651531518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd14109060 0x7fbd14109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.477+0000 7fbd1b902640 1 -- 172.21.15.129:0/651531518 >> 172.21.15.129:0/651531518 conn(0x7fbd1406d750 msgr2=0x7fbd1406dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:34.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.485+0000 7fbd1b902640 1 -- 172.21.15.129:0/651531518 shutdown_connections 2024-05-20T18:04:34.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.489+0000 7fbd1b902640 1 -- 172.21.15.129:0/651531518 wait complete. 2024-05-20T18:04:34.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.490+0000 7fbd1b902640 1 Processor -- start 2024-05-20T18:04:34.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.491+0000 7fbd1b902640 1 -- start start 2024-05-20T18:04:34.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.491+0000 7fbd1b902640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd14109060 0x7fbd141135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.491+0000 7fbd1b902640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd1411c0d0 0x7fbd14113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.492+0000 7fbd1b902640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14114020 0x7fbd141bcf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.492+0000 7fbd1b902640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbd141119b0 con 0x7fbd1411c0d0 2024-05-20T18:04:34.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.492+0000 7fbd1b902640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd14111830 con 0x7fbd14109060 2024-05-20T18:04:34.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.492+0000 7fbd1b902640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbd14111b30 con 0x7fbd14114020 2024-05-20T18:04:34.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.499+0000 7fbd1969e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd14109060 0x7fbd141135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.499+0000 7fbd19e9f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14114020 0x7fbd141bcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.504+0000 7fbd19e9f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14114020 0x7fbd141bcf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:57996/0 (socket says 172.21.15.129:57996) 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.504+0000 7fbd19e9f640 1 -- 172.21.15.129:0/1860327568 learned_addr learned my addr 172.21.15.129:0/1860327568 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.504+0000 7fbd18e9d640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd1411c0d0 0x7fbd14113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.509+0000 7fbd19e9f640 1 -- 172.21.15.129:0/1860327568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd14109060 msgr2=0x7fbd141135a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.509+0000 7fbd19e9f640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd14109060 0x7fbd141135a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.509+0000 7fbd19e9f640 1 -- 172.21.15.129:0/1860327568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd1411c0d0 msgr2=0x7fbd14113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.509+0000 7fbd19e9f640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd1411c0d0 0x7fbd14113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.509+0000 7fbd19e9f640 1 -- 172.21.15.129:0/1860327568 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd141bd550 con 0x7fbd14114020 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.515+0000 7fbd19e9f640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14114020 0x7fbd141bcf60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbd0400d750 tx=0x7fbd0400db50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.517+0000 7fbd0a7fc640 1 -- 172.21.15.129:0/1860327568 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd04017070 con 0x7fbd14114020 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.517+0000 7fbd1b902640 1 -- 172.21.15.129:0/1860327568 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd141bd840 con 0x7fbd14114020 2024-05-20T18:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.518+0000 7fbd1b902640 1 -- 172.21.15.129:0/1860327568 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd141bdd60 con 0x7fbd14114020 2024-05-20T18:04:34.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.525+0000 7fbd0a7fc640 1 -- 172.21.15.129:0/1860327568 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd040097c0 con 0x7fbd14114020 2024-05-20T18:04:34.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.525+0000 7fbd0a7fc640 1 -- 172.21.15.129:0/1860327568 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd04020060 con 0x7fbd14114020 2024-05-20T18:04:34.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.526+0000 7fbd227a9740 1 Processor -- start 2024-05-20T18:04:34.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.526+0000 7fbd227a9740 1 -- start start 2024-05-20T18:04:34.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.526+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04e0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.526+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01194d0 0x5619e04e1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.527+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 0x5619e04e17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.527+0000 7fbd227a9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5619e0446d00 con 0x5619e01194d0 2024-05-20T18:04:34.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.527+0000 7fbd227a9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5619e03ee4c0 con 0x5619e0142390 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.527+0000 7fbd227a9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5619e0442cc0 con 0x5619e01216b0 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.527+0000 7fbd18e9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04e0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.527+0000 7fbd18e9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04e0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:48558/0 (socket says 172.21.15.129:48558) 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.527+0000 7fbd18e9d640 1 -- 172.21.15.129:0/2112548953 learned_addr learned my addr 172.21.15.129:0/2112548953 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.528+0000 7fbd18e9d640 1 -- 172.21.15.129:0/2112548953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 msgr2=0x5619e04e17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.528+0000 7fbd18e9d640 1 --2- 172.21.15.129:0/2112548953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 0x5619e04e17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.528+0000 7fbd18e9d640 1 -- 172.21.15.129:0/2112548953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01194d0 msgr2=0x5619e04e1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.528+0000 7fbd18e9d640 1 --2- 172.21.15.129:0/2112548953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01194d0 0x5619e04e1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.529+0000 7fbd18e9d640 1 -- 172.21.15.129:0/2112548953 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619e02247d0 con 0x5619e0142390 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.538+0000 7fbd18e9d640 1 --2- 172.21.15.129:0/2112548953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04e0d30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c006010 tx=0x7fbd0c02fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e111df6e348398cc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.539+0000 7fbd0a7fc640 1 -- 172.21.15.129:0/1860327568 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fbd04004c60 con 0x7fbd14114020 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.541+0000 7fbcff7fe640 1 -- 172.21.15.129:0/2112548953 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c034040 con 0x5619e0142390 2024-05-20T18:04:34.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.541+0000 7fbd0a7fc640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd000a2d00 0x7fbd000a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.541+0000 7fbcff7fe640 1 -- 172.21.15.129:0/2112548953 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd0c041070 con 0x5619e0142390 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.541+0000 7fbd0a7fc640 1 -- 172.21.15.129:0/1860327568 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fbd040cecb0 con 0x7fbd14114020 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.542+0000 7fbd227a9740 1 -- 172.21.15.129:0/2112548953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 msgr2=0x5619e04e0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.542+0000 7fbd227a9740 1 --2- 172.21.15.129:0/2112548953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04e0d30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c006010 tx=0x7fbd0c02fc30 comp rx=0 tx=0).stop 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.543+0000 7fbd1969e640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd000a2d00 0x7fbd000a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.544+0000 7fbd227a9740 1 -- 172.21.15.129:0/2112548953 shutdown_connections 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.544+0000 7fbd227a9740 1 --2- 172.21.15.129:0/2112548953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 0x5619e04e17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.544+0000 7fbd227a9740 1 --2- 172.21.15.129:0/2112548953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01194d0 0x5619e04e1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.544+0000 7fbd227a9740 1 --2- 172.21.15.129:0/2112548953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04e0d30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.544+0000 7fbd227a9740 1 -- 172.21.15.129:0/2112548953 >> 172.21.15.129:0/2112548953 conn(0x5619e01364e0 msgr2=0x5619e04dd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.547+0000 7fbd227a9740 1 -- 172.21.15.129:0/2112548953 shutdown_connections 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.547+0000 7fbd227a9740 1 -- 172.21.15.129:0/2112548953 wait complete. 2024-05-20T18:04:34.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.548+0000 7fbd227a9740 1 -- start start 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.548+0000 7fbd227a9740 1 Processor -- start 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.549+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01194d0 0x5619e04eb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.549+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 0x5619e04eb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.549+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04f1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.549+0000 7fbd227a9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5619e04ee010 con 0x5619e01194d0 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.549+0000 7fbd227a9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5619e0401b90 con 0x5619e0142390 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.549+0000 7fbd227a9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5619e04ee190 con 0x5619e01216b0 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.549+0000 7fbd227a9740 10 client.0 _ll_register_callbacks cb 0x5619e043dda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.550+0000 7fbd1969e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01194d0 0x5619e04eb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.550+0000 7fbd1969e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01194d0 0x5619e04eb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57862/0 (socket says 172.21.15.129:57862) 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.550+0000 7fbd1969e640 1 -- 172.21.15.129:0/4188162788 learned_addr learned my addr 172.21.15.129:0/4188162788 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.551+0000 7fbd1969e640 1 -- 172.21.15.129:0/4188162788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 msgr2=0x5619e04eb8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:04:34.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.551+0000 7fbd1969e640 1 --2- 172.21.15.129:0/4188162788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 0x5619e04eb8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.551+0000 7fbd1969e640 1 -- 172.21.15.129:0/4188162788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 msgr2=0x5619e04f1570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:04:34.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.551+0000 7fbd1969e640 1 --2- 172.21.15.129:0/4188162788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04f1570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.551+0000 7fbd1969e640 1 -- 172.21.15.129:0/4188162788 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619e015a3f0 con 0x5619e01194d0 2024-05-20T18:04:34.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.551+0000 7fbd1969e640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd000a2d00 0x7fbd000a5210 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbd10006aa0 tx=0x7fbd10006a30 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:34.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.552+0000 7fbd1969e640 1 --2- 172.21.15.129:0/4188162788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01194d0 0x5619e04eb390 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x5619e04de820 tx=0x7fbd100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490bd02e28f308e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:34.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.558+0000 7fbcfcff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:04:34.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.558+0000 7fbcfcff9640 1 -- 172.21.15.129:0/4188162788 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd10023020 con 0x5619e01194d0 2024-05-20T18:04:34.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.558+0000 7fbd227a9740 1 -- 172.21.15.129:0/4188162788 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd0c041070 con 0x5619e01194d0 2024-05-20T18:04:34.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.558+0000 7fbcfcff9640 1 -- 172.21.15.129:0/4188162788 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fbd10013510 con 0x5619e01194d0 2024-05-20T18:04:34.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.564+0000 7fbcfcff9640 1 -- 172.21.15.129:0/4188162788 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbd1001b040 con 0x5619e01194d0 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.565+0000 7fbd227a9740 10 client.16281 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.565+0000 7fbd227a9740 1 -- 172.21.15.129:0/4188162788 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5619e011a0f0 con 0x5619e01194d0 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.570+0000 7fbcfcff9640 1 -- 172.21.15.129:0/4188162788 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd0c041070 con 0x5619e01194d0 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.570+0000 7fbcfcff9640 1 -- 172.21.15.129:0/4188162788 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fbd10030020 con 0x5619e01194d0 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.570+0000 7fbd227a9740 10 client.16281 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.570+0000 7fbd227a9740 10 client.16281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.570+0000 7fbd227a9740 20 client.16281 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.570+0000 7fbd227a9740 1 --2- 172.21.15.129:0/4188162788 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619e0100f60 0x5619e04fde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.570+0000 7fbd227a9740 4 client.16281 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.570+0000 7fbd227a9740 1 -- 172.21.15.129:0/4188162788 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619e0454430 con 0x5619e0100f60 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.574+0000 7fbd18e9d640 1 --2- 172.21.15.129:0/4188162788 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619e0100f60 0x5619e04fde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.577+0000 7fbcfcff9640 10 client.16281 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.577+0000 7fbd18e9d640 1 --2- 172.21.15.129:0/4188162788 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619e0100f60 0x5619e04fde60 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fe8e1f6a4f481d9 server_cookie=a90bca94e61588a9 in_seq=0 out_seq=0 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.582+0000 7fbcfcff9640 1 -- 172.21.15.129:0/4188162788 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5619e0454430 con 0x5619e0100f60 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.582+0000 7fbcfcff9640 10 client.16281 handle_command_reply: tid=0 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.582+0000 7fbd227a9740 1 client.16281 shutdown 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.596+0000 7fbd227a9740 1 -- 172.21.15.129:0/4188162788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01194d0 msgr2=0x5619e04eb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.596+0000 7fbd227a9740 1 --2- 172.21.15.129:0/4188162788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01194d0 0x5619e04eb390 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x5619e04de820 tx=0x7fbd100046c0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.606+0000 7fbd227a9740 1 -- 172.21.15.129:0/4188162788 shutdown_connections 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.606+0000 7fbd227a9740 1 --2- 172.21.15.129:0/4188162788 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619e0100f60 0x5619e04fde60 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.606+0000 7fbd227a9740 1 --2- 172.21.15.129:0/4188162788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04f1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.606+0000 7fbd227a9740 1 --2- 172.21.15.129:0/4188162788 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 0x5619e04eb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.606+0000 7fbd227a9740 1 --2- 172.21.15.129:0/4188162788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01194d0 0x5619e04eb390 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.606+0000 7fbcfcff9640 0 client.16281 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.606+0000 7fbd227a9740 1 -- 172.21.15.129:0/4188162788 >> 172.21.15.129:0/4188162788 conn(0x5619e01364e0 msgr2=0x5619e04e0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.608+0000 7fbd227a9740 1 -- 172.21.15.129:0/4188162788 shutdown_connections 2024-05-20T18:04:34.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.610+0000 7fbd227a9740 1 -- 172.21.15.129:0/4188162788 wait complete. 2024-05-20T18:04:34.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.611+0000 7fbd227a9740 20 client.16281 trim_cache size 0 max 16384 2024-05-20T18:04:34.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.672+0000 7fbd227a9740 1 Processor -- start 2024-05-20T18:04:34.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.672+0000 7fbd227a9740 1 -- start start 2024-05-20T18:04:34.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.673+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e01194d0 0x5619e04f14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.673+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 0x5619e04dc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.673+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e0142390 0x5619e0503620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.673+0000 7fbd227a9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf800c920 con 0x5619e0142390 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.673+0000 7fbd227a9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf800f1a0 con 0x5619e01194d0 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.673+0000 7fbd227a9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf800cf90 con 0x5619e01216b0 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.675+0000 7fbd18e9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 0x5619e04dc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.675+0000 7fbd18e9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 0x5619e04dc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:58008/0 (socket says 172.21.15.129:58008) 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.675+0000 7fbd18e9d640 1 -- 172.21.15.129:0/1165549727 learned_addr learned my addr 172.21.15.129:0/1165549727 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.678+0000 7fbd19e9f640 1 --2- 172.21.15.129:0/1165549727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e0142390 0x5619e0503620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.679+0000 7fbd1969e640 1 --2- 172.21.15.129:0/1165549727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e01194d0 0x5619e04f14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.679+0000 7fbd19e9f640 1 -- 172.21.15.129:0/1165549727 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 msgr2=0x5619e04dc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:34.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.679+0000 7fbd19e9f640 1 --2- 172.21.15.129:0/1165549727 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 0x5619e04dc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.679+0000 7fbd19e9f640 1 -- 172.21.15.129:0/1165549727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e01194d0 msgr2=0x5619e04f14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:34.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.679+0000 7fbd19e9f640 1 --2- 172.21.15.129:0/1165549727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e01194d0 0x5619e04f14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.679+0000 7fbd19e9f640 1 -- 172.21.15.129:0/1165549727 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619e04e96d0 con 0x5619e0142390 2024-05-20T18:04:34.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.680+0000 7fbd19e9f640 1 --2- 172.21.15.129:0/1165549727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e0142390 0x5619e0503620 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fbd040cc3f0 tx=0x7fbd040974e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98c190bb352daed server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:34.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.685+0000 7fbcfeffd640 1 -- 172.21.15.129:0/1165549727 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd040994c0 con 0x5619e0142390 2024-05-20T18:04:34.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.685+0000 7fbcfeffd640 1 -- 172.21.15.129:0/1165549727 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd040cc670 con 0x5619e0142390 2024-05-20T18:04:34.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.685+0000 7fbcfeffd640 1 -- 172.21.15.129:0/1165549727 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd040ccb40 con 0x5619e0142390 2024-05-20T18:04:34.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.687+0000 7fbd227a9740 1 -- 172.21.15.129:0/1165549727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e0142390 msgr2=0x5619e0503620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:34.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.687+0000 7fbd227a9740 1 --2- 172.21.15.129:0/1165549727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e0142390 0x5619e0503620 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fbd040cc3f0 tx=0x7fbd040974e0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.687+0000 7fbd227a9740 1 -- 172.21.15.129:0/1165549727 shutdown_connections 2024-05-20T18:04:34.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.687+0000 7fbd227a9740 1 --2- 172.21.15.129:0/1165549727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e0142390 0x5619e0503620 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.687+0000 7fbd227a9740 1 --2- 172.21.15.129:0/1165549727 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01216b0 0x5619e04dc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.687+0000 7fbd227a9740 1 --2- 172.21.15.129:0/1165549727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e01194d0 0x5619e04f14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.687+0000 7fbd227a9740 1 -- 172.21.15.129:0/1165549727 >> 172.21.15.129:0/1165549727 conn(0x5619e01364e0 msgr2=0x5619e04dbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.687+0000 7fbd227a9740 1 -- 172.21.15.129:0/1165549727 shutdown_connections 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.689+0000 7fbd227a9740 1 -- 172.21.15.129:0/1165549727 wait complete. 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.690+0000 7fbd227a9740 1 -- start start 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.690+0000 7fbd227a9740 1 Processor -- start 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.691+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01194d0 0x5619e04f7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.691+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01216b0 0x5619e04f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.691+0000 7fbd227a9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04f3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.691+0000 7fbd227a9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf800d660 con 0x5619e01216b0 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.691+0000 7fbd227a9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf800ead0 con 0x5619e0142390 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.691+0000 7fbd227a9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf800dd30 con 0x5619e01194d0 2024-05-20T18:04:34.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.691+0000 7fbd227a9740 10 client.0 _ll_register_callbacks cb 0x7fbd10009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:34.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.694+0000 7fbd1969e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01194d0 0x5619e04f7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.694+0000 7fbd1969e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01194d0 0x5619e04f7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:58014/0 (socket says 172.21.15.129:58014) 2024-05-20T18:04:34.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.694+0000 7fbd1969e640 1 -- 172.21.15.129:0/3730935125 learned_addr learned my addr 172.21.15.129:0/3730935125 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:34.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.694+0000 7fbd19e9f640 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.696+0000 7fbd18e9d640 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01216b0 0x5619e04f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.697+0000 7fbd1969e640 1 -- 172.21.15.129:0/3730935125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 msgr2=0x5619e04f3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:34.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.697+0000 7fbd1969e640 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04f3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.697+0000 7fbd1969e640 1 -- 172.21.15.129:0/3730935125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01216b0 msgr2=0x5619e04f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:34.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.697+0000 7fbd1969e640 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01216b0 0x5619e04f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.697+0000 7fbd1969e640 1 -- 172.21.15.129:0/3730935125 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619e04f3da0 con 0x5619e01194d0 2024-05-20T18:04:34.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.699+0000 7fbd1969e640 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01194d0 0x5619e04f7c90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009150 tx=0x7fbd1001f720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=151a20467811bf2d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:34.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.699+0000 7fbcfcff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:34.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.702+0000 7fbcfcff9640 1 -- 172.21.15.129:0/3730935125 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd10023040 con 0x5619e01194d0 2024-05-20T18:04:34.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.702+0000 7fbcfcff9640 1 -- 172.21.15.129:0/3730935125 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fbd1001fd20 con 0x5619e01194d0 2024-05-20T18:04:34.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.703+0000 7fbd227a9740 1 -- 172.21.15.129:0/3730935125 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd040cc670 con 0x5619e01194d0 2024-05-20T18:04:34.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.705+0000 7fbcfcff9640 1 -- 172.21.15.129:0/3730935125 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbd100059c0 con 0x5619e01194d0 2024-05-20T18:04:34.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.711+0000 7fbd227a9740 10 client.26056 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:34.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.711+0000 7fbd227a9740 1 -- 172.21.15.129:0/3730935125 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5619e0566280 con 0x5619e01194d0 2024-05-20T18:04:34.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.714+0000 7fbcfcff9640 1 -- 172.21.15.129:0/3730935125 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd040cc670 con 0x5619e01194d0 2024-05-20T18:04:34.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.715+0000 7fbcfcff9640 1 -- 172.21.15.129:0/3730935125 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fbd10030040 con 0x5619e01194d0 2024-05-20T18:04:34.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.715+0000 7fbd227a9740 10 client.26056 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:34.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.715+0000 7fbd227a9740 10 client.26056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:34.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.715+0000 7fbd227a9740 20 client.26056 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:34.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.715+0000 7fbd227a9740 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619e00d63a0 0x5619e0568a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:34.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.716+0000 7fbd227a9740 4 client.26056 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:04:34.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.716+0000 7fbd227a9740 1 -- 172.21.15.129:0/3730935125 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5619e044e780 con 0x5619e00d63a0 2024-05-20T18:04:34.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.717+0000 7fbd18e9d640 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619e00d63a0 0x5619e0568a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:34.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.719+0000 7fbd18e9d640 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619e00d63a0 0x5619e0568a10 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a506d37cb2348df4 server_cookie=6a37f2c3d567ec35 in_seq=0 out_seq=0 2024-05-20T18:04:34.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.719+0000 7fbcfcff9640 10 client.26056 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:34.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.721+0000 7fbcfcff9640 1 -- 172.21.15.129:0/3730935125 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5619e044e780 con 0x5619e00d63a0 2024-05-20T18:04:34.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.721+0000 7fbcfcff9640 10 client.26056 handle_command_reply: tid=0 2024-05-20T18:04:34.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.722+0000 7fbd227a9740 1 client.26056 shutdown 2024-05-20T18:04:34.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.724+0000 7fbd227a9740 1 -- 172.21.15.129:0/3730935125 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01194d0 msgr2=0x5619e04f7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:34.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.724+0000 7fbd227a9740 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01194d0 0x5619e04f7c90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009150 tx=0x7fbd1001f720 comp rx=0 tx=0).stop 2024-05-20T18:04:34.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.725+0000 7fbd227a9740 1 -- 172.21.15.129:0/3730935125 shutdown_connections 2024-05-20T18:04:34.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.726+0000 7fbd227a9740 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619e00d63a0 0x5619e0568a10 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.726+0000 7fbd227a9740 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619e0142390 0x5619e04f3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.726+0000 7fbd227a9740 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619e01216b0 0x5619e04f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.726+0000 7fbd227a9740 1 --2- 172.21.15.129:0/3730935125 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619e01194d0 0x5619e04f7c90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.726+0000 7fbd227a9740 1 -- 172.21.15.129:0/3730935125 >> 172.21.15.129:0/3730935125 conn(0x5619e01364e0 msgr2=0x5619e04e4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:34.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.726+0000 7fbcfcff9640 0 client.26056 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:34.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.728+0000 7fbd227a9740 1 -- 172.21.15.129:0/3730935125 shutdown_connections 2024-05-20T18:04:34.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.729+0000 7fbd227a9740 1 -- 172.21.15.129:0/3730935125 wait complete. 2024-05-20T18:04:34.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.729+0000 7fbd227a9740 20 client.26056 trim_cache size 0 max 16384 2024-05-20T18:04:34.728 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:04:34.728 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:04:34.728 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:04:34.728 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:04:34.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.742+0000 7fbd1b902640 1 -- 172.21.15.129:0/1860327568 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd000a2d00 msgr2=0x7fbd000a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:34.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.742+0000 7fbd1b902640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd000a2d00 0x7fbd000a5210 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbd10006aa0 tx=0x7fbd10006a30 comp rx=0 tx=0).stop 2024-05-20T18:04:34.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.742+0000 7fbd1b902640 1 -- 172.21.15.129:0/1860327568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14114020 msgr2=0x7fbd141bcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:34.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.742+0000 7fbd1b902640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14114020 0x7fbd141bcf60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbd0400d750 tx=0x7fbd0400db50 comp rx=0 tx=0).stop 2024-05-20T18:04:34.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.742+0000 7fbd1b902640 1 -- 172.21.15.129:0/1860327568 shutdown_connections 2024-05-20T18:04:34.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.742+0000 7fbd1b902640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd000a2d00 0x7fbd000a5210 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.742+0000 7fbd1b902640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd14114020 0x7fbd141bcf60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.742+0000 7fbd1b902640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd1411c0d0 0x7fbd14113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.742+0000 7fbd1b902640 1 --2- 172.21.15.129:0/1860327568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd14109060 0x7fbd141135a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:34.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.742+0000 7fbd1b902640 1 -- 172.21.15.129:0/1860327568 >> 172.21.15.129:0/1860327568 conn(0x7fbd1406d750 msgr2=0x7fbd14119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:34.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.745+0000 7fbd1b902640 1 -- 172.21.15.129:0/1860327568 shutdown_connections 2024-05-20T18:04:34.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:34.746+0000 7fbd1b902640 1 -- 172.21.15.129:0/1860327568 wait complete. 2024-05-20T18:04:34.808 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:04:34.808 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:04:35.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.378+0000 7f578759e640 1 -- 172.21.15.129:0/3294403209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f578810e890 msgr2=0x7f578810eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:35.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.378+0000 7f578759e640 1 --2- 172.21.15.129:0/3294403209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f578810e890 0x7f578810eca0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5778009900 tx=0x7f577802a440 comp rx=0 tx=0).stop 2024-05-20T18:04:35.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.391+0000 7f578759e640 1 -- 172.21.15.129:0/3294403209 shutdown_connections 2024-05-20T18:04:35.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.391+0000 7f578759e640 1 --2- 172.21.15.129:0/3294403209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5788109090 0x7f5788109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:35.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.391+0000 7f578759e640 1 --2- 172.21.15.129:0/3294403209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f578810f270 0x7f5788108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:35.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.391+0000 7f578759e640 1 --2- 172.21.15.129:0/3294403209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f578810e890 0x7f578810eca0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:35.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.391+0000 7f578759e640 1 -- 172.21.15.129:0/3294403209 >> 172.21.15.129:0/3294403209 conn(0x7f578806d580 msgr2=0x7f578806d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:35.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.392+0000 7f578759e640 1 -- 172.21.15.129:0/3294403209 shutdown_connections 2024-05-20T18:04:35.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.395+0000 7f578759e640 1 -- 172.21.15.129:0/3294403209 wait complete. 2024-05-20T18:04:35.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.397+0000 7f578759e640 1 Processor -- start 2024-05-20T18:04:35.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.398+0000 7f578759e640 1 -- start start 2024-05-20T18:04:35.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.398+0000 7f578759e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5788109090 0x7f57881a8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:35.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.398+0000 7f578759e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f578810e890 0x7f57881a8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:35.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.398+0000 7f578759e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f578810f270 0x7f57881acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:35.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.398+0000 7f578759e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5788121ad0 con 0x7f578810f270 2024-05-20T18:04:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.398+0000 7f578759e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5788121950 con 0x7f5788109090 2024-05-20T18:04:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.398+0000 7f578759e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5788121c50 con 0x7f578810e890 2024-05-20T18:04:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.399+0000 7f578659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5788109090 0x7f57881a8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.399+0000 7f578659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5788109090 0x7f57881a8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:48592/0 (socket says 172.21.15.129:48592) 2024-05-20T18:04:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.399+0000 7f578659c640 1 -- 172.21.15.129:0/1265311487 learned_addr learned my addr 172.21.15.129:0/1265311487 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:35.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.401+0000 7f578659c640 1 -- 172.21.15.129:0/1265311487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f578810e890 msgr2=0x7f57881a8990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T18:04:35.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.401+0000 7f578659c640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f578810e890 0x7f57881a8990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:35.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.401+0000 7f578659c640 1 -- 172.21.15.129:0/1265311487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f578810f270 msgr2=0x7f57881acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:35.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.401+0000 7f578659c640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f578810f270 0x7f57881acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:35.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.401+0000 7f578659c640 1 -- 172.21.15.129:0/1265311487 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57881ad290 con 0x7f5788109090 2024-05-20T18:04:35.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.406+0000 7f578659c640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5788109090 0x7f57881a8430 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f577802a9b0 tx=0x7f5778005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:35.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.413+0000 7f57777fe640 1 -- 172.21.15.129:0/1265311487 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5778005640 con 0x7f5788109090 2024-05-20T18:04:35.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.413+0000 7f578759e640 1 -- 172.21.15.129:0/1265311487 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57881ad520 con 0x7f5788109090 2024-05-20T18:04:35.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.413+0000 7f578759e640 1 -- 172.21.15.129:0/1265311487 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57881ada60 con 0x7f5788109090 2024-05-20T18:04:35.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.414+0000 7f57777fe640 1 -- 172.21.15.129:0/1265311487 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5778031400 con 0x7f5788109090 2024-05-20T18:04:35.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.414+0000 7f57777fe640 1 -- 172.21.15.129:0/1265311487 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5778004ae0 con 0x7f5788109090 2024-05-20T18:04:35.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.420+0000 7f57777fe640 1 -- 172.21.15.129:0/1265311487 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5778004c80 con 0x7f5788109090 2024-05-20T18:04:35.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.422+0000 7f57777fe640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f57700a2fe0 0x7f57700a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:35.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.422+0000 7f57777fe640 1 -- 172.21.15.129:0/1265311487 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f57780f03c0 con 0x7f5788109090 2024-05-20T18:04:35.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.431+0000 7f5785d9b640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f57700a2fe0 0x7f57700a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:35.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.431+0000 7f578759e640 1 -- 172.21.15.129:0/1265311487 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5768005290 con 0x7f5788109090 2024-05-20T18:04:35.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.452+0000 7f5785d9b640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f57700a2fe0 0x7f57700a54f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f578810ab00 tx=0x7f577c006bd0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:35.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:35.452+0000 7f57777fe640 1 -- 172.21.15.129:0/1265311487 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f577804a020 con 0x7f5788109090 2024-05-20T18:04:35.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:35 smithi154 ceph-mon[36482]: pgmap v185: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 712 MiB/s wr, 2.03k op/s 2024-05-20T18:04:35.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:35 smithi129 ceph-mon[32029]: pgmap v185: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 712 MiB/s wr, 2.03k op/s 2024-05-20T18:04:35.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:35 smithi204 ceph-mon[37076]: pgmap v185: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 712 MiB/s wr, 2.03k op/s 2024-05-20T18:04:36.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.255+0000 7f578759e640 1 -- 172.21.15.129:0/1265311487 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5768005850 con 0x7f5788109090 2024-05-20T18:04:36.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.256+0000 7f57777fe640 1 -- 172.21.15.129:0/1265311487 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11829 (secure 0 0 0) 0x7f57780bcf70 con 0x7f5788109090 2024-05-20T18:04:36.255 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:04:36.255 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:03:18.451223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/4068528573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":4068528573},{"type":"v1","addr":"172.21.15.154:6837","nonce":4068528573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:04:36.255 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 24 2024-05-20T18:04:36.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.262+0000 7f578759e640 1 -- 172.21.15.129:0/1265311487 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f57700a2fe0 msgr2=0x7f57700a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:36.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.262+0000 7f578759e640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f57700a2fe0 0x7f57700a54f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f578810ab00 tx=0x7f577c006bd0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.262+0000 7f578759e640 1 -- 172.21.15.129:0/1265311487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5788109090 msgr2=0x7f57881a8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:36.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.262+0000 7f578759e640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5788109090 0x7f57881a8430 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f577802a9b0 tx=0x7f5778005040 comp rx=0 tx=0).stop 2024-05-20T18:04:36.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.262+0000 7f578759e640 1 -- 172.21.15.129:0/1265311487 shutdown_connections 2024-05-20T18:04:36.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.263+0000 7f578759e640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f57700a2fe0 0x7f57700a54f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.263+0000 7f578759e640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f578810f270 0x7f57881acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.263+0000 7f578759e640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f578810e890 0x7f57881a8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.263+0000 7f578759e640 1 --2- 172.21.15.129:0/1265311487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5788109090 0x7f57881a8430 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.263+0000 7f578759e640 1 -- 172.21.15.129:0/1265311487 >> 172.21.15.129:0/1265311487 conn(0x7f578806d580 msgr2=0x7f57880728c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:36.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.263+0000 7f578759e640 1 -- 172.21.15.129:0/1265311487 shutdown_connections 2024-05-20T18:04:36.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.263+0000 7f578759e640 1 -- 172.21.15.129:0/1265311487 wait complete. 2024-05-20T18:04:36.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:04:36.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:36 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1265311487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:04:36.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:36 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1265311487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:04:36.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:36 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1265311487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:04:36.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.931+0000 7faba4b3e640 1 -- 172.21.15.129:0/1934015997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faba011c0d0 msgr2=0x7faba011e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:36.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.931+0000 7faba4b3e640 1 --2- 172.21.15.129:0/1934015997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faba011c0d0 0x7faba011e4f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fab90009900 tx=0x7fab9002a4d0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.936+0000 7faba4b3e640 1 -- 172.21.15.129:0/1934015997 shutdown_connections 2024-05-20T18:04:36.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.936+0000 7faba4b3e640 1 --2- 172.21.15.129:0/1934015997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faba011c0d0 0x7faba011e4f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.936+0000 7faba4b3e640 1 --2- 172.21.15.129:0/1934015997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faba0119770 0x7faba011bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.936+0000 7faba4b3e640 1 --2- 172.21.15.129:0/1934015997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faba0108f70 0x7faba0109380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.936+0000 7faba4b3e640 1 -- 172.21.15.129:0/1934015997 >> 172.21.15.129:0/1934015997 conn(0x7faba006d5b0 msgr2=0x7faba006da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:36.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.947+0000 7faba4b3e640 1 -- 172.21.15.129:0/1934015997 shutdown_connections 2024-05-20T18:04:36.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.949+0000 7faba4b3e640 1 -- 172.21.15.129:0/1934015997 wait complete. 2024-05-20T18:04:36.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.950+0000 7faba4b3e640 1 Processor -- start 2024-05-20T18:04:36.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.952+0000 7faba4b3e640 1 -- start start 2024-05-20T18:04:36.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.952+0000 7faba4b3e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faba0108f70 0x7faba01a8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:36.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.952+0000 7faba4b3e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faba0119770 0x7faba01a8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:36.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.952+0000 7faba4b3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faba011c0d0 0x7faba01acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:36.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.952+0000 7faba4b3e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faba0121a90 con 0x7faba0108f70 2024-05-20T18:04:36.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.952+0000 7faba4b3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faba0121910 con 0x7faba011c0d0 2024-05-20T18:04:36.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.952+0000 7faba4b3e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faba0121c10 con 0x7faba0119770 2024-05-20T18:04:36.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.953+0000 7fab9e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faba0108f70 0x7faba01a8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:36.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.953+0000 7fab9e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faba0108f70 0x7faba01a8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57902/0 (socket says 172.21.15.129:57902) 2024-05-20T18:04:36.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.954+0000 7fab9e59c640 1 -- 172.21.15.129:0/2168746929 learned_addr learned my addr 172.21.15.129:0/2168746929 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:36.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.954+0000 7fab9dd9b640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faba0119770 0x7faba01a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:36.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.954+0000 7fab9ed9d640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faba011c0d0 0x7faba01acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:36.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.956+0000 7fab9dd9b640 1 -- 172.21.15.129:0/2168746929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faba011c0d0 msgr2=0x7faba01acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:36.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.956+0000 7fab9dd9b640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faba011c0d0 0x7faba01acc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.956+0000 7fab9dd9b640 1 -- 172.21.15.129:0/2168746929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faba0108f70 msgr2=0x7faba01a8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:36.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.956+0000 7fab9dd9b640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faba0108f70 0x7faba01a8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.956+0000 7fab9dd9b640 1 -- 172.21.15.129:0/2168746929 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba01ad290 con 0x7faba0119770 2024-05-20T18:04:36.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.957+0000 7fab9dd9b640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faba0119770 0x7faba01a8990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fab94009fd0 tx=0x7fab9400b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:36.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.958+0000 7fab8f7fe640 1 -- 172.21.15.129:0/2168746929 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94015020 con 0x7faba0119770 2024-05-20T18:04:36.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.958+0000 7faba4b3e640 1 -- 172.21.15.129:0/2168746929 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba01ad580 con 0x7faba0119770 2024-05-20T18:04:36.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.958+0000 7faba4b3e640 1 -- 172.21.15.129:0/2168746929 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba01b4f00 con 0x7faba0119770 2024-05-20T18:04:36.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.968+0000 7fabab9ed740 1 Processor -- start 2024-05-20T18:04:36.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.966+0000 7fab8f7fe640 1 -- 172.21.15.129:0/2168746929 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab94011d10 con 0x7faba0119770 2024-05-20T18:04:36.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.969+0000 7fab8f7fe640 1 -- 172.21.15.129:0/2168746929 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9400f2e0 con 0x7faba0119770 2024-05-20T18:04:36.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.969+0000 7fab8f7fe640 1 -- 172.21.15.129:0/2168746929 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fab94018020 con 0x7faba0119770 2024-05-20T18:04:36.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.970+0000 7fab8f7fe640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fab880a2e70 0x7fab880a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:36.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.971+0000 7fab8f7fe640 1 -- 172.21.15.129:0/2168746929 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fab940cfc30 con 0x7faba0119770 2024-05-20T18:04:36.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.971+0000 7fabab9ed740 1 -- start start 2024-05-20T18:04:36.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.972+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c7af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:36.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.972+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c7b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:36.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.972+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c7ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:36.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.972+0000 7fabab9ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5619f1be0c40 con 0x5619f18dc390 2024-05-20T18:04:36.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.972+0000 7fabab9ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5619f1b88580 con 0x5619f18bb6b0 2024-05-20T18:04:36.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.972+0000 7fabab9ed740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5619f1bdcc00 con 0x5619f18b34d0 2024-05-20T18:04:36.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.973+0000 7fab9e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c7af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:36.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.974+0000 7fab9e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c7af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57918/0 (socket says 172.21.15.129:57918) 2024-05-20T18:04:36.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.974+0000 7fab9e59c640 1 -- 172.21.15.129:0/499993815 learned_addr learned my addr 172.21.15.129:0/499993815 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:36.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.975+0000 7fab9ed9d640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fab880a2e70 0x7fab880a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:36.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.975+0000 7fab9ed9d640 1 --2- 172.21.15.129:0/499993815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c7ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:36.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.977+0000 7fab9e59c640 1 -- 172.21.15.129:0/499993815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 msgr2=0x5619f1c7b4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:04:36.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.977+0000 7fab9e59c640 1 --2- 172.21.15.129:0/499993815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c7b4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.977+0000 7fab9e59c640 1 -- 172.21.15.129:0/499993815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 msgr2=0x5619f1c7ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:36.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.977+0000 7fab9e59c640 1 --2- 172.21.15.129:0/499993815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c7ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.977+0000 7fab9e59c640 1 -- 172.21.15.129:0/499993815 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619f18f43f0 con 0x5619f18dc390 2024-05-20T18:04:36.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.977+0000 7fab9e59c640 1 --2- 172.21.15.129:0/499993815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c7af90 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fab98011490 tx=0x7fab98011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92666058b520f61 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:36.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.981+0000 7fab9ed9d640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fab880a2e70 0x7fab880a5380 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fab90006010 tx=0x7fab900314f0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:36.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.986+0000 7fab8cff9640 1 -- 172.21.15.129:0/499993815 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab98012770 con 0x5619f18dc390 2024-05-20T18:04:36.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.986+0000 7fab8cff9640 1 -- 172.21.15.129:0/499993815 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab98012df0 con 0x5619f18dc390 2024-05-20T18:04:36.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.987+0000 7fabab9ed740 1 -- 172.21.15.129:0/499993815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 msgr2=0x5619f1c7af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:36.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.987+0000 7fabab9ed740 1 --2- 172.21.15.129:0/499993815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c7af90 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fab98011490 tx=0x7fab98011950 comp rx=0 tx=0).stop 2024-05-20T18:04:36.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.991+0000 7fabab9ed740 1 -- 172.21.15.129:0/499993815 shutdown_connections 2024-05-20T18:04:36.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.991+0000 7fabab9ed740 1 --2- 172.21.15.129:0/499993815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c7ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.991+0000 7fabab9ed740 1 --2- 172.21.15.129:0/499993815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c7b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.991+0000 7fabab9ed740 1 --2- 172.21.15.129:0/499993815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c7af90 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:36.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.991+0000 7fabab9ed740 1 -- 172.21.15.129:0/499993815 >> 172.21.15.129:0/499993815 conn(0x5619f18d04e0 msgr2=0x5619f1c77740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:36.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:36.998+0000 7fabab9ed740 1 -- 172.21.15.129:0/499993815 shutdown_connections 2024-05-20T18:04:36.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.000+0000 7fabab9ed740 1 -- 172.21.15.129:0/499993815 wait complete. 2024-05-20T18:04:36.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.001+0000 7fabab9ed740 1 -- start start 2024-05-20T18:04:36.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.001+0000 7fabab9ed740 1 Processor -- start 2024-05-20T18:04:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.002+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c854a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.002+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c859e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.002+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c91860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:37.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.002+0000 7fabab9ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5619f1c88330 con 0x5619f18dc390 2024-05-20T18:04:37.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.002+0000 7fabab9ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5619f1b9bc40 con 0x5619f18bb6b0 2024-05-20T18:04:37.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.002+0000 7fabab9ed740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5619f1c884b0 con 0x5619f18b34d0 2024-05-20T18:04:37.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.002+0000 7fabab9ed740 10 client.0 _ll_register_callbacks cb 0x5619f18f3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:37.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.003+0000 7fab9ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c91860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:37.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.003+0000 7fab9ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c91860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57920/0 (socket says 172.21.15.129:57920) 2024-05-20T18:04:37.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.003+0000 7fab9ed9d640 1 -- 172.21.15.129:0/4081447791 learned_addr learned my addr 172.21.15.129:0/4081447791 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:37.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.005+0000 7fab9e59c640 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c854a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:37.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.005+0000 7fab9dd9b640 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c859e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:37.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.008+0000 7fab9e59c640 1 -- 172.21.15.129:0/4081447791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 msgr2=0x5619f1c859e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:37.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.008+0000 7fab9e59c640 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c859e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.008+0000 7fab9e59c640 1 -- 172.21.15.129:0/4081447791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 msgr2=0x5619f1c91860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:37.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.008+0000 7fab9e59c640 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c91860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.008+0000 7fab9e59c640 1 -- 172.21.15.129:0/4081447791 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619f19be7d0 con 0x5619f18b34d0 2024-05-20T18:04:37.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.008+0000 7fab9ed9d640 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c91860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:04:37.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.010+0000 7fab9e59c640 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c854a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x5619f1c78100 tx=0x7fab9801a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2433f9380f8e5a9f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:37.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.010+0000 7fab85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:37.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.010+0000 7fab85ffb640 1 -- 172.21.15.129:0/4081447791 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab98028070 con 0x5619f18b34d0 2024-05-20T18:04:37.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.010+0000 7fabab9ed740 1 -- 172.21.15.129:0/4081447791 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab98012df0 con 0x5619f18b34d0 2024-05-20T18:04:37.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.010+0000 7fab85ffb640 1 -- 172.21.15.129:0/4081447791 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fab98003d80 con 0x5619f18b34d0 2024-05-20T18:04:37.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.011+0000 7fab85ffb640 1 -- 172.21.15.129:0/4081447791 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fab98031050 con 0x5619f18b34d0 2024-05-20T18:04:37.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.011+0000 7fabab9ed740 10 client.26062 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:37.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.011+0000 7fabab9ed740 1 -- 172.21.15.129:0/4081447791 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5619f1c949b0 con 0x5619f18b34d0 2024-05-20T18:04:37.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.012+0000 7fab85ffb640 1 -- 172.21.15.129:0/4081447791 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab98012df0 con 0x5619f18b34d0 2024-05-20T18:04:37.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.012+0000 7fab85ffb640 1 -- 172.21.15.129:0/4081447791 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fab9802b020 con 0x5619f18b34d0 2024-05-20T18:04:37.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.012+0000 7fabab9ed740 10 client.26062 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:37.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.012+0000 7fabab9ed740 10 client.26062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:37.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.012+0000 7fabab9ed740 20 client.26062 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:37.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.012+0000 7fabab9ed740 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619f189af60 0x5619f1c98390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:37.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.012+0000 7fabab9ed740 4 client.26062 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:04:37.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.012+0000 7fabab9ed740 1 -- 172.21.15.129:0/4081447791 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619f1bee370 con 0x5619f189af60 2024-05-20T18:04:37.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.021+0000 7fab9dd9b640 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619f189af60 0x5619f1c98390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:37.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.025+0000 7fab9dd9b640 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619f189af60 0x5619f1c98390 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=536eed7b61983461 server_cookie=3c5633560d309a0c in_seq=0 out_seq=0 2024-05-20T18:04:37.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.025+0000 7fab85ffb640 10 client.26062 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:37.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.027+0000 7fab85ffb640 1 -- 172.21.15.129:0/4081447791 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5619f1bee370 con 0x5619f189af60 2024-05-20T18:04:37.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.027+0000 7fab85ffb640 10 client.26062 handle_command_reply: tid=0 2024-05-20T18:04:37.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.028+0000 7fabab9ed740 1 client.26062 shutdown 2024-05-20T18:04:37.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.031+0000 7fabab9ed740 1 -- 172.21.15.129:0/4081447791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 msgr2=0x5619f1c854a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:37.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.031+0000 7fabab9ed740 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c854a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x5619f1c78100 tx=0x7fab9801a620 comp rx=0 tx=0).stop 2024-05-20T18:04:37.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.031+0000 7fabab9ed740 1 -- 172.21.15.129:0/4081447791 shutdown_connections 2024-05-20T18:04:37.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.031+0000 7fabab9ed740 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619f189af60 0x5619f1c98390 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.031+0000 7fabab9ed740 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c859e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.031+0000 7fabab9ed740 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18dc390 0x5619f1c91860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.031+0000 7fabab9ed740 1 --2- 172.21.15.129:0/4081447791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c854a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.031+0000 7fabab9ed740 1 -- 172.21.15.129:0/4081447791 >> 172.21.15.129:0/4081447791 conn(0x5619f18d04e0 msgr2=0x5619f1c7ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:37.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.031+0000 7fab85ffb640 0 client.26062 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:37.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.032+0000 7fabab9ed740 1 -- 172.21.15.129:0/4081447791 shutdown_connections 2024-05-20T18:04:37.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.032+0000 7fabab9ed740 1 -- 172.21.15.129:0/4081447791 wait complete. 2024-05-20T18:04:37.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.037+0000 7fabab9ed740 20 client.26062 trim_cache size 0 max 16384 2024-05-20T18:04:37.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.139+0000 7fabab9ed740 1 Processor -- start 2024-05-20T18:04:37.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.140+0000 7fabab9ed740 1 -- start start 2024-05-20T18:04:37.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.151+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18b34d0 0x5619f1c77810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:37.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.152+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c7fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:37.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.152+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18dc390 0x5619f1c91440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:37.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.152+0000 7fabab9ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fab8000c920 con 0x5619f18b34d0 2024-05-20T18:04:37.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.153+0000 7fabab9ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fab8000f1a0 con 0x5619f18bb6b0 2024-05-20T18:04:37.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.153+0000 7fabab9ed740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fab8000cf90 con 0x5619f18dc390 2024-05-20T18:04:37.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.154+0000 7fab9ed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18dc390 0x5619f1c91440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.154+0000 7fab9e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18b34d0 0x5619f1c77810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.154+0000 7fab9e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18b34d0 0x5619f1c77810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57932/0 (socket says 172.21.15.129:57932) 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.154+0000 7fab9e59c640 1 -- 172.21.15.129:0/3879819631 learned_addr learned my addr 172.21.15.129:0/3879819631 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.154+0000 7fab9dd9b640 1 --2- 172.21.15.129:0/3879819631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c7fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.156+0000 7fab9ed9d640 1 -- 172.21.15.129:0/3879819631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 msgr2=0x5619f1c7fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.156+0000 7fab9ed9d640 1 --2- 172.21.15.129:0/3879819631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c7fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.156+0000 7fab9ed9d640 1 -- 172.21.15.129:0/3879819631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18b34d0 msgr2=0x5619f1c77810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.156+0000 7fab9ed9d640 1 --2- 172.21.15.129:0/3879819631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18b34d0 0x5619f1c77810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.156+0000 7fab9ed9d640 1 -- 172.21.15.129:0/3879819631 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619f1c7ff90 con 0x5619f18dc390 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.158+0000 7fab9ed9d640 1 --2- 172.21.15.129:0/3879819631 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18dc390 0x5619f1c91440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x5619f1c72e60 tx=0x7fab9003f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=538651aae09f78b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.158+0000 7fab87fff640 1 -- 172.21.15.129:0/3879819631 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab90035050 con 0x5619f18dc390 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.159+0000 7fab87fff640 1 -- 172.21.15.129:0/3879819631 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab900325c0 con 0x5619f18dc390 2024-05-20T18:04:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.161+0000 7fab87fff640 1 -- 172.21.15.129:0/3879819631 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9004a680 con 0x5619f18dc390 2024-05-20T18:04:37.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.164+0000 7fabab9ed740 1 -- 172.21.15.129:0/3879819631 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18dc390 msgr2=0x5619f1c91440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:37.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.164+0000 7fabab9ed740 1 --2- 172.21.15.129:0/3879819631 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18dc390 0x5619f1c91440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x5619f1c72e60 tx=0x7fab9003f000 comp rx=0 tx=0).stop 2024-05-20T18:04:37.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.174+0000 7fabab9ed740 1 -- 172.21.15.129:0/3879819631 shutdown_connections 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.174+0000 7fabab9ed740 1 --2- 172.21.15.129:0/3879819631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18bb6b0 0x5619f1c7fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.174+0000 7fabab9ed740 1 --2- 172.21.15.129:0/3879819631 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18dc390 0x5619f1c91440 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.174+0000 7fabab9ed740 1 --2- 172.21.15.129:0/3879819631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18b34d0 0x5619f1c77810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.174+0000 7fabab9ed740 1 -- 172.21.15.129:0/3879819631 >> 172.21.15.129:0/3879819631 conn(0x5619f18d04e0 msgr2=0x5619f1c77100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.174+0000 7fabab9ed740 1 -- 172.21.15.129:0/3879819631 shutdown_connections 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.174+0000 7fabab9ed740 1 -- 172.21.15.129:0/3879819631 wait complete. 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.175+0000 7fabab9ed740 1 -- start start 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.175+0000 7fabab9ed740 1 Processor -- start 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.176+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c8d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.177+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18bb6b0 0x5619f1c88780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.177+0000 7fabab9ed740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18dc390 0x5619f1c88cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:37.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.177+0000 7fabab9ed740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fab8000d660 con 0x5619f18bb6b0 2024-05-20T18:04:37.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.177+0000 7fabab9ed740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fab8000ead0 con 0x5619f18dc390 2024-05-20T18:04:37.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.177+0000 7fabab9ed740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fab8000dd30 con 0x5619f18b34d0 2024-05-20T18:04:37.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.178+0000 7fabab9ed740 10 client.0 _ll_register_callbacks cb 0x7fab98009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:37.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.179+0000 7fab9ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18dc390 0x5619f1c88cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:37.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.180+0000 7fab9ed9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18dc390 0x5619f1c88cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:48636/0 (socket says 172.21.15.129:48636) 2024-05-20T18:04:37.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.180+0000 7fab9ed9d640 1 -- 172.21.15.129:0/157955930 learned_addr learned my addr 172.21.15.129:0/157955930 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:37.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.180+0000 7fab9e59c640 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c8d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:37.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.183+0000 7fab9ed9d640 1 -- 172.21.15.129:0/157955930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 msgr2=0x5619f1c8d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:37.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.183+0000 7fab9ed9d640 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c8d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.184+0000 7fab9ed9d640 1 -- 172.21.15.129:0/157955930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18bb6b0 msgr2=0x5619f1c88780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:37.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.184+0000 7fab9ed9d640 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18bb6b0 0x5619f1c88780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.184+0000 7fab9ed9d640 1 -- 172.21.15.129:0/157955930 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619f1c89200 con 0x5619f18dc390 2024-05-20T18:04:37.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.185+0000 7fab9ed9d640 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18dc390 0x5619f1c88cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fab900096c0 tx=0x7fab90032e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cad4fc0e9e8fa83c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:37.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.189+0000 7fab85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:04:37.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.190+0000 7fab85ffb640 1 -- 172.21.15.129:0/157955930 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab90060060 con 0x5619f18dc390 2024-05-20T18:04:37.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.190+0000 7fabab9ed740 1 -- 172.21.15.129:0/157955930 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab900325c0 con 0x5619f18dc390 2024-05-20T18:04:37.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.190+0000 7fab85ffb640 1 -- 172.21.15.129:0/157955930 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fab90004840 con 0x5619f18dc390 2024-05-20T18:04:37.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.191+0000 7fab85ffb640 1 -- 172.21.15.129:0/157955930 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fab90004a40 con 0x5619f18dc390 2024-05-20T18:04:37.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.193+0000 7fabab9ed740 10 client.26027 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:37.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.194+0000 7fabab9ed740 1 -- 172.21.15.129:0/157955930 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5619f1cff2f0 con 0x5619f18dc390 2024-05-20T18:04:37.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.198+0000 7fab85ffb640 1 -- 172.21.15.129:0/157955930 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab900325c0 con 0x5619f18dc390 2024-05-20T18:04:37.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.198+0000 7fab85ffb640 1 -- 172.21.15.129:0/157955930 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fab9003a460 con 0x5619f18dc390 2024-05-20T18:04:37.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.198+0000 7fabab9ed740 10 client.26027 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:37.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.198+0000 7fabab9ed740 10 client.26027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:37.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.198+0000 7fabab9ed740 20 client.26027 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:37.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.198+0000 7fabab9ed740 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619f18703a0 0x5619f1d01950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:37.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.198+0000 7fabab9ed740 4 client.26027 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:04:37.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.198+0000 7fabab9ed740 1 -- 172.21.15.129:0/157955930 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5619f1be86c0 con 0x5619f18703a0 2024-05-20T18:04:37.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.201+0000 7fab9e59c640 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619f18703a0 0x5619f1d01950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:37.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.208+0000 7fab9e59c640 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619f18703a0 0x5619f1d01950 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6de4facc7088dd95 server_cookie=52457df1d78a98f1 in_seq=0 out_seq=0 2024-05-20T18:04:37.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.211+0000 7fab85ffb640 10 client.26027 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:37.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.214+0000 7fab85ffb640 1 -- 172.21.15.129:0/157955930 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5619f1be86c0 con 0x5619f18703a0 2024-05-20T18:04:37.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.214+0000 7fab85ffb640 10 client.26027 handle_command_reply: tid=0 2024-05-20T18:04:37.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.216+0000 7fabab9ed740 1 client.26027 shutdown 2024-05-20T18:04:37.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.223+0000 7fabab9ed740 1 -- 172.21.15.129:0/157955930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18dc390 msgr2=0x5619f1c88cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:37.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.223+0000 7fabab9ed740 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18dc390 0x5619f1c88cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fab900096c0 tx=0x7fab90032e80 comp rx=0 tx=0).stop 2024-05-20T18:04:37.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.223+0000 7fabab9ed740 1 -- 172.21.15.129:0/157955930 shutdown_connections 2024-05-20T18:04:37.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.223+0000 7fabab9ed740 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5619f18703a0 0x5619f1d01950 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.223+0000 7fabab9ed740 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619f18dc390 0x5619f1c88cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.223+0000 7fabab9ed740 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619f18bb6b0 0x5619f1c88780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.223+0000 7fabab9ed740 1 --2- 172.21.15.129:0/157955930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5619f18b34d0 0x5619f1c8d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.223+0000 7fabab9ed740 1 -- 172.21.15.129:0/157955930 >> 172.21.15.129:0/157955930 conn(0x5619f18d04e0 msgr2=0x5619f1c86870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.225+0000 7fab85ffb640 0 client.26027 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:37.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.228+0000 7fabab9ed740 1 -- 172.21.15.129:0/157955930 shutdown_connections 2024-05-20T18:04:37.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.231+0000 7fabab9ed740 1 -- 172.21.15.129:0/157955930 wait complete. 2024-05-20T18:04:37.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.233+0000 7fabab9ed740 20 client.26027 trim_cache size 0 max 16384 2024-05-20T18:04:37.235 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:04:37.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.240+0000 7faba4b3e640 1 -- 172.21.15.129:0/2168746929 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fab880a2e70 msgr2=0x7fab880a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:37.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.240+0000 7faba4b3e640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fab880a2e70 0x7fab880a5380 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fab90006010 tx=0x7fab900314f0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.240+0000 7faba4b3e640 1 -- 172.21.15.129:0/2168746929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faba0119770 msgr2=0x7faba01a8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:37.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.240+0000 7faba4b3e640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faba0119770 0x7faba01a8990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fab94009fd0 tx=0x7fab9400b870 comp rx=0 tx=0).stop 2024-05-20T18:04:37.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.240+0000 7faba4b3e640 1 -- 172.21.15.129:0/2168746929 shutdown_connections 2024-05-20T18:04:37.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.240+0000 7faba4b3e640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fab880a2e70 0x7fab880a5380 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.240+0000 7faba4b3e640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faba011c0d0 0x7faba01acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.240+0000 7faba4b3e640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faba0119770 0x7faba01a8990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.240+0000 7faba4b3e640 1 --2- 172.21.15.129:0/2168746929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faba0108f70 0x7faba01a8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:37.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.240+0000 7faba4b3e640 1 -- 172.21.15.129:0/2168746929 >> 172.21.15.129:0/2168746929 conn(0x7faba006d5b0 msgr2=0x7faba0119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:37.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.241+0000 7faba4b3e640 1 -- 172.21.15.129:0/2168746929 shutdown_connections 2024-05-20T18:04:37.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:37.241+0000 7faba4b3e640 1 -- 172.21.15.129:0/2168746929 wait complete. 2024-05-20T18:04:37.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:37 smithi204 ceph-mon[37076]: pgmap v186: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 922 MiB/s wr, 2.40k op/s 2024-05-20T18:04:38.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:37 smithi154 ceph-mon[36482]: pgmap v186: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 922 MiB/s wr, 2.40k op/s 2024-05-20T18:04:38.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:37 smithi129 ceph-mon[32029]: pgmap v186: 97 pgs: 97 active+clean; 49 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 922 MiB/s wr, 2.40k op/s 2024-05-20T18:04:38.306 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:04:38.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:04:38.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:04:38.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.815+0000 7f2174dd5640 1 -- 172.21.15.129:0/3858874959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2170108f70 msgr2=0x7f2170109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:38.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.815+0000 7f2174dd5640 1 --2- 172.21.15.129:0/3858874959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2170108f70 0x7f2170109380 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2164009930 tx=0x7f216402a420 comp rx=0 tx=0).stop 2024-05-20T18:04:38.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.816+0000 7f2174dd5640 1 -- 172.21.15.129:0/3858874959 shutdown_connections 2024-05-20T18:04:38.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.816+0000 7f2174dd5640 1 --2- 172.21.15.129:0/3858874959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f217011c0d0 0x7f217011e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.816+0000 7f2174dd5640 1 --2- 172.21.15.129:0/3858874959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2170119770 0x7f217011bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.816+0000 7f2174dd5640 1 --2- 172.21.15.129:0/3858874959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2170108f70 0x7f2170109380 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.816+0000 7f2174dd5640 1 -- 172.21.15.129:0/3858874959 >> 172.21.15.129:0/3858874959 conn(0x7f217006e600 msgr2=0x7f217006ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.816+0000 7f2174dd5640 1 -- 172.21.15.129:0/3858874959 shutdown_connections 2024-05-20T18:04:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.816+0000 7f2174dd5640 1 -- 172.21.15.129:0/3858874959 wait complete. 2024-05-20T18:04:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.817+0000 7f2174dd5640 1 Processor -- start 2024-05-20T18:04:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.818+0000 7f2174dd5640 1 -- start start 2024-05-20T18:04:38.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.818+0000 7f2174dd5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2170108f70 0x7f21701a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:38.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.818+0000 7f2174dd5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2170119770 0x7f21701a8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:38.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.818+0000 7f2174dd5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217011c0d0 0x7f21701acce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:38.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.818+0000 7f2174dd5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2170121aa0 con 0x7f2170108f70 2024-05-20T18:04:38.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.818+0000 7f2174dd5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2170121920 con 0x7f217011c0d0 2024-05-20T18:04:38.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.818+0000 7f2174dd5640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2170121c20 con 0x7f2170119770 2024-05-20T18:04:38.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.822+0000 7f216ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217011c0d0 0x7f21701acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:38.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.822+0000 7f216ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217011c0d0 0x7f21701acce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:48664/0 (socket says 172.21.15.129:48664) 2024-05-20T18:04:38.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.822+0000 7f216ffff640 1 -- 172.21.15.129:0/797706008 learned_addr learned my addr 172.21.15.129:0/797706008 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:38.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.822+0000 7f216effd640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2170119770 0x7f21701a8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:38.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.822+0000 7f216f7fe640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2170108f70 0x7f21701a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:38.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.823+0000 7f216f7fe640 1 -- 172.21.15.129:0/797706008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2170119770 msgr2=0x7f21701a8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:38.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.823+0000 7f216f7fe640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2170119770 0x7f21701a8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.823+0000 7f216f7fe640 1 -- 172.21.15.129:0/797706008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217011c0d0 msgr2=0x7f21701acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:38.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.823+0000 7f216f7fe640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217011c0d0 0x7f21701acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.823+0000 7f216f7fe640 1 -- 172.21.15.129:0/797706008 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21701ad320 con 0x7f2170108f70 2024-05-20T18:04:38.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.823+0000 7f216f7fe640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2170108f70 0x7f21701a84c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f2164009850 tx=0x7f2164039520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:38.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.823+0000 7f216ffff640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217011c0d0 0x7f21701acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:04:38.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.825+0000 7f216cff9640 1 -- 172.21.15.129:0/797706008 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164035070 con 0x7f2170108f70 2024-05-20T18:04:38.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.825+0000 7f2174dd5640 1 -- 172.21.15.129:0/797706008 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21701ad5b0 con 0x7f2170108f70 2024-05-20T18:04:38.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.825+0000 7f2174dd5640 1 -- 172.21.15.129:0/797706008 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21701ada70 con 0x7f2170108f70 2024-05-20T18:04:38.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.825+0000 7f216cff9640 1 -- 172.21.15.129:0/797706008 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2164051030 con 0x7f2170108f70 2024-05-20T18:04:38.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.825+0000 7f216cff9640 1 -- 172.21.15.129:0/797706008 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21640035f0 con 0x7f2170108f70 2024-05-20T18:04:38.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.836+0000 7f217cfb1740 1 Processor -- start 2024-05-20T18:04:38.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.841+0000 7f217cfb1740 1 -- start start 2024-05-20T18:04:38.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.847+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1aa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:38.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.847+0000 7f216cff9640 1 -- 172.21.15.129:0/797706008 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f21640398a0 con 0x7f2170108f70 2024-05-20T18:04:38.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.847+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62ede2550 0x55b62f1aad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:38.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.847+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62edea730 0x55b62f1ab2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:38.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.847+0000 7f217cfb1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b62f10f990 con 0x55b62ee0b410 2024-05-20T18:04:38.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.847+0000 7f217cfb1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b62f0b75b0 con 0x55b62ede2550 2024-05-20T18:04:38.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.847+0000 7f217cfb1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b62f10b950 con 0x55b62edea730 2024-05-20T18:04:38.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.848+0000 7f216f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62ede2550 0x55b62f1aad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:38.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.848+0000 7f216f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62ede2550 0x55b62f1aad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:48672/0 (socket says 172.21.15.129:48672) 2024-05-20T18:04:38.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.848+0000 7f216f7fe640 1 -- 172.21.15.129:0/2214595137 learned_addr learned my addr 172.21.15.129:0/2214595137 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.850+0000 7f216effd640 1 --2- 172.21.15.129:0/2214595137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62edea730 0x55b62f1ab2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.850+0000 7f216cff9640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21580a2f10 0x7f21580a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.850+0000 7f216cff9640 1 -- 172.21.15.129:0/797706008 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f2164101220 con 0x7f2170108f70 2024-05-20T18:04:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.850+0000 7f216effd640 1 -- 172.21.15.129:0/2214595137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62ede2550 msgr2=0x55b62f1aad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.850+0000 7f216effd640 1 --2- 172.21.15.129:0/2214595137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62ede2550 0x55b62f1aad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.850+0000 7f216effd640 1 -- 172.21.15.129:0/2214595137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 msgr2=0x55b62f1aa830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.850+0000 7f216effd640 1 --2- 172.21.15.129:0/2214595137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.850+0000 7f216effd640 1 -- 172.21.15.129:0/2214595137 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b62eeeb7e0 con 0x55b62edea730 2024-05-20T18:04:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.851+0000 7f216effd640 1 --2- 172.21.15.129:0/2214595137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62edea730 0x55b62f1ab2b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f217006b2d0 tx=0x7f2168010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f1e8ff6028632f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.852+0000 7f216ffff640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21580a2f10 0x7f21580a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.852+0000 7f2155ffb640 1 -- 172.21.15.129:0/2214595137 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2168016070 con 0x55b62edea730 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.852+0000 7f2155ffb640 1 -- 172.21.15.129:0/2214595137 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2168012930 con 0x55b62edea730 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.852+0000 7f2155ffb640 1 -- 172.21.15.129:0/2214595137 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2168011ae0 con 0x55b62edea730 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.854+0000 7f217cfb1740 1 -- 172.21.15.129:0/2214595137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62edea730 msgr2=0x55b62f1ab2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.854+0000 7f217cfb1740 1 --2- 172.21.15.129:0/2214595137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62edea730 0x55b62f1ab2b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f217006b2d0 tx=0x7f2168010800 comp rx=0 tx=0).stop 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.855+0000 7f217cfb1740 1 -- 172.21.15.129:0/2214595137 shutdown_connections 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.855+0000 7f217cfb1740 1 --2- 172.21.15.129:0/2214595137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62ede2550 0x55b62f1aad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.855+0000 7f217cfb1740 1 --2- 172.21.15.129:0/2214595137 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62edea730 0x55b62f1ab2b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.855+0000 7f217cfb1740 1 --2- 172.21.15.129:0/2214595137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1aa830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.855+0000 7f217cfb1740 1 -- 172.21.15.129:0/2214595137 >> 172.21.15.129:0/2214595137 conn(0x55b62edff560 msgr2=0x55b62f1a6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:38.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.855+0000 7f216ffff640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21580a2f10 0x7f21580a5420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2160005db0 tx=0x7f2160005e60 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:38.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.859+0000 7f217cfb1740 1 -- 172.21.15.129:0/2214595137 shutdown_connections 2024-05-20T18:04:38.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.868+0000 7f217cfb1740 1 -- 172.21.15.129:0/2214595137 wait complete. 2024-05-20T18:04:38.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.869+0000 7f217cfb1740 1 -- start start 2024-05-20T18:04:38.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.875+0000 7f217cfb1740 1 Processor -- start 2024-05-20T18:04:38.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.878+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edea730 0x55b62f1b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:38.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.878+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62ee0b410 0x55b62f1b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:38.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.879+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62edca1f0 0x55b62f1b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:38.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.879+0000 7f217cfb1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b62f1b7b80 con 0x55b62edca1f0 2024-05-20T18:04:38.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.879+0000 7f217cfb1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b62f0cab40 con 0x55b62edea730 2024-05-20T18:04:38.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.879+0000 7f217cfb1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b62f1b7d00 con 0x55b62ee0b410 2024-05-20T18:04:38.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.879+0000 7f217cfb1740 10 client.0 _ll_register_callbacks cb 0x55b62f0bf0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:38.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.881+0000 7f216f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edea730 0x55b62f1b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:38.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.881+0000 7f216f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edea730 0x55b62f1b68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:48680/0 (socket says 172.21.15.129:48680) 2024-05-20T18:04:38.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.881+0000 7f216f7fe640 1 -- 172.21.15.129:0/3201483263 learned_addr learned my addr 172.21.15.129:0/3201483263 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:38.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.882+0000 7f216effd640 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62ee0b410 0x55b62f1b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:38.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.883+0000 7f216f7fe640 1 -- 172.21.15.129:0/3201483263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62ee0b410 msgr2=0x55b62f1b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:38.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.883+0000 7f216f7fe640 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62ee0b410 0x55b62f1b6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.883+0000 7f216f7fe640 1 -- 172.21.15.129:0/3201483263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62edca1f0 msgr2=0x55b62f1b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:38.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.883+0000 7f216f7fe640 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62edca1f0 0x55b62f1b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.883+0000 7f216f7fe640 1 -- 172.21.15.129:0/3201483263 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b62ef4bf70 con 0x55b62edea730 2024-05-20T18:04:38.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.887+0000 7f216f7fe640 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edea730 0x55b62f1b68e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2164006010 tx=0x7f21640045a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2790e42494a14bef server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:38.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.893+0000 7f214effd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:04:38.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.893+0000 7f214effd640 1 -- 172.21.15.129:0/3201483263 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216405b070 con 0x55b62edea730 2024-05-20T18:04:38.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.893+0000 7f217cfb1740 1 -- 172.21.15.129:0/3201483263 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2168012930 con 0x55b62edea730 2024-05-20T18:04:38.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.893+0000 7f214effd640 1 -- 172.21.15.129:0/3201483263 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f21640feb80 con 0x55b62edea730 2024-05-20T18:04:38.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.898+0000 7f214effd640 1 -- 172.21.15.129:0/3201483263 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f21640c98e0 con 0x55b62edea730 2024-05-20T18:04:38.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.898+0000 7f217cfb1740 10 client.26039 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:38.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.898+0000 7f217cfb1740 1 -- 172.21.15.129:0/3201483263 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b62ede3170 con 0x55b62edea730 2024-05-20T18:04:38.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.908+0000 7f214effd640 1 -- 172.21.15.129:0/3201483263 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2168012930 con 0x55b62edea730 2024-05-20T18:04:38.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.908+0000 7f214effd640 1 -- 172.21.15.129:0/3201483263 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f216403a040 con 0x55b62edea730 2024-05-20T18:04:38.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.908+0000 7f217cfb1740 10 client.26039 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:38.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.908+0000 7f217cfb1740 10 client.26039 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:38.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.908+0000 7f217cfb1740 20 client.26039 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:38.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.908+0000 7f217cfb1740 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b62ed9f630 0x55b62f1c7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:38.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.909+0000 7f217cfb1740 4 client.26039 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:04:38.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.909+0000 7f217cfb1740 1 -- 172.21.15.129:0/3201483263 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b62f11d0c0 con 0x55b62ed9f630 2024-05-20T18:04:38.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.909+0000 7f216effd640 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b62ed9f630 0x55b62f1c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:38.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.925+0000 7f216effd640 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b62ed9f630 0x55b62f1c7560 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=335cc5add1f9e648 server_cookie=351de0bd24a4676a in_seq=0 out_seq=0 2024-05-20T18:04:38.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.925+0000 7f214effd640 10 client.26039 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:38.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.929+0000 7f214effd640 1 -- 172.21.15.129:0/3201483263 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b62f11d0c0 con 0x55b62ed9f630 2024-05-20T18:04:38.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.929+0000 7f214effd640 10 client.26039 handle_command_reply: tid=0 2024-05-20T18:04:38.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.929+0000 7f217cfb1740 1 client.26039 shutdown 2024-05-20T18:04:38.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.931+0000 7f217cfb1740 1 -- 172.21.15.129:0/3201483263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edea730 msgr2=0x55b62f1b68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:38.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.931+0000 7f217cfb1740 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edea730 0x55b62f1b68e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2164006010 tx=0x7f21640045a0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.931+0000 7f217cfb1740 1 -- 172.21.15.129:0/3201483263 shutdown_connections 2024-05-20T18:04:38.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.931+0000 7f217cfb1740 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b62ed9f630 0x55b62f1c7560 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.931+0000 7f217cfb1740 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62edca1f0 0x55b62f1b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.931+0000 7f217cfb1740 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62ee0b410 0x55b62f1b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.931+0000 7f217cfb1740 1 --2- 172.21.15.129:0/3201483263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edea730 0x55b62f1b68e0 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2164006010 tx=0x7f21640045a0 comp rx=0 tx=0).stop 2024-05-20T18:04:38.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.931+0000 7f217cfb1740 1 -- 172.21.15.129:0/3201483263 >> 172.21.15.129:0/3201483263 conn(0x55b62edff560 msgr2=0x55b62f1a58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:38.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.935+0000 7f214effd640 0 client.26039 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:38.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.937+0000 7f217cfb1740 1 -- 172.21.15.129:0/3201483263 shutdown_connections 2024-05-20T18:04:38.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.937+0000 7f217cfb1740 1 -- 172.21.15.129:0/3201483263 wait complete. 2024-05-20T18:04:38.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:38.937+0000 7f217cfb1740 20 client.26039 trim_cache size 0 max 16384 2024-05-20T18:04:39.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.019+0000 7f217cfb1740 1 Processor -- start 2024-05-20T18:04:39.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.024+0000 7f217cfb1740 1 -- start start 2024-05-20T18:04:39.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.024+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edca1f0 0x55b62f1b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.024+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.025+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62f1cd260 0x55b62f1bd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.025+0000 7f217cfb1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f215c00c8c0 con 0x55b62ee0b410 2024-05-20T18:04:39.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.025+0000 7f217cfb1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f215c00f140 con 0x55b62edca1f0 2024-05-20T18:04:39.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.025+0000 7f217cfb1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f215c00cf30 con 0x55b62f1cd260 2024-05-20T18:04:39.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.025+0000 7f216effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.025+0000 7f216effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57974/0 (socket says 172.21.15.129:57974) 2024-05-20T18:04:39.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.025+0000 7f216effd640 1 -- 172.21.15.129:0/3402199755 learned_addr learned my addr 172.21.15.129:0/3402199755 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:39.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.027+0000 7f216effd640 1 -- 172.21.15.129:0/3402199755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62f1cd260 msgr2=0x55b62f1bd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:39.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.027+0000 7f216effd640 1 --2- 172.21.15.129:0/3402199755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62f1cd260 0x55b62f1bd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.027+0000 7f216effd640 1 -- 172.21.15.129:0/3402199755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edca1f0 msgr2=0x55b62f1b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:39.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.027+0000 7f216effd640 1 --2- 172.21.15.129:0/3402199755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edca1f0 0x55b62f1b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.027+0000 7f216effd640 1 -- 172.21.15.129:0/3402199755 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b62f1b06b0 con 0x55b62ee0b410 2024-05-20T18:04:39.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.034+0000 7f216effd640 1 --2- 172.21.15.129:0/3402199755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1ccd20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2168005fd0 tx=0x7f21680113e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722a821640dcdb5b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:39.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.036+0000 7f21557fa640 1 -- 172.21.15.129:0/3402199755 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2168016040 con 0x55b62ee0b410 2024-05-20T18:04:39.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.036+0000 7f21557fa640 1 -- 172.21.15.129:0/3402199755 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2168004be0 con 0x55b62ee0b410 2024-05-20T18:04:39.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.036+0000 7f21557fa640 1 -- 172.21.15.129:0/3402199755 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2168012850 con 0x55b62ee0b410 2024-05-20T18:04:39.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.037+0000 7f217cfb1740 1 -- 172.21.15.129:0/3402199755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 msgr2=0x55b62f1ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:39.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.038+0000 7f217cfb1740 1 --2- 172.21.15.129:0/3402199755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1ccd20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2168005fd0 tx=0x7f21680113e0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.038+0000 7f217cfb1740 1 -- 172.21.15.129:0/3402199755 shutdown_connections 2024-05-20T18:04:39.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.038+0000 7f217cfb1740 1 --2- 172.21.15.129:0/3402199755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62f1cd260 0x55b62f1bd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.038+0000 7f217cfb1740 1 --2- 172.21.15.129:0/3402199755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1ccd20 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.038+0000 7f217cfb1740 1 --2- 172.21.15.129:0/3402199755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edca1f0 0x55b62f1b36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.038+0000 7f217cfb1740 1 -- 172.21.15.129:0/3402199755 >> 172.21.15.129:0/3402199755 conn(0x55b62edff560 msgr2=0x55b62f1c74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:39.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.039+0000 7f217cfb1740 1 -- 172.21.15.129:0/3402199755 shutdown_connections 2024-05-20T18:04:39.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.039+0000 7f217cfb1740 1 -- 172.21.15.129:0/3402199755 wait complete. 2024-05-20T18:04:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.040+0000 7f217cfb1740 1 -- start start 2024-05-20T18:04:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.040+0000 7f217cfb1740 1 Processor -- start 2024-05-20T18:04:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.041+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edca1f0 0x55b62f1bcf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.044+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1bd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.044+0000 7f217cfb1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62f1cd260 0x55b62f1bb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.044+0000 7f217cfb1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f215c00d600 con 0x55b62ee0b410 2024-05-20T18:04:39.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.044+0000 7f217cfb1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f215c00ea70 con 0x55b62edca1f0 2024-05-20T18:04:39.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.044+0000 7f217cfb1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f215c00dcd0 con 0x55b62f1cd260 2024-05-20T18:04:39.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.044+0000 7f217cfb1740 10 client.0 _ll_register_callbacks cb 0x7f2164005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:39.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.045+0000 7f216f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edca1f0 0x55b62f1bcf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.045+0000 7f216f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edca1f0 0x55b62f1bcf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:48696/0 (socket says 172.21.15.129:48696) 2024-05-20T18:04:39.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.045+0000 7f216f7fe640 1 -- 172.21.15.129:0/1138541039 learned_addr learned my addr 172.21.15.129:0/1138541039 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:39.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.046+0000 7f216ffff640 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62f1cd260 0x55b62f1bb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.046+0000 7f216effd640 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1bd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.046+0000 7f216f7fe640 1 -- 172.21.15.129:0/1138541039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62f1cd260 msgr2=0x55b62f1bb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.046+0000 7f216f7fe640 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62f1cd260 0x55b62f1bb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.046+0000 7f216f7fe640 1 -- 172.21.15.129:0/1138541039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 msgr2=0x55b62f1bd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.046+0000 7f216f7fe640 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1bd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.046+0000 7f216f7fe640 1 -- 172.21.15.129:0/1138541039 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b62f1bba60 con 0x55b62edca1f0 2024-05-20T18:04:39.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.052+0000 7f216f7fe640 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edca1f0 0x55b62f1bcf10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2164101550 tx=0x7f21640314c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8704fc9b14b6abd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:39.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.055+0000 7f214effd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:04:39.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.055+0000 7f214effd640 1 -- 172.21.15.129:0/1138541039 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216405b070 con 0x55b62edca1f0 2024-05-20T18:04:39.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.055+0000 7f214effd640 1 -- 172.21.15.129:0/1138541039 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f2164039d90 con 0x55b62edca1f0 2024-05-20T18:04:39.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.057+0000 7f217cfb1740 1 -- 172.21.15.129:0/1138541039 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2168004be0 con 0x55b62edca1f0 2024-05-20T18:04:39.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.061+0000 7f214effd640 1 -- 172.21.15.129:0/1138541039 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f21640319b0 con 0x55b62edca1f0 2024-05-20T18:04:39.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.062+0000 7f217cfb1740 10 client.26042 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:39.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.062+0000 7f217cfb1740 1 -- 172.21.15.129:0/1138541039 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b62f230300 con 0x55b62edca1f0 2024-05-20T18:04:39.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.068+0000 7f214effd640 1 -- 172.21.15.129:0/1138541039 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2168004be0 con 0x55b62edca1f0 2024-05-20T18:04:39.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.070+0000 7f214effd640 1 -- 172.21.15.129:0/1138541039 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f21640ccb20 con 0x55b62edca1f0 2024-05-20T18:04:39.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.071+0000 7f217cfb1740 10 client.26042 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:39.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.071+0000 7f217cfb1740 10 client.26042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:39.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.071+0000 7f217cfb1740 20 client.26042 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:39.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.071+0000 7f217cfb1740 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b62f230830 0x55b62f232c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.071+0000 7f217cfb1740 4 client.26042 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:04:39.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.071+0000 7f217cfb1740 1 -- 172.21.15.129:0/1138541039 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b62f117410 con 0x55b62f230830 2024-05-20T18:04:39.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.073+0000 7f216effd640 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b62f230830 0x55b62f232c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.075+0000 7f216effd640 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b62f230830 0x55b62f232c50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6431dcbefa613b3b server_cookie=2947ca0d2b25bc66 in_seq=0 out_seq=0 2024-05-20T18:04:39.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.077+0000 7f214effd640 10 client.26042 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:39.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:38 smithi154 ceph-mon[36482]: pgmap v187: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 756 MiB/s wr, 1.84k op/s 2024-05-20T18:04:39.165 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:38 smithi129 ceph-mon[32029]: pgmap v187: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 756 MiB/s wr, 1.84k op/s 2024-05-20T18:04:39.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:38 smithi204 ceph-mon[37076]: pgmap v187: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 756 MiB/s wr, 1.84k op/s 2024-05-20T18:04:39.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.280+0000 7f214effd640 1 -- 172.21.15.129:0/1138541039 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b62f117410 con 0x55b62f230830 2024-05-20T18:04:39.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.280+0000 7f214effd640 10 client.26042 handle_command_reply: tid=0 2024-05-20T18:04:39.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.280+0000 7f217cfb1740 1 client.26042 shutdown 2024-05-20T18:04:39.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.281+0000 7f217cfb1740 1 -- 172.21.15.129:0/1138541039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edca1f0 msgr2=0x55b62f1bcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:39.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.281+0000 7f217cfb1740 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edca1f0 0x55b62f1bcf10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2164101550 tx=0x7f21640314c0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.281+0000 7f217cfb1740 1 -- 172.21.15.129:0/1138541039 shutdown_connections 2024-05-20T18:04:39.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.281+0000 7f217cfb1740 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b62f230830 0x55b62f232c50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.281+0000 7f217cfb1740 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b62f1cd260 0x55b62f1bb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.281+0000 7f217cfb1740 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b62ee0b410 0x55b62f1bd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.281+0000 7f217cfb1740 1 --2- 172.21.15.129:0/1138541039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b62edca1f0 0x55b62f1bcf10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.282+0000 7f217cfb1740 1 -- 172.21.15.129:0/1138541039 >> 172.21.15.129:0/1138541039 conn(0x55b62edff560 msgr2=0x55b62f1c74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.282+0000 7f214effd640 0 client.26042 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.282+0000 7f217cfb1740 1 -- 172.21.15.129:0/1138541039 shutdown_connections 2024-05-20T18:04:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.282+0000 7f217cfb1740 1 -- 172.21.15.129:0/1138541039 wait complete. 2024-05-20T18:04:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.282+0000 7f217cfb1740 20 client.26042 trim_cache size 0 max 16384 2024-05-20T18:04:39.281 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:04:39.281 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:04:39.281 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "5cbbd97d-3f50-4011-b1c1-5208267f6401", 2024-05-20T18:04:39.281 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:04:39.282 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:04:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.284+0000 7f2174dd5640 1 -- 172.21.15.129:0/797706008 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21580a2f10 msgr2=0x7f21580a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.284+0000 7f2174dd5640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21580a2f10 0x7f21580a5420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2160005db0 tx=0x7f2160005e60 comp rx=0 tx=0).stop 2024-05-20T18:04:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.284+0000 7f2174dd5640 1 -- 172.21.15.129:0/797706008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2170108f70 msgr2=0x7f21701a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.284+0000 7f2174dd5640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2170108f70 0x7f21701a84c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f2164009850 tx=0x7f2164039520 comp rx=0 tx=0).stop 2024-05-20T18:04:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.287+0000 7f2174dd5640 1 -- 172.21.15.129:0/797706008 shutdown_connections 2024-05-20T18:04:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.287+0000 7f2174dd5640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21580a2f10 0x7f21580a5420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.287+0000 7f2174dd5640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217011c0d0 0x7f21701acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.287+0000 7f2174dd5640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2170119770 0x7f21701a8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.287+0000 7f2174dd5640 1 --2- 172.21.15.129:0/797706008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2170108f70 0x7f21701a84c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.287+0000 7f2174dd5640 1 -- 172.21.15.129:0/797706008 >> 172.21.15.129:0/797706008 conn(0x7f217006e600 msgr2=0x7f2170119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:39.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.290+0000 7f2174dd5640 1 -- 172.21.15.129:0/797706008 shutdown_connections 2024-05-20T18:04:39.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.290+0000 7f2174dd5640 1 -- 172.21.15.129:0/797706008 wait complete. 2024-05-20T18:04:39.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:04:39.333 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:04:39.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.857+0000 7f016da51640 1 -- 172.21.15.129:0/3534431745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0168101aa0 msgr2=0x7f0168101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:39.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.857+0000 7f016da51640 1 --2- 172.21.15.129:0/3534431745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0168101aa0 0x7f0168101eb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0158009900 tx=0x7f015802e820 comp rx=0 tx=0).stop 2024-05-20T18:04:39.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.863+0000 7f016da51640 1 -- 172.21.15.129:0/3534431745 shutdown_connections 2024-05-20T18:04:39.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.863+0000 7f016da51640 1 --2- 172.21.15.129:0/3534431745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0168101aa0 0x7f0168101eb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.863+0000 7f016da51640 1 --2- 172.21.15.129:0/3534431745 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f016810c330 0x7f016810e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.863+0000 7f016da51640 1 --2- 172.21.15.129:0/3534431745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01681079d0 0x7f0168101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.863+0000 7f016da51640 1 -- 172.21.15.129:0/3534431745 >> 172.21.15.129:0/3534431745 conn(0x7f01680770b0 msgr2=0x7f01680774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:39.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.872+0000 7f016da51640 1 -- 172.21.15.129:0/3534431745 shutdown_connections 2024-05-20T18:04:39.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.872+0000 7f016da51640 1 -- 172.21.15.129:0/3534431745 wait complete. 2024-05-20T18:04:39.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.873+0000 7f016da51640 1 Processor -- start 2024-05-20T18:04:39.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.874+0000 7f016da51640 1 -- start start 2024-05-20T18:04:39.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.875+0000 7f016da51640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0168101aa0 0x7f0168117a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.875+0000 7f016da51640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01681079d0 0x7f0168110bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.875+0000 7f016da51640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f016810c330 0x7f0168111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.875+0000 7f016da51640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f01681194b0 con 0x7f01681079d0 2024-05-20T18:04:39.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.875+0000 7f016da51640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0168119330 con 0x7f0168101aa0 2024-05-20T18:04:39.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.875+0000 7f016da51640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0168119630 con 0x7f016810c330 2024-05-20T18:04:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.877+0000 7f0166ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0168101aa0 0x7f0168117a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.877+0000 7f0166ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0168101aa0 0x7f0168117a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:57758/0 (socket says 172.21.15.129:57758) 2024-05-20T18:04:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.877+0000 7f0166ffd640 1 -- 172.21.15.129:0/2031235801 learned_addr learned my addr 172.21.15.129:0/2031235801 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.878+0000 7f01677fe640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f016810c330 0x7f0168111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.878+0000 7f0166ffd640 1 -- 172.21.15.129:0/2031235801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f016810c330 msgr2=0x7f0168111150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.878+0000 7f0166ffd640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f016810c330 0x7f0168111150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.878+0000 7f0166ffd640 1 -- 172.21.15.129:0/2031235801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01681079d0 msgr2=0x7f0168110bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.878+0000 7f0166ffd640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01681079d0 0x7f0168110bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.878+0000 7f0166ffd640 1 -- 172.21.15.129:0/2031235801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0168111970 con 0x7f0168101aa0 2024-05-20T18:04:39.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.889+0000 7f0166ffd640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0168101aa0 0x7f0168117a90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f015c00b450 tx=0x7f015c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.890+0000 7f016ca4f640 1 -- 172.21.15.129:0/2031235801 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c015020 con 0x7f0168101aa0 2024-05-20T18:04:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.890+0000 7f016ca4f640 1 -- 172.21.15.129:0/2031235801 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015c011d70 con 0x7f0168101aa0 2024-05-20T18:04:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.890+0000 7f016ca4f640 1 -- 172.21.15.129:0/2031235801 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c00f340 con 0x7f0168101aa0 2024-05-20T18:04:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.892+0000 7f016da51640 1 -- 172.21.15.129:0/2031235801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0168115670 con 0x7f0168101aa0 2024-05-20T18:04:39.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.893+0000 7f016da51640 1 -- 172.21.15.129:0/2031235801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0168115b40 con 0x7f0168101aa0 2024-05-20T18:04:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.905+0000 7f01749cf740 1 Processor -- start 2024-05-20T18:04:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.905+0000 7f01749cf740 1 -- start start 2024-05-20T18:04:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.905+0000 7f01749cf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c58390 0x55ebb4ff6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.906+0000 7f01749cf740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c2f4d0 0x55ebb4ff7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.906+0000 7f01749cf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c376b0 0x55ebb4ff77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.906+0000 7f01749cf740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ebb4f5cd00 con 0x55ebb4c2f4d0 2024-05-20T18:04:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.906+0000 7f01749cf740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ebb4f044c0 con 0x55ebb4c376b0 2024-05-20T18:04:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.906+0000 7f01749cf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ebb4f58cc0 con 0x55ebb4c58390 2024-05-20T18:04:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.906+0000 7f01667fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c376b0 0x55ebb4ff77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.906+0000 7f01667fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c376b0 0x55ebb4ff77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:57768/0 (socket says 172.21.15.129:57768) 2024-05-20T18:04:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.906+0000 7f01667fc640 1 -- 172.21.15.129:0/2308304897 learned_addr learned my addr 172.21.15.129:0/2308304897 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.907+0000 7f01677fe640 1 --2- 172.21.15.129:0/2308304897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c58390 0x55ebb4ff6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.910+0000 7f01667fc640 1 -- 172.21.15.129:0/2308304897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c58390 msgr2=0x55ebb4ff6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.910+0000 7f01667fc640 1 --2- 172.21.15.129:0/2308304897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c58390 0x55ebb4ff6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.910+0000 7f01667fc640 1 -- 172.21.15.129:0/2308304897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c2f4d0 msgr2=0x55ebb4ff7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.910+0000 7f01667fc640 1 --2- 172.21.15.129:0/2308304897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c2f4d0 0x55ebb4ff7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.910+0000 7f01667fc640 1 -- 172.21.15.129:0/2308304897 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebb4d3a7d0 con 0x55ebb4c376b0 2024-05-20T18:04:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.910+0000 7f016ca4f640 1 -- 172.21.15.129:0/2031235801 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f015c018020 con 0x7f0168101aa0 2024-05-20T18:04:39.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.912+0000 7f016ca4f640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f01500a2dc0 0x7f01500a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.912+0000 7f016ca4f640 1 -- 172.21.15.129:0/2031235801 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f015c0d0e30 con 0x7f0168101aa0 2024-05-20T18:04:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.913+0000 7f01667fc640 1 --2- 172.21.15.129:0/2308304897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c376b0 0x55ebb4ff77b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f015802ee10 tx=0x7f01580364b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e54a8c0b6d3a1b28 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.914+0000 7f01677fe640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f01500a2dc0 0x7f01500a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.918+0000 7f014dffb640 1 -- 172.21.15.129:0/2308304897 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158044070 con 0x55ebb4c376b0 2024-05-20T18:04:39.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.918+0000 7f014dffb640 1 -- 172.21.15.129:0/2308304897 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0158036c50 con 0x55ebb4c376b0 2024-05-20T18:04:39.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.918+0000 7f014dffb640 1 -- 172.21.15.129:0/2308304897 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01580102b0 con 0x55ebb4c376b0 2024-05-20T18:04:39.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.920+0000 7f01749cf740 1 -- 172.21.15.129:0/2308304897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c376b0 msgr2=0x55ebb4ff77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:39.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.920+0000 7f01749cf740 1 --2- 172.21.15.129:0/2308304897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c376b0 0x55ebb4ff77b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f015802ee10 tx=0x7f01580364b0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.921+0000 7f01749cf740 1 -- 172.21.15.129:0/2308304897 shutdown_connections 2024-05-20T18:04:39.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.921+0000 7f01749cf740 1 --2- 172.21.15.129:0/2308304897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c376b0 0x55ebb4ff77b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.921+0000 7f01749cf740 1 --2- 172.21.15.129:0/2308304897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c2f4d0 0x55ebb4ff7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.921+0000 7f01749cf740 1 --2- 172.21.15.129:0/2308304897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c58390 0x55ebb4ff6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.921+0000 7f01749cf740 1 -- 172.21.15.129:0/2308304897 >> 172.21.15.129:0/2308304897 conn(0x55ebb4c4c4e0 msgr2=0x55ebb4ff34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.925+0000 7f01749cf740 1 -- 172.21.15.129:0/2308304897 shutdown_connections 2024-05-20T18:04:39.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.925+0000 7f01749cf740 1 -- 172.21.15.129:0/2308304897 wait complete. 2024-05-20T18:04:39.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.926+0000 7f01749cf740 1 -- start start 2024-05-20T18:04:39.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.927+0000 7f01749cf740 1 Processor -- start 2024-05-20T18:04:39.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.929+0000 7f01749cf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 0x55ebb5002e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.929+0000 7f01749cf740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c376b0 0x55ebb5003370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.929+0000 7f01749cf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb50012b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.930+0000 7f01749cf740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ebb5004020 con 0x55ebb4c376b0 2024-05-20T18:04:39.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.930+0000 7f01749cf740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ebb4f17b90 con 0x55ebb4c58390 2024-05-20T18:04:39.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.930+0000 7f01749cf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ebb50041a0 con 0x55ebb4c2f4d0 2024-05-20T18:04:39.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.930+0000 7f0166ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 0x55ebb5002e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.930+0000 7f0166ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 0x55ebb5002e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:56926/0 (socket says 172.21.15.129:56926) 2024-05-20T18:04:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.930+0000 7f0166ffd640 1 -- 172.21.15.129:0/4055364077 learned_addr learned my addr 172.21.15.129:0/4055364077 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.930+0000 7f01677fe640 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb50012b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.931+0000 7f0166ffd640 1 -- 172.21.15.129:0/4055364077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 msgr2=0x55ebb50012b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:39.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.931+0000 7f0166ffd640 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb50012b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.931+0000 7f0166ffd640 1 -- 172.21.15.129:0/4055364077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c376b0 msgr2=0x55ebb5003370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:39.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.931+0000 7f0166ffd640 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c376b0 0x55ebb5003370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.931+0000 7f0166ffd640 1 -- 172.21.15.129:0/4055364077 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebb4c703f0 con 0x55ebb4c2f4d0 2024-05-20T18:04:39.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.933+0000 7f01749cf740 10 client.0 _ll_register_callbacks cb 0x55ebb4f53da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:39.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.934+0000 7f0166ffd640 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 0x55ebb5002e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f015c00be10 tx=0x7f015c009280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73b009308598386e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:39.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.934+0000 7f01749cf740 1 -- 172.21.15.129:0/4055364077 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0158036c50 con 0x55ebb4c2f4d0 2024-05-20T18:04:39.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.936+0000 7f01677fe640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f01500a2dc0 0x7f01500a52d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x55ebb4ff3ea0 tx=0x7f016000f040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:39.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.938+0000 7f0146ffd640 10 client.26089 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:39.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.938+0000 7f0146ffd640 1 -- 172.21.15.129:0/4055364077 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c0a1800 con 0x55ebb4c2f4d0 2024-05-20T18:04:39.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.938+0000 7f0146ffd640 1 -- 172.21.15.129:0/4055364077 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f015c099ac0 con 0x55ebb4c2f4d0 2024-05-20T18:04:39.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.938+0000 7f0146ffd640 1 -- 172.21.15.129:0/4055364077 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f015c0a60c0 con 0x55ebb4c2f4d0 2024-05-20T18:04:39.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.938+0000 7f01749cf740 10 client.26089 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:39.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.939+0000 7f01749cf740 1 -- 172.21.15.129:0/4055364077 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({fsmap=24}) -- 0x55ebb4c300f0 con 0x55ebb4c2f4d0 2024-05-20T18:04:39.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.940+0000 7f0146ffd640 1 -- 172.21.15.129:0/4055364077 <== mon.1 v2:172.21.15.204:3300/0 4 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f015c022060 con 0x55ebb4c2f4d0 2024-05-20T18:04:39.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.941+0000 7f01749cf740 10 client.26089 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:39.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.941+0000 7f01749cf740 10 client.26089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:39.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.941+0000 7f01749cf740 20 client.26089 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:39.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.942+0000 7f01749cf740 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ebb4c16f60 0x55ebb5013790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:39.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.942+0000 7f01749cf740 4 client.26089 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:04:39.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.942+0000 7f01749cf740 1 -- 172.21.15.129:0/4055364077 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0158036c50 con 0x55ebb4c16f60 2024-05-20T18:04:39.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.945+0000 7f01667fc640 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ebb4c16f60 0x55ebb5013790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:39.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.952+0000 7f01667fc640 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ebb4c16f60 0x55ebb5013790 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fb415ab5125683f server_cookie=c8cec1c3d8ee6b7a in_seq=0 out_seq=0 2024-05-20T18:04:39.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.952+0000 7f0146ffd640 10 client.26089 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:39.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.955+0000 7f0146ffd640 1 -- 172.21.15.129:0/4055364077 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f0158036c50 con 0x55ebb4c16f60 2024-05-20T18:04:39.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.955+0000 7f0146ffd640 10 client.26089 handle_command_reply: tid=0 2024-05-20T18:04:39.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.956+0000 7f01749cf740 1 client.26089 shutdown 2024-05-20T18:04:39.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.956+0000 7f01749cf740 1 -- 172.21.15.129:0/4055364077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 msgr2=0x55ebb5002e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:39.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.956+0000 7f01749cf740 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 0x55ebb5002e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f015c00be10 tx=0x7f015c009280 comp rx=0 tx=0).stop 2024-05-20T18:04:39.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.956+0000 7f01749cf740 1 -- 172.21.15.129:0/4055364077 shutdown_connections 2024-05-20T18:04:39.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.956+0000 7f01749cf740 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ebb4c16f60 0x55ebb5013790 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.957+0000 7f01749cf740 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb50012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.957+0000 7f01749cf740 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c376b0 0x55ebb5003370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.957+0000 7f01749cf740 1 --2- 172.21.15.129:0/4055364077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 0x55ebb5002e30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:39.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.957+0000 7f01749cf740 1 -- 172.21.15.129:0/4055364077 >> 172.21.15.129:0/4055364077 conn(0x55ebb4c4c4e0 msgr2=0x55ebb4ff8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:39.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.957+0000 7f0146ffd640 0 client.26089 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:39.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.957+0000 7f01749cf740 1 -- 172.21.15.129:0/4055364077 shutdown_connections 2024-05-20T18:04:39.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.957+0000 7f01749cf740 1 -- 172.21.15.129:0/4055364077 wait complete. 2024-05-20T18:04:39.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:39.958+0000 7f01749cf740 20 client.26089 trim_cache size 0 max 16384 2024-05-20T18:04:40.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.043+0000 7f01749cf740 1 Processor -- start 2024-05-20T18:04:40.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.044+0000 7f01749cf740 1 -- start start 2024-05-20T18:04:40.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.044+0000 7f01749cf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 0x55ebb50001f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:40.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.045+0000 7f01749cf740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c376b0 0x55ebb50030e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:40.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.045+0000 7f01749cf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb5018f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:40.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.046+0000 7f0166ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 0x55ebb50001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:40.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.046+0000 7f0166ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 0x55ebb50001f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:56932/0 (socket says 172.21.15.129:56932) 2024-05-20T18:04:40.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.046+0000 7f01667fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c376b0 0x55ebb50030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:40.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.046+0000 7f01667fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c376b0 0x55ebb50030e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40058/0 (socket says 172.21.15.129:40058) 2024-05-20T18:04:40.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.046+0000 7f0166ffd640 1 -- 172.21.15.129:0/589421639 learned_addr learned my addr 172.21.15.129:0/589421639 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:40.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.046+0000 7f01749cf740 1 -- 172.21.15.129:0/589421639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0168071410 con 0x55ebb4c376b0 2024-05-20T18:04:40.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.046+0000 7f01749cf740 1 -- 172.21.15.129:0/589421639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f01681c69d0 con 0x55ebb4c58390 2024-05-20T18:04:40.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.048+0000 7f01677fe640 1 --2- 172.21.15.129:0/589421639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb5018f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:40.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.048+0000 7f01749cf740 1 -- 172.21.15.129:0/589421639 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0168071a80 con 0x55ebb4c2f4d0 2024-05-20T18:04:40.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.049+0000 7f01667fc640 1 -- 172.21.15.129:0/589421639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 msgr2=0x55ebb50001f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:40.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.049+0000 7f01667fc640 1 --2- 172.21.15.129:0/589421639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 0x55ebb50001f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.049+0000 7f01667fc640 1 -- 172.21.15.129:0/589421639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 msgr2=0x55ebb5018f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:40.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.049+0000 7f01667fc640 1 --2- 172.21.15.129:0/589421639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb5018f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.049+0000 7f01667fc640 1 -- 172.21.15.129:0/589421639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebb4ffbc60 con 0x55ebb4c376b0 2024-05-20T18:04:40.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.051+0000 7f01667fc640 1 --2- 172.21.15.129:0/589421639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c376b0 0x55ebb50030e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f01580096c0 tx=0x7f0158036410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d245db7089dcdd server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.068+0000 7f014d7fa640 1 -- 172.21.15.129:0/589421639 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158050070 con 0x55ebb4c376b0 2024-05-20T18:04:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.068+0000 7f014d7fa640 1 -- 172.21.15.129:0/589421639 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0158036690 con 0x55ebb4c376b0 2024-05-20T18:04:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.070+0000 7f01749cf740 1 -- 172.21.15.129:0/589421639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c376b0 msgr2=0x55ebb50030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.070+0000 7f01749cf740 1 --2- 172.21.15.129:0/589421639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c376b0 0x55ebb50030e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f01580096c0 tx=0x7f0158036410 comp rx=0 tx=0).stop 2024-05-20T18:04:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.071+0000 7f01749cf740 1 -- 172.21.15.129:0/589421639 shutdown_connections 2024-05-20T18:04:40.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.071+0000 7f01749cf740 1 --2- 172.21.15.129:0/589421639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb5018f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.071+0000 7f01749cf740 1 --2- 172.21.15.129:0/589421639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c376b0 0x55ebb50030e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.071+0000 7f01749cf740 1 --2- 172.21.15.129:0/589421639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4c2f4d0 0x55ebb50001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.071+0000 7f01749cf740 1 -- 172.21.15.129:0/589421639 >> 172.21.15.129:0/589421639 conn(0x55ebb4c4c4e0 msgr2=0x55ebb50029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:40.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.071+0000 7f01749cf740 1 -- 172.21.15.129:0/589421639 shutdown_connections 2024-05-20T18:04:40.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.071+0000 7f01749cf740 1 -- 172.21.15.129:0/589421639 wait complete. 2024-05-20T18:04:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.072+0000 7f01749cf740 1 -- start start 2024-05-20T18:04:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.072+0000 7f01749cf740 1 Processor -- start 2024-05-20T18:04:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.074+0000 7f01749cf740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c2f4d0 0x55ebb5006630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.074+0000 7f01749cf740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb5006b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.074+0000 7f01749cf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4bec3a0 0x55ebb5079b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:40.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.075+0000 7f01749cf740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0168072150 con 0x55ebb4c2f4d0 2024-05-20T18:04:40.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.075+0000 7f01667fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb5006b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:40.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.075+0000 7f01667fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb5006b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:57792/0 (socket says 172.21.15.129:57792) 2024-05-20T18:04:40.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.075+0000 7f01667fc640 1 -- 172.21.15.129:0/383333559 learned_addr learned my addr 172.21.15.129:0/383333559 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:40.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.075+0000 7f01749cf740 1 -- 172.21.15.129:0/383333559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f01680735c0 con 0x55ebb4c58390 2024-05-20T18:04:40.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.075+0000 7f01677fe640 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4bec3a0 0x55ebb5079b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:40.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.075+0000 7f01749cf740 1 -- 172.21.15.129:0/383333559 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0168072820 con 0x55ebb4bec3a0 2024-05-20T18:04:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.076+0000 7f01749cf740 10 client.0 _ll_register_callbacks cb 0x7f015c0103e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.077+0000 7f01677fe640 1 -- 172.21.15.129:0/383333559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 msgr2=0x55ebb5006b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.077+0000 7f01677fe640 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb5006b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.077+0000 7f01677fe640 1 -- 172.21.15.129:0/383333559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c2f4d0 msgr2=0x55ebb5006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:40.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.077+0000 7f01677fe640 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c2f4d0 0x55ebb5006630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.077+0000 7f01677fe640 1 -- 172.21.15.129:0/383333559 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebb507a040 con 0x55ebb4bec3a0 2024-05-20T18:04:40.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.081+0000 7f01677fe640 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4bec3a0 0x55ebb5079b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f016001d000 tx=0x7f0160004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f488d691355e4787 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:40.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.081+0000 7f0146ffd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:40.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.081+0000 7f0146ffd640 1 -- 172.21.15.129:0/383333559 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160029030 con 0x55ebb4bec3a0 2024-05-20T18:04:40.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.082+0000 7f01749cf740 1 -- 172.21.15.129:0/383333559 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ebb4f6a430 con 0x55ebb4bec3a0 2024-05-20T18:04:40.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.082+0000 7f0146ffd640 1 -- 172.21.15.129:0/383333559 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f0160004b20 con 0x55ebb4bec3a0 2024-05-20T18:04:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.082+0000 7f0146ffd640 1 -- 172.21.15.129:0/383333559 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0160033050 con 0x55ebb4bec3a0 2024-05-20T18:04:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.083+0000 7f01749cf740 10 client.26095 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.083+0000 7f01749cf740 1 -- 172.21.15.129:0/383333559 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ebb507b270 con 0x55ebb4bec3a0 2024-05-20T18:04:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.084+0000 7f0146ffd640 1 -- 172.21.15.129:0/383333559 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ebb4f6a430 con 0x55ebb4bec3a0 2024-05-20T18:04:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.086+0000 7f0146ffd640 1 -- 172.21.15.129:0/383333559 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f0160024040 con 0x55ebb4bec3a0 2024-05-20T18:04:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.086+0000 7f01749cf740 10 client.26095 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.086+0000 7f01749cf740 10 client.26095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.086+0000 7f01749cf740 20 client.26095 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.087+0000 7f01749cf740 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ebb507b990 0x55ebb507ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.087+0000 7f01749cf740 4 client.26095 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:04:40.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.087+0000 7f01749cf740 1 -- 172.21.15.129:0/383333559 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ebb4f64780 con 0x55ebb507b990 2024-05-20T18:04:40.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.092+0000 7f0166ffd640 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ebb507b990 0x55ebb507ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:40.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.104+0000 7f0166ffd640 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ebb507b990 0x55ebb507ddb0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11e5765397e142a5 server_cookie=2f83b10e1fac0c15 in_seq=0 out_seq=0 2024-05-20T18:04:40.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.104+0000 7f0146ffd640 10 client.26095 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.109+0000 7f0146ffd640 1 -- 172.21.15.129:0/383333559 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55ebb4f64780 con 0x55ebb507b990 2024-05-20T18:04:40.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.110+0000 7f0146ffd640 10 client.26095 handle_command_reply: tid=0 2024-05-20T18:04:40.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.113+0000 7f01749cf740 1 client.26095 shutdown 2024-05-20T18:04:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.117+0000 7f01749cf740 1 -- 172.21.15.129:0/383333559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4bec3a0 msgr2=0x55ebb5079b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.117+0000 7f01749cf740 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4bec3a0 0x55ebb5079b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f016001d000 tx=0x7f0160004ab0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.117+0000 7f01749cf740 1 -- 172.21.15.129:0/383333559 shutdown_connections 2024-05-20T18:04:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.117+0000 7f01749cf740 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ebb507b990 0x55ebb507ddb0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.117+0000 7f01749cf740 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebb4c58390 0x55ebb5006b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.117+0000 7f01749cf740 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ebb4bec3a0 0x55ebb5079b00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.117+0000 7f01749cf740 1 --2- 172.21.15.129:0/383333559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb4c2f4d0 0x55ebb5006630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.117+0000 7f01749cf740 1 -- 172.21.15.129:0/383333559 >> 172.21.15.129:0/383333559 conn(0x55ebb4c4c4e0 msgr2=0x55ebb50007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:40.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.118+0000 7f0146ffd640 0 client.26095 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:40.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.121+0000 7f01749cf740 1 -- 172.21.15.129:0/383333559 shutdown_connections 2024-05-20T18:04:40.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.122+0000 7f01749cf740 1 -- 172.21.15.129:0/383333559 wait complete. 2024-05-20T18:04:40.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.123+0000 7f01749cf740 20 client.26095 trim_cache size 0 max 16384 2024-05-20T18:04:40.124 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:04:40.125 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "scrub active (2 inodes in the stack)", 2024-05-20T18:04:40.125 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:04:40.125 INFO:teuthology.orchestra.run.smithi129.stdout: "5cbbd97d-3f50-4011-b1c1-5208267f6401": { 2024-05-20T18:04:40.125 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:04:40.125 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "5cbbd97d-3f50-4011-b1c1-5208267f6401", 2024-05-20T18:04:40.125 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:04:40.125 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:04:40.125 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:04:40.125 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:04:40.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.137+0000 7f016da51640 1 -- 172.21.15.129:0/2031235801 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f01500a2dc0 msgr2=0x7f01500a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:40.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.137+0000 7f016da51640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f01500a2dc0 0x7f01500a52d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x55ebb4ff3ea0 tx=0x7f016000f040 comp rx=0 tx=0).stop 2024-05-20T18:04:40.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.137+0000 7f016da51640 1 -- 172.21.15.129:0/2031235801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0168101aa0 msgr2=0x7f0168117a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:40.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.137+0000 7f016da51640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0168101aa0 0x7f0168117a90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f015c00b450 tx=0x7f015c00b910 comp rx=0 tx=0).stop 2024-05-20T18:04:40.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.137+0000 7f016da51640 1 -- 172.21.15.129:0/2031235801 shutdown_connections 2024-05-20T18:04:40.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.137+0000 7f016da51640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f01500a2dc0 0x7f01500a52d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.137+0000 7f016da51640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f016810c330 0x7f0168111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.137+0000 7f016da51640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01681079d0 0x7f0168110bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.137+0000 7f016da51640 1 --2- 172.21.15.129:0/2031235801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0168101aa0 0x7f0168117a90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:40.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.137+0000 7f016da51640 1 -- 172.21.15.129:0/2031235801 >> 172.21.15.129:0/2031235801 conn(0x7f01680770b0 msgr2=0x7f0168103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:40.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.138+0000 7f016da51640 1 -- 172.21.15.129:0/2031235801 shutdown_connections 2024-05-20T18:04:40.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:04:40.138+0000 7f016da51640 1 -- 172.21.15.129:0/2031235801 wait complete. 2024-05-20T18:04:40.192 INFO:tasks.cephfs.filesystem:scrub status for tag:5cbbd97d-3f50-4011-b1c1-5208267f6401 - {'path': '/', 'tag': '5cbbd97d-3f50-4011-b1c1-5208267f6401', 'options': 'recursive,force'} 2024-05-20T18:04:41.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:41 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:04:41.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:41 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:41.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:41 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:04:41.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:41 smithi154 ceph-mon[36482]: pgmap v188: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 963 MiB/s wr, 2.23k op/s 2024-05-20T18:04:41.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:41 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:04:41.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:41 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:41.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:41 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:04:41.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:41 smithi129 ceph-mon[32029]: pgmap v188: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 963 MiB/s wr, 2.23k op/s 2024-05-20T18:04:41.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:41 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:04:41.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:41 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:41.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:41 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:04:41.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:41 smithi204 ceph-mon[37076]: pgmap v188: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 963 MiB/s wr, 2.23k op/s 2024-05-20T18:04:43.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:43 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:43.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:43 smithi129 ceph-mon[32029]: pgmap v189: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 726 MiB/s wr, 1.63k op/s 2024-05-20T18:04:43.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:43 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:43.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:43 smithi204 ceph-mon[37076]: pgmap v189: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 726 MiB/s wr, 1.63k op/s 2024-05-20T18:04:43.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:43 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:04:43.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:43 smithi154 ceph-mon[36482]: pgmap v189: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 726 MiB/s wr, 1.63k op/s 2024-05-20T18:04:45.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:45 smithi154 ceph-mon[36482]: pgmap v190: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 742 MiB/s wr, 1.69k op/s 2024-05-20T18:04:45.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:45 smithi129 ceph-mon[32029]: pgmap v190: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 742 MiB/s wr, 1.69k op/s 2024-05-20T18:04:45.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:45 smithi204 ceph-mon[37076]: pgmap v190: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 742 MiB/s wr, 1.69k op/s 2024-05-20T18:04:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.874+0000 7f002b08b640 1 -- 172.21.15.129:0/363016323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0024119770 msgr2=0x7f002411bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.874+0000 7f002b08b640 1 --2- 172.21.15.129:0/363016323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0024119770 0x7f002411bb90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0014009900 tx=0x7f001402a4d0 comp rx=0 tx=0).stop 2024-05-20T18:04:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.905+0000 7f002b08b640 1 -- 172.21.15.129:0/363016323 shutdown_connections 2024-05-20T18:04:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.905+0000 7f002b08b640 1 --2- 172.21.15.129:0/363016323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f002411c0d0 0x7f002411e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.905+0000 7f002b08b640 1 --2- 172.21.15.129:0/363016323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0024119770 0x7f002411bb90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.905+0000 7f002b08b640 1 --2- 172.21.15.129:0/363016323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0024109080 0x7f0024109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.905+0000 7f002b08b640 1 -- 172.21.15.129:0/363016323 >> 172.21.15.129:0/363016323 conn(0x7f002406d780 msgr2=0x7f002406dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.918+0000 7f002b08b640 1 -- 172.21.15.129:0/363016323 shutdown_connections 2024-05-20T18:04:45.915 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.918+0000 7f002b08b640 1 -- 172.21.15.129:0/363016323 wait complete. 2024-05-20T18:04:45.918 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.920+0000 7f002b08b640 1 Processor -- start 2024-05-20T18:04:45.919 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.920+0000 7f002b08b640 1 -- start start 2024-05-20T18:04:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.921+0000 7f002b08b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0024109080 0x7f00241b69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.921+0000 7f002b08b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0024119770 0x7f00241b6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.921+0000 7f002b08b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f002411c0d0 0x7f0024113d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.921+0000 7f002b08b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00241122a0 con 0x7f002411c0d0 2024-05-20T18:04:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.921+0000 7f002b08b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0024112120 con 0x7f0024109080 2024-05-20T18:04:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.921+0000 7f002b08b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0024112420 con 0x7f0024119770 2024-05-20T18:04:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.922+0000 7f0023fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0024119770 0x7f00241b6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.922+0000 7f0023fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0024119770 0x7f00241b6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:56954/0 (socket says 172.21.15.129:56954) 2024-05-20T18:04:45.920 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.922+0000 7f0023fff640 1 -- 172.21.15.129:0/368781770 learned_addr learned my addr 172.21.15.129:0/368781770 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.922+0000 7f0028e27640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0024109080 0x7f00241b69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.923+0000 7f0028e27640 1 -- 172.21.15.129:0/368781770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0024119770 msgr2=0x7f00241b6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.923+0000 7f0028e27640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0024119770 0x7f00241b6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.923+0000 7f0028e27640 1 -- 172.21.15.129:0/368781770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f002411c0d0 msgr2=0x7f0024113d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:45.921 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.923+0000 7f0028e27640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f002411c0d0 0x7f0024113d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:45.922 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.923+0000 7f0028e27640 1 -- 172.21.15.129:0/368781770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00241143c0 con 0x7f0024109080 2024-05-20T18:04:45.929 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.928+0000 7f0028e27640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0024109080 0x7f00241b69a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f000c00d6d0 tx=0x7f000c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:45.940 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.942+0000 7f0021ffb640 1 -- 172.21.15.129:0/368781770 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c017070 con 0x7f0024109080 2024-05-20T18:04:45.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.943+0000 7f002b08b640 1 -- 172.21.15.129:0/368781770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00241146b0 con 0x7f0024109080 2024-05-20T18:04:45.943 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.943+0000 7f002b08b640 1 -- 172.21.15.129:0/368781770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0024126980 con 0x7f0024109080 2024-05-20T18:04:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.951+0000 7f0021ffb640 1 -- 172.21.15.129:0/368781770 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f000c009820 con 0x7f0024109080 2024-05-20T18:04:45.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.952+0000 7f0021ffb640 1 -- 172.21.15.129:0/368781770 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c004d00 con 0x7f0024109080 2024-05-20T18:04:45.953 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.955+0000 7f0021ffb640 1 -- 172.21.15.129:0/368781770 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f000c010460 con 0x7f0024109080 2024-05-20T18:04:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.960+0000 7f003200d740 1 Processor -- start 2024-05-20T18:04:45.958 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.960+0000 7f003200d740 1 -- start start 2024-05-20T18:04:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.962+0000 7f003200d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678e7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:45.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.963+0000 7f003200d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56086751f2f0 0x5608678e75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.963+0000 7f003200d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5608675274d0 0x5608678e7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.963+0000 7f003200d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56086784c340 con 0x56086751f2f0 2024-05-20T18:04:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.963+0000 7f003200d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5608677f4290 con 0x5608675274d0 2024-05-20T18:04:45.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.963+0000 7f0029628640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:45.963 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.963+0000 7f0029628640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678e7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:56964/0 (socket says 172.21.15.129:56964) 2024-05-20T18:04:45.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.963+0000 7f0029628640 1 -- 172.21.15.129:0/109751742 learned_addr learned my addr 172.21.15.129:0/109751742 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.964+0000 7f0021ffb640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f00100a2f00 0x7f00100a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.965+0000 7f0021ffb640 1 -- 172.21.15.129:0/368781770 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f000c0cee40 con 0x7f0024109080 2024-05-20T18:04:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.965+0000 7f003200d740 1 -- 172.21.15.129:0/109751742 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560867848300 con 0x5608675481b0 2024-05-20T18:04:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.966+0000 7f0023fff640 1 --2- 172.21.15.129:0/109751742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5608675274d0 0x5608678e7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.967+0000 7f0028e27640 1 --2- 172.21.15.129:0/109751742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56086751f2f0 0x5608678e75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:45.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.967+0000 7f0028e27640 1 -- 172.21.15.129:0/109751742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 msgr2=0x5608678e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:45.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.967+0000 7f0028e27640 1 --2- 172.21.15.129:0/109751742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678e7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:45.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.967+0000 7f0028e27640 1 -- 172.21.15.129:0/109751742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5608675274d0 msgr2=0x5608678e7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:45.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.967+0000 7f0028e27640 1 --2- 172.21.15.129:0/109751742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5608675274d0 0x5608678e7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:45.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.967+0000 7f0029628640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f00100a2f00 0x7f00100a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:45.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.968+0000 7f0028e27640 1 -- 172.21.15.129:0/109751742 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608678188e0 con 0x56086751f2f0 2024-05-20T18:04:45.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.968+0000 7f0028e27640 1 --2- 172.21.15.129:0/109751742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56086751f2f0 0x5608678e75c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f000c0cc520 tx=0x7f000c004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13fb0ddf3eabb75 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.988+0000 7f001affd640 1 -- 172.21.15.129:0/109751742 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c020040 con 0x56086751f2f0 2024-05-20T18:04:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.988+0000 7f001affd640 1 -- 172.21.15.129:0/109751742 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f000c09b9f0 con 0x56086751f2f0 2024-05-20T18:04:45.992 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:45.988+0000 7f001affd640 1 -- 172.21.15.129:0/109751742 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c0099d0 con 0x56086751f2f0 2024-05-20T18:04:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.001+0000 7f003200d740 1 -- 172.21.15.129:0/109751742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56086751f2f0 msgr2=0x5608678e75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:45.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.001+0000 7f003200d740 1 --2- 172.21.15.129:0/109751742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56086751f2f0 0x5608678e75c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f000c0cc520 tx=0x7f000c004f20 comp rx=0 tx=0).stop 2024-05-20T18:04:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.006+0000 7f003200d740 1 -- 172.21.15.129:0/109751742 shutdown_connections 2024-05-20T18:04:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.007+0000 7f003200d740 1 --2- 172.21.15.129:0/109751742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5608675274d0 0x5608678e7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.007+0000 7f003200d740 1 --2- 172.21.15.129:0/109751742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56086751f2f0 0x5608678e75c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.007+0000 7f003200d740 1 --2- 172.21.15.129:0/109751742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.007+0000 7f003200d740 1 -- 172.21.15.129:0/109751742 >> 172.21.15.129:0/109751742 conn(0x56086753c300 msgr2=0x5608678e3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.011+0000 7f003200d740 1 -- 172.21.15.129:0/109751742 shutdown_connections 2024-05-20T18:04:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.011+0000 7f003200d740 1 -- 172.21.15.129:0/109751742 wait complete. 2024-05-20T18:04:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.012+0000 7f003200d740 1 -- start start 2024-05-20T18:04:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.012+0000 7f003200d740 1 Processor -- start 2024-05-20T18:04:46.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.014+0000 7f003200d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 0x5608678f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:46.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.014+0000 7f003200d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x5608678f3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:46.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.014+0000 7f003200d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678f1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:46.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.014+0000 7f003200d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5608678f4150 con 0x5608675274d0 2024-05-20T18:04:46.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.014+0000 7f003200d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560867807790 con 0x56086751f2f0 2024-05-20T18:04:46.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.014+0000 7f003200d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5608678f42d0 con 0x5608675481b0 2024-05-20T18:04:46.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.015+0000 7f003200d740 10 client.0 _ll_register_callbacks cb 0x560867842a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:46.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.020+0000 7f0028e27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 0x5608678f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.021+0000 7f0028e27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 0x5608678f30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:57840/0 (socket says 172.21.15.129:57840) 2024-05-20T18:04:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.022+0000 7f0023fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x5608678f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.022+0000 7f0023fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x5608678f3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40096/0 (socket says 172.21.15.129:40096) 2024-05-20T18:04:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.022+0000 7f0023fff640 1 -- 172.21.15.129:0/1282798392 learned_addr learned my addr 172.21.15.129:0/1282798392 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.025+0000 7f0029628640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f00100a2f00 0x7f00100a5410 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x5608678e41f0 tx=0x7f001c009500 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.025+0000 7f0029628640 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:46.025 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.027+0000 7f0029628640 1 -- 172.21.15.129:0/1282798392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 msgr2=0x5608678f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:46.025 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.027+0000 7f0029628640 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 0x5608678f30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.025 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.027+0000 7f0029628640 1 -- 172.21.15.129:0/1282798392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 msgr2=0x5608678f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.027+0000 7f0029628640 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x5608678f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.027+0000 7f0029628640 1 -- 172.21.15.129:0/1282798392 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560867560210 con 0x5608675481b0 2024-05-20T18:04:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.028+0000 7f0028e27640 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 0x5608678f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:04:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.029+0000 7f0029628640 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678f1590 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f001c0049a0 tx=0x7f001c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11ea9b5fbb535dc8 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:46.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.034+0000 7f0003fff640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:04:46.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.034+0000 7f0003fff640 1 -- 172.21.15.129:0/1282798392 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c018070 con 0x5608675481b0 2024-05-20T18:04:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.035+0000 7f003200d740 1 -- 172.21.15.129:0/1282798392 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f000c09b9f0 con 0x5608675481b0 2024-05-20T18:04:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.035+0000 7f0003fff640 1 -- 172.21.15.129:0/1282798392 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f001c00fda0 con 0x5608675481b0 2024-05-20T18:04:46.033 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.035+0000 7f0003fff640 1 -- 172.21.15.129:0/1282798392 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f001c027400 con 0x5608675481b0 2024-05-20T18:04:46.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.038+0000 7f003200d740 10 client.26107 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:46.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.038+0000 7f003200d740 1 -- 172.21.15.129:0/1282798392 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56086762c620 con 0x5608675481b0 2024-05-20T18:04:46.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.039+0000 7f0003fff640 1 -- 172.21.15.129:0/1282798392 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f001c027a80 con 0x5608675481b0 2024-05-20T18:04:46.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.039+0000 7f0003fff640 1 -- 172.21.15.129:0/1282798392 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f001c01f970 con 0x5608675481b0 2024-05-20T18:04:46.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.039+0000 7f003200d740 10 client.26107 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:46.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.039+0000 7f003200d740 10 client.26107 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:46.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.039+0000 7f003200d740 20 client.26107 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:46.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.039+0000 7f003200d740 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560867506d80 0x560867903de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:46.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.039+0000 7f003200d740 4 client.26107 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:04:46.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.039+0000 7f003200d740 1 -- 172.21.15.129:0/1282798392 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f000c09b9f0 con 0x560867506d80 2024-05-20T18:04:46.052 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.055+0000 7f0028e27640 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560867506d80 0x560867903de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.077+0000 7f0003fff640 10 client.26107 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.077+0000 7f0028e27640 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560867506d80 0x560867903de0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4975a3906eb58b38 server_cookie=17371cf269becf10 in_seq=0 out_seq=0 2024-05-20T18:04:46.080 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.083+0000 7f0003fff640 1 -- 172.21.15.129:0/1282798392 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f000c09b9f0 con 0x560867506d80 2024-05-20T18:04:46.080 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.083+0000 7f0003fff640 10 client.26107 handle_command_reply: tid=0 2024-05-20T18:04:46.080 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.083+0000 7f003200d740 1 client.26107 shutdown 2024-05-20T18:04:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.084+0000 7f003200d740 1 -- 172.21.15.129:0/1282798392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 msgr2=0x5608678f1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:46.083 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.084+0000 7f003200d740 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678f1590 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f001c0049a0 tx=0x7f001c004a40 comp rx=0 tx=0).stop 2024-05-20T18:04:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.089+0000 7f003200d740 1 -- 172.21.15.129:0/1282798392 shutdown_connections 2024-05-20T18:04:46.087 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.089+0000 7f003200d740 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560867506d80 0x560867903de0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.089+0000 7f003200d740 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678f1590 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.089+0000 7f003200d740 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x5608678f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.089+0000 7f003200d740 1 --2- 172.21.15.129:0/1282798392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 0x5608678f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.089+0000 7f003200d740 1 -- 172.21.15.129:0/1282798392 >> 172.21.15.129:0/1282798392 conn(0x56086753c300 msgr2=0x5608678e7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.089+0000 7f0003fff640 0 client.26107 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.090+0000 7f003200d740 1 -- 172.21.15.129:0/1282798392 shutdown_connections 2024-05-20T18:04:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.090+0000 7f003200d740 1 -- 172.21.15.129:0/1282798392 wait complete. 2024-05-20T18:04:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.090+0000 7f003200d740 20 client.26107 trim_cache size 0 max 16384 2024-05-20T18:04:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.146+0000 7f003200d740 1 Processor -- start 2024-05-20T18:04:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.146+0000 7f003200d740 1 -- start start 2024-05-20T18:04:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.147+0000 7f003200d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56086751f2f0 0x5608679095e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.147+0000 7f003200d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x560867909b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.148+0000 7f003200d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5608675481b0 0x5608678f9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.148+0000 7f003200d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f000400c8c0 con 0x5608675274d0 2024-05-20T18:04:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.148+0000 7f003200d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f000400f140 con 0x5608675481b0 2024-05-20T18:04:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.148+0000 7f003200d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f000400cf30 con 0x56086751f2f0 2024-05-20T18:04:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.148+0000 7f0028e27640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56086751f2f0 0x5608679095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.148+0000 7f0028e27640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56086751f2f0 0x5608679095e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:56980/0 (socket says 172.21.15.129:56980) 2024-05-20T18:04:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.148+0000 7f0028e27640 1 -- 172.21.15.129:0/67891780 learned_addr learned my addr 172.21.15.129:0/67891780 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.148+0000 7f0029628640 1 --2- 172.21.15.129:0/67891780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5608675481b0 0x5608678f9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.150+0000 7f0023fff640 1 --2- 172.21.15.129:0/67891780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x560867909b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.151+0000 7f0029628640 1 -- 172.21.15.129:0/67891780 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56086751f2f0 msgr2=0x5608679095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.151+0000 7f0029628640 1 --2- 172.21.15.129:0/67891780 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56086751f2f0 0x5608679095e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.151+0000 7f0029628640 1 -- 172.21.15.129:0/67891780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 msgr2=0x560867909b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.151+0000 7f0029628640 1 --2- 172.21.15.129:0/67891780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x560867909b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.151+0000 7f0029628640 1 -- 172.21.15.129:0/67891780 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608678e7fa0 con 0x5608675481b0 2024-05-20T18:04:46.149 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.152+0000 7f0029628640 1 --2- 172.21.15.129:0/67891780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5608675481b0 0x5608678f9590 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f001c01ff30 tx=0x7f001c0098e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ab8bdc082822145 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.155+0000 7f001a7fc640 1 -- 172.21.15.129:0/67891780 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c018070 con 0x5608675481b0 2024-05-20T18:04:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.155+0000 7f001a7fc640 1 -- 172.21.15.129:0/67891780 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f001c01f8e0 con 0x5608675481b0 2024-05-20T18:04:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.157+0000 7f003200d740 1 -- 172.21.15.129:0/67891780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5608675481b0 msgr2=0x5608678f9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.157+0000 7f003200d740 1 --2- 172.21.15.129:0/67891780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5608675481b0 0x5608678f9590 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f001c01ff30 tx=0x7f001c0098e0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.163+0000 7f003200d740 1 -- 172.21.15.129:0/67891780 shutdown_connections 2024-05-20T18:04:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.163+0000 7f003200d740 1 --2- 172.21.15.129:0/67891780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5608675481b0 0x5608678f9590 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.163+0000 7f003200d740 1 --2- 172.21.15.129:0/67891780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x560867909b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.163+0000 7f003200d740 1 --2- 172.21.15.129:0/67891780 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56086751f2f0 0x5608679095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.163+0000 7f003200d740 1 -- 172.21.15.129:0/67891780 >> 172.21.15.129:0/67891780 conn(0x56086753c300 msgr2=0x5608678efe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.169+0000 7f003200d740 1 -- 172.21.15.129:0/67891780 shutdown_connections 2024-05-20T18:04:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.174+0000 7f003200d740 1 -- 172.21.15.129:0/67891780 wait complete. 2024-05-20T18:04:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.174+0000 7f003200d740 1 -- start start 2024-05-20T18:04:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.175+0000 7f003200d740 1 Processor -- start 2024-05-20T18:04:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.176+0000 7f003200d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 0x560867963510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.176+0000 7f003200d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x560867963a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.176+0000 7f003200d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678f8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.176+0000 7f003200d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f000400d600 con 0x5608675274d0 2024-05-20T18:04:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.176+0000 7f003200d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f000400ea70 con 0x56086751f2f0 2024-05-20T18:04:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.176+0000 7f003200d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f000400dcd0 con 0x5608675481b0 2024-05-20T18:04:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.176+0000 7f003200d740 10 client.0 _ll_register_callbacks cb 0x7f001c0126b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:04:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.178+0000 7f0029628640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.180+0000 7f0028e27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 0x560867963510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.180+0000 7f0029628640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678f8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:56990/0 (socket says 172.21.15.129:56990) 2024-05-20T18:04:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.180+0000 7f0029628640 1 -- 172.21.15.129:0/317012997 learned_addr learned my addr 172.21.15.129:0/317012997 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:04:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.180+0000 7f0023fff640 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x560867963a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.184+0000 7f0028e27640 1 -- 172.21.15.129:0/317012997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 msgr2=0x5608678f8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.184+0000 7f0028e27640 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678f8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.184+0000 7f0028e27640 1 -- 172.21.15.129:0/317012997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 msgr2=0x560867963a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:04:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.184+0000 7f0028e27640 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x560867963a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.184+0000 7f0028e27640 1 -- 172.21.15.129:0/317012997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608678f9030 con 0x56086751f2f0 2024-05-20T18:04:46.190 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.189+0000 7f0028e27640 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 0x560867963510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f000c0cf170 tx=0x7f000c099530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c3b1b62081ac797 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:04:46.190 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.190+0000 7f0003fff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:04:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.195+0000 7f0003fff640 1 -- 172.21.15.129:0/317012997 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c020040 con 0x56086751f2f0 2024-05-20T18:04:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.195+0000 7f003200d740 1 -- 172.21.15.129:0/317012997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f001c01f8e0 con 0x56086751f2f0 2024-05-20T18:04:46.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.195+0000 7f0003fff640 1 -- 172.21.15.129:0/317012997 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f000c0b4130 con 0x56086751f2f0 2024-05-20T18:04:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.225+0000 7f0003fff640 1 -- 172.21.15.129:0/317012997 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f000c09b9f0 con 0x56086751f2f0 2024-05-20T18:04:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.226+0000 7f003200d740 10 client.26075 fetch_fsmap learned FSMap version 24 2024-05-20T18:04:46.224 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.226+0000 7f003200d740 1 -- 172.21.15.129:0/317012997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56086796bb70 con 0x56086751f2f0 2024-05-20T18:04:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.228+0000 7f0003fff640 1 -- 172.21.15.129:0/317012997 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f001c01f8e0 con 0x56086751f2f0 2024-05-20T18:04:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.228+0000 7f0003fff640 1 -- 172.21.15.129:0/317012997 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f000c09a740 con 0x56086751f2f0 2024-05-20T18:04:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.228+0000 7f003200d740 10 client.26075 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:04:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.228+0000 7f003200d740 10 client.26075 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:04:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.228+0000 7f003200d740 20 client.26075 populate_metadata read hostname 'smithi129' 2024-05-20T18:04:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.228+0000 7f003200d740 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5608674dc1c0 0x56086796dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:04:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.228+0000 7f003200d740 4 client.26075 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:04:46.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.228+0000 7f003200d740 1 -- 172.21.15.129:0/317012997 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560867859a70 con 0x5608674dc1c0 2024-05-20T18:04:46.233 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.234+0000 7f0023fff640 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5608674dc1c0 0x56086796dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:04:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.250+0000 7f0023fff640 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5608674dc1c0 0x56086796dec0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a1aeff65494dd2f server_cookie=6c65ce6f6e12722a in_seq=0 out_seq=0 2024-05-20T18:04:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:46.252+0000 7f0003fff640 10 client.26075 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:46.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:46 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:04:46.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:46 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:04:46.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:46 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:04:47.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:47 smithi154 ceph-mon[36482]: pgmap v191: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 944 MiB/s wr, 2.11k op/s 2024-05-20T18:04:47.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:47 smithi129 ceph-mon[32029]: pgmap v191: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 944 MiB/s wr, 2.11k op/s 2024-05-20T18:04:47.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:47 smithi204 ceph-mon[37076]: pgmap v191: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 944 MiB/s wr, 2.11k op/s 2024-05-20T18:04:49.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:49 smithi154 ceph-mon[36482]: pgmap v192: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 754 MiB/s wr, 1.63k op/s 2024-05-20T18:04:49.672 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:49 smithi129 ceph-mon[32029]: pgmap v192: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 754 MiB/s wr, 1.63k op/s 2024-05-20T18:04:49.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:49 smithi204 ceph-mon[37076]: pgmap v192: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 754 MiB/s wr, 1.63k op/s 2024-05-20T18:04:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.830+0000 7f0003fff640 1 -- 172.21.15.129:0/317012997 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560867859a70 con 0x5608674dc1c0 2024-05-20T18:04:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.830+0000 7f0003fff640 10 client.26075 handle_command_reply: tid=0 2024-05-20T18:04:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.830+0000 7f003200d740 1 client.26075 shutdown 2024-05-20T18:04:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.833+0000 7f003200d740 1 -- 172.21.15.129:0/317012997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 msgr2=0x560867963510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.833+0000 7f003200d740 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 0x560867963510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f000c0cf170 tx=0x7f000c099530 comp rx=0 tx=0).stop 2024-05-20T18:04:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.834+0000 7f003200d740 1 -- 172.21.15.129:0/317012997 shutdown_connections 2024-05-20T18:04:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.834+0000 7f003200d740 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5608674dc1c0 0x56086796dec0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.834+0000 7f003200d740 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5608675481b0 0x5608678f8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.834+0000 7f003200d740 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5608675274d0 0x560867963a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.834+0000 7f003200d740 1 --2- 172.21.15.129:0/317012997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56086751f2f0 0x560867963510 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.834+0000 7f003200d740 1 -- 172.21.15.129:0/317012997 >> 172.21.15.129:0/317012997 conn(0x56086753c300 msgr2=0x5608678efdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.834+0000 7f0003fff640 0 client.26075 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:04:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.835+0000 7f003200d740 1 -- 172.21.15.129:0/317012997 shutdown_connections 2024-05-20T18:04:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.835+0000 7f003200d740 1 -- 172.21.15.129:0/317012997 wait complete. 2024-05-20T18:04:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.835+0000 7f003200d740 20 client.26075 trim_cache size 0 max 16384 2024-05-20T18:04:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:04:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:04:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:04:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:04:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.839+0000 7f002b08b640 1 -- 172.21.15.129:0/368781770 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f00100a2f00 msgr2=0x7f00100a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.839+0000 7f002b08b640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f00100a2f00 0x7f00100a5410 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x5608678e41f0 tx=0x7f001c009500 comp rx=0 tx=0).stop 2024-05-20T18:04:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.839+0000 7f002b08b640 1 -- 172.21.15.129:0/368781770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0024109080 msgr2=0x7f00241b69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:04:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.839+0000 7f002b08b640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0024109080 0x7f00241b69a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f000c00d6d0 tx=0x7f000c00db90 comp rx=0 tx=0).stop 2024-05-20T18:04:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.839+0000 7f002b08b640 1 -- 172.21.15.129:0/368781770 shutdown_connections 2024-05-20T18:04:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.839+0000 7f002b08b640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f00100a2f00 0x7f00100a5410 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.840+0000 7f002b08b640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f002411c0d0 0x7f0024113d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:49.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.840+0000 7f002b08b640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0024119770 0x7f00241b6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.840+0000 7f002b08b640 1 --2- 172.21.15.129:0/368781770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0024109080 0x7f00241b69a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:04:49.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.840+0000 7f002b08b640 1 -- 172.21.15.129:0/368781770 >> 172.21.15.129:0/368781770 conn(0x7f002406d780 msgr2=0x7f0024119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:04:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.849+0000 7f002b08b640 1 -- 172.21.15.129:0/368781770 shutdown_connections 2024-05-20T18:04:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:04:49.849+0000 7f002b08b640 1 -- 172.21.15.129:0/368781770 wait complete. 2024-05-20T18:04:51.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:51 smithi154 ceph-mon[36482]: pgmap v193: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 957 MiB/s wr, 2.06k op/s 2024-05-20T18:04:51.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:51 smithi129 ceph-mon[32029]: pgmap v193: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 957 MiB/s wr, 2.06k op/s 2024-05-20T18:04:51.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:51 smithi204 ceph-mon[37076]: pgmap v193: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 957 MiB/s wr, 2.06k op/s 2024-05-20T18:04:53.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:53 smithi154 ceph-mon[36482]: pgmap v194: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 729 MiB/s wr, 1.55k op/s 2024-05-20T18:04:53.668 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:53 smithi129 ceph-mon[32029]: pgmap v194: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 729 MiB/s wr, 1.55k op/s 2024-05-20T18:04:53.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:53 smithi204 ceph-mon[37076]: pgmap v194: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 729 MiB/s wr, 1.55k op/s 2024-05-20T18:04:55.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:55 smithi154 ceph-mon[36482]: pgmap v195: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 751 MiB/s wr, 1.60k op/s 2024-05-20T18:04:55.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:55 smithi129 ceph-mon[32029]: pgmap v195: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 751 MiB/s wr, 1.60k op/s 2024-05-20T18:04:55.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:55 smithi204 ceph-mon[37076]: pgmap v195: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 751 MiB/s wr, 1.60k op/s 2024-05-20T18:04:57.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:57 smithi154 ceph-mon[36482]: pgmap v196: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 953 MiB/s wr, 1.99k op/s 2024-05-20T18:04:57.668 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:57 smithi129 ceph-mon[32029]: pgmap v196: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 953 MiB/s wr, 1.99k op/s 2024-05-20T18:04:57.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:57 smithi204 ceph-mon[37076]: pgmap v196: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 953 MiB/s wr, 1.99k op/s 2024-05-20T18:04:59.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:04:59 smithi154 ceph-mon[36482]: pgmap v197: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 755 MiB/s wr, 1.58k op/s 2024-05-20T18:04:59.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:04:59 smithi129 ceph-mon[32029]: pgmap v197: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 755 MiB/s wr, 1.58k op/s 2024-05-20T18:04:59.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:04:59 smithi204 ceph-mon[37076]: pgmap v197: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 755 MiB/s wr, 1.58k op/s 2024-05-20T18:05:01.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:01 smithi154 ceph-mon[36482]: pgmap v198: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 965 MiB/s wr, 2.05k op/s 2024-05-20T18:05:01.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:01 smithi129 ceph-mon[32029]: pgmap v198: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 965 MiB/s wr, 2.05k op/s 2024-05-20T18:05:01.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:01 smithi204 ceph-mon[37076]: pgmap v198: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 965 MiB/s wr, 2.05k op/s 2024-05-20T18:05:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.132+0000 7f21d0bd7640 1 -- 172.21.15.129:0/1551373367 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4031dc0 con 0x7f21cc1032a0 2024-05-20T18:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.133+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1551373367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21cc1032a0 msgr2=0x7f21cc10f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.133+0000 7f21d3e3d640 1 --2- 172.21.15.129:0/1551373367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21cc1032a0 0x7f21cc10f670 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f21c4009ca0 tx=0x7f21c402a8e0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.142+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1551373367 shutdown_connections 2024-05-20T18:05:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.142+0000 7f21d3e3d640 1 --2- 172.21.15.129:0/1551373367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21cc1032a0 0x7f21cc10f670 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.142+0000 7f21d3e3d640 1 --2- 172.21.15.129:0/1551373367 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21cc108ff0 0x7f21cc102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.142+0000 7f21d3e3d640 1 --2- 172.21.15.129:0/1551373367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21cc108b50 0x7f21cc1026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.142+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1551373367 >> 172.21.15.129:0/1551373367 conn(0x7f21cc078250 msgr2=0x7f21cc078690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.154+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1551373367 shutdown_connections 2024-05-20T18:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.154+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1551373367 wait complete. 2024-05-20T18:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.155+0000 7f21d3e3d640 1 Processor -- start 2024-05-20T18:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.155+0000 7f21d3e3d640 1 -- start start 2024-05-20T18:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.155+0000 7f21d3e3d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21cc1032a0 0x7f21cc072ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.156+0000 7f21d3e3d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21cc108b50 0x7f21cc073550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.156+0000 7f21d3e3d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21cc108ff0 0x7f21cc06d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.156+0000 7f21d3e3d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc06b6f0 con 0x7f21cc108ff0 2024-05-20T18:05:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.156+0000 7f21d3e3d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc06b570 con 0x7f21cc1032a0 2024-05-20T18:05:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.156+0000 7f21d3e3d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc06b870 con 0x7f21cc108b50 2024-05-20T18:05:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.160+0000 7f21d1bd9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21cc1032a0 0x7f21cc072ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.160+0000 7f21d23da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21cc108ff0 0x7f21cc06d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.160+0000 7f21d23da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21cc108ff0 0x7f21cc06d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45584/0 (socket says 172.21.15.129:45584) 2024-05-20T18:05:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.160+0000 7f21d23da640 1 -- 172.21.15.129:0/1859443508 learned_addr learned my addr 172.21.15.129:0/1859443508 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.160+0000 7f21d13d8640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21cc108b50 0x7f21cc073550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.162+0000 7f21d23da640 1 -- 172.21.15.129:0/1859443508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21cc108b50 msgr2=0x7f21cc073550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.162+0000 7f21d23da640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21cc108b50 0x7f21cc073550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.162+0000 7f21d23da640 1 -- 172.21.15.129:0/1859443508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21cc1032a0 msgr2=0x7f21cc072ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.162+0000 7f21d23da640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21cc1032a0 0x7f21cc072ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.162+0000 7f21d23da640 1 -- 172.21.15.129:0/1859443508 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21cc06d830 con 0x7f21cc108ff0 2024-05-20T18:05:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.169+0000 7f21d23da640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21cc108ff0 0x7f21cc06d100 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005e30 tx=0x7f21c40093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.170+0000 7f21c2ffd640 1 -- 172.21.15.129:0/1859443508 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4030b70 con 0x7f21cc108ff0 2024-05-20T18:05:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.170+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1859443508 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21cc06dac0 con 0x7f21cc108ff0 2024-05-20T18:05:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.170+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1859443508 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21cc19b860 con 0x7f21cc108ff0 2024-05-20T18:05:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.171+0000 7f21c2ffd640 1 -- 172.21.15.129:0/1859443508 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21c4036020 con 0x7f21cc108ff0 2024-05-20T18:05:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.174+0000 7f21c2ffd640 1 -- 172.21.15.129:0/1859443508 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4045410 con 0x7f21cc108ff0 2024-05-20T18:05:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.182+0000 7f21c2ffd640 1 -- 172.21.15.129:0/1859443508 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f21c4043020 con 0x7f21cc108ff0 2024-05-20T18:05:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.194+0000 7f21dadc1740 1 Processor -- start 2024-05-20T18:05:03.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.197+0000 7f21c2ffd640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21b80b3be0 0x7f21b80b60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.198+0000 7f21c2ffd640 1 -- 172.21.15.129:0/1859443508 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f21c40f04a0 con 0x7f21cc108ff0 2024-05-20T18:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.199+0000 7f21dadc1740 1 -- start start 2024-05-20T18:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.199+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba947080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.199+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba9475c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.199+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba947b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.199+0000 7f21dadc1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ffba8ac340 con 0x55ffba5874d0 2024-05-20T18:05:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.199+0000 7f21dadc1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ffba854290 con 0x55ffba57f2f0 2024-05-20T18:05:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.199+0000 7f21dadc1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ffba8a8300 con 0x55ffba5a81b0 2024-05-20T18:05:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.200+0000 7f21d23da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba947b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.200+0000 7f21d13d8640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21b80b3be0 0x7f21b80b60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.202+0000 7f21d1bd9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba947080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.202+0000 7f21d23da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba947b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45590/0 (socket says 172.21.15.129:45590) 2024-05-20T18:05:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.203+0000 7f21d23da640 1 -- 172.21.15.129:0/2103582341 learned_addr learned my addr 172.21.15.129:0/2103582341 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.205+0000 7f21d13d8640 1 --2- 172.21.15.129:0/2103582341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba9475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.206+0000 7f21d23da640 1 -- 172.21.15.129:0/2103582341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 msgr2=0x55ffba947080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.206+0000 7f21d23da640 1 --2- 172.21.15.129:0/2103582341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba947080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.206+0000 7f21d23da640 1 -- 172.21.15.129:0/2103582341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 msgr2=0x55ffba9475c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.206+0000 7f21d23da640 1 --2- 172.21.15.129:0/2103582341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba9475c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.206+0000 7f21d23da640 1 -- 172.21.15.129:0/2103582341 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffba8788e0 con 0x55ffba5874d0 2024-05-20T18:05:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.207+0000 7f21d13d8640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21b80b3be0 0x7f21b80b60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0ff3a0 tx=0x7f21bc009df0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.207+0000 7f21d23da640 1 --2- 172.21.15.129:0/2103582341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba947b00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f21c40c5090 tx=0x7f21c40bc370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d0f9087bd8a73f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.208+0000 7f21b7fff640 1 -- 172.21.15.129:0/2103582341 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c40c0d00 con 0x55ffba5874d0 2024-05-20T18:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.208+0000 7f21b7fff640 1 -- 172.21.15.129:0/2103582341 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21c40c55c0 con 0x55ffba5874d0 2024-05-20T18:05:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.208+0000 7f21b7fff640 1 -- 172.21.15.129:0/2103582341 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c40bcbb0 con 0x55ffba5874d0 2024-05-20T18:05:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.211+0000 7f21dadc1740 1 -- 172.21.15.129:0/2103582341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 msgr2=0x55ffba947b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.211+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2103582341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba947b00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f21c40c5090 tx=0x7f21c40bc370 comp rx=0 tx=0).stop 2024-05-20T18:05:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.213+0000 7f21dadc1740 1 -- 172.21.15.129:0/2103582341 shutdown_connections 2024-05-20T18:05:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.213+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2103582341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba9475c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.213+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2103582341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba947b00 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.213+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2103582341 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba947080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.213+0000 7f21dadc1740 1 -- 172.21.15.129:0/2103582341 >> 172.21.15.129:0/2103582341 conn(0x55ffba59c300 msgr2=0x55ffba943830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.215+0000 7f21dadc1740 1 -- 172.21.15.129:0/2103582341 shutdown_connections 2024-05-20T18:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.216+0000 7f21dadc1740 1 -- 172.21.15.129:0/2103582341 wait complete. 2024-05-20T18:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.217+0000 7f21dadc1740 1 -- start start 2024-05-20T18:05:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.217+0000 7f21dadc1740 1 Processor -- start 2024-05-20T18:05:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.218+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba953220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.219+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba953760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.219+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba951630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.219+0000 7f21dadc1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ffba953f70 con 0x55ffba5874d0 2024-05-20T18:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.219+0000 7f21dadc1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ffba867790 con 0x55ffba57f2f0 2024-05-20T18:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.219+0000 7f21dadc1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ffba9540f0 con 0x55ffba5a81b0 2024-05-20T18:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.219+0000 7f21dadc1740 10 client.0 _ll_register_callbacks cb 0x55ffba8a2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.220+0000 7f21d23da640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba951630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.220+0000 7f21d23da640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba951630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:33014/0 (socket says 172.21.15.129:33014) 2024-05-20T18:05:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.220+0000 7f21d23da640 1 -- 172.21.15.129:0/2627293153 learned_addr learned my addr 172.21.15.129:0/2627293153 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.224+0000 7f21d1bd9640 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba953220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.225+0000 7f21d13d8640 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba953760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.225+0000 7f21d23da640 1 -- 172.21.15.129:0/2627293153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 msgr2=0x55ffba953220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.225+0000 7f21d23da640 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba953220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.226+0000 7f21d23da640 1 -- 172.21.15.129:0/2627293153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 msgr2=0x55ffba953760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.226+0000 7f21d23da640 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba953760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.226+0000 7f21d23da640 1 -- 172.21.15.129:0/2627293153 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffba5c0210 con 0x55ffba5a81b0 2024-05-20T18:05:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.228+0000 7f21d23da640 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba951630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f21c4009dc0 tx=0x7f21c40edba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b08a7e13e694911 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.229+0000 7f21b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:05:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.230+0000 7f21b57fa640 1 -- 172.21.15.129:0/2627293153 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4030e50 con 0x55ffba5a81b0 2024-05-20T18:05:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.230+0000 7f21dadc1740 1 -- 172.21.15.129:0/2627293153 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21c40c55c0 con 0x55ffba5a81b0 2024-05-20T18:05:03.231 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.230+0000 7f21b57fa640 1 -- 172.21.15.129:0/2627293153 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f21c40db270 con 0x55ffba5a81b0 2024-05-20T18:05:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.234+0000 7f21b57fa640 1 -- 172.21.15.129:0/2627293153 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f21c4087050 con 0x55ffba5a81b0 2024-05-20T18:05:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.234+0000 7f21dadc1740 10 client.26122 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.234+0000 7f21dadc1740 1 -- 172.21.15.129:0/2627293153 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ffba68c620 con 0x55ffba5a81b0 2024-05-20T18:05:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.238+0000 7f21b57fa640 1 -- 172.21.15.129:0/2627293153 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21c40c55c0 con 0x55ffba5a81b0 2024-05-20T18:05:03.239 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.239+0000 7f21b57fa640 1 -- 172.21.15.129:0/2627293153 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f21c40c5130 con 0x55ffba5a81b0 2024-05-20T18:05:03.239 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.240+0000 7f21dadc1740 10 client.26122 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:03.239 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.240+0000 7f21dadc1740 10 client.26122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:03.239 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.240+0000 7f21dadc1740 20 client.26122 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:03.239 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.240+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ffba566d80 0x55ffba9636c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.240+0000 7f21dadc1740 4 client.26122 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:03.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.240+0000 7f21dadc1740 1 -- 172.21.15.129:0/2627293153 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffba8b9a70 con 0x55ffba566d80 2024-05-20T18:05:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.242+0000 7f21d1bd9640 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ffba566d80 0x55ffba9636c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.245+0000 7f21d1bd9640 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ffba566d80 0x55ffba9636c0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=733c177e0f6602de server_cookie=ae667d605bcb5f7a in_seq=0 out_seq=0 2024-05-20T18:05:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.245+0000 7f21b57fa640 10 client.26122 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.247+0000 7f21b57fa640 1 -- 172.21.15.129:0/2627293153 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ffba8b9a70 con 0x55ffba566d80 2024-05-20T18:05:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.247+0000 7f21b57fa640 10 client.26122 handle_command_reply: tid=0 2024-05-20T18:05:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.247+0000 7f21dadc1740 1 client.26122 shutdown 2024-05-20T18:05:03.251 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.251+0000 7f21dadc1740 1 -- 172.21.15.129:0/2627293153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 msgr2=0x55ffba951630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.251+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba951630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f21c4009dc0 tx=0x7f21c40edba0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.253+0000 7f21dadc1740 1 -- 172.21.15.129:0/2627293153 shutdown_connections 2024-05-20T18:05:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.253+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ffba566d80 0x55ffba9636c0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.253+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba951630 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.253+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba953760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.253+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2627293153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba953220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.253+0000 7f21dadc1740 1 -- 172.21.15.129:0/2627293153 >> 172.21.15.129:0/2627293153 conn(0x55ffba59c300 msgr2=0x55ffba948da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:03.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.255+0000 7f21b57fa640 0 client.26122 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.256+0000 7f21dadc1740 1 -- 172.21.15.129:0/2627293153 shutdown_connections 2024-05-20T18:05:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.257+0000 7f21dadc1740 1 -- 172.21.15.129:0/2627293153 wait complete. 2024-05-20T18:05:03.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.257+0000 7f21dadc1740 20 client.26122 trim_cache size 0 max 16384 2024-05-20T18:05:03.282 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.284+0000 7f21dadc1740 1 Processor -- start 2024-05-20T18:05:03.283 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.284+0000 7f21dadc1740 1 -- start start 2024-05-20T18:05:03.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.285+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba9439d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.285+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba968ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.285+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba969400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.285+0000 7f21d13d8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba968ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.285+0000 7f21d13d8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba968ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45604/0 (socket says 172.21.15.129:45604) 2024-05-20T18:05:03.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.285+0000 7f21d1bd9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba9439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.285+0000 7f21d1bd9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba9439d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59952/0 (socket says 172.21.15.129:59952) 2024-05-20T18:05:03.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.285+0000 7f21d13d8640 1 -- 172.21.15.129:0/753656040 learned_addr learned my addr 172.21.15.129:0/753656040 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:03.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.285+0000 7f21dadc1740 1 -- 172.21.15.129:0/753656040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f21b000c680 con 0x55ffba5874d0 2024-05-20T18:05:03.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.285+0000 7f21dadc1740 1 -- 172.21.15.129:0/753656040 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21b000ee50 con 0x55ffba57f2f0 2024-05-20T18:05:03.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.285+0000 7f21dadc1740 1 -- 172.21.15.129:0/753656040 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f21b000cc40 con 0x55ffba5a81b0 2024-05-20T18:05:03.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.286+0000 7f21d23da640 1 --2- 172.21.15.129:0/753656040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba969400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.286+0000 7f21d13d8640 1 -- 172.21.15.129:0/753656040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 msgr2=0x55ffba969400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:03.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.286+0000 7f21d13d8640 1 --2- 172.21.15.129:0/753656040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba969400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.287+0000 7f21d13d8640 1 -- 172.21.15.129:0/753656040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 msgr2=0x55ffba9439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:03.286 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.287+0000 7f21d13d8640 1 --2- 172.21.15.129:0/753656040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba9439d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.287 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.287+0000 7f21d13d8640 1 -- 172.21.15.129:0/753656040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffba566fb0 con 0x55ffba5874d0 2024-05-20T18:05:03.287 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.287+0000 7f21d23da640 1 --2- 172.21.15.129:0/753656040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba969400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:05:03.287 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.287+0000 7f21d1bd9640 1 --2- 172.21.15.129:0/753656040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba9439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:05:03.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.287+0000 7f21d13d8640 1 --2- 172.21.15.129:0/753656040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba968ec0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f21cc109420 tx=0x7f21bc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b967310766018d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:03.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.288+0000 7f21b77fe640 1 -- 172.21.15.129:0/753656040 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc005020 con 0x55ffba5874d0 2024-05-20T18:05:03.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.288+0000 7f21b77fe640 1 -- 172.21.15.129:0/753656040 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21bc023070 con 0x55ffba5874d0 2024-05-20T18:05:03.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.288+0000 7f21b77fe640 1 -- 172.21.15.129:0/753656040 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0116b0 con 0x55ffba5874d0 2024-05-20T18:05:03.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.289+0000 7f21dadc1740 1 -- 172.21.15.129:0/753656040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 msgr2=0x55ffba968ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:03.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.289+0000 7f21dadc1740 1 --2- 172.21.15.129:0/753656040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba968ec0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f21cc109420 tx=0x7f21bc019000 comp rx=0 tx=0).stop 2024-05-20T18:05:03.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.289+0000 7f21dadc1740 1 -- 172.21.15.129:0/753656040 shutdown_connections 2024-05-20T18:05:03.289 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.289+0000 7f21dadc1740 1 --2- 172.21.15.129:0/753656040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba969400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.289+0000 7f21dadc1740 1 --2- 172.21.15.129:0/753656040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba5874d0 0x55ffba968ec0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.289+0000 7f21dadc1740 1 --2- 172.21.15.129:0/753656040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba57f2f0 0x55ffba9439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.289+0000 7f21dadc1740 1 -- 172.21.15.129:0/753656040 >> 172.21.15.129:0/753656040 conn(0x55ffba59c300 msgr2=0x55ffba9432c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:03.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.290+0000 7f21dadc1740 1 -- 172.21.15.129:0/753656040 shutdown_connections 2024-05-20T18:05:03.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.290+0000 7f21dadc1740 1 -- 172.21.15.129:0/753656040 wait complete. 2024-05-20T18:05:03.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.290+0000 7f21dadc1740 1 -- start start 2024-05-20T18:05:03.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.290+0000 7f21dadc1740 1 Processor -- start 2024-05-20T18:05:03.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba57f2f0 0x55ffba9546f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba5874d0 0x55ffba954c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.290 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21dadc1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba9c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21d1bd9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba57f2f0 0x55ffba9546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21d1bd9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba57f2f0 0x55ffba9546f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45608/0 (socket says 172.21.15.129:45608) 2024-05-20T18:05:03.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21dadc1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f21b000d310 con 0x55ffba57f2f0 2024-05-20T18:05:03.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21d1bd9640 1 -- 172.21.15.129:0/2470478595 learned_addr learned my addr 172.21.15.129:0/2470478595 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:03.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21dadc1740 1 -- 172.21.15.129:0/2470478595 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21b000e780 con 0x55ffba5874d0 2024-05-20T18:05:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21dadc1740 1 -- 172.21.15.129:0/2470478595 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f21b000d9e0 con 0x55ffba5a81b0 2024-05-20T18:05:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21dadc1740 10 client.0 _ll_register_callbacks cb 0x7f21c40bc7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21d13d8640 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba5874d0 0x55ffba954c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21d1bd9640 1 -- 172.21.15.129:0/2470478595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 msgr2=0x55ffba9c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21d1bd9640 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba9c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21d1bd9640 1 -- 172.21.15.129:0/2470478595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba5874d0 msgr2=0x55ffba954c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21d1bd9640 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba5874d0 0x55ffba954c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.291+0000 7f21d1bd9640 1 -- 172.21.15.129:0/2470478595 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffba9c9fa0 con 0x55ffba57f2f0 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.292+0000 7f21d13d8640 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba5874d0 0x55ffba954c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.292+0000 7f21d1bd9640 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba57f2f0 0x55ffba9546f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x55ffba950e90 tx=0x7f21c801f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94b8c85b8c30296 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.292+0000 7f21b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.292+0000 7f21b57fa640 1 -- 172.21.15.129:0/2470478595 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c801c810 con 0x55ffba57f2f0 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.292+0000 7f21b57fa640 1 -- 172.21.15.129:0/2470478595 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f21c801c9b0 con 0x55ffba57f2f0 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.292+0000 7f21dadc1740 1 -- 172.21.15.129:0/2470478595 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21bc023070 con 0x55ffba57f2f0 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.292+0000 7f21b57fa640 1 -- 172.21.15.129:0/2470478595 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f21c801cbb0 con 0x55ffba57f2f0 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.292+0000 7f21dadc1740 10 client.16353 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.292+0000 7f21dadc1740 1 -- 172.21.15.129:0/2470478595 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ffba9cb270 con 0x55ffba57f2f0 2024-05-20T18:05:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.293+0000 7f21b57fa640 1 -- 172.21.15.129:0/2470478595 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21bc023070 con 0x55ffba57f2f0 2024-05-20T18:05:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.293+0000 7f21b57fa640 1 -- 172.21.15.129:0/2470478595 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f21c8026020 con 0x55ffba57f2f0 2024-05-20T18:05:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.293+0000 7f21dadc1740 10 client.16353 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.293+0000 7f21dadc1740 10 client.16353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.293+0000 7f21dadc1740 20 client.16353 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.293+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ffba53c1c0 0x55ffba9cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.293+0000 7f21dadc1740 4 client.16353 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:05:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.293+0000 7f21dadc1740 1 -- 172.21.15.129:0/2470478595 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ffba8b3dc0 con 0x55ffba53c1c0 2024-05-20T18:05:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.293+0000 7f21d13d8640 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ffba53c1c0 0x55ffba9cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.294+0000 7f21d13d8640 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ffba53c1c0 0x55ffba9cdaa0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ffea8ec300e4cbf server_cookie=47be704330800c8e in_seq=0 out_seq=0 2024-05-20T18:05:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:03.294+0000 7f21b57fa640 10 client.16353 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:03.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:03 smithi154 ceph-mon[36482]: pgmap v199: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 739 MiB/s wr, 1.55k op/s 2024-05-20T18:05:03.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:03 smithi129 ceph-mon[32029]: pgmap v199: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 739 MiB/s wr, 1.55k op/s 2024-05-20T18:05:03.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:03 smithi204 ceph-mon[37076]: pgmap v199: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 739 MiB/s wr, 1.55k op/s 2024-05-20T18:05:04.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:04 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:05:04.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:04 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:05:04.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:04 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:05:05.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:05 smithi154 ceph-mon[36482]: pgmap v200: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 763 MiB/s wr, 1.63k op/s 2024-05-20T18:05:05.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:05 smithi129 ceph-mon[32029]: pgmap v200: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 763 MiB/s wr, 1.63k op/s 2024-05-20T18:05:05.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:05 smithi204 ceph-mon[37076]: pgmap v200: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 763 MiB/s wr, 1.63k op/s 2024-05-20T18:05:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.098+0000 7f21b57fa640 1 -- 172.21.15.129:0/2470478595 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ffba8b3dc0 con 0x55ffba53c1c0 2024-05-20T18:05:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.098+0000 7f21b57fa640 10 client.16353 handle_command_reply: tid=0 2024-05-20T18:05:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.098+0000 7f21dadc1740 1 client.16353 shutdown 2024-05-20T18:05:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.098+0000 7f21dadc1740 1 -- 172.21.15.129:0/2470478595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba57f2f0 msgr2=0x55ffba9546f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.098+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba57f2f0 0x55ffba9546f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x55ffba950e90 tx=0x7f21c801f490 comp rx=0 tx=0).stop 2024-05-20T18:05:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.098+0000 7f21dadc1740 1 -- 172.21.15.129:0/2470478595 shutdown_connections 2024-05-20T18:05:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.098+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ffba53c1c0 0x55ffba9cdaa0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.098+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffba5874d0 0x55ffba954c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.098+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ffba5a81b0 0x55ffba9c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.098+0000 7f21dadc1740 1 --2- 172.21.15.129:0/2470478595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ffba57f2f0 0x55ffba9546f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.098+0000 7f21dadc1740 1 -- 172.21.15.129:0/2470478595 >> 172.21.15.129:0/2470478595 conn(0x55ffba59c300 msgr2=0x55ffba950ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.099+0000 7f21b57fa640 0 client.16353 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.099+0000 7f21dadc1740 1 -- 172.21.15.129:0/2470478595 shutdown_connections 2024-05-20T18:05:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.099+0000 7f21dadc1740 1 -- 172.21.15.129:0/2470478595 wait complete. 2024-05-20T18:05:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.099+0000 7f21dadc1740 20 client.16353 trim_cache size 0 max 16384 2024-05-20T18:05:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:05:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:05:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:05:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:05:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.100+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1859443508 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21b80b3be0 msgr2=0x7f21b80b60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.100+0000 7f21d3e3d640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21b80b3be0 0x7f21b80b60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0ff3a0 tx=0x7f21bc009df0 comp rx=0 tx=0).stop 2024-05-20T18:05:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.100+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1859443508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21cc108ff0 msgr2=0x7f21cc06d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.100+0000 7f21d3e3d640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21cc108ff0 0x7f21cc06d100 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f21c4005e30 tx=0x7f21c40093d0 comp rx=0 tx=0).stop 2024-05-20T18:05:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.101+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1859443508 shutdown_connections 2024-05-20T18:05:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.101+0000 7f21d3e3d640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f21b80b3be0 0x7f21b80b60f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.101+0000 7f21d3e3d640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21cc108ff0 0x7f21cc06d100 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.101+0000 7f21d3e3d640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f21cc108b50 0x7f21cc073550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.101+0000 7f21d3e3d640 1 --2- 172.21.15.129:0/1859443508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21cc1032a0 0x7f21cc072ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.101+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1859443508 >> 172.21.15.129:0/1859443508 conn(0x7f21cc078250 msgr2=0x7f21cc0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:06.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.101+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1859443508 shutdown_connections 2024-05-20T18:05:06.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:06.101+0000 7f21d3e3d640 1 -- 172.21.15.129:0/1859443508 wait complete. 2024-05-20T18:05:07.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:07 smithi154 ceph-mon[36482]: pgmap v201: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 958 MiB/s wr, 2.20k op/s 2024-05-20T18:05:07.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:07 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:05:07.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:07 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:05:07.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:07 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:05:07.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:07 smithi129 ceph-mon[32029]: pgmap v201: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 958 MiB/s wr, 2.20k op/s 2024-05-20T18:05:07.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:07 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:05:07.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:07 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:05:07.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:07 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:05:07.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:07 smithi204 ceph-mon[37076]: pgmap v201: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 958 MiB/s wr, 2.20k op/s 2024-05-20T18:05:07.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:07 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:05:07.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:07 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:05:07.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:07 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:05:09.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:09 smithi154 ceph-mon[36482]: pgmap v202: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 755 MiB/s wr, 2.00k op/s 2024-05-20T18:05:09.671 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:09 smithi129 ceph-mon[32029]: pgmap v202: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 755 MiB/s wr, 2.00k op/s 2024-05-20T18:05:09.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:09 smithi204 ceph-mon[37076]: pgmap v202: 97 pgs: 97 active+clean; 50 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 755 MiB/s wr, 2.00k op/s 2024-05-20T18:05:10.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:05:10.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:05:10.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.578+0000 7f9b95e51640 1 -- 172.21.15.129:0/3966103215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b90119770 msgr2=0x7f9b9011bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.578+0000 7f9b95e51640 1 --2- 172.21.15.129:0/3966103215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b90119770 0x7f9b9011bb90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9b7801a940 tx=0x7f9b7803b4e0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.578+0000 7f9b95e51640 1 -- 172.21.15.129:0/3966103215 shutdown_connections 2024-05-20T18:05:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.578+0000 7f9b95e51640 1 --2- 172.21.15.129:0/3966103215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b9011c0d0 0x7f9b9011e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.578+0000 7f9b95e51640 1 --2- 172.21.15.129:0/3966103215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b90119770 0x7f9b9011bb90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.578+0000 7f9b95e51640 1 --2- 172.21.15.129:0/3966103215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b90109060 0x7f9b90109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.578+0000 7f9b95e51640 1 -- 172.21.15.129:0/3966103215 >> 172.21.15.129:0/3966103215 conn(0x7f9b9006d750 msgr2=0x7f9b9006dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.578+0000 7f9b95e51640 1 -- 172.21.15.129:0/3966103215 shutdown_connections 2024-05-20T18:05:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.578+0000 7f9b95e51640 1 -- 172.21.15.129:0/3966103215 wait complete. 2024-05-20T18:05:10.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.579+0000 7f9b95e51640 1 Processor -- start 2024-05-20T18:05:10.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.580+0000 7f9b95e51640 1 -- start start 2024-05-20T18:05:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.580+0000 7f9b95e51640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90109060 0x7f9b90119120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.580+0000 7f9b95e51640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b90119770 0x7f9b90115250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.580+0000 7f9b95e51640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b9011c0d0 0x7f9b901157d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.580+0000 7f9b95e51640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9b90113940 con 0x7f9b90109060 2024-05-20T18:05:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b8ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b9011c0d0 0x7f9b901157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b87fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b90119770 0x7f9b90115250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b8ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b9011c0d0 0x7f9b901157d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55098/0 (socket says 172.21.15.129:55098) 2024-05-20T18:05:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b8ffff640 1 -- 172.21.15.129:0/3449741046 learned_addr learned my addr 172.21.15.129:0/3449741046 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90109060 0x7f9b90119120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b95e51640 1 -- 172.21.15.129:0/3449741046 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901137c0 con 0x7f9b9011c0d0 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b95e51640 1 -- 172.21.15.129:0/3449741046 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9b90113ac0 con 0x7f9b90119770 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b8ffff640 1 -- 172.21.15.129:0/3449741046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b90119770 msgr2=0x7f9b90115250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b8ffff640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b90119770 0x7f9b90115250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b8ffff640 1 -- 172.21.15.129:0/3449741046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90109060 msgr2=0x7f9b90119120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b8ffff640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90109060 0x7f9b90119120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.581+0000 7f9b8ffff640 1 -- 172.21.15.129:0/3449741046 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b90115f60 con 0x7f9b9011c0d0 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.582+0000 7f9b8ffff640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b9011c0d0 0x7f9b901157d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800cce0 tx=0x7f9b8800a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.582+0000 7f9b8d7fa640 1 -- 172.21.15.129:0/3449741046 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8800f450 con 0x7f9b9011c0d0 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.582+0000 7f9b8d7fa640 1 -- 172.21.15.129:0/3449741046 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b8800fad0 con 0x7f9b9011c0d0 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.582+0000 7f9b8d7fa640 1 -- 172.21.15.129:0/3449741046 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88018c10 con 0x7f9b9011c0d0 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.582+0000 7f9b95e51640 1 -- 172.21.15.129:0/3449741046 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b901abfa0 con 0x7f9b9011c0d0 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.582+0000 7f9b95e51640 1 -- 172.21.15.129:0/3449741046 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b901ac3b0 con 0x7f9b9011c0d0 2024-05-20T18:05:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.586+0000 7f9b8d7fa640 1 -- 172.21.15.129:0/3449741046 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9b8800f5f0 con 0x7f9b9011c0d0 2024-05-20T18:05:10.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.587+0000 7f9b8d7fa640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b7c0a2f00 0x7f9b7c0a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.587+0000 7f9b8d7fa640 1 -- 172.21.15.129:0/3449741046 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f9b880cfa40 con 0x7f9b9011c0d0 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.587+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b7c0a2f00 0x7f9b7c0a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.588+0000 7f9b9cdce740 1 Processor -- start 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.588+0000 7f9b9cdce740 1 -- start start 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.588+0000 7f9b9cdce740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e22770e390 0x55e227aacd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.588+0000 7f9b9cdce740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227aad270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.589+0000 7f9b9cdce740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2276ed6b0 0x55e227aad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.589+0000 7f9b9cdce740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e227a12d00 con 0x55e22770e390 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.589+0000 7f9b8f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e22770e390 0x55e227aacd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.589+0000 7f9b8f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e22770e390 0x55e227aacd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60082/0 (socket says 172.21.15.129:60082) 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.589+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/3904148979 learned_addr learned my addr 172.21.15.129:0/3904148979 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.589+0000 7f9b87fff640 1 --2- 172.21.15.129:0/3904148979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227aad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.589+0000 7f9b8ffff640 1 --2- 172.21.15.129:0/3904148979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2276ed6b0 0x55e227aad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.589+0000 7f9b9cdce740 1 -- 172.21.15.129:0/3904148979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e2279ba4c0 con 0x55e2276e54d0 2024-05-20T18:05:10.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.589+0000 7f9b9cdce740 1 -- 172.21.15.129:0/3904148979 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e227a0ecc0 con 0x55e2276ed6b0 2024-05-20T18:05:10.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.589+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b7c0a2f00 0x7f9b7c0a5410 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009bd0 tx=0x7f9b80005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:10.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.590+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/3904148979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2276ed6b0 msgr2=0x55e227aad7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:10.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.590+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/3904148979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2276ed6b0 0x55e227aad7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.590+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/3904148979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 msgr2=0x55e227aad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:10.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.590+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/3904148979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227aad270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.590+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/3904148979 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2277f07d0 con 0x55e22770e390 2024-05-20T18:05:10.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.590+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/3904148979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e22770e390 0x55e227aacd30 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000e620 tx=0x7f9b80014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39b0ee1c373bf58 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:10.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.590+0000 7f9b85ffb640 1 -- 172.21.15.129:0/3904148979 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80005020 con 0x55e22770e390 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.590+0000 7f9b85ffb640 1 -- 172.21.15.129:0/3904148979 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b800047a0 con 0x55e22770e390 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.590+0000 7f9b85ffb640 1 -- 172.21.15.129:0/3904148979 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80026880 con 0x55e22770e390 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.592+0000 7f9b9cdce740 1 -- 172.21.15.129:0/3904148979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e22770e390 msgr2=0x55e227aacd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.592+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/3904148979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e22770e390 0x55e227aacd30 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000e620 tx=0x7f9b80014000 comp rx=0 tx=0).stop 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.592+0000 7f9b9cdce740 1 -- 172.21.15.129:0/3904148979 shutdown_connections 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.592+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/3904148979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227aad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.592+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/3904148979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2276ed6b0 0x55e227aad7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.592+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/3904148979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e22770e390 0x55e227aacd30 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.592+0000 7f9b9cdce740 1 -- 172.21.15.129:0/3904148979 >> 172.21.15.129:0/3904148979 conn(0x55e2277024e0 msgr2=0x55e227aa94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.592+0000 7f9b9cdce740 1 -- 172.21.15.129:0/3904148979 shutdown_connections 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.592+0000 7f9b9cdce740 1 -- 172.21.15.129:0/3904148979 wait complete. 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.593+0000 7f9b9cdce740 1 -- start start 2024-05-20T18:05:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.593+0000 7f9b9cdce740 1 Processor -- start 2024-05-20T18:05:10.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b9cdce740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227ab8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b9cdce740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276ed6b0 0x55e227ab9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b8f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227ab8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b8f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227ab8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55114/0 (socket says 172.21.15.129:55114) 2024-05-20T18:05:10.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1901527531 learned_addr learned my addr 172.21.15.129:0/1901527531 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:10.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e22770e390 0x55e227ab73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b87fff640 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276ed6b0 0x55e227ab9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1901527531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e227ab9fc0 con 0x55e2276ed6b0 2024-05-20T18:05:10.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1901527531 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e2279cdb90 con 0x55e2276e54d0 2024-05-20T18:05:10.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1901527531 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e227aba140 con 0x55e22770e390 2024-05-20T18:05:10.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b8ffff640 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e22770e390 0x55e227ab73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.594+0000 7f9b9cdce740 10 client.0 _ll_register_callbacks cb 0x55e227a09da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:10.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1901527531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e22770e390 msgr2=0x55e227ab73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:10.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e22770e390 0x55e227ab73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1901527531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276ed6b0 msgr2=0x55e227ab9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:10.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276ed6b0 0x55e227ab9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1901527531 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2277263f0 con 0x55e2276e54d0 2024-05-20T18:05:10.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1901527531 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b800047a0 con 0x55e2276e54d0 2024-05-20T18:05:10.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227ab8e50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000a010 tx=0x7f9b8000ede0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc98bebcadc5aedf server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:10.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b6effd640 10 client.26102 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:05:10.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b6effd640 1 -- 172.21.15.129:0/1901527531 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80026b40 con 0x55e2276e54d0 2024-05-20T18:05:10.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b6effd640 1 -- 172.21.15.129:0/1901527531 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f9b80026ce0 con 0x55e2276e54d0 2024-05-20T18:05:10.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b6effd640 1 -- 172.21.15.129:0/1901527531 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9b80004950 con 0x55e2276e54d0 2024-05-20T18:05:10.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b9cdce740 10 client.26102 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:10.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.595+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1901527531 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({fsmap=24}) -- 0x55e2276e60f0 con 0x55e2276e54d0 2024-05-20T18:05:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.596+0000 7f9b6effd640 1 -- 172.21.15.129:0/1901527531 <== mon.2 v2:172.21.15.154:3300/0 4 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f9b80004af0 con 0x55e2276e54d0 2024-05-20T18:05:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.596+0000 7f9b9cdce740 10 client.26102 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.596+0000 7f9b9cdce740 10 client.26102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:10.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.596+0000 7f9b9cdce740 20 client.26102 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:10.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.596+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2276ccf60 0x55e227ac9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.596+0000 7f9b9cdce740 4 client.26102 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:10.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.596+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1901527531 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e227a20430 con 0x55e2276ccf60 2024-05-20T18:05:10.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.596+0000 7f9b87fff640 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2276ccf60 0x55e227ac9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.597+0000 7f9b87fff640 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2276ccf60 0x55e227ac9900 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e35ab05b208e8f8 server_cookie=9b644dd23ed3a149 in_seq=0 out_seq=0 2024-05-20T18:05:10.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.597+0000 7f9b6effd640 10 client.26102 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:10.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.598+0000 7f9b6effd640 1 -- 172.21.15.129:0/1901527531 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e227a20430 con 0x55e2276ccf60 2024-05-20T18:05:10.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.598+0000 7f9b6effd640 10 client.26102 handle_command_reply: tid=0 2024-05-20T18:05:10.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.598+0000 7f9b9cdce740 1 client.26102 shutdown 2024-05-20T18:05:10.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1901527531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 msgr2=0x55e227ab8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:10.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227ab8e50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9b8000a010 tx=0x7f9b8000ede0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1901527531 shutdown_connections 2024-05-20T18:05:10.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2276ccf60 0x55e227ac9900 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e22770e390 0x55e227ab73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276ed6b0 0x55e227ab9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1901527531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227ab8e50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b6effd640 0 client.26102 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:10.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1901527531 >> 172.21.15.129:0/1901527531 conn(0x55e2277024e0 msgr2=0x55e227aaea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:10.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1901527531 shutdown_connections 2024-05-20T18:05:10.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1901527531 wait complete. 2024-05-20T18:05:10.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.600+0000 7f9b9cdce740 20 client.26102 trim_cache size 0 max 16384 2024-05-20T18:05:10.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.628+0000 7f9b9cdce740 1 Processor -- start 2024-05-20T18:05:10.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.628+0000 7f9b9cdce740 1 -- start start 2024-05-20T18:05:10.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.629+0000 7f9b9cdce740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227abca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.629+0000 7f9b9cdce740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276ed6b0 0x55e227ab09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.629+0000 7f9b9cdce740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e22770e390 0x55e227ab0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.630+0000 7f9b9cdce740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901b2e30 con 0x55e2276ed6b0 2024-05-20T18:05:10.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.630+0000 7f9b9cdce740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901b56b0 con 0x55e2276e54d0 2024-05-20T18:05:10.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.630+0000 7f9b9cdce740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901b34a0 con 0x55e22770e390 2024-05-20T18:05:10.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.630+0000 7f9b8f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227abca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.630+0000 7f9b8f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227abca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55126/0 (socket says 172.21.15.129:55126) 2024-05-20T18:05:10.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.630+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1049085480 learned_addr learned my addr 172.21.15.129:0/1049085480 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:10.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.630+0000 7f9b8ffff640 1 --2- 172.21.15.129:0/1049085480 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e22770e390 0x55e227ab0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.634+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1049085480 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e22770e390 msgr2=0x55e227ab0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:10.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.634+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/1049085480 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e22770e390 0x55e227ab0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.634+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1049085480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276ed6b0 msgr2=0x55e227ab09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:10.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.634+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/1049085480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276ed6b0 0x55e227ab09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.634+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1049085480 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e227aae820 con 0x55e2276e54d0 2024-05-20T18:05:10.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.634+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/1049085480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227abca70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009220 tx=0x7f9b80029000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfa4998fe02c8984 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:10.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.635+0000 7f9b857fa640 1 -- 172.21.15.129:0/1049085480 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8003c070 con 0x55e2276e54d0 2024-05-20T18:05:10.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.635+0000 7f9b857fa640 1 -- 172.21.15.129:0/1049085480 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b800047a0 con 0x55e2276e54d0 2024-05-20T18:05:10.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.635+0000 7f9b857fa640 1 -- 172.21.15.129:0/1049085480 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80026b40 con 0x55e2276e54d0 2024-05-20T18:05:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.637+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1049085480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 msgr2=0x55e227abca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.637+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1049085480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227abca70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009220 tx=0x7f9b80029000 comp rx=0 tx=0).stop 2024-05-20T18:05:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.637+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1049085480 shutdown_connections 2024-05-20T18:05:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.637+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1049085480 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e22770e390 0x55e227ab0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.637+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1049085480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276ed6b0 0x55e227ab09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.637+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1049085480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2276e54d0 0x55e227abca70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.637+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1049085480 >> 172.21.15.129:0/1049085480 conn(0x55e2277024e0 msgr2=0x55e227abc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:10.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.637+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1049085480 shutdown_connections 2024-05-20T18:05:10.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.638+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1049085480 wait complete. 2024-05-20T18:05:10.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.638+0000 7f9b9cdce740 1 -- start start 2024-05-20T18:05:10.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.639+0000 7f9b9cdce740 1 Processor -- start 2024-05-20T18:05:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.640+0000 7f9b9cdce740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276e54d0 0x55e227b29e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.641+0000 7f9b8f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276e54d0 0x55e227b29e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.641+0000 7f9b8f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276e54d0 0x55e227b29e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60116/0 (socket says 172.21.15.129:60116) 2024-05-20T18:05:10.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.641+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1701271091 learned_addr learned my addr 172.21.15.129:0/1701271091 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:10.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.642+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2276ed6b0 0x55e227b2a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.642+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e22770e390 0x55e227ac0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.642+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1701271091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901b3b70 con 0x55e2276e54d0 2024-05-20T18:05:10.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.642+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1701271091 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901b4fe0 con 0x55e22770e390 2024-05-20T18:05:10.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.642+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1701271091 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901b4240 con 0x55e2276ed6b0 2024-05-20T18:05:10.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.642+0000 7f9b87fff640 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2276ed6b0 0x55e227b2a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.643+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1701271091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2276ed6b0 msgr2=0x55e227b2a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:10.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.643+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2276ed6b0 0x55e227b2a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.643+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1701271091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e22770e390 msgr2=0x55e227ac0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:10.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.643+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e22770e390 0x55e227ac0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.643+0000 7f9b8f7fe640 1 -- 172.21.15.129:0/1701271091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e227ac13d0 con 0x55e2276e54d0 2024-05-20T18:05:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.643+0000 7f9b9cdce740 10 client.0 _ll_register_callbacks cb 0x7f9b80005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.643+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1701271091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b800047a0 con 0x55e2276e54d0 2024-05-20T18:05:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.643+0000 7f9b8f7fe640 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276e54d0 0x55e227b29e70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9b80004b50 tx=0x7f9b8000e590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d6d9f229a5ea02 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.644+0000 7f9b6effd640 10 client.16365 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:05:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.644+0000 7f9b6effd640 1 -- 172.21.15.129:0/1701271091 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8003c040 con 0x55e2276e54d0 2024-05-20T18:05:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.644+0000 7f9b6effd640 1 -- 172.21.15.129:0/1701271091 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f9b80026830 con 0x55e2276e54d0 2024-05-20T18:05:10.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.644+0000 7f9b6effd640 1 -- 172.21.15.129:0/1701271091 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9b800052a0 con 0x55e2276e54d0 2024-05-20T18:05:10.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.647+0000 7f9b9cdce740 10 client.16365 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:10.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.647+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1701271091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({fsmap=24}) -- 0x55e227b31290 con 0x55e2276e54d0 2024-05-20T18:05:10.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.652+0000 7f9b6effd640 1 -- 172.21.15.129:0/1701271091 <== mon.0 v2:172.21.15.129:3300/0 4 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f9b80005440 con 0x55e2276e54d0 2024-05-20T18:05:10.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.652+0000 7f9b9cdce740 10 client.16365 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:10.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.652+0000 7f9b9cdce740 10 client.16365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:10.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.652+0000 7f9b9cdce740 20 client.16365 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:10.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.652+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2276a23a0 0x55e227b338f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:10.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.652+0000 7f9b9cdce740 4 client.16365 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:05:10.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.652+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1701271091 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e227a1a780 con 0x55e2276a23a0 2024-05-20T18:05:10.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.653+0000 7f9b87fff640 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2276a23a0 0x55e227b338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:10.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.659+0000 7f9b87fff640 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2276a23a0 0x55e227b338f0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b44ec9fb7024db57 server_cookie=35b5f803a48435e7 in_seq=0 out_seq=0 2024-05-20T18:05:10.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.659+0000 7f9b6effd640 10 client.16365 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:10.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.661+0000 7f9b6effd640 1 -- 172.21.15.129:0/1701271091 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e227a1a780 con 0x55e2276a23a0 2024-05-20T18:05:10.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.661+0000 7f9b6effd640 10 client.16365 handle_command_reply: tid=0 2024-05-20T18:05:10.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.661+0000 7f9b9cdce740 1 client.16365 shutdown 2024-05-20T18:05:10.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.662+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1701271091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276e54d0 msgr2=0x55e227b29e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:10.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.662+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276e54d0 0x55e227b29e70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9b80004b50 tx=0x7f9b8000e590 comp rx=0 tx=0).stop 2024-05-20T18:05:10.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.665+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1701271091 shutdown_connections 2024-05-20T18:05:10.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.665+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e2276a23a0 0x55e227b338f0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.665+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e22770e390 0x55e227ac0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.665+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2276ed6b0 0x55e227b2a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.665+0000 7f9b6effd640 0 client.16365 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:10.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.665+0000 7f9b9cdce740 1 --2- 172.21.15.129:0/1701271091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e2276e54d0 0x55e227b29e70 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.665+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1701271091 >> 172.21.15.129:0/1701271091 conn(0x55e2277024e0 msgr2=0x55e227ab5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:10.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.665+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1701271091 shutdown_connections 2024-05-20T18:05:10.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.665+0000 7f9b9cdce740 1 -- 172.21.15.129:0/1701271091 wait complete. 2024-05-20T18:05:10.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.666+0000 7f9b9cdce740 20 client.16365 trim_cache size 0 max 16384 2024-05-20T18:05:10.665 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:05:10.665 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:05:10.665 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:05:10.665 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:05:10.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.667+0000 7f9b95e51640 1 -- 172.21.15.129:0/3449741046 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b7c0a2f00 msgr2=0x7f9b7c0a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.667+0000 7f9b95e51640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b7c0a2f00 0x7f9b7c0a5410 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009bd0 tx=0x7f9b80005f90 comp rx=0 tx=0).stop 2024-05-20T18:05:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.667+0000 7f9b95e51640 1 -- 172.21.15.129:0/3449741046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b9011c0d0 msgr2=0x7f9b901157d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.667+0000 7f9b95e51640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b9011c0d0 0x7f9b901157d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800cce0 tx=0x7f9b8800a620 comp rx=0 tx=0).stop 2024-05-20T18:05:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.667+0000 7f9b95e51640 1 -- 172.21.15.129:0/3449741046 shutdown_connections 2024-05-20T18:05:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.667+0000 7f9b95e51640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9b7c0a2f00 0x7f9b7c0a5410 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.667+0000 7f9b95e51640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b9011c0d0 0x7f9b901157d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.667+0000 7f9b95e51640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b90119770 0x7f9b90115250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.667+0000 7f9b95e51640 1 --2- 172.21.15.129:0/3449741046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90109060 0x7f9b90119120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.667+0000 7f9b95e51640 1 -- 172.21.15.129:0/3449741046 >> 172.21.15.129:0/3449741046 conn(0x7f9b9006d750 msgr2=0x7f9b9011b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.667+0000 7f9b95e51640 1 -- 172.21.15.129:0/3449741046 shutdown_connections 2024-05-20T18:05:10.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:10.668+0000 7f9b95e51640 1 -- 172.21.15.129:0/3449741046 wait complete. 2024-05-20T18:05:10.704 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:05:10.705 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:05:11.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.019+0000 7fdfee73e640 1 -- 172.21.15.129:0/2705702734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe811c0d0 msgr2=0x7fdfe811e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:11.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.019+0000 7fdfee73e640 1 --2- 172.21.15.129:0/2705702734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe811c0d0 0x7fdfe811e4f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc01a940 tx=0x7fdfdc03b520 comp rx=0 tx=0).stop 2024-05-20T18:05:11.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.022+0000 7fdfee73e640 1 -- 172.21.15.129:0/2705702734 shutdown_connections 2024-05-20T18:05:11.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.022+0000 7fdfee73e640 1 --2- 172.21.15.129:0/2705702734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe811c0d0 0x7fdfe811e4f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.022+0000 7fdfee73e640 1 --2- 172.21.15.129:0/2705702734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe8119770 0x7fdfe811bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.022+0000 7fdfee73e640 1 --2- 172.21.15.129:0/2705702734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfe8109060 0x7fdfe8109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.022+0000 7fdfee73e640 1 -- 172.21.15.129:0/2705702734 >> 172.21.15.129:0/2705702734 conn(0x7fdfe806d760 msgr2=0x7fdfe806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:11.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.027+0000 7fdfee73e640 1 -- 172.21.15.129:0/2705702734 shutdown_connections 2024-05-20T18:05:11.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.027+0000 7fdfee73e640 1 -- 172.21.15.129:0/2705702734 wait complete. 2024-05-20T18:05:11.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.029+0000 7fdfee73e640 1 Processor -- start 2024-05-20T18:05:11.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.029+0000 7fdfee73e640 1 -- start start 2024-05-20T18:05:11.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.030+0000 7fdfee73e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8109060 0x7fdfe8115580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.030+0000 7fdfee73e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe8119770 0x7fdfe8115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.030+0000 7fdfee73e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfe811c0d0 0x7fdfe8116000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.030+0000 7fdfee73e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe81139d0 con 0x7fdfe8119770 2024-05-20T18:05:11.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.030+0000 7fdfee73e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe8113850 con 0x7fdfe811c0d0 2024-05-20T18:05:11.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.030+0000 7fdfee73e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe8113b50 con 0x7fdfe8109060 2024-05-20T18:05:11.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.031+0000 7fdfe7fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8109060 0x7fdfe8115580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.031+0000 7fdfe7fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8109060 0x7fdfe8115580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:43580/0 (socket says 172.21.15.129:43580) 2024-05-20T18:05:11.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.031+0000 7fdfe7fff640 1 -- 172.21.15.129:0/3911075225 learned_addr learned my addr 172.21.15.129:0/3911075225 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:11.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.031+0000 7fdfe77fe640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe8119770 0x7fdfe8115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.031+0000 7fdfeccdb640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfe811c0d0 0x7fdfe8116000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.031+0000 7fdfe7fff640 1 -- 172.21.15.129:0/3911075225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfe811c0d0 msgr2=0x7fdfe8116000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:11.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.031+0000 7fdfe7fff640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfe811c0d0 0x7fdfe8116000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.031+0000 7fdfe7fff640 1 -- 172.21.15.129:0/3911075225 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe8119770 msgr2=0x7fdfe8115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:11.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.031+0000 7fdfe7fff640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe8119770 0x7fdfe8115ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.031+0000 7fdfe7fff640 1 -- 172.21.15.129:0/3911075225 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe8079e30 con 0x7fdfe8109060 2024-05-20T18:05:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.032+0000 7fdfe77fe640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe8119770 0x7fdfe8115ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:05:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.032+0000 7fdfeccdb640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfe811c0d0 0x7fdfe8116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:05:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.032+0000 7fdfe7fff640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8109060 0x7fdfe8115580 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000ba50 tx=0x7fdfe000bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.032+0000 7fdfe57fa640 1 -- 172.21.15.129:0/3911075225 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0019070 con 0x7fdfe8109060 2024-05-20T18:05:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.032+0000 7fdfe57fa640 1 -- 172.21.15.129:0/3911075225 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfe000cc10 con 0x7fdfe8109060 2024-05-20T18:05:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.032+0000 7fdfe57fa640 1 -- 172.21.15.129:0/3911075225 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe00076a0 con 0x7fdfe8109060 2024-05-20T18:05:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.033+0000 7fdfee73e640 1 -- 172.21.15.129:0/3911075225 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe807a120 con 0x7fdfe8109060 2024-05-20T18:05:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.033+0000 7fdfee73e640 1 -- 172.21.15.129:0/3911075225 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe807a620 con 0x7fdfe8109060 2024-05-20T18:05:11.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.035+0000 7fdfee73e640 1 -- 172.21.15.129:0/3911075225 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd4005290 con 0x7fdfe8109060 2024-05-20T18:05:11.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.038+0000 7fdfe57fa640 1 -- 172.21.15.129:0/3911075225 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fdfe001c070 con 0x7fdfe8109060 2024-05-20T18:05:11.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.039+0000 7fdfe57fa640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdfd00a2f00 0x7fdfd00a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.039+0000 7fdfe57fa640 1 -- 172.21.15.129:0/3911075225 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fdfe00ce8c0 con 0x7fdfe8109060 2024-05-20T18:05:11.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.040+0000 7fdfe77fe640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdfd00a2f00 0x7fdfd00a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.041+0000 7fdfe77fe640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdfd00a2f00 0x7fdfd00a5410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8006fd0 tx=0x7fdfd8008040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:11.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.044+0000 7fdfe57fa640 1 -- 172.21.15.129:0/3911075225 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdfe009b580 con 0x7fdfe8109060 2024-05-20T18:05:11.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.431+0000 7fdfee73e640 1 -- 172.21.15.129:0/3911075225 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfd4005850 con 0x7fdfe8109060 2024-05-20T18:05:11.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.442+0000 7fdfe57fa640 1 -- 172.21.15.129:0/3911075225 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11829 (secure 0 0 0) 0x7fdfe009ae80 con 0x7fdfe8109060 2024-05-20T18:05:11.440 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:05:11.441 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:03:18.451223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/4068528573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":4068528573},{"type":"v1","addr":"172.21.15.154:6837","nonce":4068528573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:05:11.441 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 24 2024-05-20T18:05:11.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.453+0000 7fdfceffd640 1 -- 172.21.15.129:0/3911075225 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdfd00a2f00 msgr2=0x7fdfd00a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:11.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.453+0000 7fdfceffd640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdfd00a2f00 0x7fdfd00a5410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8006fd0 tx=0x7fdfd8008040 comp rx=0 tx=0).stop 2024-05-20T18:05:11.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.453+0000 7fdfceffd640 1 -- 172.21.15.129:0/3911075225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8109060 msgr2=0x7fdfe8115580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:11.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.453+0000 7fdfceffd640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8109060 0x7fdfe8115580 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000ba50 tx=0x7fdfe000bf10 comp rx=0 tx=0).stop 2024-05-20T18:05:11.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.454+0000 7fdfceffd640 1 -- 172.21.15.129:0/3911075225 shutdown_connections 2024-05-20T18:05:11.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.454+0000 7fdfceffd640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdfd00a2f00 0x7fdfd00a5410 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.454+0000 7fdfceffd640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfe811c0d0 0x7fdfe8116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.454+0000 7fdfceffd640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfe8119770 0x7fdfe8115ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.454+0000 7fdfceffd640 1 --2- 172.21.15.129:0/3911075225 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdfe8109060 0x7fdfe8115580 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.454+0000 7fdfceffd640 1 -- 172.21.15.129:0/3911075225 >> 172.21.15.129:0/3911075225 conn(0x7fdfe806d760 msgr2=0x7fdfe8119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:11.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.454+0000 7fdfceffd640 1 -- 172.21.15.129:0/3911075225 shutdown_connections 2024-05-20T18:05:11.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.454+0000 7fdfceffd640 1 -- 172.21.15.129:0/3911075225 wait complete. 2024-05-20T18:05:11.500 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:05:11.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:11 smithi154 ceph-mon[36482]: pgmap v203: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 922 MiB/s wr, 3.81k op/s 2024-05-20T18:05:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.915+0000 7f1674557640 1 -- 172.21.15.129:0/1709546734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c109060 msgr2=0x7f166c109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.915+0000 7f1674557640 1 --2- 172.21.15.129:0/1709546734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c109060 0x7f166c109470 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f165c009900 tx=0x7f165c02a4d0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:11 smithi129 ceph-mon[32029]: pgmap v203: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 922 MiB/s wr, 3.81k op/s 2024-05-20T18:05:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.919+0000 7f1674557640 1 -- 172.21.15.129:0/1709546734 shutdown_connections 2024-05-20T18:05:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.919+0000 7f1674557640 1 --2- 172.21.15.129:0/1709546734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f166c11c0d0 0x7f166c11e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.919+0000 7f1674557640 1 --2- 172.21.15.129:0/1709546734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c119770 0x7f166c11bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.919+0000 7f1674557640 1 --2- 172.21.15.129:0/1709546734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c109060 0x7f166c109470 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.919+0000 7f1674557640 1 -- 172.21.15.129:0/1709546734 >> 172.21.15.129:0/1709546734 conn(0x7f166c06d750 msgr2=0x7f166c06dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.921+0000 7f1674557640 1 -- 172.21.15.129:0/1709546734 shutdown_connections 2024-05-20T18:05:11.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.921+0000 7f1674557640 1 -- 172.21.15.129:0/1709546734 wait complete. 2024-05-20T18:05:11.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.923+0000 7f1674557640 1 Processor -- start 2024-05-20T18:05:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.923+0000 7f1674557640 1 -- start start 2024-05-20T18:05:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.923+0000 7f1674557640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f166c109060 0x7f166c1133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.923+0000 7f1674557640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c119770 0x7f166c113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.923+0000 7f1674557640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c11c0d0 0x7f166c113e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.923+0000 7f1674557640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f166c111970 con 0x7f166c11c0d0 2024-05-20T18:05:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.923+0000 7f1674557640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f166c1117f0 con 0x7f166c109060 2024-05-20T18:05:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.923+0000 7f1674557640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f166c111af0 con 0x7f166c119770 2024-05-20T18:05:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.924+0000 7f1672af4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c11c0d0 0x7f166c113e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.924+0000 7f1672af4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c11c0d0 0x7f166c113e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60160/0 (socket says 172.21.15.129:60160) 2024-05-20T18:05:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.924+0000 7f1672af4640 1 -- 172.21.15.129:0/2947430183 learned_addr learned my addr 172.21.15.129:0/2947430183 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.925+0000 7f1671af2640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c119770 0x7f166c113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:11 smithi204 ceph-mon[37076]: pgmap v203: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 922 MiB/s wr, 3.81k op/s 2024-05-20T18:05:11.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.926+0000 7f1672af4640 1 -- 172.21.15.129:0/2947430183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c119770 msgr2=0x7f166c113900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:11.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.926+0000 7f1672af4640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c119770 0x7f166c113900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.926+0000 7f1672af4640 1 -- 172.21.15.129:0/2947430183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f166c109060 msgr2=0x7f166c1133c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:11.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.926+0000 7f1672af4640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f166c109060 0x7f166c1133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.926+0000 7f1672af4640 1 -- 172.21.15.129:0/2947430183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166c079db0 con 0x7f166c11c0d0 2024-05-20T18:05:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.932+0000 7f1672af4640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c11c0d0 0x7f166c113e40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f166400eaa0 tx=0x7f166400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.933+0000 7f16637fe640 1 -- 172.21.15.129:0/2947430183 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166400cdb0 con 0x7f166c11c0d0 2024-05-20T18:05:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.933+0000 7f1674557640 1 -- 172.21.15.129:0/2947430183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166c07a0a0 con 0x7f166c11c0d0 2024-05-20T18:05:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.933+0000 7f1674557640 1 -- 172.21.15.129:0/2947430183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f166c07a5f0 con 0x7f166c11c0d0 2024-05-20T18:05:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.934+0000 7f16637fe640 1 -- 172.21.15.129:0/2947430183 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1664005020 con 0x7f166c11c0d0 2024-05-20T18:05:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.934+0000 7f16637fe640 1 -- 172.21.15.129:0/2947430183 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16640116d0 con 0x7f166c11c0d0 2024-05-20T18:05:11.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.939+0000 7f167b401740 1 Processor -- start 2024-05-20T18:05:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.942+0000 7f167b401740 1 -- start start 2024-05-20T18:05:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.943+0000 7f167b401740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd820390 0x5640ddbbed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.943+0000 7f167b401740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd7f74d0 0x5640ddbbf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.943+0000 7f167b401740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7ff6b0 0x5640ddbbf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.943+0000 7f167b401740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5640ddb24c40 con 0x5640dd7ff6b0 2024-05-20T18:05:11.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.943+0000 7f167b401740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5640ddacc580 con 0x5640dd7f74d0 2024-05-20T18:05:11.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.943+0000 7f167b401740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5640ddb20c00 con 0x5640dd820390 2024-05-20T18:05:11.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.943+0000 7f16637fe640 1 -- 172.21.15.129:0/2947430183 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f1664011870 con 0x7f166c11c0d0 2024-05-20T18:05:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.945+0000 7f16637fe640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16580a2f00 0x7f16580a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.945+0000 7f16637fe640 1 -- 172.21.15.129:0/2947430183 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f16640cfa80 con 0x7f166c11c0d0 2024-05-20T18:05:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.946+0000 7f1671af2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd820390 0x5640ddbbed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.947+0000 7f1671af2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd820390 0x5640ddbbed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:43600/0 (socket says 172.21.15.129:43600) 2024-05-20T18:05:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.947+0000 7f1671af2640 1 -- 172.21.15.129:0/1555264723 learned_addr learned my addr 172.21.15.129:0/1555264723 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.947+0000 7f1671af2640 1 --2- 172.21.15.129:0/1555264723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7ff6b0 0x5640ddbbf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.947+0000 7f1672af4640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16580a2f00 0x7f16580a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.948+0000 7f16722f3640 1 --2- 172.21.15.129:0/1555264723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd7f74d0 0x5640ddbbf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.948+0000 7f1671af2640 1 -- 172.21.15.129:0/1555264723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd820390 msgr2=0x5640ddbbed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.948+0000 7f1671af2640 1 --2- 172.21.15.129:0/1555264723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd820390 0x5640ddbbed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.948+0000 7f1671af2640 1 -- 172.21.15.129:0/1555264723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd7f74d0 msgr2=0x5640ddbbf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:11.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.948+0000 7f1671af2640 1 --2- 172.21.15.129:0/1555264723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd7f74d0 0x5640ddbbf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.948+0000 7f1671af2640 1 -- 172.21.15.129:0/1555264723 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640dd8383f0 con 0x5640dd7ff6b0 2024-05-20T18:05:11.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.950+0000 7f1671af2640 1 --2- 172.21.15.129:0/1555264723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7ff6b0 0x5640ddbbf7a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f166800ced0 tx=0x7f1668004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58e5773b1e91f7d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:11.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.950+0000 7f1672af4640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16580a2f00 0x7f16580a5410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f166400efd0 tx=0x7f166400cf70 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:11.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.950+0000 7f1660bfe640 1 -- 172.21.15.129:0/1555264723 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16680142d0 con 0x5640dd7ff6b0 2024-05-20T18:05:11.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.950+0000 7f1660bfe640 1 -- 172.21.15.129:0/1555264723 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16680119b0 con 0x5640dd7ff6b0 2024-05-20T18:05:11.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.950+0000 7f1660bfe640 1 -- 172.21.15.129:0/1555264723 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16680129f0 con 0x5640dd7ff6b0 2024-05-20T18:05:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.952+0000 7f167b401740 1 -- 172.21.15.129:0/1555264723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7ff6b0 msgr2=0x5640ddbbf7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.952+0000 7f167b401740 1 --2- 172.21.15.129:0/1555264723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7ff6b0 0x5640ddbbf7a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f166800ced0 tx=0x7f1668004730 comp rx=0 tx=0).stop 2024-05-20T18:05:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.954+0000 7f167b401740 1 -- 172.21.15.129:0/1555264723 shutdown_connections 2024-05-20T18:05:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.954+0000 7f167b401740 1 --2- 172.21.15.129:0/1555264723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd7f74d0 0x5640ddbbf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.954+0000 7f167b401740 1 --2- 172.21.15.129:0/1555264723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7ff6b0 0x5640ddbbf7a0 secure :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f166800ced0 tx=0x7f1668004730 comp rx=0 tx=0).stop 2024-05-20T18:05:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.954+0000 7f167b401740 1 --2- 172.21.15.129:0/1555264723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd820390 0x5640ddbbed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.954+0000 7f167b401740 1 -- 172.21.15.129:0/1555264723 >> 172.21.15.129:0/1555264723 conn(0x5640dd8144e0 msgr2=0x5640ddbbb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.961+0000 7f167b401740 1 -- 172.21.15.129:0/1555264723 shutdown_connections 2024-05-20T18:05:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.963+0000 7f167b401740 1 -- 172.21.15.129:0/1555264723 wait complete. 2024-05-20T18:05:11.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.964+0000 7f167b401740 1 -- start start 2024-05-20T18:05:11.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.964+0000 7f167b401740 1 Processor -- start 2024-05-20T18:05:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.965+0000 7f167b401740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd7f74d0 0x5640ddbcad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.965+0000 7f167b401740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd820390 0x5640ddbcb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.965+0000 7f167b401740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbc91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.965+0000 7f167b401740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5640ddbcc070 con 0x5640dd7def60 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.965+0000 7f167b401740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5640ddadfc40 con 0x5640dd820390 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.965+0000 7f167b401740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5640ddbcc1f0 con 0x5640dd7f74d0 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.965+0000 7f167b401740 10 client.0 _ll_register_callbacks cb 0x5640dd837be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.966+0000 7f1671af2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbc91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.966+0000 7f1671af2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbc91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60188/0 (socket says 172.21.15.129:60188) 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.966+0000 7f1671af2640 1 -- 172.21.15.129:0/3413726391 learned_addr learned my addr 172.21.15.129:0/3413726391 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.967+0000 7f1671af2640 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd7f74d0 0x5640ddbcad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.967+0000 7f1671af2640 1 -- 172.21.15.129:0/3413726391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd7f74d0 msgr2=0x5640ddbcad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.967+0000 7f1671af2640 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd7f74d0 0x5640ddbcad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.967+0000 7f1671af2640 1 -- 172.21.15.129:0/3413726391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd820390 msgr2=0x5640ddbcb280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:05:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.967+0000 7f1671af2640 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd820390 0x5640ddbcb280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:11.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.967+0000 7f1671af2640 1 -- 172.21.15.129:0/3413726391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640dd9027d0 con 0x5640dd7def60 2024-05-20T18:05:11.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.973+0000 7f1671af2640 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbc91e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f166800ca00 tx=0x7f1668012590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8caceadd265e51a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:11.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.974+0000 7f1655ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:05:11.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.974+0000 7f1655ffb640 1 -- 172.21.15.129:0/3413726391 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16680117d0 con 0x5640dd7def60 2024-05-20T18:05:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.974+0000 7f167b401740 1 -- 172.21.15.129:0/3413726391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16680119b0 con 0x5640dd7def60 2024-05-20T18:05:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.974+0000 7f1655ffb640 1 -- 172.21.15.129:0/3413726391 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f1668004030 con 0x5640dd7def60 2024-05-20T18:05:11.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.975+0000 7f1655ffb640 1 -- 172.21.15.129:0/3413726391 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1668004230 con 0x5640dd7def60 2024-05-20T18:05:11.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.975+0000 7f167b401740 10 client.16377 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:11.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.975+0000 7f167b401740 1 -- 172.21.15.129:0/3413726391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5640ddbd8920 con 0x5640dd7def60 2024-05-20T18:05:11.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.980+0000 7f1655ffb640 1 -- 172.21.15.129:0/3413726391 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16680119b0 con 0x5640dd7def60 2024-05-20T18:05:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.980+0000 7f1655ffb640 1 -- 172.21.15.129:0/3413726391 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f1668003aa0 con 0x5640dd7def60 2024-05-20T18:05:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.980+0000 7f167b401740 10 client.16377 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.980+0000 7f167b401740 10 client.16377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.980+0000 7f167b401740 20 client.16377 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.980+0000 7f167b401740 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5640dd7b43a0 0x5640ddbdc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.980+0000 7f167b401740 4 client.16377 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.980+0000 7f167b401740 1 -- 172.21.15.129:0/3413726391 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640ddb32370 con 0x5640dd7b43a0 2024-05-20T18:05:11.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.982+0000 7f16722f3640 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5640dd7b43a0 0x5640ddbdc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:11.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:11.999+0000 7f16722f3640 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5640dd7b43a0 0x5640ddbdc180 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ced89772201e819d server_cookie=edaf4e7ce9da6d13 in_seq=0 out_seq=0 2024-05-20T18:05:11.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.000+0000 7f1655ffb640 10 client.16377 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:12.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.002+0000 7f1655ffb640 1 -- 172.21.15.129:0/3413726391 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5640ddb32370 con 0x5640dd7b43a0 2024-05-20T18:05:12.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.002+0000 7f1655ffb640 10 client.16377 handle_command_reply: tid=0 2024-05-20T18:05:12.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.002+0000 7f167b401740 1 client.16377 shutdown 2024-05-20T18:05:12.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.007+0000 7f167b401740 1 -- 172.21.15.129:0/3413726391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 msgr2=0x5640ddbc91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:12.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.007+0000 7f167b401740 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbc91e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f166800ca00 tx=0x7f1668012590 comp rx=0 tx=0).stop 2024-05-20T18:05:12.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.010+0000 7f167b401740 1 -- 172.21.15.129:0/3413726391 shutdown_connections 2024-05-20T18:05:12.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.010+0000 7f167b401740 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5640dd7b43a0 0x5640ddbdc180 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.010+0000 7f167b401740 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd820390 0x5640ddbcb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.010+0000 7f167b401740 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbc91e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.010+0000 7f167b401740 1 --2- 172.21.15.129:0/3413726391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd7f74d0 0x5640ddbcad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.010+0000 7f167b401740 1 -- 172.21.15.129:0/3413726391 >> 172.21.15.129:0/3413726391 conn(0x5640dd8144e0 msgr2=0x5640ddbc0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:12.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.010+0000 7f1655ffb640 0 client.16377 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:12.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.014+0000 7f167b401740 1 -- 172.21.15.129:0/3413726391 shutdown_connections 2024-05-20T18:05:12.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.015+0000 7f167b401740 1 -- 172.21.15.129:0/3413726391 wait complete. 2024-05-20T18:05:12.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.016+0000 7f167b401740 20 client.16377 trim_cache size 0 max 16384 2024-05-20T18:05:12.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.058+0000 7f167b401740 1 Processor -- start 2024-05-20T18:05:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.059+0000 7f167b401740 1 -- start start 2024-05-20T18:05:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.059+0000 7f167b401740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd7def60 0x5640ddbbab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.059+0000 7f167b401740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7f74d0 0x5640ddbbb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.059+0000 7f167b401740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd820390 0x5640ddbc6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.059+0000 7f167b401740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f165000c8b0 con 0x5640dd7f74d0 2024-05-20T18:05:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.059+0000 7f167b401740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f165000f130 con 0x5640dd820390 2024-05-20T18:05:12.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.059+0000 7f167b401740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f165000cf20 con 0x5640dd7def60 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.059+0000 7f16722f3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7f74d0 0x5640ddbbb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.060+0000 7f16722f3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7f74d0 0x5640ddbbb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60204/0 (socket says 172.21.15.129:60204) 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.060+0000 7f16722f3640 1 -- 172.21.15.129:0/2246337334 learned_addr learned my addr 172.21.15.129:0/2246337334 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.060+0000 7f16722f3640 1 -- 172.21.15.129:0/2246337334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd7def60 msgr2=0x5640ddbbab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.060+0000 7f1671af2640 1 --2- 172.21.15.129:0/2246337334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd7def60 0x5640ddbbab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.060+0000 7f16722f3640 1 --2- 172.21.15.129:0/2246337334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd7def60 0x5640ddbbab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.060+0000 7f16722f3640 1 -- 172.21.15.129:0/2246337334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd820390 msgr2=0x5640ddbc6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.060+0000 7f16722f3640 1 --2- 172.21.15.129:0/2246337334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd820390 0x5640ddbc6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.060+0000 7f16722f3640 1 -- 172.21.15.129:0/2246337334 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640ddbb62a0 con 0x5640dd7f74d0 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.060+0000 7f1671af2640 1 --2- 172.21.15.129:0/2246337334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd7def60 0x5640ddbbab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.061+0000 7f16722f3640 1 --2- 172.21.15.129:0/2246337334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7f74d0 0x5640ddbbb090 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f165c033040 tx=0x7f165c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f415d40ab9d4b8b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.061+0000 7f1657fff640 1 -- 172.21.15.129:0/2246337334 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c004930 con 0x5640dd7f74d0 2024-05-20T18:05:12.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.061+0000 7f1657fff640 1 -- 172.21.15.129:0/2246337334 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f165c004ad0 con 0x5640dd7f74d0 2024-05-20T18:05:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.061+0000 7f1657fff640 1 -- 172.21.15.129:0/2246337334 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c043250 con 0x5640dd7f74d0 2024-05-20T18:05:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.062+0000 7f167b401740 1 -- 172.21.15.129:0/2246337334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7f74d0 msgr2=0x5640ddbbb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.062+0000 7f167b401740 1 --2- 172.21.15.129:0/2246337334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7f74d0 0x5640ddbbb090 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f165c033040 tx=0x7f165c000bc0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.062+0000 7f167b401740 1 -- 172.21.15.129:0/2246337334 shutdown_connections 2024-05-20T18:05:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.062+0000 7f167b401740 1 --2- 172.21.15.129:0/2246337334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd820390 0x5640ddbc6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.062+0000 7f167b401740 1 --2- 172.21.15.129:0/2246337334 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7f74d0 0x5640ddbbb090 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.062+0000 7f167b401740 1 --2- 172.21.15.129:0/2246337334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd7def60 0x5640ddbbab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.062+0000 7f167b401740 1 -- 172.21.15.129:0/2246337334 >> 172.21.15.129:0/2246337334 conn(0x5640dd8144e0 msgr2=0x5640ddbc7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.062+0000 7f167b401740 1 -- 172.21.15.129:0/2246337334 shutdown_connections 2024-05-20T18:05:12.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.062+0000 7f167b401740 1 -- 172.21.15.129:0/2246337334 wait complete. 2024-05-20T18:05:12.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.063+0000 7f167b401740 1 -- start start 2024-05-20T18:05:12.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.063+0000 7f167b401740 1 Processor -- start 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.063+0000 7f167b401740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.063+0000 7f167b401740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd7f74d0 0x5640ddbce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f167b401740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd820390 0x5640ddbced60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f1671af2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f1671af2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60214/0 (socket says 172.21.15.129:60214) 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f1671af2640 1 -- 172.21.15.129:0/1035452274 learned_addr learned my addr 172.21.15.129:0/1035452274 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f167b401740 1 -- 172.21.15.129:0/1035452274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f165000d5f0 con 0x5640dd7def60 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f167b401740 1 -- 172.21.15.129:0/1035452274 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f165000ea60 con 0x5640dd7f74d0 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f167b401740 1 -- 172.21.15.129:0/1035452274 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f165000dcc0 con 0x5640dd820390 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f167b401740 10 client.0 _ll_register_callbacks cb 0x7f1668014690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f1671af2640 1 -- 172.21.15.129:0/1035452274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd820390 msgr2=0x5640ddbced60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f1671af2640 1 --2- 172.21.15.129:0/1035452274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd820390 0x5640ddbced60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f1671af2640 1 -- 172.21.15.129:0/1035452274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd7f74d0 msgr2=0x5640ddbce820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:05:12.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f1671af2640 1 --2- 172.21.15.129:0/1035452274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd7f74d0 0x5640ddbce820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.064+0000 7f1671af2640 1 -- 172.21.15.129:0/1035452274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640ddc42620 con 0x5640dd7def60 2024-05-20T18:05:12.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.065+0000 7f1671af2640 1 --2- 172.21.15.129:0/1035452274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbce2e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f16680124c0 tx=0x7f1668011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9b9a5d64f9dd1e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:12.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.065+0000 7f1655ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:05:12.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.065+0000 7f1655ffb640 1 -- 172.21.15.129:0/1035452274 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668011e70 con 0x5640dd7def60 2024-05-20T18:05:12.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.065+0000 7f1655ffb640 1 -- 172.21.15.129:0/1035452274 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f1668024040 con 0x5640dd7def60 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.065+0000 7f167b401740 1 -- 172.21.15.129:0/1035452274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f165c004ad0 con 0x5640dd7def60 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.065+0000 7f1655ffb640 1 -- 172.21.15.129:0/1035452274 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f166800f580 con 0x5640dd7def60 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.065+0000 7f167b401740 10 client.16383 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.065+0000 7f167b401740 1 -- 172.21.15.129:0/1035452274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5640ddc437f0 con 0x5640dd7def60 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.065+0000 7f1655ffb640 1 -- 172.21.15.129:0/1035452274 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f165c004ad0 con 0x5640dd7def60 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.066+0000 7f1655ffb640 1 -- 172.21.15.129:0/1035452274 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f166802e070 con 0x5640dd7def60 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.066+0000 7f167b401740 10 client.16383 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.066+0000 7f167b401740 10 client.16383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.066+0000 7f167b401740 20 client.16383 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.066+0000 7f167b401740 1 --2- 172.21.15.129:0/1035452274 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5640ddc43e80 0x5640ddc462a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.066+0000 7f167b401740 4 client.16383 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.066+0000 7f167b401740 1 -- 172.21.15.129:0/1035452274 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5640ddb2c6c0 con 0x5640ddc43e80 2024-05-20T18:05:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.067+0000 7f16722f3640 1 --2- 172.21.15.129:0/1035452274 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5640ddc43e80 0x5640ddc462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:12.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.071+0000 7f1655ffb640 10 client.16383 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:12.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.071+0000 7f16722f3640 1 --2- 172.21.15.129:0/1035452274 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5640ddc43e80 0x5640ddc462a0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d135f7d4176e865e server_cookie=853b471658852290 in_seq=0 out_seq=0 2024-05-20T18:05:12.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f1655ffb640 1 -- 172.21.15.129:0/1035452274 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5640ddb2c6c0 con 0x5640ddc43e80 2024-05-20T18:05:12.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f1655ffb640 10 client.16383 handle_command_reply: tid=0 2024-05-20T18:05:12.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f167b401740 1 client.16383 shutdown 2024-05-20T18:05:12.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f167b401740 1 -- 172.21.15.129:0/1035452274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 msgr2=0x5640ddbce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:12.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f167b401740 1 --2- 172.21.15.129:0/1035452274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbce2e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f16680124c0 tx=0x7f1668011ae0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f167b401740 1 -- 172.21.15.129:0/1035452274 shutdown_connections 2024-05-20T18:05:12.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f167b401740 1 --2- 172.21.15.129:0/1035452274 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5640ddc43e80 0x5640ddc462a0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f167b401740 1 --2- 172.21.15.129:0/1035452274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640dd7f74d0 0x5640ddbce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f1655ffb640 0 client.16383 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:12.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f167b401740 1 --2- 172.21.15.129:0/1035452274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5640dd820390 0x5640ddbced60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f167b401740 1 --2- 172.21.15.129:0/1035452274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5640dd7def60 0x5640ddbce2e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.072+0000 7f167b401740 1 -- 172.21.15.129:0/1035452274 >> 172.21.15.129:0/1035452274 conn(0x5640dd8144e0 msgr2=0x5640ddbc8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:12.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.073+0000 7f167b401740 1 -- 172.21.15.129:0/1035452274 shutdown_connections 2024-05-20T18:05:12.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.073+0000 7f167b401740 1 -- 172.21.15.129:0/1035452274 wait complete. 2024-05-20T18:05:12.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.073+0000 7f167b401740 20 client.16383 trim_cache size 0 max 16384 2024-05-20T18:05:12.073 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:05:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.074+0000 7f1674557640 1 -- 172.21.15.129:0/2947430183 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16580a2f00 msgr2=0x7f16580a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.074+0000 7f1674557640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16580a2f00 0x7f16580a5410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f166400efd0 tx=0x7f166400cf70 comp rx=0 tx=0).stop 2024-05-20T18:05:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.074+0000 7f1674557640 1 -- 172.21.15.129:0/2947430183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c11c0d0 msgr2=0x7f166c113e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.074+0000 7f1674557640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c11c0d0 0x7f166c113e40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f166400eaa0 tx=0x7f166400ef60 comp rx=0 tx=0).stop 2024-05-20T18:05:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.074+0000 7f1674557640 1 -- 172.21.15.129:0/2947430183 shutdown_connections 2024-05-20T18:05:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.074+0000 7f1674557640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16580a2f00 0x7f16580a5410 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.074+0000 7f1674557640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f166c11c0d0 0x7f166c113e40 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.074+0000 7f1674557640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f166c119770 0x7f166c113900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.074+0000 7f1674557640 1 --2- 172.21.15.129:0/2947430183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f166c109060 0x7f166c1133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.074+0000 7f1674557640 1 -- 172.21.15.129:0/2947430183 >> 172.21.15.129:0/2947430183 conn(0x7f166c06d750 msgr2=0x7f166c119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.074+0000 7f1674557640 1 -- 172.21.15.129:0/2947430183 shutdown_connections 2024-05-20T18:05:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:12.075+0000 7f1674557640 1 -- 172.21.15.129:0/2947430183 wait complete. 2024-05-20T18:05:12.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:12 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3911075225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:05:12.916 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:12 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3911075225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:05:12.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:12 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3911075225' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:05:13.091 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:05:13.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:05:13.091 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:05:13.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.412+0000 7f830308b640 1 -- 172.21.15.129:0/3767307015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82fc101990 msgr2=0x7f82fc101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:13.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.412+0000 7f830308b640 1 --2- 172.21.15.129:0/3767307015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82fc101990 0x7f82fc101da0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009900 tx=0x7f82f002a440 comp rx=0 tx=0).stop 2024-05-20T18:05:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.418+0000 7f830308b640 1 -- 172.21.15.129:0/3767307015 shutdown_connections 2024-05-20T18:05:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.418+0000 7f830308b640 1 --2- 172.21.15.129:0/3767307015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82fc1063e0 0x7f82fc1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.418+0000 7f830308b640 1 --2- 172.21.15.129:0/3767307015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82fc106e10 0x7f82fc1072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.418+0000 7f830308b640 1 --2- 172.21.15.129:0/3767307015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82fc101990 0x7f82fc101da0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.418+0000 7f830308b640 1 -- 172.21.15.129:0/3767307015 >> 172.21.15.129:0/3767307015 conn(0x7f82fc0770b0 msgr2=0x7f82fc0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:13.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.429+0000 7f830308b640 1 -- 172.21.15.129:0/3767307015 shutdown_connections 2024-05-20T18:05:13.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.433+0000 7f830308b640 1 -- 172.21.15.129:0/3767307015 wait complete. 2024-05-20T18:05:13.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.434+0000 7f830308b640 1 Processor -- start 2024-05-20T18:05:13.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.434+0000 7f830308b640 1 -- start start 2024-05-20T18:05:13.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.435+0000 7f830308b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82fc101990 0x7f82fc19ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.436+0000 7f8300e27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82fc101990 0x7f82fc19ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.436+0000 7f8300e27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82fc101990 0x7f82fc19ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55238/0 (socket says 172.21.15.129:55238) 2024-05-20T18:05:13.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.439+0000 7f830308b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82fc1063e0 0x7f82fc1a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.439+0000 7f830308b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82fc106e10 0x7f82fc1a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.439+0000 7f830308b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc1194d0 con 0x7f82fc1063e0 2024-05-20T18:05:13.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.439+0000 7f830308b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc119350 con 0x7f82fc101990 2024-05-20T18:05:13.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.439+0000 7f830308b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc119650 con 0x7f82fc106e10 2024-05-20T18:05:13.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.439+0000 7f8300e27640 1 -- 172.21.15.129:0/1690066892 learned_addr learned my addr 172.21.15.129:0/1690066892 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:13.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.440+0000 7f8300e27640 1 -- 172.21.15.129:0/1690066892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82fc106e10 msgr2=0x7f82fc1a4730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:05:13.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.440+0000 7f8300e27640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82fc106e10 0x7f82fc1a4730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.440+0000 7f8300e27640 1 -- 172.21.15.129:0/1690066892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82fc1063e0 msgr2=0x7f82fc1a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:13.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.440+0000 7f8300e27640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82fc1063e0 0x7f82fc1a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.440+0000 7f8300e27640 1 -- 172.21.15.129:0/1690066892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82fc1a4d70 con 0x7f82fc101990 2024-05-20T18:05:13.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.441+0000 7f8300e27640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82fc101990 0x7f82fc19ff10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009a20 tx=0x7f82f0003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:13.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.442+0000 7f82f9ffb640 1 -- 172.21.15.129:0/1690066892 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f003f070 con 0x7f82fc101990 2024-05-20T18:05:13.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.442+0000 7f82f9ffb640 1 -- 172.21.15.129:0/1690066892 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f0003e40 con 0x7f82fc101990 2024-05-20T18:05:13.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.442+0000 7f82f9ffb640 1 -- 172.21.15.129:0/1690066892 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0035030 con 0x7f82fc101990 2024-05-20T18:05:13.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.442+0000 7f830308b640 1 -- 172.21.15.129:0/1690066892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82fc1a5000 con 0x7f82fc101990 2024-05-20T18:05:13.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.442+0000 7f830308b640 1 -- 172.21.15.129:0/1690066892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82fc1a54a0 con 0x7f82fc101990 2024-05-20T18:05:13.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.447+0000 7f830a00d740 1 Processor -- start 2024-05-20T18:05:13.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.448+0000 7f830a00d740 1 -- start start 2024-05-20T18:05:13.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.448+0000 7f830a00d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d62b410 0x557b9d9ca620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.448+0000 7f830a00d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 0x557b9d9cab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.448+0000 7f830a00d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d60a730 0x557b9d9cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.448+0000 7f830a00d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557b9d92f990 con 0x557b9d62b410 2024-05-20T18:05:13.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.448+0000 7f830a00d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557b9d8d75b0 con 0x557b9d60a730 2024-05-20T18:05:13.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.448+0000 7f830a00d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557b9d92b950 con 0x557b9d602550 2024-05-20T18:05:13.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.449+0000 7f82f9ffb640 1 -- 172.21.15.129:0/1690066892 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f82f0042070 con 0x7f82fc101990 2024-05-20T18:05:13.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.449+0000 7f8300e27640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 0x557b9d9cab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.450+0000 7f8300e27640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 0x557b9d9cab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:43678/0 (socket says 172.21.15.129:43678) 2024-05-20T18:05:13.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.450+0000 7f8300e27640 1 -- 172.21.15.129:0/715884840 learned_addr learned my addr 172.21.15.129:0/715884840 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:13.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.451+0000 7f8300e27640 1 -- 172.21.15.129:0/715884840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d60a730 msgr2=0x557b9d9cb0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:05:13.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.451+0000 7f8300e27640 1 --2- 172.21.15.129:0/715884840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d60a730 0x557b9d9cb0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.451+0000 7f8300e27640 1 -- 172.21.15.129:0/715884840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d62b410 msgr2=0x557b9d9ca620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:05:13.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.451+0000 7f8300e27640 1 --2- 172.21.15.129:0/715884840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d62b410 0x557b9d9ca620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.451+0000 7f8300e27640 1 -- 172.21.15.129:0/715884840 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b9d70b7e0 con 0x557b9d602550 2024-05-20T18:05:13.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.452+0000 7f8300e27640 1 --2- 172.21.15.129:0/715884840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 0x557b9d9cab60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f82f00ed3f0 tx=0x7f82f00f3f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fac736556e674896 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:13.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.455+0000 7f82f9ffb640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f82e40a2e40 0x7f82e40a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.456+0000 7f82f9ffb640 1 -- 172.21.15.129:0/1690066892 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f82f00efdd0 con 0x7f82fc101990 2024-05-20T18:05:13.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.456+0000 7f82e2ffd640 1 -- 172.21.15.129:0/715884840 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f00ff070 con 0x557b9d602550 2024-05-20T18:05:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.457+0000 7f82e2ffd640 1 -- 172.21.15.129:0/715884840 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f00f8e20 con 0x557b9d602550 2024-05-20T18:05:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.457+0000 7f82e2ffd640 1 -- 172.21.15.129:0/715884840 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f00f4a70 con 0x557b9d602550 2024-05-20T18:05:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.459+0000 7f830a00d740 1 -- 172.21.15.129:0/715884840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 msgr2=0x557b9d9cab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.459+0000 7f830a00d740 1 --2- 172.21.15.129:0/715884840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 0x557b9d9cab60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f82f00ed3f0 tx=0x7f82f00f3f90 comp rx=0 tx=0).stop 2024-05-20T18:05:13.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.464+0000 7f8301628640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f82e40a2e40 0x7f82e40a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.470+0000 7f8301628640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f82e40a2e40 0x7f82e40a5350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x557b9d9c7790 tx=0x7f82ec006c20 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:13.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.470+0000 7f830a00d740 1 -- 172.21.15.129:0/715884840 shutdown_connections 2024-05-20T18:05:13.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.470+0000 7f830a00d740 1 --2- 172.21.15.129:0/715884840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d60a730 0x557b9d9cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.470+0000 7f830a00d740 1 --2- 172.21.15.129:0/715884840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 0x557b9d9cab60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.470+0000 7f830a00d740 1 --2- 172.21.15.129:0/715884840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d62b410 0x557b9d9ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.470+0000 7f830a00d740 1 -- 172.21.15.129:0/715884840 >> 172.21.15.129:0/715884840 conn(0x557b9d61f560 msgr2=0x557b9d9c6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:13.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.471+0000 7f830a00d740 1 -- 172.21.15.129:0/715884840 shutdown_connections 2024-05-20T18:05:13.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.474+0000 7f830a00d740 1 -- 172.21.15.129:0/715884840 wait complete. 2024-05-20T18:05:13.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.475+0000 7f830a00d740 1 -- start start 2024-05-20T18:05:13.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.475+0000 7f830a00d740 1 Processor -- start 2024-05-20T18:05:13.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.477+0000 7f830a00d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 0x557b9d9d6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.477+0000 7f830a00d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d60a730 0x557b9d9d6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.477+0000 7f830a00d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 0x557b9d9d4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.477+0000 7f830a00d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557b9d9e8740 con 0x557b9d60a730 2024-05-20T18:05:13.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.477+0000 7f830a00d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557b9d8eab40 con 0x557b9d602550 2024-05-20T18:05:13.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.477+0000 7f830a00d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557b9d9e88c0 con 0x557b9d62b410 2024-05-20T18:05:13.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.478+0000 7f830a00d740 10 client.0 _ll_register_callbacks cb 0x557b9d8df0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:13.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.480+0000 7f8300e27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 0x557b9d9d6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.480+0000 7f8301628640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 0x557b9d9d4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.481+0000 7f82fbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d60a730 0x557b9d9d6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.481+0000 7f8301628640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 0x557b9d9d4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:43694/0 (socket says 172.21.15.129:43694) 2024-05-20T18:05:13.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.481+0000 7f8301628640 1 -- 172.21.15.129:0/3445133501 learned_addr learned my addr 172.21.15.129:0/3445133501 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:13.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.482+0000 7f8301628640 1 -- 172.21.15.129:0/3445133501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 msgr2=0x557b9d9d6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:13.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.482+0000 7f8301628640 1 --2- 172.21.15.129:0/3445133501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 0x557b9d9d6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.482+0000 7f8301628640 1 -- 172.21.15.129:0/3445133501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d60a730 msgr2=0x557b9d9d6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:13.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.482+0000 7f8301628640 1 --2- 172.21.15.129:0/3445133501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d60a730 0x557b9d9d6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.482+0000 7f8301628640 1 -- 172.21.15.129:0/3445133501 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b9d76bf70 con 0x557b9d62b410 2024-05-20T18:05:13.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.483+0000 7f8301628640 1 --2- 172.21.15.129:0/3445133501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 0x557b9d9d4b70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f82ec013f40 tx=0x7f82ec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38689d5456c07516 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:13.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.490+0000 7f82dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:05:13.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.490+0000 7f82dbfff640 1 -- 172.21.15.129:0/3445133501 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec01d5f0 con 0x557b9d62b410 2024-05-20T18:05:13.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.490+0000 7f830a00d740 1 -- 172.21.15.129:0/3445133501 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82f00f8e20 con 0x557b9d62b410 2024-05-20T18:05:13.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.490+0000 7f82dbfff640 1 -- 172.21.15.129:0/3445133501 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f82ec01d790 con 0x557b9d62b410 2024-05-20T18:05:13.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.491+0000 7f82dbfff640 1 -- 172.21.15.129:0/3445133501 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f82ec02b050 con 0x557b9d62b410 2024-05-20T18:05:13.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.492+0000 7f830a00d740 10 client.26167 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:13.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.492+0000 7f830a00d740 1 -- 172.21.15.129:0/3445133501 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557b9d603170 con 0x557b9d62b410 2024-05-20T18:05:13.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.493+0000 7f82dbfff640 1 -- 172.21.15.129:0/3445133501 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f00f8e20 con 0x557b9d62b410 2024-05-20T18:05:13.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.495+0000 7f82dbfff640 1 -- 172.21.15.129:0/3445133501 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f82ec012070 con 0x557b9d62b410 2024-05-20T18:05:13.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.495+0000 7f830a00d740 10 client.26167 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:13.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.495+0000 7f830a00d740 10 client.26167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:13.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.495+0000 7f830a00d740 20 client.26167 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:13.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.495+0000 7f830a00d740 1 --2- 172.21.15.129:0/3445133501 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557b9d5ea1f0 0x557b9d9f84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.495+0000 7f830a00d740 4 client.26167 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:13.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.495+0000 7f830a00d740 1 -- 172.21.15.129:0/3445133501 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b9d93d0c0 con 0x557b9d5ea1f0 2024-05-20T18:05:13.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.496+0000 7f8300e27640 1 --2- 172.21.15.129:0/3445133501 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557b9d5ea1f0 0x557b9d9f84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.497+0000 7f8300e27640 1 --2- 172.21.15.129:0/3445133501 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557b9d5ea1f0 0x557b9d9f84f0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14775193793c65c8 server_cookie=f0ca12e502186e52 in_seq=0 out_seq=0 2024-05-20T18:05:13.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.497+0000 7f82dbfff640 10 client.26167 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:13.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.499+0000 7f82dbfff640 1 -- 172.21.15.129:0/3445133501 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557b9d93d0c0 con 0x557b9d5ea1f0 2024-05-20T18:05:13.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.499+0000 7f82dbfff640 10 client.26167 handle_command_reply: tid=0 2024-05-20T18:05:13.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.499+0000 7f830a00d740 1 client.26167 shutdown 2024-05-20T18:05:13.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.501+0000 7f830a00d740 1 -- 172.21.15.129:0/3445133501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 msgr2=0x557b9d9d4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:13.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.501+0000 7f830a00d740 1 --2- 172.21.15.129:0/3445133501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 0x557b9d9d4b70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f82ec013f40 tx=0x7f82ec0046c0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.501+0000 7f830a00d740 1 -- 172.21.15.129:0/3445133501 shutdown_connections 2024-05-20T18:05:13.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.501+0000 7f830a00d740 1 --2- 172.21.15.129:0/3445133501 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557b9d5ea1f0 0x557b9d9f84f0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.501+0000 7f830a00d740 1 --2- 172.21.15.129:0/3445133501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 0x557b9d9d4b70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.501+0000 7f830a00d740 1 --2- 172.21.15.129:0/3445133501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d60a730 0x557b9d9d6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.501+0000 7f830a00d740 1 --2- 172.21.15.129:0/3445133501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 0x557b9d9d6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.501+0000 7f830a00d740 1 -- 172.21.15.129:0/3445133501 >> 172.21.15.129:0/3445133501 conn(0x557b9d61f560 msgr2=0x557b9d9ca5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:13.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.501+0000 7f82dbfff640 0 client.26167 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:13.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.502+0000 7f830a00d740 1 -- 172.21.15.129:0/3445133501 shutdown_connections 2024-05-20T18:05:13.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.502+0000 7f830a00d740 1 -- 172.21.15.129:0/3445133501 wait complete. 2024-05-20T18:05:13.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.504+0000 7f830a00d740 20 client.26167 trim_cache size 0 max 16384 2024-05-20T18:05:13.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.567+0000 7f830a00d740 1 Processor -- start 2024-05-20T18:05:13.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.569+0000 7f830a00d740 1 -- start start 2024-05-20T18:05:13.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.569+0000 7f830a00d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 0x557b9d9ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.570+0000 7f830a00d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d60a730 0x557b9d9c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.570+0000 7f830a00d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 0x557b9d9fdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.570+0000 7f830a00d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f82e800c7e0 con 0x557b9d60a730 2024-05-20T18:05:13.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.570+0000 7f830a00d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f82e800efb0 con 0x557b9d602550 2024-05-20T18:05:13.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.570+0000 7f830a00d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82e800cda0 con 0x557b9d62b410 2024-05-20T18:05:13.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.571+0000 7f8300e27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 0x557b9d9ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.572+0000 7f8300e27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 0x557b9d9ce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55254/0 (socket says 172.21.15.129:55254) 2024-05-20T18:05:13.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.572+0000 7f8300e27640 1 -- 172.21.15.129:0/11997013 learned_addr learned my addr 172.21.15.129:0/11997013 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:13.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.573+0000 7f8301628640 1 --2- 172.21.15.129:0/11997013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 0x557b9d9fdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.574+0000 7f82fbfff640 1 --2- 172.21.15.129:0/11997013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d60a730 0x557b9d9c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.578+0000 7f82fbfff640 1 -- 172.21.15.129:0/11997013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 msgr2=0x557b9d9fdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:13.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.578+0000 7f82fbfff640 1 --2- 172.21.15.129:0/11997013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 0x557b9d9fdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.578+0000 7f82fbfff640 1 -- 172.21.15.129:0/11997013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 msgr2=0x557b9d9ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:13.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.578+0000 7f82fbfff640 1 --2- 172.21.15.129:0/11997013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 0x557b9d9ce980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.578+0000 7f82fbfff640 1 -- 172.21.15.129:0/11997013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b9d9cc110 con 0x557b9d60a730 2024-05-20T18:05:13.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.582+0000 7f8300e27640 1 --2- 172.21.15.129:0/11997013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 0x557b9d9ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:05:13.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.583+0000 7f82fbfff640 1 --2- 172.21.15.129:0/11997013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d60a730 0x557b9d9c6ea0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0fffc0 tx=0x7f82f400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1a773b36aa568a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:13.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.587+0000 7f82e27fc640 1 -- 172.21.15.129:0/11997013 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f401a070 con 0x557b9d60a730 2024-05-20T18:05:13.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.587+0000 7f82e27fc640 1 -- 172.21.15.129:0/11997013 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f4009e30 con 0x557b9d60a730 2024-05-20T18:05:13.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.588+0000 7f82e27fc640 1 -- 172.21.15.129:0/11997013 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f400a760 con 0x557b9d60a730 2024-05-20T18:05:13.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.588+0000 7f830a00d740 1 -- 172.21.15.129:0/11997013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d60a730 msgr2=0x557b9d9c6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:13.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.588+0000 7f830a00d740 1 --2- 172.21.15.129:0/11997013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d60a730 0x557b9d9c6ea0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0fffc0 tx=0x7f82f400c7f0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.588+0000 7f830a00d740 1 -- 172.21.15.129:0/11997013 shutdown_connections 2024-05-20T18:05:13.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.588+0000 7f830a00d740 1 --2- 172.21.15.129:0/11997013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d62b410 0x557b9d9fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.588+0000 7f830a00d740 1 --2- 172.21.15.129:0/11997013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d60a730 0x557b9d9c6ea0 secure :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0fffc0 tx=0x7f82f400c7f0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.588+0000 7f830a00d740 1 --2- 172.21.15.129:0/11997013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d602550 0x557b9d9ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.588+0000 7f830a00d740 1 -- 172.21.15.129:0/11997013 >> 172.21.15.129:0/11997013 conn(0x557b9d61f560 msgr2=0x557b9d9c6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:13.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.591+0000 7f830a00d740 1 -- 172.21.15.129:0/11997013 shutdown_connections 2024-05-20T18:05:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.593+0000 7f830a00d740 1 -- 172.21.15.129:0/11997013 wait complete. 2024-05-20T18:05:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.594+0000 7f830a00d740 1 -- start start 2024-05-20T18:05:13.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.595+0000 7f830a00d740 1 Processor -- start 2024-05-20T18:05:13.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.600+0000 7f830a00d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 0x557b9d9edce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.601+0000 7f8300e27640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 0x557b9d9edce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.602+0000 7f8300e27640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 0x557b9d9edce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:43708/0 (socket says 172.21.15.129:43708) 2024-05-20T18:05:13.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.604+0000 7f830a00d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d62b410 0x557b9d9ee220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.604+0000 7f830a00d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d5bf630 0x557b9d9eb160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.604+0000 7f830a00d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f82e800d470 con 0x557b9d62b410 2024-05-20T18:05:13.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.604+0000 7f830a00d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f82e800e8e0 con 0x557b9d5bf630 2024-05-20T18:05:13.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.604+0000 7f830a00d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82e800db40 con 0x557b9d602550 2024-05-20T18:05:13.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.604+0000 7f830a00d740 10 client.0 _ll_register_callbacks cb 0x7f82ec0092e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:13.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.604+0000 7f8300e27640 1 -- 172.21.15.129:0/2780494165 learned_addr learned my addr 172.21.15.129:0/2780494165 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:13.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.605+0000 7f82fbfff640 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d62b410 0x557b9d9ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.605+0000 7f82fbfff640 1 -- 172.21.15.129:0/2780494165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 msgr2=0x557b9d9edce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:13.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.605+0000 7f82fbfff640 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 0x557b9d9edce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.605+0000 7f82fbfff640 1 -- 172.21.15.129:0/2780494165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d5bf630 msgr2=0x557b9d9eb160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:05:13.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.605+0000 7f82fbfff640 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d5bf630 0x557b9d9eb160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.605+0000 7f82fbfff640 1 -- 172.21.15.129:0/2780494165 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b9d9eb8d0 con 0x557b9d62b410 2024-05-20T18:05:13.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.606+0000 7f82fbfff640 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d62b410 0x557b9d9ee220 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f82f400b000 tx=0x7f82f4015ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d64a2519e4bfdb server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:13.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.606+0000 7f82dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:05:13.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.607+0000 7f82dbfff640 1 -- 172.21.15.129:0/2780494165 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f401a040 con 0x557b9d62b410 2024-05-20T18:05:13.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.607+0000 7f82dbfff640 1 -- 172.21.15.129:0/2780494165 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f82f4012040 con 0x557b9d62b410 2024-05-20T18:05:13.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.607+0000 7f830a00d740 1 -- 172.21.15.129:0/2780494165 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82f4009e30 con 0x557b9d62b410 2024-05-20T18:05:13.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.608+0000 7f82dbfff640 1 -- 172.21.15.129:0/2780494165 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f82f400a4e0 con 0x557b9d62b410 2024-05-20T18:05:13.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.609+0000 7f830a00d740 10 client.16392 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:13.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.609+0000 7f830a00d740 1 -- 172.21.15.129:0/2780494165 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557b9da603f0 con 0x557b9d62b410 2024-05-20T18:05:13.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.613+0000 7f82dbfff640 1 -- 172.21.15.129:0/2780494165 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f4009e30 con 0x557b9d62b410 2024-05-20T18:05:13.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.613+0000 7f82dbfff640 1 -- 172.21.15.129:0/2780494165 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f82f401f3b0 con 0x557b9d62b410 2024-05-20T18:05:13.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.613+0000 7f830a00d740 10 client.16392 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:13.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.614+0000 7f830a00d740 10 client.16392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:13.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.614+0000 7f830a00d740 20 client.16392 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:13.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.614+0000 7f830a00d740 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557b9da60910 0x557b9da62d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:13.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.614+0000 7f830a00d740 4 client.16392 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:05:13.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.614+0000 7f830a00d740 1 -- 172.21.15.129:0/2780494165 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557b9d937410 con 0x557b9da60910 2024-05-20T18:05:13.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.615+0000 7f8300e27640 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557b9da60910 0x557b9da62d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:13.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.618+0000 7f8300e27640 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557b9da60910 0x557b9da62d30 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddd999fbfab5eae5 server_cookie=4b922b042444cf45 in_seq=0 out_seq=0 2024-05-20T18:05:13.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.619+0000 7f82dbfff640 10 client.16392 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:13.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.621+0000 7f82dbfff640 1 -- 172.21.15.129:0/2780494165 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557b9d937410 con 0x557b9da60910 2024-05-20T18:05:13.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.621+0000 7f82dbfff640 10 client.16392 handle_command_reply: tid=0 2024-05-20T18:05:13.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.623+0000 7f830a00d740 1 client.16392 shutdown 2024-05-20T18:05:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.630+0000 7f830a00d740 1 -- 172.21.15.129:0/2780494165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d62b410 msgr2=0x557b9d9ee220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.630+0000 7f830a00d740 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d62b410 0x557b9d9ee220 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f82f400b000 tx=0x7f82f4015ad0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.632+0000 7f830a00d740 1 -- 172.21.15.129:0/2780494165 shutdown_connections 2024-05-20T18:05:13.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.632+0000 7f830a00d740 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557b9da60910 0x557b9da62d30 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.632+0000 7f830a00d740 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557b9d5bf630 0x557b9d9eb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.632+0000 7f830a00d740 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b9d62b410 0x557b9d9ee220 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.632+0000 7f830a00d740 1 --2- 172.21.15.129:0/2780494165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557b9d602550 0x557b9d9edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.632+0000 7f830a00d740 1 -- 172.21.15.129:0/2780494165 >> 172.21.15.129:0/2780494165 conn(0x557b9d61f560 msgr2=0x557b9d9ca3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:13.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.632+0000 7f82dbfff640 0 client.16392 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:13.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.633+0000 7f830a00d740 1 -- 172.21.15.129:0/2780494165 shutdown_connections 2024-05-20T18:05:13.637 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:05:13.637 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:05:13.637 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "fcee4f7f-401b-4912-b86c-d321f73df4fe", 2024-05-20T18:05:13.637 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:05:13.637 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:05:13.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.636+0000 7f830a00d740 1 -- 172.21.15.129:0/2780494165 wait complete. 2024-05-20T18:05:13.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.637+0000 7f830a00d740 20 client.16392 trim_cache size 0 max 16384 2024-05-20T18:05:13.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.639+0000 7f830308b640 1 -- 172.21.15.129:0/1690066892 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f82e40a2e40 msgr2=0x7f82e40a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:13.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.639+0000 7f830308b640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f82e40a2e40 0x7f82e40a5350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x557b9d9c7790 tx=0x7f82ec006c20 comp rx=0 tx=0).stop 2024-05-20T18:05:13.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.639+0000 7f830308b640 1 -- 172.21.15.129:0/1690066892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82fc101990 msgr2=0x7f82fc19ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:13.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.639+0000 7f830308b640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82fc101990 0x7f82fc19ff10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009a20 tx=0x7f82f0003cd0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.640+0000 7f830308b640 1 -- 172.21.15.129:0/1690066892 shutdown_connections 2024-05-20T18:05:13.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.640+0000 7f830308b640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f82e40a2e40 0x7f82e40a5350 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.640+0000 7f830308b640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82fc106e10 0x7f82fc1a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.640+0000 7f830308b640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82fc1063e0 0x7f82fc1a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.640+0000 7f830308b640 1 --2- 172.21.15.129:0/1690066892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82fc101990 0x7f82fc19ff10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:13.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.640+0000 7f830308b640 1 -- 172.21.15.129:0/1690066892 >> 172.21.15.129:0/1690066892 conn(0x7f82fc0770b0 msgr2=0x7f82fc104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:13.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.642+0000 7f830308b640 1 -- 172.21.15.129:0/1690066892 shutdown_connections 2024-05-20T18:05:13.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:13.642+0000 7f830308b640 1 -- 172.21.15.129:0/1690066892 wait complete. 2024-05-20T18:05:13.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:05:13.685 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:05:13.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:13 smithi129 ceph-mon[32029]: pgmap v204: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 644 MiB/s wr, 3.21k op/s 2024-05-20T18:05:13.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:13 smithi154 ceph-mon[36482]: pgmap v204: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 644 MiB/s wr, 3.21k op/s 2024-05-20T18:05:13.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:13 smithi204 ceph-mon[37076]: pgmap v204: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 644 MiB/s wr, 3.21k op/s 2024-05-20T18:05:14.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.019+0000 7f5cd311d640 1 -- 172.21.15.129:0/1752463220 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ccc108f70 msgr2=0x7f5ccc109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:14.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.019+0000 7f5cd311d640 1 --2- 172.21.15.129:0/1752463220 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ccc108f70 0x7f5ccc109380 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009900 tx=0x7f5cbc02a4d0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.026+0000 7f5cd311d640 1 -- 172.21.15.129:0/1752463220 shutdown_connections 2024-05-20T18:05:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.026+0000 7f5cd311d640 1 --2- 172.21.15.129:0/1752463220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ccc119770 0x7f5ccc11bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.026+0000 7f5cd311d640 1 --2- 172.21.15.129:0/1752463220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ccc11c0d0 0x7f5ccc11e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.026+0000 7f5cd311d640 1 --2- 172.21.15.129:0/1752463220 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ccc108f70 0x7f5ccc109380 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.026+0000 7f5cd311d640 1 -- 172.21.15.129:0/1752463220 >> 172.21.15.129:0/1752463220 conn(0x7f5ccc06d5b0 msgr2=0x7f5ccc06da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.032+0000 7f5cd311d640 1 -- 172.21.15.129:0/1752463220 shutdown_connections 2024-05-20T18:05:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.032+0000 7f5cd311d640 1 -- 172.21.15.129:0/1752463220 wait complete. 2024-05-20T18:05:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.033+0000 7f5cd311d640 1 Processor -- start 2024-05-20T18:05:14.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.034+0000 7f5cd311d640 1 -- start start 2024-05-20T18:05:14.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.034+0000 7f5cd311d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ccc108f70 0x7f5ccc1a8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.034+0000 7f5cd311d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ccc119770 0x7f5ccc1a8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.034+0000 7f5cd311d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ccc11c0d0 0x7f5ccc1acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.034+0000 7f5cd311d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc121aa0 con 0x7f5ccc108f70 2024-05-20T18:05:14.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.034+0000 7f5cd311d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc121920 con 0x7f5ccc119770 2024-05-20T18:05:14.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.034+0000 7f5cd311d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc121c20 con 0x7f5ccc11c0d0 2024-05-20T18:05:14.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.040+0000 7f5cd0eb9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ccc108f70 0x7f5ccc1a8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.040+0000 7f5cd0eb9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ccc108f70 0x7f5ccc1a8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60282/0 (socket says 172.21.15.129:60282) 2024-05-20T18:05:14.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.040+0000 7f5cd0eb9640 1 -- 172.21.15.129:0/1013257423 learned_addr learned my addr 172.21.15.129:0/1013257423 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:14.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.042+0000 7f5cd16ba640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ccc11c0d0 0x7f5ccc1acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.043+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ccc119770 0x7f5ccc1a8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.045+0000 7f5ccbfff640 1 -- 172.21.15.129:0/1013257423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ccc11c0d0 msgr2=0x7f5ccc1acc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:14.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.045+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ccc11c0d0 0x7f5ccc1acc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.045+0000 7f5ccbfff640 1 -- 172.21.15.129:0/1013257423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ccc108f70 msgr2=0x7f5ccc1a8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:14.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.045+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ccc108f70 0x7f5ccc1a8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.045+0000 7f5ccbfff640 1 -- 172.21.15.129:0/1013257423 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ccc1ad260 con 0x7f5ccc119770 2024-05-20T18:05:14.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.047+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ccc119770 0x7f5ccc1a8960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400cce0 tx=0x7f5cc40075f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:14.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.049+0000 7f5cc9ffb640 1 -- 172.21.15.129:0/1013257423 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4004b20 con 0x7f5ccc119770 2024-05-20T18:05:14.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.049+0000 7f5cd311d640 1 -- 172.21.15.129:0/1013257423 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ccc1ad550 con 0x7f5ccc119770 2024-05-20T18:05:14.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.050+0000 7f5cc9ffb640 1 -- 172.21.15.129:0/1013257423 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cc4004cc0 con 0x7f5ccc119770 2024-05-20T18:05:14.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.050+0000 7f5cc9ffb640 1 -- 172.21.15.129:0/1013257423 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc400f6b0 con 0x7f5ccc119770 2024-05-20T18:05:14.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.050+0000 7f5cd311d640 1 -- 172.21.15.129:0/1013257423 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ccc110f60 con 0x7f5ccc119770 2024-05-20T18:05:14.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.056+0000 7f5cd9fbd740 1 Processor -- start 2024-05-20T18:05:14.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.057+0000 7f5cd9fbd740 1 -- start start 2024-05-20T18:05:14.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.057+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17d10390 0x55df180aef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.057+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 0x55df180af470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.057+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 0x55df180af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.057+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55df18014d00 con 0x55df17d10390 2024-05-20T18:05:14.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.057+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55df17fbc4c0 con 0x55df17ce74d0 2024-05-20T18:05:14.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.057+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55df18010cc0 con 0x55df17cef6b0 2024-05-20T18:05:14.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.061+0000 7f5cc9ffb640 1 -- 172.21.15.129:0/1013257423 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5cc400f970 con 0x7f5ccc119770 2024-05-20T18:05:14.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.062+0000 7f5cc9ffb640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5cb40a2f00 0x7f5cb40a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.063+0000 7f5cc9ffb640 1 -- 172.21.15.129:0/1013257423 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f5cc40cf3d0 con 0x7f5ccc119770 2024-05-20T18:05:14.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.064+0000 7f5cd0eb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 0x55df180af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.064+0000 7f5ccbfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 0x55df180af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.064+0000 7f5cd0eb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 0x55df180af470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55304/0 (socket says 172.21.15.129:55304) 2024-05-20T18:05:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.064+0000 7f5cd0eb9640 1 -- 172.21.15.129:0/532670736 learned_addr learned my addr 172.21.15.129:0/532670736 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.065+0000 7f5ccbfff640 1 -- 172.21.15.129:0/532670736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 msgr2=0x55df180af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.065+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/532670736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 0x55df180af470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.065+0000 7f5ccbfff640 1 -- 172.21.15.129:0/532670736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17d10390 msgr2=0x55df180aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.065+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/532670736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17d10390 0x55df180aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.065+0000 7f5ccbfff640 1 -- 172.21.15.129:0/532670736 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df17df27d0 con 0x55df17cef6b0 2024-05-20T18:05:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.065+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/532670736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 0x55df180af9b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40ccab0 tx=0x7f5cc409b650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deabd5b04c088b6a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:14.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.066+0000 7f5cb2ffd640 1 -- 172.21.15.129:0/532670736 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40b4070 con 0x55df17cef6b0 2024-05-20T18:05:14.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.066+0000 7f5cb2ffd640 1 -- 172.21.15.129:0/532670736 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cc409bf80 con 0x55df17cef6b0 2024-05-20T18:05:14.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.066+0000 7f5cb2ffd640 1 -- 172.21.15.129:0/532670736 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc401d030 con 0x55df17cef6b0 2024-05-20T18:05:14.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.068+0000 7f5cd16ba640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5cb40a2f00 0x7f5cb40a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.068+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/532670736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 msgr2=0x55df180af9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:14.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.068+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/532670736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 0x55df180af9b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40ccab0 tx=0x7f5cc409b650 comp rx=0 tx=0).stop 2024-05-20T18:05:14.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.070+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/532670736 shutdown_connections 2024-05-20T18:05:14.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.070+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/532670736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 0x55df180af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.070+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/532670736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 0x55df180af9b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.070+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/532670736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17d10390 0x55df180aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.070+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/532670736 >> 172.21.15.129:0/532670736 conn(0x55df17d044e0 msgr2=0x55df180ab6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:14.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.070+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/532670736 shutdown_connections 2024-05-20T18:05:14.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.070+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/532670736 wait complete. 2024-05-20T18:05:14.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.071+0000 7f5cd9fbd740 1 -- start start 2024-05-20T18:05:14.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.071+0000 7f5cd9fbd740 1 Processor -- start 2024-05-20T18:05:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.072+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 0x55df180bb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.076+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17cef6b0 0x55df180bb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.076+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17d10390 0x55df180b9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.076+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55df180bc280 con 0x55df17cef6b0 2024-05-20T18:05:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.076+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55df17fcfb90 con 0x55df17ce74d0 2024-05-20T18:05:14.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.076+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55df180bc400 con 0x55df17d10390 2024-05-20T18:05:14.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.076+0000 7f5cd9fbd740 10 client.0 _ll_register_callbacks cb 0x55df1800bda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:14.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.078+0000 7f5cd0eb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 0x55df180bb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.078+0000 7f5cd0eb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 0x55df180bb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55306/0 (socket says 172.21.15.129:55306) 2024-05-20T18:05:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.078+0000 7f5cd16ba640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17d10390 0x55df180b9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.078+0000 7f5cd0eb9640 1 -- 172.21.15.129:0/689597545 learned_addr learned my addr 172.21.15.129:0/689597545 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.080+0000 7f5cd0eb9640 1 -- 172.21.15.129:0/689597545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17d10390 msgr2=0x55df180b9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.080+0000 7f5cd0eb9640 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17d10390 0x55df180b9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.080+0000 7f5cd0eb9640 1 -- 172.21.15.129:0/689597545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17cef6b0 msgr2=0x55df180bb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.080+0000 7f5cd0eb9640 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17cef6b0 0x55df180bb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.080+0000 7f5cd0eb9640 1 -- 172.21.15.129:0/689597545 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df17d283f0 con 0x55df17ce74d0 2024-05-20T18:05:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.082+0000 7f5cd16ba640 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17d10390 0x55df180b9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:05:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.083+0000 7f5cd16ba640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5cb40a2f00 0x7f5cb40a5410 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000b590 tx=0x7f5cc000b520 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:14.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.084+0000 7f5cd0eb9640 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 0x55df180bb070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0314e0 tx=0x7f5cbc0043d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77b576d576455d40 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:14.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.084+0000 7f5cabfff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:05:14.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.085+0000 7f5cabfff640 1 -- 172.21.15.129:0/689597545 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc035030 con 0x55df17ce74d0 2024-05-20T18:05:14.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.085+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/689597545 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cc409bf80 con 0x55df17ce74d0 2024-05-20T18:05:14.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.085+0000 7f5cabfff640 1 -- 172.21.15.129:0/689597545 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f5cbc005450 con 0x55df17ce74d0 2024-05-20T18:05:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.095+0000 7f5cabfff640 1 -- 172.21.15.129:0/689597545 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5cbc005650 con 0x55df17ce74d0 2024-05-20T18:05:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.095+0000 7f5cd9fbd740 10 client.26126 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.095+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/689597545 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55df17ce80f0 con 0x55df17ce74d0 2024-05-20T18:05:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.096+0000 7f5cabfff640 1 -- 172.21.15.129:0/689597545 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cc409bf80 con 0x55df17ce74d0 2024-05-20T18:05:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.096+0000 7f5cabfff640 1 -- 172.21.15.129:0/689597545 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f5cbc003a30 con 0x55df17ce74d0 2024-05-20T18:05:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.096+0000 7f5cd9fbd740 10 client.26126 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.096+0000 7f5cd9fbd740 10 client.26126 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.096+0000 7f5cd9fbd740 20 client.26126 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.096+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55df17ccef60 0x55df180cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.096+0000 7f5cd9fbd740 4 client.26126 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:14.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.096+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/689597545 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df18022430 con 0x55df17ccef60 2024-05-20T18:05:14.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.100+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55df17ccef60 0x55df180cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.101+0000 7f5cabfff640 10 client.26126 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:14.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.106+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55df17ccef60 0x55df180cbf10 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f03cff77a85fc79a server_cookie=be32915b5b02a73c in_seq=0 out_seq=0 2024-05-20T18:05:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.107+0000 7f5cabfff640 1 -- 172.21.15.129:0/689597545 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55df18022430 con 0x55df17ccef60 2024-05-20T18:05:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.107+0000 7f5cabfff640 10 client.26126 handle_command_reply: tid=0 2024-05-20T18:05:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.107+0000 7f5cd9fbd740 1 client.26126 shutdown 2024-05-20T18:05:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.123+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/689597545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 msgr2=0x55df180bb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:14.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.123+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 0x55df180bb070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0314e0 tx=0x7f5cbc0043d0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.124+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/689597545 shutdown_connections 2024-05-20T18:05:14.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.124+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55df17ccef60 0x55df180cbf10 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.124+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17d10390 0x55df180b9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.124+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17cef6b0 0x55df180bb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.124+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/689597545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17ce74d0 0x55df180bb070 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.124+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/689597545 >> 172.21.15.129:0/689597545 conn(0x55df17d044e0 msgr2=0x55df180b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:14.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.124+0000 7f5cabfff640 0 client.26126 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.130+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/689597545 shutdown_connections 2024-05-20T18:05:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.132+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/689597545 wait complete. 2024-05-20T18:05:14.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.137+0000 7f5cd9fbd740 20 client.26126 trim_cache size 0 max 16384 2024-05-20T18:05:14.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.193+0000 7f5cd9fbd740 1 Processor -- start 2024-05-20T18:05:14.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.195+0000 7f5cd9fbd740 1 -- start start 2024-05-20T18:05:14.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.195+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17ce74d0 0x55df180bb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.195+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17cef6b0 0x55df180d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.195+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 0x55df180d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.195+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc1c74c0 con 0x55df17cef6b0 2024-05-20T18:05:14.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.195+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc1c6720 con 0x55df17d10390 2024-05-20T18:05:14.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.195+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc117dc0 con 0x55df17ce74d0 2024-05-20T18:05:14.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.196+0000 7f5cd16ba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 0x55df180d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.196+0000 7f5cd16ba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 0x55df180d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55308/0 (socket says 172.21.15.129:55308) 2024-05-20T18:05:14.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.196+0000 7f5cd16ba640 1 -- 172.21.15.129:0/3952549091 learned_addr learned my addr 172.21.15.129:0/3952549091 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:14.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.196+0000 7f5cd0eb9640 1 --2- 172.21.15.129:0/3952549091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17ce74d0 0x55df180bb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.197+0000 7f5cd16ba640 1 -- 172.21.15.129:0/3952549091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17ce74d0 msgr2=0x55df180bb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:14.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.197+0000 7f5cd16ba640 1 --2- 172.21.15.129:0/3952549091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17ce74d0 0x55df180bb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.197+0000 7f5cd16ba640 1 -- 172.21.15.129:0/3952549091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17cef6b0 msgr2=0x55df180d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:14.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.197+0000 7f5cd16ba640 1 --2- 172.21.15.129:0/3952549091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17cef6b0 0x55df180d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.197+0000 7f5cd16ba640 1 -- 172.21.15.129:0/3952549091 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df180b7ad0 con 0x55df17d10390 2024-05-20T18:05:14.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.198+0000 7f5cd16ba640 1 --2- 172.21.15.129:0/3952549091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 0x55df180d1c10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0010ee0 tx=0x7f5cc0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=821ab41045de8d9f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:14.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.198+0000 7f5cb27fc640 1 -- 172.21.15.129:0/3952549091 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc001e620 con 0x55df17d10390 2024-05-20T18:05:14.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.199+0000 7f5cb27fc640 1 -- 172.21.15.129:0/3952549091 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cc001eca0 con 0x55df17d10390 2024-05-20T18:05:14.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.199+0000 7f5cb27fc640 1 -- 172.21.15.129:0/3952549091 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0026c70 con 0x55df17d10390 2024-05-20T18:05:14.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.200+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/3952549091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 msgr2=0x55df180d1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:14.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.200+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/3952549091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 0x55df180d1c10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0010ee0 tx=0x7f5cc0016000 comp rx=0 tx=0).stop 2024-05-20T18:05:14.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.201+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/3952549091 shutdown_connections 2024-05-20T18:05:14.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.201+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/3952549091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 0x55df180d1c10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.201+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/3952549091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17cef6b0 0x55df180d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.201+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/3952549091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17ce74d0 0x55df180bb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.201+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/3952549091 >> 172.21.15.129:0/3952549091 conn(0x55df17d044e0 msgr2=0x55df180baca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:14.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.203+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/3952549091 shutdown_connections 2024-05-20T18:05:14.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.203+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/3952549091 wait complete. 2024-05-20T18:05:14.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.203+0000 7f5cd9fbd740 1 -- start start 2024-05-20T18:05:14.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.204+0000 7f5cd9fbd740 1 Processor -- start 2024-05-20T18:05:14.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.205+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17ce74d0 0x55df180c14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.205+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 0x55df180c19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.205+0000 7f5cd9fbd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 0x55df180be960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.205+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc118470 con 0x55df17ce74d0 2024-05-20T18:05:14.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.205+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc1c6050 con 0x55df17d10390 2024-05-20T18:05:14.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.205+0000 7f5cd9fbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc118b40 con 0x55df17cef6b0 2024-05-20T18:05:14.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.205+0000 7f5cd9fbd740 10 client.0 _ll_register_callbacks cb 0x7f5cbc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:14.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.206+0000 7f5cd16ba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 0x55df180be960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.206+0000 7f5cd16ba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 0x55df180be960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55320/0 (socket says 172.21.15.129:55320) 2024-05-20T18:05:14.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.206+0000 7f5cd16ba640 1 -- 172.21.15.129:0/2282046860 learned_addr learned my addr 172.21.15.129:0/2282046860 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:14.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.206+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 0x55df180c19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.206+0000 7f5cd0eb9640 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17ce74d0 0x55df180c14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.207+0000 7f5ccbfff640 1 -- 172.21.15.129:0/2282046860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 msgr2=0x55df180be960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:14.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.207+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 0x55df180be960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.207+0000 7f5ccbfff640 1 -- 172.21.15.129:0/2282046860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17ce74d0 msgr2=0x55df180c14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:14.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.207+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17ce74d0 0x55df180c14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.207+0000 7f5ccbfff640 1 -- 172.21.15.129:0/2282046860 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df180befa0 con 0x55df17cef6b0 2024-05-20T18:05:14.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.207+0000 7f5ccbfff640 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 0x55df180c19e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4099b30 tx=0x7f5cc40cf620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8776c08d6cde67e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:14.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.207+0000 7f5cabfff640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:05:14.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.208+0000 7f5cabfff640 1 -- 172.21.15.129:0/2282046860 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc401d050 con 0x55df17cef6b0 2024-05-20T18:05:14.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.208+0000 7f5cabfff640 1 -- 172.21.15.129:0/2282046860 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f5cc40ccd30 con 0x55df17cef6b0 2024-05-20T18:05:14.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.211+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/2282046860 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cc001eca0 con 0x55df17cef6b0 2024-05-20T18:05:14.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.213+0000 7f5cabfff640 1 -- 172.21.15.129:0/2282046860 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5cc40b4040 con 0x55df17cef6b0 2024-05-20T18:05:14.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.214+0000 7f5cd9fbd740 10 client.26188 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:14.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.214+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/2282046860 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55df18133bc0 con 0x55df17cef6b0 2024-05-20T18:05:14.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.214+0000 7f5cabfff640 1 -- 172.21.15.129:0/2282046860 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cc001eca0 con 0x55df17cef6b0 2024-05-20T18:05:14.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.216+0000 7f5cabfff640 1 -- 172.21.15.129:0/2282046860 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f5cc409cbb0 con 0x55df17cef6b0 2024-05-20T18:05:14.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.218+0000 7f5cd9fbd740 10 client.26188 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:14.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.218+0000 7f5cd9fbd740 10 client.26188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:14.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.218+0000 7f5cd9fbd740 20 client.26188 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:14.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.218+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55df17ca43a0 0x55df18135f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:14.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.218+0000 7f5cd9fbd740 4 client.26188 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:05:14.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.218+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/2282046860 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55df1801c780 con 0x55df17ca43a0 2024-05-20T18:05:14.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.224+0000 7f5cd0eb9640 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55df17ca43a0 0x55df18135f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:14.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.225+0000 7f5cd0eb9640 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55df17ca43a0 0x55df18135f30 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57b5a225cc7bb18d server_cookie=7d84be1ab5a88dcd in_seq=0 out_seq=0 2024-05-20T18:05:14.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.225+0000 7f5cabfff640 10 client.26188 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:14.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.231+0000 7f5cabfff640 1 -- 172.21.15.129:0/2282046860 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55df1801c780 con 0x55df17ca43a0 2024-05-20T18:05:14.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.231+0000 7f5cabfff640 10 client.26188 handle_command_reply: tid=0 2024-05-20T18:05:14.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.231+0000 7f5cd9fbd740 1 client.26188 shutdown 2024-05-20T18:05:14.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.232+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/2282046860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 msgr2=0x55df180c19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:14.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.232+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 0x55df180c19e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4099b30 tx=0x7f5cc40cf620 comp rx=0 tx=0).stop 2024-05-20T18:05:14.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.232+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/2282046860 shutdown_connections 2024-05-20T18:05:14.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.232+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55df17ca43a0 0x55df18135f30 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.232+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df17d10390 0x55df180be960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.232+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55df17cef6b0 0x55df180c19e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.232+0000 7f5cd9fbd740 1 --2- 172.21.15.129:0/2282046860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df17ce74d0 0x55df180c14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.232+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/2282046860 >> 172.21.15.129:0/2282046860 conn(0x55df17d044e0 msgr2=0x55df180b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.232+0000 7f5cabfff640 0 client.26188 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.233+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/2282046860 shutdown_connections 2024-05-20T18:05:14.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.233+0000 7f5cd9fbd740 1 -- 172.21.15.129:0/2282046860 wait complete. 2024-05-20T18:05:14.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.235+0000 7f5cd9fbd740 20 client.26188 trim_cache size 0 max 16384 2024-05-20T18:05:14.236 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:05:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:05:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:05:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: "fcee4f7f-401b-4912-b86c-d321f73df4fe": { 2024-05-20T18:05:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:05:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "fcee4f7f-401b-4912-b86c-d321f73df4fe", 2024-05-20T18:05:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:05:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:05:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:05:14.237 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:05:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.238+0000 7f5cd311d640 1 -- 172.21.15.129:0/1013257423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5cb40a2f00 msgr2=0x7f5cb40a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.238+0000 7f5cd311d640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5cb40a2f00 0x7f5cb40a5410 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000b590 tx=0x7f5cc000b520 comp rx=0 tx=0).stop 2024-05-20T18:05:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.238+0000 7f5cd311d640 1 -- 172.21.15.129:0/1013257423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ccc119770 msgr2=0x7f5ccc1a8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.238+0000 7f5cd311d640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ccc119770 0x7f5ccc1a8960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400cce0 tx=0x7f5cc40075f0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.238+0000 7f5cd311d640 1 -- 172.21.15.129:0/1013257423 shutdown_connections 2024-05-20T18:05:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.238+0000 7f5cd311d640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5cb40a2f00 0x7f5cb40a5410 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.238+0000 7f5cd311d640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ccc119770 0x7f5ccc1a8960 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.238+0000 7f5cd311d640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ccc11c0d0 0x7f5ccc1acc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.238+0000 7f5cd311d640 1 --2- 172.21.15.129:0/1013257423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ccc108f70 0x7f5ccc1a8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.238+0000 7f5cd311d640 1 -- 172.21.15.129:0/1013257423 >> 172.21.15.129:0/1013257423 conn(0x7f5ccc06d5b0 msgr2=0x7f5ccc119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:14.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.239+0000 7f5cd311d640 1 -- 172.21.15.129:0/1013257423 shutdown_connections 2024-05-20T18:05:14.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:14.239+0000 7f5cd311d640 1 -- 172.21.15.129:0/1013257423 wait complete. 2024-05-20T18:05:14.285 INFO:tasks.cephfs.filesystem:scrub status for tag:fcee4f7f-401b-4912-b86c-d321f73df4fe - {'path': '/', 'tag': 'fcee4f7f-401b-4912-b86c-d321f73df4fe', 'options': 'recursive,force'} 2024-05-20T18:05:14.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:14 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:05:14.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:14 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:14.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:14 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:05:14.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:14 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:14 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:05:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:14 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:14 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:05:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:14 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:14 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:05:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:14 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:14 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:05:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:14 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:15.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:15 smithi154 ceph-mon[36482]: pgmap v205: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 689 MiB/s wr, 3.66k op/s 2024-05-20T18:05:15.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:15 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:05:15.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:15 smithi129 ceph-mon[32029]: pgmap v205: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 689 MiB/s wr, 3.66k op/s 2024-05-20T18:05:15.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:15 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:05:15.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:15 smithi204 ceph-mon[37076]: pgmap v205: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 689 MiB/s wr, 3.66k op/s 2024-05-20T18:05:15.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:15 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:05:17.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:17 smithi154 ceph-mon[36482]: pgmap v206: 97 pgs: 97 active+clean; 53 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 861 MiB/s wr, 4.75k op/s 2024-05-20T18:05:17.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:17 smithi129 ceph-mon[32029]: pgmap v206: 97 pgs: 97 active+clean; 53 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 861 MiB/s wr, 4.75k op/s 2024-05-20T18:05:17.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:17 smithi204 ceph-mon[37076]: pgmap v206: 97 pgs: 97 active+clean; 53 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 861 MiB/s wr, 4.75k op/s 2024-05-20T18:05:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.433+0000 7f1fdeffd640 1 -- 172.21.15.129:0/4028323697 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40319b0 con 0x7f1fe010e890 2024-05-20T18:05:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.433+0000 7f1fe53db640 1 -- 172.21.15.129:0/4028323697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe010e890 msgr2=0x7f1fe010eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.433+0000 7f1fe53db640 1 --2- 172.21.15.129:0/4028323697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe010e890 0x7f1fe010eca0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009900 tx=0x7f1fd402a410 comp rx=0 tx=0).stop 2024-05-20T18:05:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.435+0000 7f1fe53db640 1 -- 172.21.15.129:0/4028323697 shutdown_connections 2024-05-20T18:05:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.435+0000 7f1fe53db640 1 --2- 172.21.15.129:0/4028323697 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fe0109090 0x7f1fe0109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.435+0000 7f1fe53db640 1 --2- 172.21.15.129:0/4028323697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe010f270 0x7f1fe0108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.435+0000 7f1fe53db640 1 --2- 172.21.15.129:0/4028323697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe010e890 0x7f1fe010eca0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.435+0000 7f1fe53db640 1 -- 172.21.15.129:0/4028323697 >> 172.21.15.129:0/4028323697 conn(0x7f1fe006e690 msgr2=0x7f1fe006eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:19.439 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.438+0000 7f1fe53db640 1 -- 172.21.15.129:0/4028323697 shutdown_connections 2024-05-20T18:05:19.442 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.443+0000 7f1fe53db640 1 -- 172.21.15.129:0/4028323697 wait complete. 2024-05-20T18:05:19.442 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.444+0000 7f1fe53db640 1 Processor -- start 2024-05-20T18:05:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.448+0000 7f1fe53db640 1 -- start start 2024-05-20T18:05:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.448+0000 7f1fe53db640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fe0109090 0x7f1fe011ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.446 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.448+0000 7f1fe53db640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe010e890 0x7f1fe0119090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.448+0000 7f1fe53db640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe010f270 0x7f1fe01195d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.448+0000 7f1fe53db640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe0111530 con 0x7f1fe010e890 2024-05-20T18:05:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.448+0000 7f1fe53db640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe01113b0 con 0x7f1fe010f270 2024-05-20T18:05:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.448+0000 7f1fe53db640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe01116b0 con 0x7f1fe0109090 2024-05-20T18:05:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.449+0000 7f1fe4bda640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe010f270 0x7f1fe01195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.449+0000 7f1fe4bda640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe010f270 0x7f1fe01195d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55348/0 (socket says 172.21.15.129:55348) 2024-05-20T18:05:19.447 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.449+0000 7f1fe4bda640 1 -- 172.21.15.129:0/1906723355 learned_addr learned my addr 172.21.15.129:0/1906723355 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.450+0000 7f1fe4bda640 1 -- 172.21.15.129:0/1906723355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fe0109090 msgr2=0x7f1fe011ef30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:05:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.450+0000 7f1fe4bda640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fe0109090 0x7f1fe011ef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.450+0000 7f1fe4bda640 1 -- 172.21.15.129:0/1906723355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe010e890 msgr2=0x7f1fe0119090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.450+0000 7f1fe4bda640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe010e890 0x7f1fe0119090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.450+0000 7f1fe4bda640 1 -- 172.21.15.129:0/1906723355 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe0119d00 con 0x7f1fe010f270 2024-05-20T18:05:19.448 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.450+0000 7f1fe4bda640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe010f270 0x7f1fe01195d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800eb10 tx=0x7f1fd800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.450+0000 7f1fdd7fa640 1 -- 172.21.15.129:0/1906723355 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8004b20 con 0x7f1fe010f270 2024-05-20T18:05:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.451+0000 7f1fe53db640 1 -- 172.21.15.129:0/1906723355 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe01bd210 con 0x7f1fe010f270 2024-05-20T18:05:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.451+0000 7f1fe53db640 1 -- 172.21.15.129:0/1906723355 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe01bd670 con 0x7f1fe010f270 2024-05-20T18:05:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.455+0000 7f1fed5c3740 1 Processor -- start 2024-05-20T18:05:19.453 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.456+0000 7f1fdd7fa640 1 -- 172.21.15.129:0/1906723355 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fd8004cc0 con 0x7f1fe010f270 2024-05-20T18:05:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.456+0000 7f1fed5c3740 1 -- start start 2024-05-20T18:05:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.457+0000 7f1fed5c3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec91d1b0 0x5593eccbc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.458+0000 7f1fed5c3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 0x5593eccbc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.458+0000 7f1fed5c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccbccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.458+0000 7f1fed5c3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5593ecc21340 con 0x5593ec8f42f0 2024-05-20T18:05:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.458+0000 7f1fed5c3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5593ecbc9290 con 0x5593ec8fc4d0 2024-05-20T18:05:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.458+0000 7f1fed5c3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5593ecc1d300 con 0x5593ec91d1b0 2024-05-20T18:05:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.459+0000 7f1fdf7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec91d1b0 0x5593eccbc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.459+0000 7f1fdf7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec91d1b0 0x5593eccbc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:43774/0 (socket says 172.21.15.129:43774) 2024-05-20T18:05:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.459+0000 7f1fdf7fe640 1 -- 172.21.15.129:0/747188011 learned_addr learned my addr 172.21.15.129:0/747188011 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.459+0000 7f1fdf7fe640 1 --2- 172.21.15.129:0/747188011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccbccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.458 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.459+0000 7f1fdf7fe640 1 -- 172.21.15.129:0/747188011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 msgr2=0x5593eccbccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:19.458 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.459+0000 7f1fdf7fe640 1 --2- 172.21.15.129:0/747188011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccbccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.458 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.459+0000 7f1fdf7fe640 1 -- 172.21.15.129:0/747188011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 msgr2=0x5593eccbc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:19.465 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.467+0000 7f1fdffff640 1 --2- 172.21.15.129:0/747188011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 0x5593eccbc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.468+0000 7f1fdd7fa640 1 -- 172.21.15.129:0/1906723355 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80107b0 con 0x7f1fe010f270 2024-05-20T18:05:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.468+0000 7f1fdd7fa640 1 -- 172.21.15.129:0/1906723355 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f1fd8010990 con 0x7f1fe010f270 2024-05-20T18:05:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.468+0000 7f1fdf7fe640 1 --2- 172.21.15.129:0/747188011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 0x5593eccbc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.468+0000 7f1fdf7fe640 1 -- 172.21.15.129:0/747188011 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593ecbed8e0 con 0x5593ec91d1b0 2024-05-20T18:05:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.468+0000 7f1fdffff640 1 --2- 172.21.15.129:0/747188011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 0x5593eccbc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:05:19.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.469+0000 7f1fdd7fa640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fc80a2da0 0x7f1fc80a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.470+0000 7f1fdd7fa640 1 -- 172.21.15.129:0/1906723355 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f1fd80d0c10 con 0x7f1fe010f270 2024-05-20T18:05:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.478+0000 7f1fdf7fe640 1 --2- 172.21.15.129:0/747188011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec91d1b0 0x5593eccbc270 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0013780 tx=0x7f1fd00095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4418ccc6602c2d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.478+0000 7f1fdffff640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fc80a2da0 0x7f1fc80a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.478 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.480+0000 7f1fdffff640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fc80a2da0 0x7f1fc80a52b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4000c00 tx=0x7f1fd4005d50 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:19.478 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.480+0000 7f1fc67fc640 1 -- 172.21.15.129:0/747188011 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0013070 con 0x5593ec91d1b0 2024-05-20T18:05:19.478 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.480+0000 7f1fc67fc640 1 -- 172.21.15.129:0/747188011 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fd0017070 con 0x5593ec91d1b0 2024-05-20T18:05:19.478 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.480+0000 7f1fc67fc640 1 -- 172.21.15.129:0/747188011 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00047a0 con 0x5593ec91d1b0 2024-05-20T18:05:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.482+0000 7f1fed5c3740 1 -- 172.21.15.129:0/747188011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec91d1b0 msgr2=0x5593eccbc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.482+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/747188011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec91d1b0 0x5593eccbc270 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0013780 tx=0x7f1fd00095b0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.482+0000 7f1fed5c3740 1 -- 172.21.15.129:0/747188011 shutdown_connections 2024-05-20T18:05:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.482+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/747188011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccbccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.482+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/747188011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 0x5593eccbc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.482+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/747188011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec91d1b0 0x5593eccbc270 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.483+0000 7f1fed5c3740 1 -- 172.21.15.129:0/747188011 >> 172.21.15.129:0/747188011 conn(0x5593ec911300 msgr2=0x5593eccb8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.483+0000 7f1fed5c3740 1 -- 172.21.15.129:0/747188011 shutdown_connections 2024-05-20T18:05:19.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.483+0000 7f1fed5c3740 1 -- 172.21.15.129:0/747188011 wait complete. 2024-05-20T18:05:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.484+0000 7f1fed5c3740 1 -- start start 2024-05-20T18:05:19.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.484+0000 7f1fed5c3740 1 Processor -- start 2024-05-20T18:05:19.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.485+0000 7f1fed5c3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 0x5593eccc6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.486+0000 7f1fed5c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccc6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.486+0000 7f1fdffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 0x5593eccc6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.486+0000 7f1fdffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 0x5593eccc6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60364/0 (socket says 172.21.15.129:60364) 2024-05-20T18:05:19.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.486+0000 7f1fdffff640 1 -- 172.21.15.129:0/4228506639 learned_addr learned my addr 172.21.15.129:0/4228506639 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:19.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.486+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec91d1b0 0x5593ecccc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.486+0000 7f1fdf7fe640 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccc6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.486+0000 7f1fed5c3740 1 -- 172.21.15.129:0/4228506639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5593eccc9340 con 0x5593ec8f42f0 2024-05-20T18:05:19.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.486+0000 7f1fed5c3740 1 -- 172.21.15.129:0/4228506639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5593ecbdc790 con 0x5593ec8fc4d0 2024-05-20T18:05:19.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.487+0000 7f1fed5c3740 1 -- 172.21.15.129:0/4228506639 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5593eccc94c0 con 0x5593ec91d1b0 2024-05-20T18:05:19.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.487+0000 7f1fed5c3740 10 client.0 _ll_register_callbacks cb 0x5593ecc17a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.492+0000 7f1fdf7fe640 1 -- 172.21.15.129:0/4228506639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec91d1b0 msgr2=0x5593ecccc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.492+0000 7f1fdf7fe640 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec91d1b0 0x5593ecccc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.492+0000 7f1fdf7fe640 1 -- 172.21.15.129:0/4228506639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 msgr2=0x5593eccc6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.492+0000 7f1fdf7fe640 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 0x5593eccc6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.492+0000 7f1fdf7fe640 1 -- 172.21.15.129:0/4228506639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593ec935210 con 0x5593ec8fc4d0 2024-05-20T18:05:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.493+0000 7f1fdf7fe640 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccc6ed0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0013780 tx=0x7f1fd0005630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca831f51a2f12246 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.494+0000 7f1fbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:05:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.494+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/4228506639 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0004440 con 0x5593ec8fc4d0 2024-05-20T18:05:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.494+0000 7f1fed5c3740 1 -- 172.21.15.129:0/4228506639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fd0017070 con 0x5593ec8fc4d0 2024-05-20T18:05:19.493 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.494+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/4228506639 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f1fd000e720 con 0x5593ec8fc4d0 2024-05-20T18:05:19.496 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.496+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/4228506639 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1fd0030050 con 0x5593ec8fc4d0 2024-05-20T18:05:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.503+0000 7f1fed5c3740 10 client.26144 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.503+0000 7f1fed5c3740 1 -- 172.21.15.129:0/4228506639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5593eca01620 con 0x5593ec8fc4d0 2024-05-20T18:05:19.505 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.506+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/4228506639 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fd0017070 con 0x5593ec8fc4d0 2024-05-20T18:05:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.506+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/4228506639 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f1fd0004c00 con 0x5593ec8fc4d0 2024-05-20T18:05:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.506+0000 7f1fed5c3740 10 client.26144 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.506+0000 7f1fed5c3740 10 client.26144 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.506+0000 7f1fed5c3740 20 client.26144 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.506+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593ec8dbd80 0x5593eccd8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.506+0000 7f1fed5c3740 4 client.26144 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:19.506 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.506+0000 7f1fed5c3740 1 -- 172.21.15.129:0/4228506639 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593ecc2ea70 con 0x5593ec8dbd80 2024-05-20T18:05:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.508+0000 7f1fdffff640 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593ec8dbd80 0x5593eccd8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.508 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.510+0000 7f1fdffff640 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593ec8dbd80 0x5593eccd8f20 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=409bcc80e6b02783 server_cookie=6b2f24dc1b7ef1c8 in_seq=0 out_seq=0 2024-05-20T18:05:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.515+0000 7f1fbf7fe640 10 client.26144 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.520+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/4228506639 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5593ecc2ea70 con 0x5593ec8dbd80 2024-05-20T18:05:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.520+0000 7f1fbf7fe640 10 client.26144 handle_command_reply: tid=0 2024-05-20T18:05:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.520+0000 7f1fed5c3740 1 client.26144 shutdown 2024-05-20T18:05:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.525+0000 7f1fed5c3740 1 -- 172.21.15.129:0/4228506639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 msgr2=0x5593eccc6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.525+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccc6ed0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0013780 tx=0x7f1fd0005630 comp rx=0 tx=0).stop 2024-05-20T18:05:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.548+0000 7f1fed5c3740 1 -- 172.21.15.129:0/4228506639 shutdown_connections 2024-05-20T18:05:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.548+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593ec8dbd80 0x5593eccd8f20 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.548+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccc6ed0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.548+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec91d1b0 0x5593ecccc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.549+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/4228506639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec8f42f0 0x5593eccc6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.549+0000 7f1fed5c3740 1 -- 172.21.15.129:0/4228506639 >> 172.21.15.129:0/4228506639 conn(0x5593ec911300 msgr2=0x5593eccbc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.549+0000 7f1fbf7fe640 0 client.26144 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:19.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.557+0000 7f1fed5c3740 1 -- 172.21.15.129:0/4228506639 shutdown_connections 2024-05-20T18:05:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.559+0000 7f1fed5c3740 1 -- 172.21.15.129:0/4228506639 wait complete. 2024-05-20T18:05:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.564+0000 7f1fed5c3740 20 client.26144 trim_cache size 0 max 16384 2024-05-20T18:05:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.631+0000 7f1fed5c3740 1 Processor -- start 2024-05-20T18:05:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.632+0000 7f1fed5c3740 1 -- start start 2024-05-20T18:05:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.632+0000 7f1fed5c3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec8f42f0 0x5593eccc05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.632+0000 7f1fed5c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccde720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.632+0000 7f1fed5c3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 0x5593eccdec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.632+0000 7f1fed5c3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc00c8b0 con 0x5593ec91d1b0 2024-05-20T18:05:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.632+0000 7f1fed5c3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc00f130 con 0x5593ec8fc4d0 2024-05-20T18:05:19.632 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.632+0000 7f1fed5c3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc00cf20 con 0x5593ec8f42f0 2024-05-20T18:05:19.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.635+0000 7f1fe4bda640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 0x5593eccdec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.635+0000 7f1fe4bda640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 0x5593eccdec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37800/0 (socket says 172.21.15.129:37800) 2024-05-20T18:05:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.635+0000 7f1fe4bda640 1 -- 172.21.15.129:0/1524017762 learned_addr learned my addr 172.21.15.129:0/1524017762 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.636+0000 7f1fe4bda640 1 -- 172.21.15.129:0/1524017762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec8f42f0 msgr2=0x5593eccc05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.636+0000 7f1fe4bda640 1 --2- 172.21.15.129:0/1524017762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec8f42f0 0x5593eccc05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.636+0000 7f1fe4bda640 1 -- 172.21.15.129:0/1524017762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 msgr2=0x5593eccde720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:05:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.636+0000 7f1fe4bda640 1 --2- 172.21.15.129:0/1524017762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccde720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.637+0000 7f1fe4bda640 1 -- 172.21.15.129:0/1524017762 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593eccb6de0 con 0x5593ec91d1b0 2024-05-20T18:05:19.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.637+0000 7f1fe4bda640 1 --2- 172.21.15.129:0/1524017762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 0x5593eccdec60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8005a00 tx=0x7f1fd8099510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98868de37182d642 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:19.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.639+0000 7f1fc5ffb640 1 -- 172.21.15.129:0/1524017762 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80b6070 con 0x5593ec91d1b0 2024-05-20T18:05:19.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.639+0000 7f1fc5ffb640 1 -- 172.21.15.129:0/1524017762 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fd809d140 con 0x5593ec91d1b0 2024-05-20T18:05:19.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.639+0000 7f1fc5ffb640 1 -- 172.21.15.129:0/1524017762 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80997b0 con 0x5593ec91d1b0 2024-05-20T18:05:19.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.641+0000 7f1fed5c3740 1 -- 172.21.15.129:0/1524017762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 msgr2=0x5593eccdec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:19.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.641+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/1524017762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 0x5593eccdec60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8005a00 tx=0x7f1fd8099510 comp rx=0 tx=0).stop 2024-05-20T18:05:19.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.644+0000 7f1fed5c3740 1 -- 172.21.15.129:0/1524017762 shutdown_connections 2024-05-20T18:05:19.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.644+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/1524017762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8fc4d0 0x5593eccde720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.647 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.644+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/1524017762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 0x5593eccdec60 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.644+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/1524017762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec8f42f0 0x5593eccc05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.648 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.644+0000 7f1fed5c3740 1 -- 172.21.15.129:0/1524017762 >> 172.21.15.129:0/1524017762 conn(0x5593ec911300 msgr2=0x5593eccbfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.653+0000 7f1fed5c3740 1 -- 172.21.15.129:0/1524017762 shutdown_connections 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.653+0000 7f1fed5c3740 1 -- 172.21.15.129:0/1524017762 wait complete. 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.653+0000 7f1fed5c3740 1 -- start start 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.654+0000 7f1fed5c3740 1 Processor -- start 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.654+0000 7f1fed5c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8f42f0 0x5593ecccda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.654+0000 7f1fed5c3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec8fc4d0 0x5593ecccdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.654+0000 7f1fed5c3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 0x5593eccce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.655+0000 7f1fed5c3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc00d5f0 con 0x5593ec91d1b0 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.655+0000 7f1fed5c3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc00ea60 con 0x5593ec8f42f0 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.655+0000 7f1fed5c3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc00dcc0 con 0x5593ec8fc4d0 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.655+0000 7f1fed5c3740 10 client.0 _ll_register_callbacks cb 0x7f1fd0009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.656+0000 7f1fe4bda640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 0x5593eccce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.656+0000 7f1fe4bda640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 0x5593eccce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37810/0 (socket says 172.21.15.129:37810) 2024-05-20T18:05:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.656+0000 7f1fe4bda640 1 -- 172.21.15.129:0/3239455380 learned_addr learned my addr 172.21.15.129:0/3239455380 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.656+0000 7f1fdffff640 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8f42f0 0x5593ecccda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.657+0000 7f1fdf7fe640 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec8fc4d0 0x5593ecccdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.658+0000 7f1fdffff640 1 -- 172.21.15.129:0/3239455380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec8fc4d0 msgr2=0x5593ecccdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.658+0000 7f1fdffff640 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec8fc4d0 0x5593ecccdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.658+0000 7f1fdffff640 1 -- 172.21.15.129:0/3239455380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 msgr2=0x5593eccce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.658+0000 7f1fdffff640 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 0x5593eccce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.658+0000 7f1fdffff640 1 -- 172.21.15.129:0/3239455380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593ecd3f2c0 con 0x5593ec8f42f0 2024-05-20T18:05:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.659+0000 7f1fdffff640 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8f42f0 0x5593ecccda30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009230 tx=0x7f1fd4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=851371686aa9541b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.659+0000 7f1fbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:05:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.660+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/3239455380 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4052030 con 0x5593ec8f42f0 2024-05-20T18:05:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.660+0000 7f1fed5c3740 1 -- 172.21.15.129:0/3239455380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fd809d140 con 0x5593ec8f42f0 2024-05-20T18:05:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.660+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/3239455380 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f1fd4005250 con 0x5593ec8f42f0 2024-05-20T18:05:19.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.663+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/3239455380 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1fd405c050 con 0x5593ec8f42f0 2024-05-20T18:05:19.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.663+0000 7f1fed5c3740 10 client.26147 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:19.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.663+0000 7f1fed5c3740 1 -- 172.21.15.129:0/3239455380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5593ecd62550 con 0x5593ec8f42f0 2024-05-20T18:05:19.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.667+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/3239455380 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fd809d140 con 0x5593ec8f42f0 2024-05-20T18:05:19.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.667+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/3239455380 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f1fd4003690 con 0x5593ec8f42f0 2024-05-20T18:05:19.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.667+0000 7f1fed5c3740 10 client.26147 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:19.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.667+0000 7f1fed5c3740 10 client.26147 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:19.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.667+0000 7f1fed5c3740 20 client.26147 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:19.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.667+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593ec8b11c0 0x5593ecd64cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:19.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.667+0000 7f1fed5c3740 4 client.26147 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:05:19.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.667+0000 7f1fed5c3740 1 -- 172.21.15.129:0/3239455380 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5593ecc28dc0 con 0x5593ec8b11c0 2024-05-20T18:05:19.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.672+0000 7f1fdf7fe640 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593ec8b11c0 0x5593ecd64cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:19.679 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.681+0000 7f1fdf7fe640 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593ec8b11c0 0x5593ecd64cc0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b0e19c6294474b2 server_cookie=a3c66860dec64293 in_seq=0 out_seq=0 2024-05-20T18:05:19.679 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:19.682+0000 7f1fbf7fe640 10 client.26147 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:19.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:19 smithi154 ceph-mon[36482]: pgmap v207: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 658 MiB/s wr, 4.33k op/s 2024-05-20T18:05:19.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:19 smithi129 ceph-mon[32029]: pgmap v207: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 658 MiB/s wr, 4.33k op/s 2024-05-20T18:05:19.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:19 smithi204 ceph-mon[37076]: pgmap v207: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 658 MiB/s wr, 4.33k op/s 2024-05-20T18:05:21.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:21 smithi154 ceph-mon[36482]: pgmap v208: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 803 MiB/s wr, 4.98k op/s 2024-05-20T18:05:21.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:21 smithi129 ceph-mon[32029]: pgmap v208: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 803 MiB/s wr, 4.98k op/s 2024-05-20T18:05:21.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:21 smithi204 ceph-mon[37076]: pgmap v208: 97 pgs: 97 active+clean; 54 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 803 MiB/s wr, 4.98k op/s 2024-05-20T18:05:23.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:23 smithi154 ceph-mon[36482]: pgmap v209: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 611 MiB/s wr, 3.19k op/s 2024-05-20T18:05:23.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:23 smithi129 ceph-mon[32029]: pgmap v209: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 611 MiB/s wr, 3.19k op/s 2024-05-20T18:05:23.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:23 smithi204 ceph-mon[37076]: pgmap v209: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 611 MiB/s wr, 3.19k op/s 2024-05-20T18:05:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.464+0000 7f1fbf7fe640 1 -- 172.21.15.129:0/3239455380 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5593ecc28dc0 con 0x5593ec8b11c0 2024-05-20T18:05:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.464+0000 7f1fbf7fe640 10 client.26147 handle_command_reply: tid=0 2024-05-20T18:05:24.463 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.464+0000 7f1fed5c3740 1 client.26147 shutdown 2024-05-20T18:05:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.467+0000 7f1fed5c3740 1 -- 172.21.15.129:0/3239455380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8f42f0 msgr2=0x5593ecccda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.467+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8f42f0 0x5593ecccda30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009230 tx=0x7f1fd4009260 comp rx=0 tx=0).stop 2024-05-20T18:05:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.467+0000 7f1fed5c3740 1 -- 172.21.15.129:0/3239455380 shutdown_connections 2024-05-20T18:05:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.467+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5593ec8b11c0 0x5593ecd64cc0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.467+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5593ec91d1b0 0x5593eccce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:24.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.467+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5593ec8fc4d0 0x5593ecccdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.467+0000 7f1fed5c3740 1 --2- 172.21.15.129:0/3239455380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593ec8f42f0 0x5593ecccda30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.467+0000 7f1fed5c3740 1 -- 172.21.15.129:0/3239455380 >> 172.21.15.129:0/3239455380 conn(0x5593ec911300 msgr2=0x5593eccc7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.467+0000 7f1fbf7fe640 0 client.26147 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.468+0000 7f1fed5c3740 1 -- 172.21.15.129:0/3239455380 shutdown_connections 2024-05-20T18:05:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.468+0000 7f1fed5c3740 1 -- 172.21.15.129:0/3239455380 wait complete. 2024-05-20T18:05:24.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.469+0000 7f1fed5c3740 20 client.26147 trim_cache size 0 max 16384 2024-05-20T18:05:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:05:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:05:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:05:24.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:05:24.469 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.471+0000 7f1fe53db640 1 -- 172.21.15.129:0/1906723355 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fc80a2da0 msgr2=0x7f1fc80a52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:24.469 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.471+0000 7f1fe53db640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fc80a2da0 0x7f1fc80a52b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4000c00 tx=0x7f1fd4005d50 comp rx=0 tx=0).stop 2024-05-20T18:05:24.469 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.471+0000 7f1fe53db640 1 -- 172.21.15.129:0/1906723355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe010f270 msgr2=0x7f1fe01195d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:24.469 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.471+0000 7f1fe53db640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe010f270 0x7f1fe01195d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800eb10 tx=0x7f1fd800c2d0 comp rx=0 tx=0).stop 2024-05-20T18:05:24.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.472+0000 7f1fe53db640 1 -- 172.21.15.129:0/1906723355 shutdown_connections 2024-05-20T18:05:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.472+0000 7f1fe53db640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1fc80a2da0 0x7f1fc80a52b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.472+0000 7f1fe53db640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe010f270 0x7f1fe01195d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.472+0000 7f1fe53db640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe010e890 0x7f1fe0119090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.472+0000 7f1fe53db640 1 --2- 172.21.15.129:0/1906723355 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fe0109090 0x7f1fe011ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.472+0000 7f1fe53db640 1 -- 172.21.15.129:0/1906723355 >> 172.21.15.129:0/1906723355 conn(0x7f1fe006e690 msgr2=0x7f1fe010c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.473+0000 7f1fe53db640 1 -- 172.21.15.129:0/1906723355 shutdown_connections 2024-05-20T18:05:24.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:24.473+0000 7f1fe53db640 1 -- 172.21.15.129:0/1906723355 wait complete. 2024-05-20T18:05:25.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:25 smithi154 ceph-mon[36482]: pgmap v210: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 667 MiB/s wr, 3.33k op/s 2024-05-20T18:05:25.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:25 smithi129 ceph-mon[32029]: pgmap v210: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 667 MiB/s wr, 3.33k op/s 2024-05-20T18:05:25.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:25 smithi204 ceph-mon[37076]: pgmap v210: 97 pgs: 97 active+clean; 55 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 667 MiB/s wr, 3.33k op/s 2024-05-20T18:05:27.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:27 smithi154 ceph-mon[36482]: pgmap v211: 97 pgs: 97 active+clean; 57 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 839 MiB/s wr, 3.12k op/s 2024-05-20T18:05:27.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:27 smithi129 ceph-mon[32029]: pgmap v211: 97 pgs: 97 active+clean; 57 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 839 MiB/s wr, 3.12k op/s 2024-05-20T18:05:27.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:27 smithi204 ceph-mon[37076]: pgmap v211: 97 pgs: 97 active+clean; 57 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 839 MiB/s wr, 3.12k op/s 2024-05-20T18:05:30.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:29 smithi154 ceph-mon[36482]: pgmap v212: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 668 MiB/s wr, 2.00k op/s 2024-05-20T18:05:30.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:29 smithi129 ceph-mon[32029]: pgmap v212: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 668 MiB/s wr, 2.00k op/s 2024-05-20T18:05:30.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:29 smithi204 ceph-mon[37076]: pgmap v212: 97 pgs: 97 active+clean; 58 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 668 MiB/s wr, 2.00k op/s 2024-05-20T18:05:32.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:31 smithi154 ceph-mon[36482]: pgmap v213: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 859 MiB/s wr, 2.27k op/s 2024-05-20T18:05:32.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:31 smithi129 ceph-mon[32029]: pgmap v213: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 859 MiB/s wr, 2.27k op/s 2024-05-20T18:05:32.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:31 smithi204 ceph-mon[37076]: pgmap v213: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 859 MiB/s wr, 2.27k op/s 2024-05-20T18:05:34.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:33 smithi154 ceph-mon[36482]: pgmap v214: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 691 MiB/s wr, 1.38k op/s 2024-05-20T18:05:34.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:33 smithi129 ceph-mon[32029]: pgmap v214: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 691 MiB/s wr, 1.38k op/s 2024-05-20T18:05:34.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:33 smithi204 ceph-mon[37076]: pgmap v214: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 691 MiB/s wr, 1.38k op/s 2024-05-20T18:05:36.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:35 smithi154 ceph-mon[36482]: pgmap v215: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 716 MiB/s wr, 1.32k op/s 2024-05-20T18:05:36.166 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:35 smithi129 ceph-mon[32029]: pgmap v215: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 716 MiB/s wr, 1.32k op/s 2024-05-20T18:05:36.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:35 smithi204 ceph-mon[37076]: pgmap v215: 97 pgs: 97 active+clean; 59 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 716 MiB/s wr, 1.32k op/s 2024-05-20T18:05:37.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:36 smithi154 ceph-mon[36482]: pgmap v216: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 933 MiB/s wr, 1.61k op/s 2024-05-20T18:05:37.169 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:36 smithi129 ceph-mon[32029]: pgmap v216: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 933 MiB/s wr, 1.61k op/s 2024-05-20T18:05:37.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:36 smithi204 ceph-mon[37076]: pgmap v216: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 933 MiB/s wr, 1.61k op/s 2024-05-20T18:05:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.869+0000 7f937d915640 1 -- 172.21.15.129:0/2947852456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937810f270 msgr2=0x7f9378108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.869+0000 7f937d915640 1 --2- 172.21.15.129:0/2947852456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937810f270 0x7f9378108b50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9370009ca0 tx=0x7f937002a8f0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.874+0000 7f937d915640 1 -- 172.21.15.129:0/2947852456 shutdown_connections 2024-05-20T18:05:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.874+0000 7f937d915640 1 --2- 172.21.15.129:0/2947852456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9378109090 0x7f9378109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.874+0000 7f937d915640 1 --2- 172.21.15.129:0/2947852456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937810f270 0x7f9378108b50 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.874+0000 7f937d915640 1 --2- 172.21.15.129:0/2947852456 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f937810e890 0x7f937810eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.874+0000 7f937d915640 1 -- 172.21.15.129:0/2947852456 >> 172.21.15.129:0/2947852456 conn(0x7f937806e690 msgr2=0x7f937806eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.875+0000 7f937d915640 1 -- 172.21.15.129:0/2947852456 shutdown_connections 2024-05-20T18:05:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.875+0000 7f937d915640 1 -- 172.21.15.129:0/2947852456 wait complete. 2024-05-20T18:05:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.876+0000 7f937d915640 1 Processor -- start 2024-05-20T18:05:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.876+0000 7f937d915640 1 -- start start 2024-05-20T18:05:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.877+0000 7f937d915640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9378109090 0x7f9378078ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.877+0000 7f937d915640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f937810e890 0x7f9378079200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.877+0000 7f937d915640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937810f270 0x7f9378073e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.877+0000 7f937d915640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9378121a90 con 0x7f937810f270 2024-05-20T18:05:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.877+0000 7f937d915640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9378121910 con 0x7f9378109090 2024-05-20T18:05:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.877+0000 7f937d915640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9378121c10 con 0x7f937810e890 2024-05-20T18:05:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.877+0000 7f937d114640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937810f270 0x7f9378073e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.878+0000 7f937d114640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937810f270 0x7f9378073e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39180/0 (socket says 172.21.15.129:39180) 2024-05-20T18:05:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.878+0000 7f937d114640 1 -- 172.21.15.129:0/3719519302 learned_addr learned my addr 172.21.15.129:0/3719519302 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.879+0000 7f937d114640 1 -- 172.21.15.129:0/3719519302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f937810e890 msgr2=0x7f9378079200 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:05:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.879+0000 7f937d114640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f937810e890 0x7f9378079200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.879+0000 7f937d114640 1 -- 172.21.15.129:0/3719519302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9378109090 msgr2=0x7f9378078ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T18:05:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.879+0000 7f937d114640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9378109090 0x7f9378078ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.879+0000 7f937d114640 1 -- 172.21.15.129:0/3719519302 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9378074650 con 0x7f937810f270 2024-05-20T18:05:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.880+0000 7f937d114640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937810f270 0x7f9378073e50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f936c00d750 tx=0x7f936c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:37.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.889+0000 7f9375ffb640 1 -- 172.21.15.129:0/3719519302 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c017070 con 0x7f937810f270 2024-05-20T18:05:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.889+0000 7f937d915640 1 -- 172.21.15.129:0/3719519302 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9378074940 con 0x7f937810f270 2024-05-20T18:05:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.889+0000 7f937d915640 1 -- 172.21.15.129:0/3719519302 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9378080ba0 con 0x7f937810f270 2024-05-20T18:05:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.895+0000 7f9385a1d740 1 Processor -- start 2024-05-20T18:05:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.904+0000 7f9385a1d740 1 -- start start 2024-05-20T18:05:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.905+0000 7f9385a1d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb911b0 0x55ab5ff30270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.905+0000 7f9385a1d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb682f0 0x55ab5ff307b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.906+0000 7f9385a1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 0x55ab5ff30cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.906+0000 7f9385a1d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ab5fe95340 con 0x55ab5fb911b0 2024-05-20T18:05:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.906+0000 7f937c913640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb682f0 0x55ab5ff307b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.906+0000 7f937c913640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb682f0 0x55ab5ff307b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55278/0 (socket says 172.21.15.129:55278) 2024-05-20T18:05:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.906+0000 7f937c913640 1 -- 172.21.15.129:0/957898679 learned_addr learned my addr 172.21.15.129:0/957898679 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.906+0000 7f9385a1d740 1 -- 172.21.15.129:0/957898679 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ab5fe3d290 con 0x55ab5fb682f0 2024-05-20T18:05:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.906+0000 7f9385a1d740 1 -- 172.21.15.129:0/957898679 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ab5fe91300 con 0x55ab5fb704d0 2024-05-20T18:05:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.906+0000 7f9375ffb640 1 -- 172.21.15.129:0/3719519302 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f936c005500 con 0x7f937810f270 2024-05-20T18:05:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.906+0000 7f9375ffb640 1 -- 172.21.15.129:0/3719519302 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c004d60 con 0x7f937810f270 2024-05-20T18:05:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.907+0000 7f937c913640 1 -- 172.21.15.129:0/957898679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 msgr2=0x55ab5ff30cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:05:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.907+0000 7f937c913640 1 --2- 172.21.15.129:0/957898679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 0x55ab5ff30cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.907+0000 7f937c913640 1 -- 172.21.15.129:0/957898679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb911b0 msgr2=0x55ab5ff30270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:05:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.907+0000 7f937c913640 1 --2- 172.21.15.129:0/957898679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb911b0 0x55ab5ff30270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.907+0000 7f937c913640 1 -- 172.21.15.129:0/957898679 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab5fe618e0 con 0x55ab5fb682f0 2024-05-20T18:05:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.907+0000 7f9375ffb640 1 -- 172.21.15.129:0/3719519302 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f936c010460 con 0x7f937810f270 2024-05-20T18:05:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.907+0000 7f937c913640 1 --2- 172.21.15.129:0/957898679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb682f0 0x55ab5ff307b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f936800ba00 tx=0x7f936800bec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=172c6511ea8dc6a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.909+0000 7f9375ffb640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f93600a2f30 0x7f93600a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:37.913 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.915+0000 7f9375ffb640 1 -- 172.21.15.129:0/3719519302 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f936c0cfe80 con 0x7f937810f270 2024-05-20T18:05:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.916+0000 7f935effd640 1 -- 172.21.15.129:0/957898679 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368016070 con 0x55ab5fb682f0 2024-05-20T18:05:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.917+0000 7f935effd640 1 -- 172.21.15.129:0/957898679 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9368005500 con 0x55ab5fb682f0 2024-05-20T18:05:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.919+0000 7f9385a1d740 1 -- 172.21.15.129:0/957898679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb682f0 msgr2=0x55ab5ff307b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.919+0000 7f9385a1d740 1 --2- 172.21.15.129:0/957898679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb682f0 0x55ab5ff307b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f936800ba00 tx=0x7f936800bec0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.919+0000 7f9385a1d740 1 -- 172.21.15.129:0/957898679 shutdown_connections 2024-05-20T18:05:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.919+0000 7f9385a1d740 1 --2- 172.21.15.129:0/957898679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb682f0 0x55ab5ff307b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.919+0000 7f9385a1d740 1 --2- 172.21.15.129:0/957898679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 0x55ab5ff30cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.919+0000 7f9385a1d740 1 --2- 172.21.15.129:0/957898679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb911b0 0x55ab5ff30270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.919+0000 7f9385a1d740 1 -- 172.21.15.129:0/957898679 >> 172.21.15.129:0/957898679 conn(0x55ab5fb85300 msgr2=0x55ab5ff2ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.920+0000 7f9385a1d740 1 -- 172.21.15.129:0/957898679 shutdown_connections 2024-05-20T18:05:37.919 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.920+0000 7f9385a1d740 1 -- 172.21.15.129:0/957898679 wait complete. 2024-05-20T18:05:37.919 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.921+0000 7f9385a1d740 1 -- start start 2024-05-20T18:05:37.919 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.921+0000 7f9385a1d740 1 Processor -- start 2024-05-20T18:05:37.920 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.922+0000 7f9385a1d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb704d0 0x55ab5ff3a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.923+0000 7f9385a1d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb911b0 0x55ab5ff3a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.923+0000 7f9385a1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff46980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.923+0000 7f937c913640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb704d0 0x55ab5ff3a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.923+0000 7f937c913640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb704d0 0x55ab5ff3a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:55288/0 (socket says 172.21.15.129:55288) 2024-05-20T18:05:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.923+0000 7f937c913640 1 -- 172.21.15.129:0/1356842474 learned_addr learned my addr 172.21.15.129:0/1356842474 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.924+0000 7f9385a1d740 1 -- 172.21.15.129:0/1356842474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ab5ff3d3f0 con 0x55ab5fb911b0 2024-05-20T18:05:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.924+0000 7f9385a1d740 1 -- 172.21.15.129:0/1356842474 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ab5fe50790 con 0x55ab5fb704d0 2024-05-20T18:05:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.924+0000 7f9385a1d740 1 -- 172.21.15.129:0/1356842474 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ab5ff3d570 con 0x55ab5fb4fd80 2024-05-20T18:05:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.924+0000 7f937d114640 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff46980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.924+0000 7f9385a1d740 10 client.0 _ll_register_callbacks cb 0x55ab5fe8ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.925+0000 7f937c913640 1 -- 172.21.15.129:0/1356842474 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb4fd80 msgr2=0x55ab5ff46980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.925+0000 7f937c913640 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff46980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.925+0000 7f937c913640 1 -- 172.21.15.129:0/1356842474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb911b0 msgr2=0x55ab5ff3a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.925+0000 7f937c913640 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb911b0 0x55ab5ff3a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.925+0000 7f937c913640 1 -- 172.21.15.129:0/1356842474 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab5fba9210 con 0x55ab5fb704d0 2024-05-20T18:05:37.929 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.931+0000 7f937c913640 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb704d0 0x55ab5ff3a3d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f936800dc10 tx=0x7f936800dc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5242ab3de4e794e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:37.929 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.931+0000 7f9377fff640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f93600a2f30 0x7f93600a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.932+0000 7f9357fff640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:05:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.934+0000 7f9357fff640 1 -- 172.21.15.129:0/1356842474 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368016070 con 0x55ab5fb704d0 2024-05-20T18:05:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.935+0000 7f9385a1d740 1 -- 172.21.15.129:0/1356842474 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ab5fea2a70 con 0x55ab5fb704d0 2024-05-20T18:05:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.935+0000 7f9357fff640 1 -- 172.21.15.129:0/1356842474 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f9368005220 con 0x55ab5fb704d0 2024-05-20T18:05:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.935+0000 7f9357fff640 1 -- 172.21.15.129:0/1356842474 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9368005420 con 0x55ab5fb704d0 2024-05-20T18:05:37.934 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.936+0000 7f9377fff640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f93600a2f30 0x7f93600a5440 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55ab5ff26be0 tx=0x7f9370000f40 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.937+0000 7f9385a1d740 10 client.26156 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.937+0000 7f9385a1d740 1 -- 172.21.15.129:0/1356842474 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ab5fc75620 con 0x55ab5fb704d0 2024-05-20T18:05:37.935 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.937+0000 7f9357fff640 1 -- 172.21.15.129:0/1356842474 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ab5fea2a70 con 0x55ab5fb704d0 2024-05-20T18:05:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.938+0000 7f9357fff640 1 -- 172.21.15.129:0/1356842474 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f9368021020 con 0x55ab5fb704d0 2024-05-20T18:05:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.938+0000 7f9385a1d740 10 client.26156 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.938+0000 7f9385a1d740 10 client.26156 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.938+0000 7f9385a1d740 20 client.26156 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.938+0000 7f9385a1d740 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ab5fb251c0 0x55ab5ff4cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.938+0000 7f9385a1d740 4 client.26156 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.938+0000 7f9385a1d740 1 -- 172.21.15.129:0/1356842474 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab5fe9cdc0 con 0x55ab5fb251c0 2024-05-20T18:05:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.939+0000 7f9377fff640 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ab5fb251c0 0x55ab5ff4cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.946+0000 7f9377fff640 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ab5fb251c0 0x55ab5ff4cd70 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d13765bbd3160370 server_cookie=2ce9544ce19676a0 in_seq=0 out_seq=0 2024-05-20T18:05:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.946+0000 7f9357fff640 10 client.26156 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:37.953 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.954+0000 7f9357fff640 1 -- 172.21.15.129:0/1356842474 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ab5fe9cdc0 con 0x55ab5fb251c0 2024-05-20T18:05:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.954+0000 7f9357fff640 10 client.26156 handle_command_reply: tid=0 2024-05-20T18:05:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.954+0000 7f9385a1d740 1 client.26156 shutdown 2024-05-20T18:05:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.955+0000 7f9385a1d740 1 -- 172.21.15.129:0/1356842474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb704d0 msgr2=0x55ab5ff3a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.955+0000 7f9385a1d740 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb704d0 0x55ab5ff3a3d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f936800dc10 tx=0x7f936800dc40 comp rx=0 tx=0).stop 2024-05-20T18:05:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.961+0000 7f9385a1d740 1 -- 172.21.15.129:0/1356842474 shutdown_connections 2024-05-20T18:05:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.961+0000 7f9385a1d740 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ab5fb251c0 0x55ab5ff4cd70 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.961+0000 7f9385a1d740 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff46980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.961+0000 7f9385a1d740 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb911b0 0x55ab5ff3a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.961+0000 7f9385a1d740 1 --2- 172.21.15.129:0/1356842474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb704d0 0x55ab5ff3a3d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.961+0000 7f9385a1d740 1 -- 172.21.15.129:0/1356842474 >> 172.21.15.129:0/1356842474 conn(0x55ab5fb85300 msgr2=0x55ab5ff31f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.962+0000 7f9357fff640 0 client.26156 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.962+0000 7f9385a1d740 1 -- 172.21.15.129:0/1356842474 shutdown_connections 2024-05-20T18:05:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.966+0000 7f9385a1d740 1 -- 172.21.15.129:0/1356842474 wait complete. 2024-05-20T18:05:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:37.966+0000 7f9385a1d740 20 client.26156 trim_cache size 0 max 16384 2024-05-20T18:05:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.027+0000 7f9385a1d740 1 Processor -- start 2024-05-20T18:05:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.030+0000 7f9385a1d740 1 -- start start 2024-05-20T18:05:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.031+0000 7f9385a1d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff3c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.032+0000 7f937c913640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff3c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.032+0000 7f937c913640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff3c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39198/0 (socket says 172.21.15.129:39198) 2024-05-20T18:05:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.032+0000 7f9385a1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 0x55ab5ff52570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.032+0000 7f937c913640 1 -- 172.21.15.129:0/908423071 learned_addr learned my addr 172.21.15.129:0/908423071 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.032+0000 7f9385a1d740 1 --2- 172.21.15.129:0/908423071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb911b0 0x55ab5ff52ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.032+0000 7f9385a1d740 1 -- 172.21.15.129:0/908423071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f936400c8d0 con 0x55ab5fb4fd80 2024-05-20T18:05:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.032+0000 7f9385a1d740 1 -- 172.21.15.129:0/908423071 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f936400f150 con 0x55ab5fb911b0 2024-05-20T18:05:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.032+0000 7f9385a1d740 1 -- 172.21.15.129:0/908423071 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f936400cf40 con 0x55ab5fb704d0 2024-05-20T18:05:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.033+0000 7f937c913640 1 -- 172.21.15.129:0/908423071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 msgr2=0x55ab5ff52570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:05:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.033+0000 7f937c913640 1 --2- 172.21.15.129:0/908423071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 0x55ab5ff52570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.033+0000 7f937c913640 1 -- 172.21.15.129:0/908423071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb911b0 msgr2=0x55ab5ff52ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:05:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.033+0000 7f937c913640 1 --2- 172.21.15.129:0/908423071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb911b0 0x55ab5ff52ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.033+0000 7f937c913640 1 -- 172.21.15.129:0/908423071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab5ff39380 con 0x55ab5fb4fd80 2024-05-20T18:05:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.034+0000 7f937c913640 1 --2- 172.21.15.129:0/908423071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff3c200 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f93680059f0 tx=0x7f9368005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9de5a45e27db98 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.043+0000 7f935e7fc640 1 -- 172.21.15.129:0/908423071 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368030070 con 0x55ab5fb4fd80 2024-05-20T18:05:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.043+0000 7f935e7fc640 1 -- 172.21.15.129:0/908423071 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9368003c50 con 0x55ab5fb4fd80 2024-05-20T18:05:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.043+0000 7f935e7fc640 1 -- 172.21.15.129:0/908423071 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368016040 con 0x55ab5fb4fd80 2024-05-20T18:05:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.048+0000 7f9385a1d740 1 -- 172.21.15.129:0/908423071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 msgr2=0x55ab5ff3c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.048+0000 7f9385a1d740 1 --2- 172.21.15.129:0/908423071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff3c200 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f93680059f0 tx=0x7f9368005fa0 comp rx=0 tx=0).stop 2024-05-20T18:05:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.053+0000 7f9385a1d740 1 -- 172.21.15.129:0/908423071 shutdown_connections 2024-05-20T18:05:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.053+0000 7f9385a1d740 1 --2- 172.21.15.129:0/908423071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb911b0 0x55ab5ff52ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.053+0000 7f9385a1d740 1 --2- 172.21.15.129:0/908423071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 0x55ab5ff52570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.053+0000 7f9385a1d740 1 --2- 172.21.15.129:0/908423071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff3c200 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.053+0000 7f9385a1d740 1 -- 172.21.15.129:0/908423071 >> 172.21.15.129:0/908423071 conn(0x55ab5fb85300 msgr2=0x55ab5ff3baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.060+0000 7f9385a1d740 1 -- 172.21.15.129:0/908423071 shutdown_connections 2024-05-20T18:05:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.064+0000 7f9385a1d740 1 -- 172.21.15.129:0/908423071 wait complete. 2024-05-20T18:05:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.065+0000 7f9385a1d740 1 -- start start 2024-05-20T18:05:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.066+0000 7f9385a1d740 1 Processor -- start 2024-05-20T18:05:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.067+0000 7f9385a1d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff42e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.067+0000 7f9385a1d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 0x55ab5ff3d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.067+0000 7f9385a1d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb911b0 0x55ab5ff3dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.067+0000 7f9385a1d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f936400d610 con 0x55ab5fb4fd80 2024-05-20T18:05:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.067+0000 7f9385a1d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f936400ea80 con 0x55ab5fb911b0 2024-05-20T18:05:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.067+0000 7f9385a1d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f936400dce0 con 0x55ab5fb704d0 2024-05-20T18:05:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.067+0000 7f9385a1d740 10 client.0 _ll_register_callbacks cb 0x7f936800b850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.068+0000 7f937c913640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff42e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:38.070 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.068+0000 7f937c913640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff42e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39200/0 (socket says 172.21.15.129:39200) 2024-05-20T18:05:38.070 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.068+0000 7f937c913640 1 -- 172.21.15.129:0/3594634300 learned_addr learned my addr 172.21.15.129:0/3594634300 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:38.070 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.070+0000 7f937c913640 1 -- 172.21.15.129:0/3594634300 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 msgr2=0x55ab5ff3d8a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:05:38.070 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.070+0000 7f937c913640 1 --2- 172.21.15.129:0/3594634300 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 0x55ab5ff3d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:38.070 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.070+0000 7f937c913640 1 -- 172.21.15.129:0/3594634300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb911b0 msgr2=0x55ab5ff3dde0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:05:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.070+0000 7f937c913640 1 --2- 172.21.15.129:0/3594634300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb911b0 0x55ab5ff3dde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.070+0000 7f937c913640 1 -- 172.21.15.129:0/3594634300 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab5ff3e3f0 con 0x55ab5fb4fd80 2024-05-20T18:05:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.072+0000 7f937c913640 1 --2- 172.21.15.129:0/3594634300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff42e30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f93680057d0 tx=0x7f936800b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711b06ad063c3145 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.073+0000 7f9357fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:05:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.083+0000 7f9357fff640 1 -- 172.21.15.129:0/3594634300 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368039070 con 0x55ab5fb4fd80 2024-05-20T18:05:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.083+0000 7f9385a1d740 1 -- 172.21.15.129:0/3594634300 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9368003c50 con 0x55ab5fb4fd80 2024-05-20T18:05:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.083+0000 7f9357fff640 1 -- 172.21.15.129:0/3594634300 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f9368010a80 con 0x55ab5fb4fd80 2024-05-20T18:05:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.097+0000 7f9357fff640 1 -- 172.21.15.129:0/3594634300 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9368043050 con 0x55ab5fb4fd80 2024-05-20T18:05:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.097+0000 7f9385a1d740 10 client.16425 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.097+0000 7f9385a1d740 1 -- 172.21.15.129:0/3594634300 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ab5ffb3ef0 con 0x55ab5fb4fd80 2024-05-20T18:05:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.109+0000 7f9357fff640 1 -- 172.21.15.129:0/3594634300 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9368003c50 con 0x55ab5fb4fd80 2024-05-20T18:05:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.109+0000 7f9357fff640 1 -- 172.21.15.129:0/3594634300 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f9368021070 con 0x55ab5fb4fd80 2024-05-20T18:05:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.109+0000 7f9385a1d740 10 client.16425 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.109+0000 7f9385a1d740 10 client.16425 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.109+0000 7f9385a1d740 20 client.16425 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.109+0000 7f9385a1d740 1 --2- 172.21.15.129:0/3594634300 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ab5ffb4600 0x55ab5ffb6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.109+0000 7f9385a1d740 4 client.16425 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:05:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.109+0000 7f9385a1d740 1 -- 172.21.15.129:0/3594634300 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab5fe1a8e0 con 0x55ab5ffb4600 2024-05-20T18:05:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.111+0000 7f9377fff640 1 --2- 172.21.15.129:0/3594634300 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ab5ffb4600 0x55ab5ffb6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.127+0000 7f9377fff640 1 --2- 172.21.15.129:0/3594634300 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ab5ffb4600 0x55ab5ffb6a20 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcb97412b404b97 server_cookie=3c1e4f93637c3c1a in_seq=0 out_seq=0 2024-05-20T18:05:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:38.128+0000 7f9357fff640 10 client.16425 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:39.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:39 smithi154 ceph-mon[36482]: pgmap v217: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 717 MiB/s wr, 1.37k op/s 2024-05-20T18:05:39.424 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:39 smithi129 ceph-mon[32029]: pgmap v217: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 717 MiB/s wr, 1.37k op/s 2024-05-20T18:05:39.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:39 smithi204 ceph-mon[37076]: pgmap v217: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 717 MiB/s wr, 1.37k op/s 2024-05-20T18:05:41.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:41 smithi154 ceph-mon[36482]: pgmap v218: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 923 MiB/s wr, 1.85k op/s 2024-05-20T18:05:41.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:41 smithi129 ceph-mon[32029]: pgmap v218: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 923 MiB/s wr, 1.85k op/s 2024-05-20T18:05:41.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:41 smithi204 ceph-mon[37076]: pgmap v218: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 923 MiB/s wr, 1.85k op/s 2024-05-20T18:05:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.275+0000 7f9357fff640 1 -- 172.21.15.129:0/3594634300 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ab5fe1a8e0 con 0x55ab5ffb4600 2024-05-20T18:05:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.275+0000 7f9357fff640 10 client.16425 handle_command_reply: tid=0 2024-05-20T18:05:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.275+0000 7f9385a1d740 1 client.16425 shutdown 2024-05-20T18:05:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.279+0000 7f9385a1d740 1 -- 172.21.15.129:0/3594634300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 msgr2=0x55ab5ff42e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.279+0000 7f9385a1d740 1 --2- 172.21.15.129:0/3594634300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff42e30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f93680057d0 tx=0x7f936800b560 comp rx=0 tx=0).stop 2024-05-20T18:05:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.279+0000 7f9385a1d740 1 -- 172.21.15.129:0/3594634300 shutdown_connections 2024-05-20T18:05:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.279+0000 7f9385a1d740 1 --2- 172.21.15.129:0/3594634300 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ab5ffb4600 0x55ab5ffb6a20 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.279+0000 7f9385a1d740 1 --2- 172.21.15.129:0/3594634300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab5fb911b0 0x55ab5ff3dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.279+0000 7f9357fff640 0 client.16425 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.279+0000 7f9385a1d740 1 --2- 172.21.15.129:0/3594634300 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ab5fb704d0 0x55ab5ff3d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.280+0000 7f9385a1d740 1 --2- 172.21.15.129:0/3594634300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ab5fb4fd80 0x55ab5ff42e30 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.280+0000 7f9385a1d740 1 -- 172.21.15.129:0/3594634300 >> 172.21.15.129:0/3594634300 conn(0x55ab5fb85300 msgr2=0x55ab5ff3c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.285+0000 7f9385a1d740 1 -- 172.21.15.129:0/3594634300 shutdown_connections 2024-05-20T18:05:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.285+0000 7f9385a1d740 1 -- 172.21.15.129:0/3594634300 wait complete. 2024-05-20T18:05:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.285+0000 7f9385a1d740 20 client.16425 trim_cache size 0 max 16384 2024-05-20T18:05:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:05:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:05:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:05:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:05:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 -- 172.21.15.129:0/3719519302 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f93600a2f30 msgr2=0x7f93600a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f93600a2f30 0x7f93600a5440 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55ab5ff26be0 tx=0x7f9370000f40 comp rx=0 tx=0).stop 2024-05-20T18:05:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 -- 172.21.15.129:0/3719519302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937810f270 msgr2=0x7f9378073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937810f270 0x7f9378073e50 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f936c00d750 tx=0x7f936c00dc10 comp rx=0 tx=0).stop 2024-05-20T18:05:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 -- 172.21.15.129:0/3719519302 shutdown_connections 2024-05-20T18:05:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f93600a2f30 0x7f93600a5440 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f937810f270 0x7f9378073e50 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f937810e890 0x7f9378079200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 --2- 172.21.15.129:0/3719519302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9378109090 0x7f9378078ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 -- 172.21.15.129:0/3719519302 >> 172.21.15.129:0/3719519302 conn(0x7f937806e690 msgr2=0x7f937810c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 -- 172.21.15.129:0/3719519302 shutdown_connections 2024-05-20T18:05:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:42.293+0000 7f937d915640 1 -- 172.21.15.129:0/3719519302 wait complete. 2024-05-20T18:05:43.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:43 smithi154 ceph-mon[36482]: pgmap v219: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 718 MiB/s wr, 1.38k op/s 2024-05-20T18:05:43.415 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:43 smithi129 ceph-mon[32029]: pgmap v219: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 718 MiB/s wr, 1.38k op/s 2024-05-20T18:05:43.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:43 smithi204 ceph-mon[37076]: pgmap v219: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 718 MiB/s wr, 1.38k op/s 2024-05-20T18:05:44.287 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:05:44.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:05:44.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.690+0000 7f7b067d5640 1 -- 172.21.15.129:0/3213165311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b00119770 msgr2=0x7f7b0011bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:44.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.690+0000 7f7b067d5640 1 --2- 172.21.15.129:0/3213165311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b00119770 0x7f7b0011bb90 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009ca0 tx=0x7f7af802a970 comp rx=0 tx=0).stop 2024-05-20T18:05:44.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.699+0000 7f7b067d5640 1 -- 172.21.15.129:0/3213165311 shutdown_connections 2024-05-20T18:05:44.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.699+0000 7f7b067d5640 1 --2- 172.21.15.129:0/3213165311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b0011c0d0 0x7f7b0011e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.699+0000 7f7b067d5640 1 --2- 172.21.15.129:0/3213165311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b00119770 0x7f7b0011bb90 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.699+0000 7f7b067d5640 1 --2- 172.21.15.129:0/3213165311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b00109060 0x7f7b00109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.699+0000 7f7b067d5640 1 -- 172.21.15.129:0/3213165311 >> 172.21.15.129:0/3213165311 conn(0x7f7b0006d750 msgr2=0x7f7b0006dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:44.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.701+0000 7f7b067d5640 1 -- 172.21.15.129:0/3213165311 shutdown_connections 2024-05-20T18:05:44.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.703+0000 7f7b067d5640 1 -- 172.21.15.129:0/3213165311 wait complete. 2024-05-20T18:05:44.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.705+0000 7f7b067d5640 1 Processor -- start 2024-05-20T18:05:44.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.705+0000 7f7b067d5640 1 -- start start 2024-05-20T18:05:44.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.706+0000 7f7b067d5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b00109060 0x7f7b001a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.706+0000 7f7b067d5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b00119770 0x7f7b001a8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.706+0000 7f7b067d5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b0011c0d0 0x7f7b001ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.706+0000 7f7b067d5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b00121d00 con 0x7f7b0011c0d0 2024-05-20T18:05:44.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.706+0000 7f7b067d5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b00121b80 con 0x7f7b00109060 2024-05-20T18:05:44.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.706+0000 7f7b067d5640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7b00121e80 con 0x7f7b00119770 2024-05-20T18:05:44.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.708+0000 7f7b04d72640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b0011c0d0 0x7f7b001ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.708+0000 7f7b04d72640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b0011c0d0 0x7f7b001ace60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49402/0 (socket says 172.21.15.129:49402) 2024-05-20T18:05:44.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.708+0000 7f7b04d72640 1 -- 172.21.15.129:0/3806201311 learned_addr learned my addr 172.21.15.129:0/3806201311 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:44.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.711+0000 7f7afffff640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b00109060 0x7f7b001a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.711+0000 7f7aff7fe640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b00119770 0x7f7b001a8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.716+0000 7f7afffff640 1 -- 172.21.15.129:0/3806201311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b00119770 msgr2=0x7f7b001a8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:44.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.716+0000 7f7afffff640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b00119770 0x7f7b001a8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.716+0000 7f7afffff640 1 -- 172.21.15.129:0/3806201311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b0011c0d0 msgr2=0x7f7b001ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:44.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.716+0000 7f7afffff640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b0011c0d0 0x7f7b001ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.717+0000 7f7afffff640 1 -- 172.21.15.129:0/3806201311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b001ad4a0 con 0x7f7b00109060 2024-05-20T18:05:44.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.722+0000 7f7afffff640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b00109060 0x7f7b001a8640 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009fd0 tx=0x7f7af000b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:44.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.725+0000 7f7afd7fa640 1 -- 172.21.15.129:0/3806201311 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af0015020 con 0x7f7b00109060 2024-05-20T18:05:44.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.726+0000 7f7b067d5640 1 -- 172.21.15.129:0/3806201311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b001ad790 con 0x7f7b00109060 2024-05-20T18:05:44.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.726+0000 7f7b067d5640 1 -- 172.21.15.129:0/3806201311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b00110f60 con 0x7f7b00109060 2024-05-20T18:05:44.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.726+0000 7f7afd7fa640 1 -- 172.21.15.129:0/3806201311 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7af0011d10 con 0x7f7b00109060 2024-05-20T18:05:44.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.726+0000 7f7afd7fa640 1 -- 172.21.15.129:0/3806201311 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af000f2e0 con 0x7f7b00109060 2024-05-20T18:05:44.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.736+0000 7f7b0d680740 1 Processor -- start 2024-05-20T18:05:44.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.736+0000 7f7b0d680740 1 -- start start 2024-05-20T18:05:44.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.737+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed4237390 0x555ed45d5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.737+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45d6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.738+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45d67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.738+0000 7f7b0d680740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555ed453bd00 con 0x555ed42166b0 2024-05-20T18:05:44.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.738+0000 7f7b0d680740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555ed44e34c0 con 0x555ed420e4d0 2024-05-20T18:05:44.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.738+0000 7f7b0d680740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x555ed4537cc0 con 0x555ed4237390 2024-05-20T18:05:44.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.738+0000 7f7afffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45d6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.738+0000 7f7afffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45d6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:46974/0 (socket says 172.21.15.129:46974) 2024-05-20T18:05:44.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.738+0000 7f7afffff640 1 -- 172.21.15.129:0/3823227801 learned_addr learned my addr 172.21.15.129:0/3823227801 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:44.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.738+0000 7f7aff7fe640 1 --2- 172.21.15.129:0/3823227801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45d67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.739+0000 7f7b04d72640 1 --2- 172.21.15.129:0/3823227801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed4237390 0x555ed45d5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.740+0000 7f7afffff640 1 -- 172.21.15.129:0/3823227801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed4237390 msgr2=0x555ed45d5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:44.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.740+0000 7f7afffff640 1 --2- 172.21.15.129:0/3823227801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed4237390 0x555ed45d5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.740+0000 7f7afffff640 1 -- 172.21.15.129:0/3823227801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 msgr2=0x555ed45d67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:44.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.740+0000 7f7afffff640 1 --2- 172.21.15.129:0/3823227801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45d67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.740+0000 7f7afffff640 1 -- 172.21.15.129:0/3823227801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ed43197d0 con 0x555ed420e4d0 2024-05-20T18:05:44.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.743+0000 7f7afd7fa640 1 -- 172.21.15.129:0/3806201311 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7af004c070 con 0x7f7b00109060 2024-05-20T18:05:44.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.746+0000 7f7afffff640 1 --2- 172.21.15.129:0/3823227801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45d6270 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7af0010920 tx=0x7f7af0011830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d32f027399db38ff server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:44.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.746+0000 7f7afd7fa640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7ae80a2dc0 0x7f7ae80a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.747+0000 7f7afd7fa640 1 -- 172.21.15.129:0/3806201311 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f7af00d0110 con 0x7f7b00109060 2024-05-20T18:05:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.747+0000 7f7aee7fc640 1 -- 172.21.15.129:0/3823227801 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af00d5070 con 0x555ed420e4d0 2024-05-20T18:05:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.747+0000 7f7aee7fc640 1 -- 172.21.15.129:0/3823227801 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7af009ca00 con 0x555ed420e4d0 2024-05-20T18:05:44.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.749+0000 7f7b0d680740 1 -- 172.21.15.129:0/3823227801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 msgr2=0x555ed45d6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:44.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.749+0000 7f7b0d680740 1 --2- 172.21.15.129:0/3823227801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45d6270 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7af0010920 tx=0x7f7af0011830 comp rx=0 tx=0).stop 2024-05-20T18:05:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.750+0000 7f7b04d72640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7ae80a2dc0 0x7f7ae80a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.750+0000 7f7b0d680740 1 -- 172.21.15.129:0/3823227801 shutdown_connections 2024-05-20T18:05:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.750+0000 7f7b0d680740 1 --2- 172.21.15.129:0/3823227801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45d6270 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.750+0000 7f7b0d680740 1 --2- 172.21.15.129:0/3823227801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45d67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.750+0000 7f7b0d680740 1 --2- 172.21.15.129:0/3823227801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed4237390 0x555ed45d5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.750+0000 7f7b0d680740 1 -- 172.21.15.129:0/3823227801 >> 172.21.15.129:0/3823227801 conn(0x555ed422b4e0 msgr2=0x555ed45d24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:44.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.753+0000 7f7b0d680740 1 -- 172.21.15.129:0/3823227801 shutdown_connections 2024-05-20T18:05:44.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.753+0000 7f7b0d680740 1 -- 172.21.15.129:0/3823227801 wait complete. 2024-05-20T18:05:44.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.754+0000 7f7b0d680740 1 -- start start 2024-05-20T18:05:44.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.754+0000 7f7b0d680740 1 Processor -- start 2024-05-20T18:05:44.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.755+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed420e4d0 0x555ed45e1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.755+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45e2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.755+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed4237390 0x555ed45e02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.755+0000 7f7b0d680740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555ed45e2e40 con 0x555ed42166b0 2024-05-20T18:05:44.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.755+0000 7f7b0d680740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555ed44f6b90 con 0x555ed4237390 2024-05-20T18:05:44.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.755+0000 7f7b0d680740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x555ed45e2fc0 con 0x555ed420e4d0 2024-05-20T18:05:44.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.755+0000 7f7b0d680740 10 client.0 _ll_register_callbacks cb 0x555ed4532da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:44.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.757+0000 7f7aff7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45e2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.757+0000 7f7aff7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45e2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49430/0 (socket says 172.21.15.129:49430) 2024-05-20T18:05:44.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.757+0000 7f7aff7fe640 1 -- 172.21.15.129:0/2813049441 learned_addr learned my addr 172.21.15.129:0/2813049441 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:44.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.757+0000 7f7afffff640 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed420e4d0 0x555ed45e1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.758+0000 7f7b04d72640 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed4237390 0x555ed45e02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.758+0000 7f7b04d72640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7ae80a2dc0 0x7f7ae80a52d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x555ed45d2ea0 tx=0x7f7af4009190 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:44.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.758+0000 7f7afffff640 1 -- 172.21.15.129:0/2813049441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed4237390 msgr2=0x555ed45e02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:44.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.758+0000 7f7afffff640 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed4237390 0x555ed45e02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.758+0000 7f7afffff640 1 -- 172.21.15.129:0/2813049441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 msgr2=0x555ed45e2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:44.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.758+0000 7f7afffff640 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45e2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.758+0000 7f7afffff640 1 -- 172.21.15.129:0/2813049441 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ed424f3f0 con 0x555ed420e4d0 2024-05-20T18:05:44.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.759+0000 7f7afffff640 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed420e4d0 0x555ed45e1e20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7af0098e60 tx=0x7f7af009c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fe522d799110d38 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:44.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.759+0000 7f7ae37fe640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:05:44.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.759+0000 7f7ae37fe640 1 -- 172.21.15.129:0/2813049441 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af00d5030 con 0x555ed420e4d0 2024-05-20T18:05:44.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.759+0000 7f7ae37fe640 1 -- 172.21.15.129:0/2813049441 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f7af0099070 con 0x555ed420e4d0 2024-05-20T18:05:44.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.759+0000 7f7b0d680740 1 -- 172.21.15.129:0/2813049441 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555ed4549430 con 0x555ed420e4d0 2024-05-20T18:05:44.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.769+0000 7f7ae37fe640 1 -- 172.21.15.129:0/2813049441 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7af00a0d00 con 0x555ed420e4d0 2024-05-20T18:05:44.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.769+0000 7f7b0d680740 10 client.26212 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:44.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.769+0000 7f7b0d680740 1 -- 172.21.15.129:0/2813049441 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555ed420f0f0 con 0x555ed420e4d0 2024-05-20T18:05:44.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.771+0000 7f7ae37fe640 1 -- 172.21.15.129:0/2813049441 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555ed4549430 con 0x555ed420e4d0 2024-05-20T18:05:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.771+0000 7f7ae37fe640 1 -- 172.21.15.129:0/2813049441 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f7af0022060 con 0x555ed420e4d0 2024-05-20T18:05:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.772+0000 7f7b0d680740 10 client.26212 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.772+0000 7f7b0d680740 10 client.26212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:44.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.772+0000 7f7b0d680740 20 client.26212 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:44.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.772+0000 7f7b0d680740 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x555ed41f5f60 0x555ed45f2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.772+0000 7f7b0d680740 4 client.26212 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:44.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.772+0000 7f7b0d680740 1 -- 172.21.15.129:0/2813049441 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ed4543780 con 0x555ed41f5f60 2024-05-20T18:05:44.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.778+0000 7f7aff7fe640 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x555ed41f5f60 0x555ed45f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.789+0000 7f7aff7fe640 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x555ed41f5f60 0x555ed45f2850 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80f6fa7ca9946d51 server_cookie=13bdbb13cb93a5af in_seq=0 out_seq=0 2024-05-20T18:05:44.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.792+0000 7f7ae37fe640 10 client.26212 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:44.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.796+0000 7f7ae37fe640 1 -- 172.21.15.129:0/2813049441 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555ed4543780 con 0x555ed41f5f60 2024-05-20T18:05:44.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.796+0000 7f7ae37fe640 10 client.26212 handle_command_reply: tid=0 2024-05-20T18:05:44.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.796+0000 7f7b0d680740 1 client.26212 shutdown 2024-05-20T18:05:44.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.801+0000 7f7b0d680740 1 -- 172.21.15.129:0/2813049441 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed420e4d0 msgr2=0x555ed45e1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:44.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.801+0000 7f7b0d680740 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed420e4d0 0x555ed45e1e20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7af0098e60 tx=0x7f7af009c850 comp rx=0 tx=0).stop 2024-05-20T18:05:44.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.801+0000 7f7b0d680740 1 -- 172.21.15.129:0/2813049441 shutdown_connections 2024-05-20T18:05:44.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.801+0000 7f7b0d680740 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x555ed41f5f60 0x555ed45f2850 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.801+0000 7f7b0d680740 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed4237390 0x555ed45e02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.801+0000 7f7b0d680740 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45e2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.801+0000 7f7b0d680740 1 --2- 172.21.15.129:0/2813049441 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed420e4d0 0x555ed45e1e20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.801+0000 7f7b0d680740 1 -- 172.21.15.129:0/2813049441 >> 172.21.15.129:0/2813049441 conn(0x555ed422b4e0 msgr2=0x555ed45d7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:44.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.801+0000 7f7ae37fe640 0 client.26212 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:44.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.803+0000 7f7b0d680740 1 -- 172.21.15.129:0/2813049441 shutdown_connections 2024-05-20T18:05:44.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.803+0000 7f7b0d680740 1 -- 172.21.15.129:0/2813049441 wait complete. 2024-05-20T18:05:44.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.804+0000 7f7b0d680740 20 client.26212 trim_cache size 0 max 16384 2024-05-20T18:05:44.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.872+0000 7f7b0d680740 1 Processor -- start 2024-05-20T18:05:44.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.872+0000 7f7b0d680740 1 -- start start 2024-05-20T18:05:44.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.873+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45df5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.874+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed42166b0 0x555ed45d7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.874+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed4237390 0x555ed45d99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.874+0000 7f7afffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45df5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.874+0000 7f7afffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45df5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:47000/0 (socket says 172.21.15.129:47000) 2024-05-20T18:05:44.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.874+0000 7f7aff7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed42166b0 0x555ed45d7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.874+0000 7f7aff7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed42166b0 0x555ed45d7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:36066/0 (socket says 172.21.15.129:36066) 2024-05-20T18:05:44.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.878+0000 7f7b04d72640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed4237390 0x555ed45d99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.878+0000 7f7b04d72640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed4237390 0x555ed45d99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49436/0 (socket says 172.21.15.129:49436) 2024-05-20T18:05:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.879+0000 7f7b0d680740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae400c800 con 0x555ed4237390 2024-05-20T18:05:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.879+0000 7f7b0d680740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae400f080 con 0x555ed420e4d0 2024-05-20T18:05:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.879+0000 7f7b0d680740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae400ce70 con 0x555ed42166b0 2024-05-20T18:05:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.882+0000 7f7afffff640 1 -- 172.21.15.129:0/3837800683 learned_addr learned my addr 172.21.15.129:0/3837800683 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:44.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.889+0000 7f7aff7fe640 1 -- 172.21.15.129:0/3837800683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 msgr2=0x555ed45df5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:44.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.889+0000 7f7aff7fe640 1 --2- 172.21.15.129:0/3837800683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45df5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.889+0000 7f7aff7fe640 1 -- 172.21.15.129:0/3837800683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed4237390 msgr2=0x555ed45d99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:44.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.889+0000 7f7aff7fe640 1 --2- 172.21.15.129:0/3837800683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed4237390 0x555ed45d99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.889+0000 7f7aff7fe640 1 -- 172.21.15.129:0/3837800683 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ed45ed0c0 con 0x555ed42166b0 2024-05-20T18:05:44.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.889+0000 7f7afffff640 1 --2- 172.21.15.129:0/3837800683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45df5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:05:44.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.890+0000 7f7aff7fe640 1 --2- 172.21.15.129:0/3837800683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed42166b0 0x555ed45d7ee0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x555ed45cd3f0 tx=0x7f7af802ae70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d81a2cdbdc88768 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:44.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.891+0000 7f7aedffb640 1 -- 172.21.15.129:0/3837800683 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af8034040 con 0x555ed42166b0 2024-05-20T18:05:44.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.892+0000 7f7aedffb640 1 -- 172.21.15.129:0/3837800683 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7af802f8a0 con 0x555ed42166b0 2024-05-20T18:05:44.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.892+0000 7f7aedffb640 1 -- 172.21.15.129:0/3837800683 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af8030c40 con 0x555ed42166b0 2024-05-20T18:05:44.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.894+0000 7f7b0d680740 1 -- 172.21.15.129:0/3837800683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed42166b0 msgr2=0x555ed45d7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:44.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.894+0000 7f7b0d680740 1 --2- 172.21.15.129:0/3837800683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed42166b0 0x555ed45d7ee0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x555ed45cd3f0 tx=0x7f7af802ae70 comp rx=0 tx=0).stop 2024-05-20T18:05:44.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.897+0000 7f7b0d680740 1 -- 172.21.15.129:0/3837800683 shutdown_connections 2024-05-20T18:05:44.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.897+0000 7f7b0d680740 1 --2- 172.21.15.129:0/3837800683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed4237390 0x555ed45d99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.897+0000 7f7b0d680740 1 --2- 172.21.15.129:0/3837800683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed42166b0 0x555ed45d7ee0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.897+0000 7f7b0d680740 1 --2- 172.21.15.129:0/3837800683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45df5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.897+0000 7f7b0d680740 1 -- 172.21.15.129:0/3837800683 >> 172.21.15.129:0/3837800683 conn(0x555ed422b4e0 msgr2=0x555ed45d77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:44.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.903+0000 7f7b0d680740 1 -- 172.21.15.129:0/3837800683 shutdown_connections 2024-05-20T18:05:44.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.904+0000 7f7b0d680740 1 -- 172.21.15.129:0/3837800683 wait complete. 2024-05-20T18:05:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.905+0000 7f7b0d680740 1 -- start start 2024-05-20T18:05:44.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.906+0000 7f7b0d680740 1 Processor -- start 2024-05-20T18:05:44.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.908+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45e8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.908+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45e57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.908+0000 7f7b0d680740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed4237390 0x555ed45e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.908+0000 7f7b0d680740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae400d540 con 0x555ed42166b0 2024-05-20T18:05:44.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.908+0000 7f7b0d680740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae400e9b0 con 0x555ed420e4d0 2024-05-20T18:05:44.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.908+0000 7f7b0d680740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae400dc10 con 0x555ed4237390 2024-05-20T18:05:44.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.908+0000 7f7b0d680740 10 client.0 _ll_register_callbacks cb 0x7f7af000bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:44.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.911+0000 7f7b04d72640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed4237390 0x555ed45e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.911+0000 7f7afffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45e8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.914+0000 7f7aff7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45e57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.914+0000 7f7aff7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45e57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49438/0 (socket says 172.21.15.129:49438) 2024-05-20T18:05:44.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.914+0000 7f7aff7fe640 1 -- 172.21.15.129:0/567910095 learned_addr learned my addr 172.21.15.129:0/567910095 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:44.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.924+0000 7f7b04d72640 1 -- 172.21.15.129:0/567910095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 msgr2=0x555ed45e8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:44.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.924+0000 7f7b04d72640 1 --2- 172.21.15.129:0/567910095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45e8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.924+0000 7f7b04d72640 1 -- 172.21.15.129:0/567910095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 msgr2=0x555ed45e57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:44.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.924+0000 7f7b04d72640 1 --2- 172.21.15.129:0/567910095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45e57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.924+0000 7f7b04d72640 1 -- 172.21.15.129:0/567910095 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ed45e6230 con 0x555ed4237390 2024-05-20T18:05:44.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.927+0000 7f7b04d72640 1 --2- 172.21.15.129:0/567910095 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed4237390 0x555ed45e5cf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7af4015000 tx=0x7f7af40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ede2322d03134e27 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:44.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.930+0000 7f7ae37fe640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:05:44.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.931+0000 7f7ae37fe640 1 -- 172.21.15.129:0/567910095 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af4023780 con 0x555ed4237390 2024-05-20T18:05:44.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.931+0000 7f7ae37fe640 1 -- 172.21.15.129:0/567910095 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f7af4023920 con 0x555ed4237390 2024-05-20T18:05:44.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.934+0000 7f7b0d680740 1 -- 172.21.15.129:0/567910095 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7af802f8a0 con 0x555ed4237390 2024-05-20T18:05:44.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.936+0000 7f7ae37fe640 1 -- 172.21.15.129:0/567910095 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7af4023b20 con 0x555ed4237390 2024-05-20T18:05:44.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.941+0000 7f7b0d680740 10 client.26218 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:44.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.941+0000 7f7b0d680740 1 -- 172.21.15.129:0/567910095 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555ed465a7e0 con 0x555ed4237390 2024-05-20T18:05:44.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.947+0000 7f7ae37fe640 1 -- 172.21.15.129:0/567910095 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7af802f8a0 con 0x555ed4237390 2024-05-20T18:05:44.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.947+0000 7f7ae37fe640 1 -- 172.21.15.129:0/567910095 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f7af402a070 con 0x555ed4237390 2024-05-20T18:05:44.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.950+0000 7f7b0d680740 10 client.26218 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:44.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.950+0000 7f7b0d680740 10 client.26218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:44.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.950+0000 7f7b0d680740 20 client.26218 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:44.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.950+0000 7f7b0d680740 1 --2- 172.21.15.129:0/567910095 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x555ed41cb3a0 0x555ed465cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:44.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.950+0000 7f7b0d680740 4 client.26218 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:05:44.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.950+0000 7f7b0d680740 1 -- 172.21.15.129:0/567910095 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555ed44c06f0 con 0x555ed41cb3a0 2024-05-20T18:05:44.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.959+0000 7f7afffff640 1 --2- 172.21.15.129:0/567910095 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x555ed41cb3a0 0x555ed465cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:44.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.972+0000 7f7ae37fe640 10 client.26218 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:44.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.972+0000 7f7afffff640 1 --2- 172.21.15.129:0/567910095 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x555ed41cb3a0 0x555ed465cf50 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8968503e19394469 server_cookie=72859e377a97e042 in_seq=0 out_seq=0 2024-05-20T18:05:44.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.978+0000 7f7ae37fe640 1 -- 172.21.15.129:0/567910095 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555ed44c06f0 con 0x555ed41cb3a0 2024-05-20T18:05:44.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.978+0000 7f7ae37fe640 10 client.26218 handle_command_reply: tid=0 2024-05-20T18:05:44.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.978+0000 7f7b0d680740 1 client.26218 shutdown 2024-05-20T18:05:44.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.980+0000 7f7b0d680740 1 -- 172.21.15.129:0/567910095 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed4237390 msgr2=0x555ed45e5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:44.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.980+0000 7f7b0d680740 1 --2- 172.21.15.129:0/567910095 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed4237390 0x555ed45e5cf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7af4015000 tx=0x7f7af40046c0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.996+0000 7f7b0d680740 1 -- 172.21.15.129:0/567910095 shutdown_connections 2024-05-20T18:05:44.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.996+0000 7f7b0d680740 1 --2- 172.21.15.129:0/567910095 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x555ed41cb3a0 0x555ed465cf50 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.996+0000 7f7b0d680740 1 --2- 172.21.15.129:0/567910095 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555ed4237390 0x555ed45e5cf0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.996+0000 7f7b0d680740 1 --2- 172.21.15.129:0/567910095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555ed42166b0 0x555ed45e57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.996+0000 7f7b0d680740 1 --2- 172.21.15.129:0/567910095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555ed420e4d0 0x555ed45e8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:44.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.996+0000 7f7b0d680740 1 -- 172.21.15.129:0/567910095 >> 172.21.15.129:0/567910095 conn(0x555ed422b4e0 msgr2=0x555ed45df300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:44.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:44.996+0000 7f7ae37fe640 0 client.26218 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:44.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.000+0000 7f7b0d680740 1 -- 172.21.15.129:0/567910095 shutdown_connections 2024-05-20T18:05:44.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.000+0000 7f7b0d680740 1 -- 172.21.15.129:0/567910095 wait complete. 2024-05-20T18:05:44.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.001+0000 7f7b0d680740 20 client.26218 trim_cache size 0 max 16384 2024-05-20T18:05:45.003 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:05:45.003 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:05:45.003 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:05:45.004 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:05:45.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.013+0000 7f7b067d5640 1 -- 172.21.15.129:0/3806201311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7ae80a2dc0 msgr2=0x7f7ae80a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:45.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.014+0000 7f7b067d5640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7ae80a2dc0 0x7f7ae80a52d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x555ed45d2ea0 tx=0x7f7af4009190 comp rx=0 tx=0).stop 2024-05-20T18:05:45.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.014+0000 7f7b067d5640 1 -- 172.21.15.129:0/3806201311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b00109060 msgr2=0x7f7b001a8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:45.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.014+0000 7f7b067d5640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b00109060 0x7f7b001a8640 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009fd0 tx=0x7f7af000b870 comp rx=0 tx=0).stop 2024-05-20T18:05:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.019+0000 7f7b067d5640 1 -- 172.21.15.129:0/3806201311 shutdown_connections 2024-05-20T18:05:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.019+0000 7f7b067d5640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7ae80a2dc0 0x7f7ae80a52d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.019+0000 7f7b067d5640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b0011c0d0 0x7f7b001ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.019+0000 7f7b067d5640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7b00119770 0x7f7b001a8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.019+0000 7f7b067d5640 1 --2- 172.21.15.129:0/3806201311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b00109060 0x7f7b001a8640 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.019+0000 7f7b067d5640 1 -- 172.21.15.129:0/3806201311 >> 172.21.15.129:0/3806201311 conn(0x7f7b0006d750 msgr2=0x7f7b00119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:45.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.023+0000 7f7b067d5640 1 -- 172.21.15.129:0/3806201311 shutdown_connections 2024-05-20T18:05:45.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.030+0000 7f7b067d5640 1 -- 172.21.15.129:0/3806201311 wait complete. 2024-05-20T18:05:45.109 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:05:45.110 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:05:45.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:45 smithi154 ceph-mon[36482]: pgmap v220: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 747 MiB/s wr, 1.43k op/s 2024-05-20T18:05:45.413 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:45 smithi129 ceph-mon[32029]: pgmap v220: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 747 MiB/s wr, 1.43k op/s 2024-05-20T18:05:45.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:45 smithi204 ceph-mon[37076]: pgmap v220: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 747 MiB/s wr, 1.43k op/s 2024-05-20T18:05:45.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.486+0000 7f52fb59e640 1 -- 172.21.15.129:0/1283711075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52fc10e890 msgr2=0x7f52fc10eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:45.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.486+0000 7f52fb59e640 1 --2- 172.21.15.129:0/1283711075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52fc10e890 0x7f52fc10eca0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009900 tx=0x7f52ec02a440 comp rx=0 tx=0).stop 2024-05-20T18:05:45.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.486+0000 7f52fb59e640 1 -- 172.21.15.129:0/1283711075 shutdown_connections 2024-05-20T18:05:45.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.486+0000 7f52fb59e640 1 --2- 172.21.15.129:0/1283711075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52fc109090 0x7f52fc109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:45.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.486+0000 7f52fb59e640 1 --2- 172.21.15.129:0/1283711075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52fc10f270 0x7f52fc108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:45.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.486+0000 7f52fb59e640 1 --2- 172.21.15.129:0/1283711075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52fc10e890 0x7f52fc10eca0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:45.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.486+0000 7f52fb59e640 1 -- 172.21.15.129:0/1283711075 >> 172.21.15.129:0/1283711075 conn(0x7f52fc06d580 msgr2=0x7f52fc06d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:45.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.487+0000 7f52fb59e640 1 -- 172.21.15.129:0/1283711075 shutdown_connections 2024-05-20T18:05:45.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.487+0000 7f52fb59e640 1 -- 172.21.15.129:0/1283711075 wait complete. 2024-05-20T18:05:45.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.488+0000 7f52fb59e640 1 Processor -- start 2024-05-20T18:05:45.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.488+0000 7f52fb59e640 1 -- start start 2024-05-20T18:05:45.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.489+0000 7f52fb59e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52fc109090 0x7f52fc1a85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:45.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.489+0000 7f52fb59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52fc10e890 0x7f52fc1a8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:45.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.489+0000 7f52fb59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52fc10f270 0x7f52fc1acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:45.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.489+0000 7f52fb59e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc121b20 con 0x7f52fc10e890 2024-05-20T18:05:45.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.489+0000 7f52fb59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc1219a0 con 0x7f52fc10f270 2024-05-20T18:05:45.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.489+0000 7f52fb59e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc121ca0 con 0x7f52fc109090 2024-05-20T18:05:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.491+0000 7f52f9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52fc10e890 0x7f52fc1a8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.491+0000 7f52f9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52fc10e890 0x7f52fc1a8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49452/0 (socket says 172.21.15.129:49452) 2024-05-20T18:05:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.491+0000 7f52f9d9b640 1 -- 172.21.15.129:0/1878305688 learned_addr learned my addr 172.21.15.129:0/1878305688 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.495+0000 7f52fa59c640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52fc109090 0x7f52fc1a85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.496+0000 7f52fad9d640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52fc10f270 0x7f52fc1acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.498+0000 7f52f9d9b640 1 -- 172.21.15.129:0/1878305688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52fc109090 msgr2=0x7f52fc1a85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.498+0000 7f52f9d9b640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52fc109090 0x7f52fc1a85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.498+0000 7f52f9d9b640 1 -- 172.21.15.129:0/1878305688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52fc10f270 msgr2=0x7f52fc1acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:45.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.498+0000 7f52f9d9b640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52fc10f270 0x7f52fc1acdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:45.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.498+0000 7f52f9d9b640 1 -- 172.21.15.129:0/1878305688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc1ad400 con 0x7f52fc10e890 2024-05-20T18:05:45.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.499+0000 7f52f9d9b640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52fc10e890 0x7f52fc1a8b00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f52f000b470 tx=0x7f52f000b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:45.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.500+0000 7f52eb7fe640 1 -- 172.21.15.129:0/1878305688 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f0015020 con 0x7f52fc10e890 2024-05-20T18:05:45.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.500+0000 7f52eb7fe640 1 -- 172.21.15.129:0/1878305688 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52f0011d70 con 0x7f52fc10e890 2024-05-20T18:05:45.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.500+0000 7f52eb7fe640 1 -- 172.21.15.129:0/1878305688 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f0015020 con 0x7f52fc10e890 2024-05-20T18:05:45.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.502+0000 7f52fb59e640 1 -- 172.21.15.129:0/1878305688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52fc1ad6f0 con 0x7f52fc10e890 2024-05-20T18:05:45.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.502+0000 7f52fb59e640 1 -- 172.21.15.129:0/1878305688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52fc1adbf0 con 0x7f52fc10e890 2024-05-20T18:05:45.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.504+0000 7f52e97fa640 1 -- 172.21.15.129:0/1878305688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52fc068b90 con 0x7f52fc10e890 2024-05-20T18:05:45.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.517+0000 7f52eb7fe640 1 -- 172.21.15.129:0/1878305688 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f52f0018020 con 0x7f52fc10e890 2024-05-20T18:05:45.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.518+0000 7f52eb7fe640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52e40a30f0 0x7f52e40a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:45.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.519+0000 7f52eb7fe640 1 -- 172.21.15.129:0/1878305688 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f52f0011690 con 0x7f52fc10e890 2024-05-20T18:05:45.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.527+0000 7f52fa59c640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52e40a30f0 0x7f52e40a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:45.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.529+0000 7f52eb7fe640 1 -- 172.21.15.129:0/1878305688 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f52fc1adbf0 con 0x7f52fc10e890 2024-05-20T18:05:45.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:45.556+0000 7f52fa59c640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52e40a30f0 0x7f52e40a5600 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f52ec006010 tx=0x7f52ec000bc0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:46.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.123+0000 7f52e97fa640 1 -- 172.21.15.129:0/1878305688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52fc1a98f0 con 0x7f52fc10e890 2024-05-20T18:05:46.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.138+0000 7f52eb7fe640 1 -- 172.21.15.129:0/1878305688 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11829 (secure 0 0 0) 0x7f52f001f020 con 0x7f52fc10e890 2024-05-20T18:05:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:05:46.139 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:03:18.451223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25867":{"gid":25867,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.154:6837/4068528573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":4068528573},{"type":"v1","addr":"172.21.15.154:6837","nonce":4068528573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:05:46.139 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 24 2024-05-20T18:05:46.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.147+0000 7f52e97fa640 1 -- 172.21.15.129:0/1878305688 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52e40a30f0 msgr2=0x7f52e40a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:46.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.147+0000 7f52e97fa640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52e40a30f0 0x7f52e40a5600 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f52ec006010 tx=0x7f52ec000bc0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.147+0000 7f52e97fa640 1 -- 172.21.15.129:0/1878305688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52fc10e890 msgr2=0x7f52fc1a8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:46.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.147+0000 7f52e97fa640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52fc10e890 0x7f52fc1a8b00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f52f000b470 tx=0x7f52f000b930 comp rx=0 tx=0).stop 2024-05-20T18:05:46.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.153+0000 7f52e97fa640 1 -- 172.21.15.129:0/1878305688 shutdown_connections 2024-05-20T18:05:46.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.154+0000 7f52e97fa640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52e40a30f0 0x7f52e40a5600 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.154+0000 7f52e97fa640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52fc10f270 0x7f52fc1acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.154+0000 7f52e97fa640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52fc10e890 0x7f52fc1a8b00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.154+0000 7f52e97fa640 1 --2- 172.21.15.129:0/1878305688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52fc109090 0x7f52fc1a85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.154+0000 7f52e97fa640 1 -- 172.21.15.129:0/1878305688 >> 172.21.15.129:0/1878305688 conn(0x7f52fc06d580 msgr2=0x7f52fc10a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:46.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.159+0000 7f52e97fa640 1 -- 172.21.15.129:0/1878305688 shutdown_connections 2024-05-20T18:05:46.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.159+0000 7f52e97fa640 1 -- 172.21.15.129:0/1878305688 wait complete. 2024-05-20T18:05:46.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:05:46.414 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:46 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1878305688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:05:46.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:46 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1878305688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:05:46.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:46 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1878305688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:05:46.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.718+0000 7f60fd0d6640 1 -- 172.21.15.129:0/170249232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60f8109090 msgr2=0x7f60f8109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:46.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.718+0000 7f60fd0d6640 1 --2- 172.21.15.129:0/170249232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60f8109090 0x7f60f8109520 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009ca0 tx=0x7f60f002a8f0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.718+0000 7f60fd0d6640 1 -- 172.21.15.129:0/170249232 shutdown_connections 2024-05-20T18:05:46.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.718+0000 7f60fd0d6640 1 --2- 172.21.15.129:0/170249232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60f8109090 0x7f60f8109520 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.718+0000 7f60fd0d6640 1 --2- 172.21.15.129:0/170249232 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60f810f270 0x7f60f8108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.718+0000 7f60fd0d6640 1 --2- 172.21.15.129:0/170249232 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60f810e890 0x7f60f810eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.718+0000 7f60fd0d6640 1 -- 172.21.15.129:0/170249232 >> 172.21.15.129:0/170249232 conn(0x7f60f806d580 msgr2=0x7f60f806d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:46.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.718+0000 7f60fd0d6640 1 -- 172.21.15.129:0/170249232 shutdown_connections 2024-05-20T18:05:46.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.718+0000 7f60fd0d6640 1 -- 172.21.15.129:0/170249232 wait complete. 2024-05-20T18:05:46.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.720+0000 7f60fd0d6640 1 Processor -- start 2024-05-20T18:05:46.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.720+0000 7f60fd0d6640 1 -- start start 2024-05-20T18:05:46.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.721+0000 7f60fd0d6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60f8109090 0x7f60f811f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.721+0000 7f60fd0d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60f810e890 0x7f60f8118260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.721+0000 7f60fd0d6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60f810f270 0x7f60f8118850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.721+0000 7f60fd0d6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f60f8121aa0 con 0x7f60f810f270 2024-05-20T18:05:46.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.721+0000 7f60fd0d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60f8121920 con 0x7f60f810e890 2024-05-20T18:05:46.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.721+0000 7f60fd0d6640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f60f8121c20 con 0x7f60f8109090 2024-05-20T18:05:46.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.721+0000 7f60fc8d5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60f810f270 0x7f60f8118850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.721+0000 7f60fc8d5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60f810f270 0x7f60f8118850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49456/0 (socket says 172.21.15.129:49456) 2024-05-20T18:05:46.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.721+0000 7f60fc8d5640 1 -- 172.21.15.129:0/617629863 learned_addr learned my addr 172.21.15.129:0/617629863 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:46.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.721+0000 7f60f77fe640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60f810e890 0x7f60f8118260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.723+0000 7f60f7fff640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60f8109090 0x7f60f811f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.723+0000 7f60f77fe640 1 -- 172.21.15.129:0/617629863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60f8109090 msgr2=0x7f60f811f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:46.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.723+0000 7f60f77fe640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60f8109090 0x7f60f811f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.723+0000 7f60f77fe640 1 -- 172.21.15.129:0/617629863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60f810f270 msgr2=0x7f60f8118850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:46.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.723+0000 7f60f77fe640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60f810f270 0x7f60f8118850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.723+0000 7f60f77fe640 1 -- 172.21.15.129:0/617629863 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f8119070 con 0x7f60f810e890 2024-05-20T18:05:46.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.730+0000 7f60f77fe640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60f810e890 0x7f60f8118260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c940 tx=0x7f60e800ce00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:46.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.740+0000 7f60f57fa640 1 -- 172.21.15.129:0/617629863 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8007bb0 con 0x7f60f810e890 2024-05-20T18:05:46.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.741+0000 7f60fd0d6640 1 -- 172.21.15.129:0/617629863 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f811cd80 con 0x7f60f810e890 2024-05-20T18:05:46.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.741+0000 7f60fd0d6640 1 -- 172.21.15.129:0/617629863 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f811d2a0 con 0x7f60f810e890 2024-05-20T18:05:46.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.741+0000 7f60f57fa640 1 -- 172.21.15.129:0/617629863 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60e8007d50 con 0x7f60f810e890 2024-05-20T18:05:46.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.741+0000 7f60f57fa640 1 -- 172.21.15.129:0/617629863 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8004e60 con 0x7f60f810e890 2024-05-20T18:05:46.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.752+0000 7f60f57fa640 1 -- 172.21.15.129:0/617629863 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f60e8005000 con 0x7f60f810e890 2024-05-20T18:05:46.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.753+0000 7f60f57fa640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f60e00a2e80 0x7f60e00a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.754+0000 7f60f57fa640 1 -- 172.21.15.129:0/617629863 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f60e80cef70 con 0x7f60f810e890 2024-05-20T18:05:46.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.754+0000 7f60f7fff640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f60e00a2e80 0x7f60e00a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.755+0000 7f60f7fff640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f60e00a2e80 0x7f60e00a5390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f60f8119ab0 tx=0x7f60ec008040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:46.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.809+0000 7f61051d7740 1 Processor -- start 2024-05-20T18:05:46.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.813+0000 7f61051d7740 1 -- start start 2024-05-20T18:05:46.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.814+0000 7f61051d7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e66424390 0x560e667c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.814+0000 7f61051d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e663fb4d0 0x560e667c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.815+0000 7f61051d7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e664036b0 0x560e667c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.815+0000 7f61051d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560e66728c40 con 0x560e663fb4d0 2024-05-20T18:05:46.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.815+0000 7f61051d7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560e666d0580 con 0x560e664036b0 2024-05-20T18:05:46.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.815+0000 7f61051d7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560e66724c00 con 0x560e66424390 2024-05-20T18:05:46.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.815+0000 7f60f77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e663fb4d0 0x560e667c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.817+0000 7f60f77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e663fb4d0 0x560e667c34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49468/0 (socket says 172.21.15.129:49468) 2024-05-20T18:05:46.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.817+0000 7f60f77fe640 1 -- 172.21.15.129:0/1221838669 learned_addr learned my addr 172.21.15.129:0/1221838669 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:46.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.818+0000 7f60f7fff640 1 --2- 172.21.15.129:0/1221838669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e66424390 0x560e667c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.818+0000 7f60fc8d5640 1 --2- 172.21.15.129:0/1221838669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e664036b0 0x560e667c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.821+0000 7f60f7fff640 1 -- 172.21.15.129:0/1221838669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e664036b0 msgr2=0x560e667c39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:46.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.821+0000 7f60f7fff640 1 --2- 172.21.15.129:0/1221838669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e664036b0 0x560e667c39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.821+0000 7f60f7fff640 1 -- 172.21.15.129:0/1221838669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e663fb4d0 msgr2=0x560e667c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:46.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.821+0000 7f60f7fff640 1 --2- 172.21.15.129:0/1221838669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e663fb4d0 0x560e667c34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.821+0000 7f60f7fff640 1 -- 172.21.15.129:0/1221838669 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e6643c3f0 con 0x560e66424390 2024-05-20T18:05:46.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.822+0000 7f60f7fff640 1 --2- 172.21.15.129:0/1221838669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e66424390 0x560e667c2f60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00c520 tx=0x7f60ec00f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32ed8c1e475f159 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:46.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.823+0000 7f60de7fc640 1 -- 172.21.15.129:0/1221838669 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec023070 con 0x560e66424390 2024-05-20T18:05:46.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.823+0000 7f60de7fc640 1 -- 172.21.15.129:0/1221838669 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60ec011da0 con 0x560e66424390 2024-05-20T18:05:46.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.823+0000 7f60de7fc640 1 -- 172.21.15.129:0/1221838669 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec013640 con 0x560e66424390 2024-05-20T18:05:46.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.825+0000 7f61051d7740 1 -- 172.21.15.129:0/1221838669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e66424390 msgr2=0x560e667c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:46.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.825+0000 7f61051d7740 1 --2- 172.21.15.129:0/1221838669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e66424390 0x560e667c2f60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00c520 tx=0x7f60ec00f000 comp rx=0 tx=0).stop 2024-05-20T18:05:46.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.831+0000 7f61051d7740 1 -- 172.21.15.129:0/1221838669 shutdown_connections 2024-05-20T18:05:46.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.831+0000 7f61051d7740 1 --2- 172.21.15.129:0/1221838669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e664036b0 0x560e667c39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.831+0000 7f61051d7740 1 --2- 172.21.15.129:0/1221838669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e663fb4d0 0x560e667c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.831+0000 7f61051d7740 1 --2- 172.21.15.129:0/1221838669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e66424390 0x560e667c2f60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.831+0000 7f61051d7740 1 -- 172.21.15.129:0/1221838669 >> 172.21.15.129:0/1221838669 conn(0x560e664184e0 msgr2=0x560e667bf710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:46.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.844+0000 7f61051d7740 1 -- 172.21.15.129:0/1221838669 shutdown_connections 2024-05-20T18:05:46.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.844+0000 7f61051d7740 1 -- 172.21.15.129:0/1221838669 wait complete. 2024-05-20T18:05:46.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.845+0000 7f61051d7740 1 -- start start 2024-05-20T18:05:46.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.845+0000 7f61051d7740 1 Processor -- start 2024-05-20T18:05:46.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.847+0000 7f61051d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e664036b0 0x560e667cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.847+0000 7f61051d7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e66424390 0x560e667cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.847+0000 7f61051d7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e663e2f60 0x560e667d9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.847+0000 7f61051d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560e667d02b0 con 0x560e664036b0 2024-05-20T18:05:46.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.847+0000 7f61051d7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560e666e3c40 con 0x560e66424390 2024-05-20T18:05:46.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.847+0000 7f61051d7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560e667d0430 con 0x560e663e2f60 2024-05-20T18:05:46.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.847+0000 7f61051d7740 10 client.0 _ll_register_callbacks cb 0x560e6643bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:46.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.848+0000 7f60f7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e664036b0 0x560e667cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.849+0000 7f60fc8d5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e663e2f60 0x560e667d9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.849+0000 7f60fc8d5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e663e2f60 0x560e667d9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:36110/0 (socket says 172.21.15.129:36110) 2024-05-20T18:05:46.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.849+0000 7f60fc8d5640 1 -- 172.21.15.129:0/3016119813 learned_addr learned my addr 172.21.15.129:0/3016119813 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:46.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.849+0000 7f60f77fe640 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e66424390 0x560e667cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.850+0000 7f60f77fe640 1 -- 172.21.15.129:0/3016119813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e663e2f60 msgr2=0x560e667d9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:46.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.850+0000 7f60f77fe640 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e663e2f60 0x560e667d9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.850+0000 7f60f77fe640 1 -- 172.21.15.129:0/3016119813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e664036b0 msgr2=0x560e667cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:46.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.850+0000 7f60f77fe640 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e664036b0 0x560e667cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.850+0000 7f60f77fe640 1 -- 172.21.15.129:0/3016119813 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e665067d0 con 0x560e66424390 2024-05-20T18:05:46.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.851+0000 7f60f77fe640 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e66424390 0x560e667cda30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f60e8097e50 tx=0x7f60e809af10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a02200432ff90dd0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.855+0000 7f60d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:05:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.855+0000 7f60d77fe640 1 -- 172.21.15.129:0/3016119813 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e80b3070 con 0x560e66424390 2024-05-20T18:05:46.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.855+0000 7f61051d7740 1 -- 172.21.15.129:0/3016119813 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60ec011da0 con 0x560e66424390 2024-05-20T18:05:46.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.856+0000 7f60d77fe640 1 -- 172.21.15.129:0/3016119813 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f60e80b4260 con 0x560e66424390 2024-05-20T18:05:46.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.858+0000 7f60d77fe640 1 -- 172.21.15.129:0/3016119813 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f60e801d040 con 0x560e66424390 2024-05-20T18:05:46.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.858+0000 7f61051d7740 10 client.26183 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:46.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.858+0000 7f61051d7740 1 -- 172.21.15.129:0/3016119813 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560e667dc840 con 0x560e66424390 2024-05-20T18:05:46.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.861+0000 7f60d77fe640 1 -- 172.21.15.129:0/3016119813 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60ec011da0 con 0x560e66424390 2024-05-20T18:05:46.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.864+0000 7f60d77fe640 1 -- 172.21.15.129:0/3016119813 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f60e80a3c60 con 0x560e66424390 2024-05-20T18:05:46.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.865+0000 7f61051d7740 10 client.26183 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.865+0000 7f61051d7740 10 client.26183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.865+0000 7f61051d7740 20 client.26183 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.865+0000 7f61051d7740 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560e663b83a0 0x560e667e02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.865+0000 7f61051d7740 4 client.26183 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.865+0000 7f61051d7740 1 -- 172.21.15.129:0/3016119813 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e66736370 con 0x560e663b83a0 2024-05-20T18:05:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.869+0000 7f60f7fff640 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560e663b83a0 0x560e667e02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.889+0000 7f60f7fff640 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560e663b83a0 0x560e667e02a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e16dc842d739a417 server_cookie=b90a3835b7bdc07 in_seq=0 out_seq=0 2024-05-20T18:05:46.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.893+0000 7f60d77fe640 10 client.26183 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:46.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.901+0000 7f60d77fe640 1 -- 172.21.15.129:0/3016119813 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560e66736370 con 0x560e663b83a0 2024-05-20T18:05:46.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.901+0000 7f60d77fe640 10 client.26183 handle_command_reply: tid=0 2024-05-20T18:05:46.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.901+0000 7f61051d7740 1 client.26183 shutdown 2024-05-20T18:05:46.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.902+0000 7f61051d7740 1 -- 172.21.15.129:0/3016119813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e66424390 msgr2=0x560e667cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:46.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.902+0000 7f61051d7740 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e66424390 0x560e667cda30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f60e8097e50 tx=0x7f60e809af10 comp rx=0 tx=0).stop 2024-05-20T18:05:46.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.903+0000 7f61051d7740 1 -- 172.21.15.129:0/3016119813 shutdown_connections 2024-05-20T18:05:46.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.903+0000 7f61051d7740 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560e663b83a0 0x560e667e02a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.903+0000 7f61051d7740 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e66424390 0x560e667cda30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.903+0000 7f61051d7740 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e663e2f60 0x560e667d9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.903+0000 7f61051d7740 1 --2- 172.21.15.129:0/3016119813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e664036b0 0x560e667cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.903+0000 7f61051d7740 1 -- 172.21.15.129:0/3016119813 >> 172.21.15.129:0/3016119813 conn(0x560e664184e0 msgr2=0x560e667c4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:46.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.903+0000 7f60d77fe640 0 client.26183 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.905+0000 7f61051d7740 1 -- 172.21.15.129:0/3016119813 shutdown_connections 2024-05-20T18:05:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.905+0000 7f61051d7740 1 -- 172.21.15.129:0/3016119813 wait complete. 2024-05-20T18:05:46.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.905+0000 7f61051d7740 20 client.26183 trim_cache size 0 max 16384 2024-05-20T18:05:46.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.963+0000 7f61051d7740 1 Processor -- start 2024-05-20T18:05:46.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.963+0000 7f61051d7740 1 -- start start 2024-05-20T18:05:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.964+0000 7f61051d7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e663e2f60 0x560e667d9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.964+0000 7f61051d7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e664036b0 0x560e667c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.964+0000 7f61051d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667c7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.964+0000 7f61051d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f60e400c6d0 con 0x560e66424390 2024-05-20T18:05:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.964+0000 7f61051d7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60e400eea0 con 0x560e664036b0 2024-05-20T18:05:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.964+0000 7f61051d7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f60e400cc90 con 0x560e663e2f60 2024-05-20T18:05:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.965+0000 7f60fc8d5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.965+0000 7f60fc8d5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667c7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49482/0 (socket says 172.21.15.129:49482) 2024-05-20T18:05:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.965+0000 7f60fc8d5640 1 -- 172.21.15.129:0/1222944102 learned_addr learned my addr 172.21.15.129:0/1222944102 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.966+0000 7f60fc8d5640 1 -- 172.21.15.129:0/1222944102 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e663e2f60 msgr2=0x560e667d9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.966+0000 7f60fc8d5640 1 --2- 172.21.15.129:0/1222944102 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e663e2f60 0x560e667d9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.966+0000 7f60fc8d5640 1 -- 172.21.15.129:0/1222944102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e664036b0 msgr2=0x560e667c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.966+0000 7f60fc8d5640 1 --2- 172.21.15.129:0/1222944102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e664036b0 0x560e667c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.966+0000 7f60fc8d5640 1 -- 172.21.15.129:0/1222944102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e667ce080 con 0x560e66424390 2024-05-20T18:05:46.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.968+0000 7f60fc8d5640 1 --2- 172.21.15.129:0/1222944102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667c7150 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f60f0003a50 tx=0x7f60f0031f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13f18a9e5dfd86 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:46.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.972+0000 7f60ddffb640 1 -- 172.21.15.129:0/1222944102 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f0031680 con 0x560e66424390 2024-05-20T18:05:46.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.973+0000 7f60ddffb640 1 -- 172.21.15.129:0/1222944102 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60f004b070 con 0x560e66424390 2024-05-20T18:05:46.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.975+0000 7f61051d7740 1 -- 172.21.15.129:0/1222944102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 msgr2=0x560e667c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:46.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.975+0000 7f61051d7740 1 --2- 172.21.15.129:0/1222944102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667c7150 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f60f0003a50 tx=0x7f60f0031f90 comp rx=0 tx=0).stop 2024-05-20T18:05:46.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.976+0000 7f61051d7740 1 -- 172.21.15.129:0/1222944102 shutdown_connections 2024-05-20T18:05:46.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.976+0000 7f61051d7740 1 --2- 172.21.15.129:0/1222944102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e664036b0 0x560e667c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.976+0000 7f61051d7740 1 --2- 172.21.15.129:0/1222944102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667c7150 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.976+0000 7f61051d7740 1 --2- 172.21.15.129:0/1222944102 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e663e2f60 0x560e667d9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:46.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.976+0000 7f61051d7740 1 -- 172.21.15.129:0/1222944102 >> 172.21.15.129:0/1222944102 conn(0x560e664184e0 msgr2=0x560e667cbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:46.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.983+0000 7f61051d7740 1 -- 172.21.15.129:0/1222944102 shutdown_connections 2024-05-20T18:05:46.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.988+0000 7f61051d7740 1 -- 172.21.15.129:0/1222944102 wait complete. 2024-05-20T18:05:46.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.990+0000 7f61051d7740 1 -- start start 2024-05-20T18:05:46.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.990+0000 7f61051d7740 1 Processor -- start 2024-05-20T18:05:46.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.993+0000 7f61051d7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e663e2f60 0x560e667be520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.993+0000 7f61051d7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e664036b0 0x560e667bea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.993+0000 7f61051d7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667d4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:46.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.993+0000 7f61051d7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f60e400d360 con 0x560e66424390 2024-05-20T18:05:46.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.993+0000 7f61051d7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60e400e7d0 con 0x560e663e2f60 2024-05-20T18:05:46.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.993+0000 7f61051d7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f60e400da30 con 0x560e664036b0 2024-05-20T18:05:46.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.993+0000 7f61051d7740 10 client.0 _ll_register_callbacks cb 0x7f60e8097780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:46.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.995+0000 7f60fc8d5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667d4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.995+0000 7f60fc8d5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667d4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49496/0 (socket says 172.21.15.129:49496) 2024-05-20T18:05:46.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.995+0000 7f60fc8d5640 1 -- 172.21.15.129:0/3617637309 learned_addr learned my addr 172.21.15.129:0/3617637309 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:46.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.997+0000 7f60f7fff640 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e663e2f60 0x560e667be520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:46.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:46.997+0000 7f60f77fe640 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e664036b0 0x560e667bea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:47.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.003+0000 7f60fc8d5640 1 -- 172.21.15.129:0/3617637309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e664036b0 msgr2=0x560e667bea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:47.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.003+0000 7f60fc8d5640 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e664036b0 0x560e667bea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:47.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.003+0000 7f60fc8d5640 1 -- 172.21.15.129:0/3617637309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e663e2f60 msgr2=0x560e667be520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.003+0000 7f60fc8d5640 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e663e2f60 0x560e667be520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.003+0000 7f60fc8d5640 1 -- 172.21.15.129:0/3617637309 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e667d4cd0 con 0x560e66424390 2024-05-20T18:05:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.004+0000 7f60fc8d5640 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667d4790 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f60f0006010 tx=0x7f60f0030fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3abbf995ab0ea6e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.005+0000 7f60d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:05:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.009+0000 7f60d77fe640 1 -- 172.21.15.129:0/3617637309 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f00307f0 con 0x560e66424390 2024-05-20T18:05:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.009+0000 7f61051d7740 1 -- 172.21.15.129:0/3617637309 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60f004b070 con 0x560e66424390 2024-05-20T18:05:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.009+0000 7f60d77fe640 1 -- 172.21.15.129:0/3617637309 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f60f0030990 con 0x560e66424390 2024-05-20T18:05:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.014+0000 7f60d77fe640 1 -- 172.21.15.129:0/3617637309 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f60f0031930 con 0x560e66424390 2024-05-20T18:05:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.014+0000 7f61051d7740 10 client.16464 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.014+0000 7f61051d7740 1 -- 172.21.15.129:0/3617637309 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560e667d5f10 con 0x560e66424390 2024-05-20T18:05:47.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.018+0000 7f60d77fe640 1 -- 172.21.15.129:0/3617637309 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60f004b070 con 0x560e66424390 2024-05-20T18:05:47.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.018+0000 7f60d77fe640 1 -- 172.21.15.129:0/3617637309 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f60f00463f0 con 0x560e66424390 2024-05-20T18:05:47.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.018+0000 7f61051d7740 10 client.16464 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:47.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.018+0000 7f61051d7740 10 client.16464 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:47.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.018+0000 7f61051d7740 20 client.16464 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:47.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.018+0000 7f61051d7740 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560e668487b0 0x560e6684abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:47.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.019+0000 7f61051d7740 4 client.16464 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:05:47.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.019+0000 7f61051d7740 1 -- 172.21.15.129:0/3617637309 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e667306c0 con 0x560e668487b0 2024-05-20T18:05:47.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.024+0000 7f60f7fff640 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560e668487b0 0x560e6684abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:47.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.025+0000 7f60f7fff640 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560e668487b0 0x560e6684abd0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d5dde8006003df6 server_cookie=a7905de9cfcc5b10 in_seq=0 out_seq=0 2024-05-20T18:05:47.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.025+0000 7f60d77fe640 10 client.16464 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:47.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.032+0000 7f60d77fe640 1 -- 172.21.15.129:0/3617637309 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e667306c0 con 0x560e668487b0 2024-05-20T18:05:47.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.032+0000 7f60d77fe640 10 client.16464 handle_command_reply: tid=0 2024-05-20T18:05:47.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.032+0000 7f61051d7740 1 client.16464 shutdown 2024-05-20T18:05:47.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.038+0000 7f61051d7740 1 -- 172.21.15.129:0/3617637309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 msgr2=0x560e667d4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:47.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.038+0000 7f61051d7740 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667d4790 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f60f0006010 tx=0x7f60f0030fb0 comp rx=0 tx=0).stop 2024-05-20T18:05:47.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.038+0000 7f61051d7740 1 -- 172.21.15.129:0/3617637309 shutdown_connections 2024-05-20T18:05:47.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.038+0000 7f61051d7740 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560e668487b0 0x560e6684abd0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:47.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.038+0000 7f61051d7740 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e66424390 0x560e667d4790 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:47.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.038+0000 7f61051d7740 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560e664036b0 0x560e667bea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:47.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.038+0000 7f61051d7740 1 --2- 172.21.15.129:0/3617637309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560e663e2f60 0x560e667be520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:47.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.038+0000 7f61051d7740 1 -- 172.21.15.129:0/3617637309 >> 172.21.15.129:0/3617637309 conn(0x560e664184e0 msgr2=0x560e667c4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:47.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.038+0000 7f60d77fe640 0 client.16464 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:47.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.038+0000 7f61051d7740 1 -- 172.21.15.129:0/3617637309 shutdown_connections 2024-05-20T18:05:47.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.041+0000 7f61051d7740 1 -- 172.21.15.129:0/3617637309 wait complete. 2024-05-20T18:05:47.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.041+0000 7f61051d7740 20 client.16464 trim_cache size 0 max 16384 2024-05-20T18:05:47.041 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:05:47.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.053+0000 7f60fd0d6640 1 -- 172.21.15.129:0/617629863 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f60e00a2e80 msgr2=0x7f60e00a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:47.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.053+0000 7f60fd0d6640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f60e00a2e80 0x7f60e00a5390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f60f8119ab0 tx=0x7f60ec008040 comp rx=0 tx=0).stop 2024-05-20T18:05:47.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.053+0000 7f60fd0d6640 1 -- 172.21.15.129:0/617629863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60f810e890 msgr2=0x7f60f8118260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:47.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.053+0000 7f60fd0d6640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60f810e890 0x7f60f8118260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c940 tx=0x7f60e800ce00 comp rx=0 tx=0).stop 2024-05-20T18:05:47.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.060+0000 7f60fd0d6640 1 -- 172.21.15.129:0/617629863 shutdown_connections 2024-05-20T18:05:47.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.060+0000 7f60fd0d6640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f60e00a2e80 0x7f60e00a5390 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:47.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.060+0000 7f60fd0d6640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60f810e890 0x7f60f8118260 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:47.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.060+0000 7f60fd0d6640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60f810f270 0x7f60f8118850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:47.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.060+0000 7f60fd0d6640 1 --2- 172.21.15.129:0/617629863 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60f8109090 0x7f60f811f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:47.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.060+0000 7f60fd0d6640 1 -- 172.21.15.129:0/617629863 >> 172.21.15.129:0/617629863 conn(0x7f60f806d580 msgr2=0x7f60f80728c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:47.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.064+0000 7f60fd0d6640 1 -- 172.21.15.129:0/617629863 shutdown_connections 2024-05-20T18:05:47.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:47.065+0000 7f60fd0d6640 1 -- 172.21.15.129:0/617629863 wait complete. 2024-05-20T18:05:47.418 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:47 smithi129 ceph-mon[32029]: pgmap v221: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 966 MiB/s wr, 1.85k op/s 2024-05-20T18:05:47.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:47 smithi154 ceph-mon[36482]: pgmap v221: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 966 MiB/s wr, 1.85k op/s 2024-05-20T18:05:47.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:47 smithi204 ceph-mon[37076]: pgmap v221: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 966 MiB/s wr, 1.85k op/s 2024-05-20T18:05:48.114 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:05:48.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:05:48.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:05:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.621+0000 7fc7576b8640 1 -- 172.21.15.129:0/3128371199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc750109060 msgr2=0x7fc750109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.621+0000 7fc7576b8640 1 --2- 172.21.15.129:0/3128371199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc750109060 0x7fc750109470 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc740009900 tx=0x7fc74002a4b0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.626+0000 7fc7576b8640 1 -- 172.21.15.129:0/3128371199 shutdown_connections 2024-05-20T18:05:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.626+0000 7fc7576b8640 1 --2- 172.21.15.129:0/3128371199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc750119770 0x7fc75011bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.626+0000 7fc7576b8640 1 --2- 172.21.15.129:0/3128371199 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc75011c0d0 0x7fc75011e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.626+0000 7fc7576b8640 1 --2- 172.21.15.129:0/3128371199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc750109060 0x7fc750109470 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.626+0000 7fc7576b8640 1 -- 172.21.15.129:0/3128371199 >> 172.21.15.129:0/3128371199 conn(0x7fc75006d750 msgr2=0x7fc75006dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.629+0000 7fc7576b8640 1 -- 172.21.15.129:0/3128371199 shutdown_connections 2024-05-20T18:05:48.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.633+0000 7fc7576b8640 1 -- 172.21.15.129:0/3128371199 wait complete. 2024-05-20T18:05:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.637+0000 7fc7576b8640 1 Processor -- start 2024-05-20T18:05:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.637+0000 7fc7576b8640 1 -- start start 2024-05-20T18:05:48.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.639+0000 7fc7576b8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc750109060 0x7fc750113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.639+0000 7fc7576b8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc750119770 0x7fc750113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.639+0000 7fc7576b8640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc75011c0d0 0x7fc750113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.639+0000 7fc7576b8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc7501119b0 con 0x7fc750109060 2024-05-20T18:05:48.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.639+0000 7fc7576b8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc750111830 con 0x7fc750119770 2024-05-20T18:05:48.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.639+0000 7fc7576b8640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc750111b30 con 0x7fc75011c0d0 2024-05-20T18:05:48.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.640+0000 7fc755c55640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc75011c0d0 0x7fc750113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.641+0000 7fc755c55640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc75011c0d0 0x7fc750113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:36140/0 (socket says 172.21.15.129:36140) 2024-05-20T18:05:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.641+0000 7fc755c55640 1 -- 172.21.15.129:0/499129917 learned_addr learned my addr 172.21.15.129:0/499129917 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:48.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.642+0000 7fc754c53640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc750119770 0x7fc750113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.643+0000 7fc754c53640 1 -- 172.21.15.129:0/499129917 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc75011c0d0 msgr2=0x7fc750113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:48.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.643+0000 7fc754c53640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc75011c0d0 0x7fc750113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.643+0000 7fc754c53640 1 -- 172.21.15.129:0/499129917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc750109060 msgr2=0x7fc750113500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:48.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.643+0000 7fc754c53640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc750109060 0x7fc750113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.643+0000 7fc754c53640 1 -- 172.21.15.129:0/499129917 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7501bd0b0 con 0x7fc750119770 2024-05-20T18:05:48.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.644+0000 7fc754c53640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc750119770 0x7fc750113a40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00b3f0 tx=0x7fc74c00b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:48.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.651+0000 7fc7467fc640 1 -- 172.21.15.129:0/499129917 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c015020 con 0x7fc750119770 2024-05-20T18:05:48.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.651+0000 7fc7467fc640 1 -- 172.21.15.129:0/499129917 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc74c011d10 con 0x7fc750119770 2024-05-20T18:05:48.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.651+0000 7fc7467fc640 1 -- 172.21.15.129:0/499129917 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c00f2e0 con 0x7fc750119770 2024-05-20T18:05:48.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.651+0000 7fc7576b8640 1 -- 172.21.15.129:0/499129917 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7501bd3a0 con 0x7fc750119770 2024-05-20T18:05:48.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.651+0000 7fc7576b8640 1 -- 172.21.15.129:0/499129917 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7501bd8f0 con 0x7fc750119770 2024-05-20T18:05:48.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.661+0000 7fc75e633740 1 Processor -- start 2024-05-20T18:05:48.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.661+0000 7fc75e633740 1 -- start start 2024-05-20T18:05:48.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.662+0000 7fc75e633740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 0x5596cca673b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.662+0000 7fc75e633740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc69f550 0x5596cca678f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.662+0000 7fc75e633740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6a7730 0x5596cca67e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.662+0000 7fc75e633740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5596cc9cc990 con 0x5596cc6c8410 2024-05-20T18:05:48.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.662+0000 7fc75e633740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5596cc9745b0 con 0x5596cc69f550 2024-05-20T18:05:48.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.662+0000 7fc75e633740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5596cc9c8950 con 0x5596cc6a7730 2024-05-20T18:05:48.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.665+0000 7fc755454640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc69f550 0x5596cca678f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.665+0000 7fc755c55640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 0x5596cca673b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.665+0000 7fc755c55640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 0x5596cca673b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49520/0 (socket says 172.21.15.129:49520) 2024-05-20T18:05:48.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.665+0000 7fc755c55640 1 -- 172.21.15.129:0/2117174460 learned_addr learned my addr 172.21.15.129:0/2117174460 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:48.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.668+0000 7fc7467fc640 1 -- 172.21.15.129:0/499129917 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc74c018020 con 0x7fc750119770 2024-05-20T18:05:48.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.668+0000 7fc755c55640 1 -- 172.21.15.129:0/2117174460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6a7730 msgr2=0x5596cca67e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:05:48.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.668+0000 7fc755c55640 1 --2- 172.21.15.129:0/2117174460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6a7730 0x5596cca67e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.668+0000 7fc755c55640 1 -- 172.21.15.129:0/2117174460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc69f550 msgr2=0x5596cca678f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:48.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.668+0000 7fc755c55640 1 --2- 172.21.15.129:0/2117174460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc69f550 0x5596cca678f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.668+0000 7fc755c55640 1 -- 172.21.15.129:0/2117174460 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596cc7a87e0 con 0x5596cc6c8410 2024-05-20T18:05:48.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.669+0000 7fc755c55640 1 --2- 172.21.15.129:0/2117174460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 0x5596cca673b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc74800e2a0 tx=0x7fc748009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707076f98d65cc33 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:48.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.670+0000 7fc7576b8640 1 -- 172.21.15.129:0/2117174460 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc748018020 con 0x5596cc6c8410 2024-05-20T18:05:48.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.670+0000 7fc7576b8640 1 -- 172.21.15.129:0/2117174460 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc748014d70 con 0x5596cc6c8410 2024-05-20T18:05:48.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.671+0000 7fc7467fc640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc73c0a2f00 0x7fc73c0a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.672+0000 7fc75e633740 1 -- 172.21.15.129:0/2117174460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 msgr2=0x5596cca673b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:48.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.672+0000 7fc75e633740 1 --2- 172.21.15.129:0/2117174460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 0x5596cca673b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc74800e2a0 tx=0x7fc748009710 comp rx=0 tx=0).stop 2024-05-20T18:05:48.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.673+0000 7fc755454640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc73c0a2f00 0x7fc73c0a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.673+0000 7fc7467fc640 1 -- 172.21.15.129:0/499129917 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fc74c0ce9a0 con 0x7fc750119770 2024-05-20T18:05:48.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.678+0000 7fc755454640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc73c0a2f00 0x7fc73c0a5410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x5596cca65300 tx=0x7fc740000bc0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:48.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.682+0000 7fc75e633740 1 -- 172.21.15.129:0/2117174460 shutdown_connections 2024-05-20T18:05:48.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.682+0000 7fc75e633740 1 --2- 172.21.15.129:0/2117174460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc69f550 0x5596cca678f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.682+0000 7fc75e633740 1 --2- 172.21.15.129:0/2117174460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6a7730 0x5596cca67e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.682+0000 7fc75e633740 1 --2- 172.21.15.129:0/2117174460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 0x5596cca673b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.682+0000 7fc75e633740 1 -- 172.21.15.129:0/2117174460 >> 172.21.15.129:0/2117174460 conn(0x5596cc6bc560 msgr2=0x5596cca63fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:48.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.688+0000 7fc75e633740 1 -- 172.21.15.129:0/2117174460 shutdown_connections 2024-05-20T18:05:48.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.690+0000 7fc75e633740 1 -- 172.21.15.129:0/2117174460 wait complete. 2024-05-20T18:05:48.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.691+0000 7fc75e633740 1 -- start start 2024-05-20T18:05:48.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.692+0000 7fc75e633740 1 Processor -- start 2024-05-20T18:05:48.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.693+0000 7fc75e633740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 0x5596cca73850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.693+0000 7fc75e633740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca73d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.693+0000 7fc75e633740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6c8410 0x5596cca71d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.693+0000 7fc75e633740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5596cca749f0 con 0x5596cc69f550 2024-05-20T18:05:48.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.694+0000 7fc75e633740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5596cc987b40 con 0x5596cc6a7730 2024-05-20T18:05:48.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.694+0000 7fc75e633740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5596cca74b70 con 0x5596cc6c8410 2024-05-20T18:05:48.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.694+0000 7fc75e633740 10 client.0 _ll_register_callbacks cb 0x5596cc97c0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:48.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.695+0000 7fc755454640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 0x5596cca73850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.695+0000 7fc755454640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 0x5596cca73850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49526/0 (socket says 172.21.15.129:49526) 2024-05-20T18:05:48.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.695+0000 7fc755454640 1 -- 172.21.15.129:0/1556022674 learned_addr learned my addr 172.21.15.129:0/1556022674 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:48.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.696+0000 7fc755c55640 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6c8410 0x5596cca71d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.696+0000 7fc754c53640 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca73d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.699+0000 7fc754c53640 1 -- 172.21.15.129:0/1556022674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6c8410 msgr2=0x5596cca71d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:48.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.699+0000 7fc754c53640 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6c8410 0x5596cca71d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.699+0000 7fc754c53640 1 -- 172.21.15.129:0/1556022674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 msgr2=0x5596cca73850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:48.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.699+0000 7fc754c53640 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 0x5596cca73850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.699+0000 7fc754c53640 1 -- 172.21.15.129:0/1556022674 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596cc808f70 con 0x5596cc6a7730 2024-05-20T18:05:48.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.705+0000 7fc754c53640 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca73d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0cc190 tx=0x7fc74c09a9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b557924a3b35163 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:48.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.705+0000 7fc738ff9640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:05:48.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.706+0000 7fc738ff9640 1 -- 172.21.15.129:0/1556022674 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c0b3070 con 0x5596cc6a7730 2024-05-20T18:05:48.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.707+0000 7fc738ff9640 1 -- 172.21.15.129:0/1556022674 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fc74c0cc300 con 0x5596cc6a7730 2024-05-20T18:05:48.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.707+0000 7fc75e633740 1 -- 172.21.15.129:0/1556022674 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc748014d70 con 0x5596cc6a7730 2024-05-20T18:05:48.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.708+0000 7fc738ff9640 1 -- 172.21.15.129:0/1556022674 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc74c0cc500 con 0x5596cc6a7730 2024-05-20T18:05:48.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.708+0000 7fc75e633740 10 client.26195 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:48.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.708+0000 7fc75e633740 1 -- 172.21.15.129:0/1556022674 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5596cc6a0170 con 0x5596cc6a7730 2024-05-20T18:05:48.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.709+0000 7fc738ff9640 1 -- 172.21.15.129:0/1556022674 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc748014d70 con 0x5596cc6a7730 2024-05-20T18:05:48.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.710+0000 7fc738ff9640 1 -- 172.21.15.129:0/1556022674 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fc74c022060 con 0x5596cc6a7730 2024-05-20T18:05:48.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.711+0000 7fc75e633740 10 client.26195 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:48.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.711+0000 7fc75e633740 10 client.26195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:48.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.711+0000 7fc75e633740 20 client.26195 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:48.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.711+0000 7fc75e633740 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596cc6871f0 0x5596cca84410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.711+0000 7fc75e633740 4 client.26195 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:48.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.711+0000 7fc75e633740 1 -- 172.21.15.129:0/1556022674 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596cc9da0c0 con 0x5596cc6871f0 2024-05-20T18:05:48.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.714+0000 7fc755454640 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596cc6871f0 0x5596cca84410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.738+0000 7fc755454640 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596cc6871f0 0x5596cca84410 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4feba1f4adff0bc server_cookie=fbde69691e3f497d in_seq=0 out_seq=0 2024-05-20T18:05:48.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.738+0000 7fc738ff9640 10 client.26195 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:48.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.745+0000 7fc738ff9640 1 -- 172.21.15.129:0/1556022674 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5596cc9da0c0 con 0x5596cc6871f0 2024-05-20T18:05:48.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.745+0000 7fc738ff9640 10 client.26195 handle_command_reply: tid=0 2024-05-20T18:05:48.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.748+0000 7fc75e633740 1 client.26195 shutdown 2024-05-20T18:05:48.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.749+0000 7fc75e633740 1 -- 172.21.15.129:0/1556022674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 msgr2=0x5596cca73d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:48.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.749+0000 7fc75e633740 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca73d90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0cc190 tx=0x7fc74c09a9c0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.750+0000 7fc75e633740 1 -- 172.21.15.129:0/1556022674 shutdown_connections 2024-05-20T18:05:48.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.750+0000 7fc75e633740 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596cc6871f0 0x5596cca84410 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.750+0000 7fc75e633740 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca73d90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.750+0000 7fc75e633740 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6c8410 0x5596cca71d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.750+0000 7fc75e633740 1 --2- 172.21.15.129:0/1556022674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 0x5596cca73850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.750+0000 7fc75e633740 1 -- 172.21.15.129:0/1556022674 >> 172.21.15.129:0/1556022674 conn(0x5596cc6bc560 msgr2=0x5596cca63180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:48.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.750+0000 7fc738ff9640 0 client.26195 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:48.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.751+0000 7fc75e633740 1 -- 172.21.15.129:0/1556022674 shutdown_connections 2024-05-20T18:05:48.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.751+0000 7fc75e633740 1 -- 172.21.15.129:0/1556022674 wait complete. 2024-05-20T18:05:48.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.756+0000 7fc75e633740 20 client.26195 trim_cache size 0 max 16384 2024-05-20T18:05:48.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.815+0000 7fc75e633740 1 Processor -- start 2024-05-20T18:05:48.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.818+0000 7fc75e633740 1 -- start start 2024-05-20T18:05:48.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.819+0000 7fc75e633740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 0x5596cca6cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.819+0000 7fc75e633740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca6d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.819+0000 7fc75e633740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6c8410 0x5596cca705f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.819+0000 7fc75e633740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc7501c4220 con 0x5596cc69f550 2024-05-20T18:05:48.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.819+0000 7fc75e633740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc7501c6aa0 con 0x5596cc6a7730 2024-05-20T18:05:48.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.819+0000 7fc75e633740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc7501c4890 con 0x5596cc6c8410 2024-05-20T18:05:48.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.820+0000 7fc755c55640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6c8410 0x5596cca705f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.820+0000 7fc755c55640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6c8410 0x5596cca705f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:36180/0 (socket says 172.21.15.129:36180) 2024-05-20T18:05:48.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.820+0000 7fc755c55640 1 -- 172.21.15.129:0/3048001953 learned_addr learned my addr 172.21.15.129:0/3048001953 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:48.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.823+0000 7fc754c53640 1 --2- 172.21.15.129:0/3048001953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca6d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.825+0000 7fc755454640 1 --2- 172.21.15.129:0/3048001953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 0x5596cca6cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.827+0000 7fc755454640 1 -- 172.21.15.129:0/3048001953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6c8410 msgr2=0x5596cca705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:48.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.827+0000 7fc755454640 1 --2- 172.21.15.129:0/3048001953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6c8410 0x5596cca705f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.827+0000 7fc755454640 1 -- 172.21.15.129:0/3048001953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 msgr2=0x5596cca6d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:48.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.827+0000 7fc755454640 1 --2- 172.21.15.129:0/3048001953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca6d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.827+0000 7fc755454640 1 -- 172.21.15.129:0/3048001953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596cca6c9f0 con 0x5596cc69f550 2024-05-20T18:05:48.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.828+0000 7fc755454640 1 --2- 172.21.15.129:0/3048001953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 0x5596cca6cf00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x5596cca84b50 tx=0x7fc74003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd95be52eddd2d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:48.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.829+0000 7fc73affd640 1 -- 172.21.15.129:0/3048001953 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740053030 con 0x5596cc69f550 2024-05-20T18:05:48.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.829+0000 7fc73affd640 1 -- 172.21.15.129:0/3048001953 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc740004c60 con 0x5596cc69f550 2024-05-20T18:05:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.829+0000 7fc73affd640 1 -- 172.21.15.129:0/3048001953 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740035040 con 0x5596cc69f550 2024-05-20T18:05:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.831+0000 7fc75e633740 1 -- 172.21.15.129:0/3048001953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 msgr2=0x5596cca6cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.831+0000 7fc75e633740 1 --2- 172.21.15.129:0/3048001953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 0x5596cca6cf00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x5596cca84b50 tx=0x7fc74003f000 comp rx=0 tx=0).stop 2024-05-20T18:05:48.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.846+0000 7fc75e633740 1 -- 172.21.15.129:0/3048001953 shutdown_connections 2024-05-20T18:05:48.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.846+0000 7fc75e633740 1 --2- 172.21.15.129:0/3048001953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca6d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.846+0000 7fc75e633740 1 --2- 172.21.15.129:0/3048001953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc6c8410 0x5596cca705f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.846+0000 7fc75e633740 1 --2- 172.21.15.129:0/3048001953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc69f550 0x5596cca6cf00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.846+0000 7fc75e633740 1 -- 172.21.15.129:0/3048001953 >> 172.21.15.129:0/3048001953 conn(0x5596cc6bc560 msgr2=0x5596cca73850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:48.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.850+0000 7fc75e633740 1 -- 172.21.15.129:0/3048001953 shutdown_connections 2024-05-20T18:05:48.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.852+0000 7fc75e633740 1 -- 172.21.15.129:0/3048001953 wait complete. 2024-05-20T18:05:48.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.853+0000 7fc75e633740 1 -- start start 2024-05-20T18:05:48.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.853+0000 7fc75e633740 1 Processor -- start 2024-05-20T18:05:48.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.856+0000 7fc75e633740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc69f550 0x5596cca7a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.857+0000 7fc75e633740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca7a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.857+0000 7fc75e633740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 0x5596cca77480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.857+0000 7fc75e633740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc7501c4f60 con 0x5596cc6c8410 2024-05-20T18:05:48.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.857+0000 7fc75e633740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc7501c63d0 con 0x5596cc6a7730 2024-05-20T18:05:48.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.857+0000 7fc75e633740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc7501c5630 con 0x5596cc69f550 2024-05-20T18:05:48.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.857+0000 7fc75e633740 10 client.0 _ll_register_callbacks cb 0x7fc74c0107f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:48.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.858+0000 7fc755454640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc69f550 0x5596cca7a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.858+0000 7fc754c53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca7a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.858+0000 7fc754c53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca7a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:47096/0 (socket says 172.21.15.129:47096) 2024-05-20T18:05:48.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.858+0000 7fc754c53640 1 -- 172.21.15.129:0/1434444200 learned_addr learned my addr 172.21.15.129:0/1434444200 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:48.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.858+0000 7fc755c55640 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 0x5596cca77480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.860+0000 7fc755454640 1 -- 172.21.15.129:0/1434444200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 msgr2=0x5596cca7a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:48.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.860+0000 7fc755454640 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca7a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.860+0000 7fc755454640 1 -- 172.21.15.129:0/1434444200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 msgr2=0x5596cca77480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.860+0000 7fc755454640 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 0x5596cca77480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.860+0000 7fc755454640 1 -- 172.21.15.129:0/1434444200 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596cca77bc0 con 0x5596cc69f550 2024-05-20T18:05:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.860+0000 7fc754c53640 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca7a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:05:48.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.862+0000 7fc755454640 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc69f550 0x5596cca7a000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc74002aaa0 tx=0x7fc740004e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=558f779f25b04f52 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:48.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.863+0000 7fc738ff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:05:48.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.867+0000 7fc738ff9640 1 -- 172.21.15.129:0/1434444200 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74005b070 con 0x5596cc69f550 2024-05-20T18:05:48.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.867+0000 7fc75e633740 1 -- 172.21.15.129:0/1434444200 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc740004c60 con 0x5596cc69f550 2024-05-20T18:05:48.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.868+0000 7fc738ff9640 1 -- 172.21.15.129:0/1434444200 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fc74004fa00 con 0x5596cc69f550 2024-05-20T18:05:48.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.874+0000 7fc738ff9640 1 -- 172.21.15.129:0/1434444200 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc740065050 con 0x5596cc69f550 2024-05-20T18:05:48.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.875+0000 7fc75e633740 10 client.26236 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:48.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.875+0000 7fc75e633740 1 -- 172.21.15.129:0/1434444200 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5596ccaec2d0 con 0x5596cc69f550 2024-05-20T18:05:48.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.878+0000 7fc738ff9640 1 -- 172.21.15.129:0/1434444200 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc740004c60 con 0x5596cc69f550 2024-05-20T18:05:48.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.878+0000 7fc738ff9640 1 -- 172.21.15.129:0/1434444200 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fc74003a040 con 0x5596cc69f550 2024-05-20T18:05:48.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.878+0000 7fc75e633740 10 client.26236 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:48.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.878+0000 7fc75e633740 10 client.26236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:48.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.878+0000 7fc75e633740 20 client.26236 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:48.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.878+0000 7fc75e633740 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596cc65c630 0x5596ccaeea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:48.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.878+0000 7fc75e633740 4 client.26236 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:05:48.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.878+0000 7fc75e633740 1 -- 172.21.15.129:0/1434444200 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5596cc9d4410 con 0x5596cc65c630 2024-05-20T18:05:48.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.880+0000 7fc754c53640 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596cc65c630 0x5596ccaeea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:48.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.888+0000 7fc754c53640 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596cc65c630 0x5596ccaeea00 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32ac83556d136e7e server_cookie=94bedf99ab9a92fd in_seq=0 out_seq=0 2024-05-20T18:05:48.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:48.889+0000 7fc738ff9640 10 client.26236 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:49.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:49 smithi154 ceph-mon[36482]: pgmap v222: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 734 MiB/s wr, 1.51k op/s 2024-05-20T18:05:49.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:49 smithi129 ceph-mon[32029]: pgmap v222: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 734 MiB/s wr, 1.51k op/s 2024-05-20T18:05:49.682 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:49 smithi204 ceph-mon[37076]: pgmap v222: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 734 MiB/s wr, 1.51k op/s 2024-05-20T18:05:49.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.699+0000 7fc738ff9640 1 -- 172.21.15.129:0/1434444200 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5596cc9d4410 con 0x5596cc65c630 2024-05-20T18:05:49.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.699+0000 7fc738ff9640 10 client.26236 handle_command_reply: tid=0 2024-05-20T18:05:49.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.699+0000 7fc75e633740 1 client.26236 shutdown 2024-05-20T18:05:49.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.704+0000 7fc75e633740 1 -- 172.21.15.129:0/1434444200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc69f550 msgr2=0x5596cca7a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:49.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.704+0000 7fc75e633740 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc69f550 0x5596cca7a000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc74002aaa0 tx=0x7fc740004e10 comp rx=0 tx=0).stop 2024-05-20T18:05:49.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.704+0000 7fc75e633740 1 -- 172.21.15.129:0/1434444200 shutdown_connections 2024-05-20T18:05:49.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.704+0000 7fc75e633740 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5596cc65c630 0x5596ccaeea00 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:49.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.704+0000 7fc75e633740 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5596cc6a7730 0x5596cca7a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:49.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.704+0000 7fc75e633740 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5596cc6c8410 0x5596cca77480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:49.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.704+0000 7fc75e633740 1 --2- 172.21.15.129:0/1434444200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5596cc69f550 0x5596cca7a000 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:49.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.704+0000 7fc75e633740 1 -- 172.21.15.129:0/1434444200 >> 172.21.15.129:0/1434444200 conn(0x5596cc6bc560 msgr2=0x5596cca73850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:49.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.705+0000 7fc738ff9640 0 client.26236 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:49.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.705+0000 7fc75e633740 1 -- 172.21.15.129:0/1434444200 shutdown_connections 2024-05-20T18:05:49.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.705+0000 7fc75e633740 1 -- 172.21.15.129:0/1434444200 wait complete. 2024-05-20T18:05:49.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.706+0000 7fc75e633740 20 client.26236 trim_cache size 0 max 16384 2024-05-20T18:05:49.709 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:05:49.709 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:05:49.709 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "eb18d02a-ae41-4971-aca3-c840c2bf68d0", 2024-05-20T18:05:49.710 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:05:49.710 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:05:49.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.726+0000 7fc73bfff640 1 -- 172.21.15.129:0/499129917 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc73c0a2f00 msgr2=0x7fc73c0a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:49.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.726+0000 7fc73bfff640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc73c0a2f00 0x7fc73c0a5410 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x5596cca65300 tx=0x7fc740000bc0 comp rx=0 tx=0).stop 2024-05-20T18:05:49.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.726+0000 7fc73bfff640 1 -- 172.21.15.129:0/499129917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc750119770 msgr2=0x7fc750113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:49.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.726+0000 7fc73bfff640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc750119770 0x7fc750113a40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00b3f0 tx=0x7fc74c00b8b0 comp rx=0 tx=0).stop 2024-05-20T18:05:49.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.732+0000 7fc73bfff640 1 -- 172.21.15.129:0/499129917 shutdown_connections 2024-05-20T18:05:49.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.732+0000 7fc73bfff640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc73c0a2f00 0x7fc73c0a5410 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:49.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.732+0000 7fc73bfff640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc750119770 0x7fc750113a40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:49.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.732+0000 7fc73bfff640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc75011c0d0 0x7fc750113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:49.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.732+0000 7fc73bfff640 1 --2- 172.21.15.129:0/499129917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc750109060 0x7fc750113500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:49.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.732+0000 7fc73bfff640 1 -- 172.21.15.129:0/499129917 >> 172.21.15.129:0/499129917 conn(0x7fc75006d750 msgr2=0x7fc750119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:49.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.733+0000 7fc73bfff640 1 -- 172.21.15.129:0/499129917 shutdown_connections 2024-05-20T18:05:49.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:49.733+0000 7fc73bfff640 1 -- 172.21.15.129:0/499129917 wait complete. 2024-05-20T18:05:49.793 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:05:49.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:05:50.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.331+0000 7f5904964640 1 -- 172.21.15.129:0/187510041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f590010f270 msgr2=0x7f5900108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:50.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.331+0000 7f5904964640 1 --2- 172.21.15.129:0/187510041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f590010f270 0x7f5900108b50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009900 tx=0x7f58f002a440 comp rx=0 tx=0).stop 2024-05-20T18:05:50.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.332+0000 7f5904964640 1 -- 172.21.15.129:0/187510041 shutdown_connections 2024-05-20T18:05:50.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.332+0000 7f5904964640 1 --2- 172.21.15.129:0/187510041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5900109090 0x7f5900109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.332+0000 7f5904964640 1 --2- 172.21.15.129:0/187510041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f590010f270 0x7f5900108b50 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.332+0000 7f5904964640 1 --2- 172.21.15.129:0/187510041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f590010e890 0x7f590010eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.332+0000 7f5904964640 1 -- 172.21.15.129:0/187510041 >> 172.21.15.129:0/187510041 conn(0x7f590006d580 msgr2=0x7f590006d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:50.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.333+0000 7f5904964640 1 -- 172.21.15.129:0/187510041 shutdown_connections 2024-05-20T18:05:50.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.333+0000 7f5904964640 1 -- 172.21.15.129:0/187510041 wait complete. 2024-05-20T18:05:50.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.335+0000 7f5904964640 1 Processor -- start 2024-05-20T18:05:50.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.335+0000 7f5904964640 1 -- start start 2024-05-20T18:05:50.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.336+0000 7f5904964640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5900109090 0x7f590011f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.336+0000 7f5904964640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f590010e890 0x7f59001182e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.336+0000 7f5904964640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f590010f270 0x7f5900118960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.336+0000 7f5904964640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5900121a90 con 0x7f590010f270 2024-05-20T18:05:50.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.336+0000 7f5904964640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5900121910 con 0x7f590010e890 2024-05-20T18:05:50.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.336+0000 7f5904964640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5900121c10 con 0x7f5900109090 2024-05-20T18:05:50.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.336+0000 7f58fed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5900109090 0x7f590011f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.336+0000 7f58fed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5900109090 0x7f590011f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:58930/0 (socket says 172.21.15.129:58930) 2024-05-20T18:05:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.336+0000 7f58fed9d640 1 -- 172.21.15.129:0/2301826015 learned_addr learned my addr 172.21.15.129:0/2301826015 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.337+0000 7f58ff59e640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f590010f270 0x7f5900118960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.337+0000 7f58fed9d640 1 -- 172.21.15.129:0/2301826015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f590010e890 msgr2=0x7f59001182e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-20T18:05:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.337+0000 7f58fed9d640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f590010e890 0x7f59001182e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.337+0000 7f58fed9d640 1 -- 172.21.15.129:0/2301826015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f590010f270 msgr2=0x7f5900118960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.337+0000 7f58fed9d640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f590010f270 0x7f5900118960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.337+0000 7f58fed9d640 1 -- 172.21.15.129:0/2301826015 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590011cce0 con 0x7f5900109090 2024-05-20T18:05:50.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.338+0000 7f58fed9d640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5900109090 0x7f590011f1c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f58f800ebc0 tx=0x7f58f800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.338+0000 7f58e7fff640 1 -- 172.21.15.129:0/2301826015 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f8010630 con 0x7f5900109090 2024-05-20T18:05:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.339+0000 7f5904964640 1 -- 172.21.15.129:0/2301826015 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f590011cfd0 con 0x7f5900109090 2024-05-20T18:05:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.339+0000 7f5904964640 1 -- 172.21.15.129:0/2301826015 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f590011d570 con 0x7f5900109090 2024-05-20T18:05:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.340+0000 7f58e7fff640 1 -- 172.21.15.129:0/2301826015 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f8010cb0 con 0x7f5900109090 2024-05-20T18:05:50.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.340+0000 7f58e7fff640 1 -- 172.21.15.129:0/2301826015 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f8018c40 con 0x7f5900109090 2024-05-20T18:05:50.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.349+0000 7f590c013740 1 Processor -- start 2024-05-20T18:05:50.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.352+0000 7f590c013740 1 -- start start 2024-05-20T18:05:50.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.353+0000 7f590c013740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c61cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.353+0000 7f590c013740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 0x55fa1c61d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.353+0000 7f590c013740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c25d6b0 0x55fa1c61d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.353+0000 7f590c013740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fa1c582d00 con 0x55fa1c2554d0 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.353+0000 7f590c013740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fa1c52a4c0 con 0x55fa1c27e390 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.353+0000 7f590c013740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fa1c57ecc0 con 0x55fa1c25d6b0 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.353+0000 7f58e7fff640 1 -- 172.21.15.129:0/2301826015 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f58f80107d0 con 0x7f5900109090 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.353+0000 7f58fe59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c25d6b0 0x55fa1c61d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.353+0000 7f58fe59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c25d6b0 0x55fa1c61d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:58938/0 (socket says 172.21.15.129:58938) 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.353+0000 7f58ff59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c61cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.353+0000 7f58fe59c640 1 -- 172.21.15.129:0/501965219 learned_addr learned my addr 172.21.15.129:0/501965219 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.354+0000 7f58e7fff640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58e80a2df0 0x7f58e80a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.355+0000 7f58e7fff640 1 -- 172.21.15.129:0/2301826015 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f58f80cfaa0 con 0x7f5900109090 2024-05-20T18:05:50.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.355+0000 7f58fed9d640 1 --2- 172.21.15.129:0/501965219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 0x55fa1c61d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.355+0000 7f58fe59c640 1 -- 172.21.15.129:0/501965219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 msgr2=0x55fa1c61cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:50.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.355+0000 7f58fe59c640 1 --2- 172.21.15.129:0/501965219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c61cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.355+0000 7f58fe59c640 1 -- 172.21.15.129:0/501965219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 msgr2=0x55fa1c61d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:50.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.355+0000 7f58fe59c640 1 --2- 172.21.15.129:0/501965219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 0x55fa1c61d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.355+0000 7f58fe59c640 1 -- 172.21.15.129:0/501965219 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa1c3607d0 con 0x55fa1c25d6b0 2024-05-20T18:05:50.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.356+0000 7f58ff59e640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58e80a2df0 0x7f58e80a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.357+0000 7f58ff59e640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58e80a2df0 0x7f58e80a5300 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55fa1c61a0a0 tx=0x7f58f4004730 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:50.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.357+0000 7f58fe59c640 1 --2- 172.21.15.129:0/501965219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c25d6b0 0x55fa1c61d9b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f58f002a960 tx=0x7f58f0009470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=115ace297bbf27f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:50.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.360+0000 7f58e57fa640 1 -- 172.21.15.129:0/501965219 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0040030 con 0x55fa1c25d6b0 2024-05-20T18:05:50.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.360+0000 7f58e57fa640 1 -- 172.21.15.129:0/501965219 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f0004c10 con 0x55fa1c25d6b0 2024-05-20T18:05:50.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.366+0000 7f58e57fa640 1 -- 172.21.15.129:0/501965219 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f003c6a0 con 0x55fa1c25d6b0 2024-05-20T18:05:50.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.367+0000 7f590c013740 1 -- 172.21.15.129:0/501965219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c25d6b0 msgr2=0x55fa1c61d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:50.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.367+0000 7f590c013740 1 --2- 172.21.15.129:0/501965219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c25d6b0 0x55fa1c61d9b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f58f002a960 tx=0x7f58f0009470 comp rx=0 tx=0).stop 2024-05-20T18:05:50.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.367+0000 7f590c013740 1 -- 172.21.15.129:0/501965219 shutdown_connections 2024-05-20T18:05:50.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.367+0000 7f590c013740 1 --2- 172.21.15.129:0/501965219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c25d6b0 0x55fa1c61d9b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.367+0000 7f590c013740 1 --2- 172.21.15.129:0/501965219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 0x55fa1c61d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.367+0000 7f590c013740 1 --2- 172.21.15.129:0/501965219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c61cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.367+0000 7f590c013740 1 -- 172.21.15.129:0/501965219 >> 172.21.15.129:0/501965219 conn(0x55fa1c2724e0 msgr2=0x55fa1c6196e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:50.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.368+0000 7f590c013740 1 -- 172.21.15.129:0/501965219 shutdown_connections 2024-05-20T18:05:50.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.368+0000 7f590c013740 1 -- 172.21.15.129:0/501965219 wait complete. 2024-05-20T18:05:50.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.369+0000 7f590c013740 1 -- start start 2024-05-20T18:05:50.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.369+0000 7f590c013740 1 Processor -- start 2024-05-20T18:05:50.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.370+0000 7f590c013740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 0x55fa1c626f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.370+0000 7f590c013740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c6274c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.370+0000 7f590c013740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c23cf60 0x55fa1c627a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.370+0000 7f590c013740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fa1c62a270 con 0x55fa1c23cf60 2024-05-20T18:05:50.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.370+0000 7f590c013740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fa1c53db90 con 0x55fa1c27e390 2024-05-20T18:05:50.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.370+0000 7f590c013740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fa1c62a3f0 con 0x55fa1c2554d0 2024-05-20T18:05:50.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.370+0000 7f590c013740 10 client.0 _ll_register_callbacks cb 0x55fa1c579da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:50.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.370+0000 7f58fed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 0x55fa1c626f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.370+0000 7f58fed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 0x55fa1c626f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:58940/0 (socket says 172.21.15.129:58940) 2024-05-20T18:05:50.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.370+0000 7f58fed9d640 1 -- 172.21.15.129:0/166753041 learned_addr learned my addr 172.21.15.129:0/166753041 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:50.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.371+0000 7f58ff59e640 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c23cf60 0x55fa1c627a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.371+0000 7f58fe59c640 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c6274c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.372+0000 7f58ff59e640 1 -- 172.21.15.129:0/166753041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 msgr2=0x55fa1c626f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:50.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.372+0000 7f58ff59e640 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 0x55fa1c626f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.372+0000 7f58ff59e640 1 -- 172.21.15.129:0/166753041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 msgr2=0x55fa1c6274c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:50.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.372+0000 7f58ff59e640 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c6274c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.372+0000 7f58ff59e640 1 -- 172.21.15.129:0/166753041 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa1c2963f0 con 0x55fa1c23cf60 2024-05-20T18:05:50.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.373+0000 7f58ff59e640 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c23cf60 0x55fa1c627a00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009bb0 tx=0x7f58f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f329e560ba87a92 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:50.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.373+0000 7f58e2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:05:50.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.373+0000 7f58e2ff5640 1 -- 172.21.15.129:0/166753041 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f4026020 con 0x55fa1c23cf60 2024-05-20T18:05:50.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.373+0000 7f590c013740 1 -- 172.21.15.129:0/166753041 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58f0004c10 con 0x55fa1c23cf60 2024-05-20T18:05:50.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.374+0000 7f58e2ff5640 1 -- 172.21.15.129:0/166753041 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f58f4022890 con 0x55fa1c23cf60 2024-05-20T18:05:50.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.374+0000 7f58e2ff5640 1 -- 172.21.15.129:0/166753041 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f58f4031050 con 0x55fa1c23cf60 2024-05-20T18:05:50.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.374+0000 7f590c013740 10 client.16491 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:50.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.374+0000 7f590c013740 1 -- 172.21.15.129:0/166753041 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fa1c2560f0 con 0x55fa1c23cf60 2024-05-20T18:05:50.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.376+0000 7f58e2ff5640 1 -- 172.21.15.129:0/166753041 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f0004c10 con 0x55fa1c23cf60 2024-05-20T18:05:50.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.376+0000 7f58e2ff5640 1 -- 172.21.15.129:0/166753041 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f58f400fc00 con 0x55fa1c23cf60 2024-05-20T18:05:50.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.376+0000 7f590c013740 10 client.16491 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:50.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.376+0000 7f590c013740 10 client.16491 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:50.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.376+0000 7f590c013740 20 client.16491 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:50.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.376+0000 7f590c013740 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fa1c2123a0 0x55fa1c639e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.376+0000 7f590c013740 4 client.16491 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:50.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.376+0000 7f590c013740 1 -- 172.21.15.129:0/166753041 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa1c590430 con 0x55fa1c2123a0 2024-05-20T18:05:50.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.378+0000 7f58fed9d640 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fa1c2123a0 0x55fa1c639e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.386+0000 7f58fed9d640 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fa1c2123a0 0x55fa1c639e30 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f8ced38bc222616 server_cookie=7da8a1881d695419 in_seq=0 out_seq=0 2024-05-20T18:05:50.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.386+0000 7f58e2ff5640 10 client.16491 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:50.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.389+0000 7f58e2ff5640 1 -- 172.21.15.129:0/166753041 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fa1c590430 con 0x55fa1c2123a0 2024-05-20T18:05:50.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.389+0000 7f58e2ff5640 10 client.16491 handle_command_reply: tid=0 2024-05-20T18:05:50.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.389+0000 7f590c013740 1 client.16491 shutdown 2024-05-20T18:05:50.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.393+0000 7f590c013740 1 -- 172.21.15.129:0/166753041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c23cf60 msgr2=0x55fa1c627a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:50.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.393+0000 7f590c013740 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c23cf60 0x55fa1c627a00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009bb0 tx=0x7f58f40046c0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.394+0000 7f590c013740 1 -- 172.21.15.129:0/166753041 shutdown_connections 2024-05-20T18:05:50.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.394+0000 7f590c013740 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fa1c2123a0 0x55fa1c639e30 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.394+0000 7f590c013740 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c6274c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.394+0000 7f590c013740 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c23cf60 0x55fa1c627a00 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.394+0000 7f590c013740 1 --2- 172.21.15.129:0/166753041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 0x55fa1c626f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.394+0000 7f590c013740 1 -- 172.21.15.129:0/166753041 >> 172.21.15.129:0/166753041 conn(0x55fa1c2724e0 msgr2=0x55fa1c61ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:50.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.394+0000 7f58e2ff5640 0 client.16491 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:50.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.394+0000 7f590c013740 1 -- 172.21.15.129:0/166753041 shutdown_connections 2024-05-20T18:05:50.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.396+0000 7f590c013740 1 -- 172.21.15.129:0/166753041 wait complete. 2024-05-20T18:05:50.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.399+0000 7f590c013740 20 client.16491 trim_cache size 0 max 16384 2024-05-20T18:05:50.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.439+0000 7f590c013740 1 Processor -- start 2024-05-20T18:05:50.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.439+0000 7f590c013740 1 -- start start 2024-05-20T18:05:50.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.440+0000 7f590c013740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c23cf60 0x55fa1c625a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.440+0000 7f590c013740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 0x55fa1c61d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.440+0000 7f58fed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c23cf60 0x55fa1c625a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.440+0000 7f58fed9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c23cf60 0x55fa1c625a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:58952/0 (socket says 172.21.15.129:58952) 2024-05-20T18:05:50.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.441+0000 7f58fe59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 0x55fa1c61d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.441+0000 7f58fe59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 0x55fa1c61d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38040/0 (socket says 172.21.15.129:38040) 2024-05-20T18:05:50.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.441+0000 7f58fed9d640 1 -- 172.21.15.129:0/2271337852 learned_addr learned my addr 172.21.15.129:0/2271337852 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:50.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.441+0000 7f590c013740 1 --2- 172.21.15.129:0/2271337852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c6274d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.442+0000 7f590c013740 1 -- 172.21.15.129:0/2271337852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec00c920 con 0x55fa1c2554d0 2024-05-20T18:05:50.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.442+0000 7f590c013740 1 -- 172.21.15.129:0/2271337852 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec00f1a0 con 0x55fa1c27e390 2024-05-20T18:05:50.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.442+0000 7f590c013740 1 -- 172.21.15.129:0/2271337852 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec00cf90 con 0x55fa1c23cf60 2024-05-20T18:05:50.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.442+0000 7f58ff59e640 1 --2- 172.21.15.129:0/2271337852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c6274d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.442+0000 7f58fed9d640 1 -- 172.21.15.129:0/2271337852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 msgr2=0x55fa1c6274d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:50.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.442+0000 7f58fed9d640 1 --2- 172.21.15.129:0/2271337852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c6274d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.442+0000 7f58fed9d640 1 -- 172.21.15.129:0/2271337852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 msgr2=0x55fa1c61d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:50.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.442+0000 7f58fed9d640 1 --2- 172.21.15.129:0/2271337852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 0x55fa1c61d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.443+0000 7f58fed9d640 1 -- 172.21.15.129:0/2271337852 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa1c613920 con 0x55fa1c23cf60 2024-05-20T18:05:50.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.443+0000 7f58fe59c640 1 --2- 172.21.15.129:0/2271337852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 0x55fa1c61d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:05:50.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.443+0000 7f58fed9d640 1 --2- 172.21.15.129:0/2271337852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c23cf60 0x55fa1c625a80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f58f809bc50 tx=0x7f58f80bc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b85e9249b326f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.464+0000 7f58e4ff9640 1 -- 172.21.15.129:0/2271337852 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f8010e60 con 0x55fa1c23cf60 2024-05-20T18:05:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.464+0000 7f58e4ff9640 1 -- 172.21.15.129:0/2271337852 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f809c650 con 0x55fa1c23cf60 2024-05-20T18:05:50.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.466+0000 7f590c013740 1 -- 172.21.15.129:0/2271337852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c23cf60 msgr2=0x55fa1c625a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:50.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.466+0000 7f590c013740 1 --2- 172.21.15.129:0/2271337852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c23cf60 0x55fa1c625a80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f58f809bc50 tx=0x7f58f80bc040 comp rx=0 tx=0).stop 2024-05-20T18:05:50.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.466+0000 7f590c013740 1 -- 172.21.15.129:0/2271337852 shutdown_connections 2024-05-20T18:05:50.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.466+0000 7f590c013740 1 --2- 172.21.15.129:0/2271337852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c27e390 0x55fa1c6274d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.466+0000 7f590c013740 1 --2- 172.21.15.129:0/2271337852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c2554d0 0x55fa1c61d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.466+0000 7f590c013740 1 --2- 172.21.15.129:0/2271337852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c23cf60 0x55fa1c625a80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.466+0000 7f590c013740 1 -- 172.21.15.129:0/2271337852 >> 172.21.15.129:0/2271337852 conn(0x55fa1c2724e0 msgr2=0x55fa1c61f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:50.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.468+0000 7f590c013740 1 -- 172.21.15.129:0/2271337852 shutdown_connections 2024-05-20T18:05:50.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.468+0000 7f590c013740 1 -- 172.21.15.129:0/2271337852 wait complete. 2024-05-20T18:05:50.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.468+0000 7f590c013740 1 -- start start 2024-05-20T18:05:50.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.469+0000 7f590c013740 1 Processor -- start 2024-05-20T18:05:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.469+0000 7f590c013740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c23cf60 0x55fa1c62a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.469+0000 7f590c013740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 0x55fa1c62abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.469+0000 7f590c013740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c27e390 0x55fa1c62b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.469+0000 7f590c013740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec00d660 con 0x55fa1c27e390 2024-05-20T18:05:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.469+0000 7f590c013740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec00ead0 con 0x55fa1c23cf60 2024-05-20T18:05:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.469+0000 7f590c013740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec00dd30 con 0x55fa1c2554d0 2024-05-20T18:05:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.469+0000 7f590c013740 10 client.0 _ll_register_callbacks cb 0x7f58f4009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.470+0000 7f58fe59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 0x55fa1c62abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.470+0000 7f58fe59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 0x55fa1c62abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:58958/0 (socket says 172.21.15.129:58958) 2024-05-20T18:05:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.470+0000 7f58fe59c640 1 -- 172.21.15.129:0/300980281 learned_addr learned my addr 172.21.15.129:0/300980281 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.470+0000 7f58fed9d640 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c23cf60 0x55fa1c62a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.471+0000 7f58fe59c640 1 -- 172.21.15.129:0/300980281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c23cf60 msgr2=0x55fa1c62a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:50.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.472+0000 7f58fe59c640 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c23cf60 0x55fa1c62a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.472+0000 7f58fe59c640 1 -- 172.21.15.129:0/300980281 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c27e390 msgr2=0x55fa1c62b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:50.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.472+0000 7f58fe59c640 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c27e390 0x55fa1c62b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.472+0000 7f58fe59c640 1 -- 172.21.15.129:0/300980281 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa1c6a01d0 con 0x55fa1c2554d0 2024-05-20T18:05:50.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.472+0000 7f58fe59c640 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 0x55fa1c62abd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f58f002a960 tx=0x7f58f002abc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=972c27874b3c2d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:50.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.473+0000 7f58e2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:05:50.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.473+0000 7f58e2ff5640 1 -- 172.21.15.129:0/300980281 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0005020 con 0x55fa1c2554d0 2024-05-20T18:05:50.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.473+0000 7f590c013740 1 -- 172.21.15.129:0/300980281 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58f809c650 con 0x55fa1c2554d0 2024-05-20T18:05:50.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.473+0000 7f58e2ff5640 1 -- 172.21.15.129:0/300980281 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7f58f00051c0 con 0x55fa1c2554d0 2024-05-20T18:05:50.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.474+0000 7f58e2ff5640 1 -- 172.21.15.129:0/300980281 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f58f00053c0 con 0x55fa1c2554d0 2024-05-20T18:05:50.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.474+0000 7f590c013740 10 client.26254 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:50.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.474+0000 7f590c013740 1 -- 172.21.15.129:0/300980281 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fa1c6a13a0 con 0x55fa1c2554d0 2024-05-20T18:05:50.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.474+0000 7f58e2ff5640 1 -- 172.21.15.129:0/300980281 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f809c650 con 0x55fa1c2554d0 2024-05-20T18:05:50.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.477+0000 7f58e2ff5640 1 -- 172.21.15.129:0/300980281 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7f58f0031440 con 0x55fa1c2554d0 2024-05-20T18:05:50.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.477+0000 7f590c013740 10 client.26254 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:50.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.477+0000 7f590c013740 10 client.26254 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:50.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.477+0000 7f590c013740 20 client.26254 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:50.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.477+0000 7f590c013740 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fa1c6a1ab0 0x55fa1c6a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:50.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.477+0000 7f590c013740 4 client.26254 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:05:50.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.477+0000 7f590c013740 1 -- 172.21.15.129:0/300980281 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fa1c58a780 con 0x55fa1c6a1ab0 2024-05-20T18:05:50.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.478+0000 7f58fed9d640 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fa1c6a1ab0 0x55fa1c6a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:50.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.480+0000 7f58fed9d640 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fa1c6a1ab0 0x55fa1c6a3ed0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d8c9188991ca392 server_cookie=9ca1cc24123e1c5 in_seq=0 out_seq=0 2024-05-20T18:05:50.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.480+0000 7f58e2ff5640 10 client.26254 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:50.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.490+0000 7f58e2ff5640 1 -- 172.21.15.129:0/300980281 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55fa1c58a780 con 0x55fa1c6a1ab0 2024-05-20T18:05:50.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.490+0000 7f58e2ff5640 10 client.26254 handle_command_reply: tid=0 2024-05-20T18:05:50.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.490+0000 7f590c013740 1 client.26254 shutdown 2024-05-20T18:05:50.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.502+0000 7f590c013740 1 -- 172.21.15.129:0/300980281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 msgr2=0x55fa1c62abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:50.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.502+0000 7f590c013740 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 0x55fa1c62abd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f58f002a960 tx=0x7f58f002abc0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.503+0000 7f590c013740 1 -- 172.21.15.129:0/300980281 shutdown_connections 2024-05-20T18:05:50.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.503+0000 7f590c013740 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55fa1c6a1ab0 0x55fa1c6a3ed0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.503+0000 7f590c013740 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa1c27e390 0x55fa1c62b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.503+0000 7f590c013740 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fa1c2554d0 0x55fa1c62abd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.503+0000 7f590c013740 1 --2- 172.21.15.129:0/300980281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fa1c23cf60 0x55fa1c62a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.503+0000 7f58e2ff5640 0 client.26254 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:50.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.503+0000 7f590c013740 1 -- 172.21.15.129:0/300980281 >> 172.21.15.129:0/300980281 conn(0x55fa1c2724e0 msgr2=0x55fa1c61f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:50.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.503+0000 7f590c013740 1 -- 172.21.15.129:0/300980281 shutdown_connections 2024-05-20T18:05:50.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.504+0000 7f590c013740 1 -- 172.21.15.129:0/300980281 wait complete. 2024-05-20T18:05:50.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.504+0000 7f590c013740 20 client.26254 trim_cache size 0 max 16384 2024-05-20T18:05:50.507 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:05:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "scrub active (1 inodes in the stack)", 2024-05-20T18:05:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:05:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: "eb18d02a-ae41-4971-aca3-c840c2bf68d0": { 2024-05-20T18:05:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:05:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "eb18d02a-ae41-4971-aca3-c840c2bf68d0", 2024-05-20T18:05:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:05:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:05:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:05:50.508 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:05:50.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.506+0000 7f5904964640 1 -- 172.21.15.129:0/2301826015 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58e80a2df0 msgr2=0x7f58e80a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:50.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.506+0000 7f5904964640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58e80a2df0 0x7f58e80a5300 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55fa1c61a0a0 tx=0x7f58f4004730 comp rx=0 tx=0).stop 2024-05-20T18:05:50.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.506+0000 7f5904964640 1 -- 172.21.15.129:0/2301826015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5900109090 msgr2=0x7f590011f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:50.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.506+0000 7f5904964640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5900109090 0x7f590011f1c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f58f800ebc0 tx=0x7f58f800c6a0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.506+0000 7f5904964640 1 -- 172.21.15.129:0/2301826015 shutdown_connections 2024-05-20T18:05:50.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.506+0000 7f5904964640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58e80a2df0 0x7f58e80a5300 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.506+0000 7f5904964640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f590010e890 0x7f59001182e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.507+0000 7f5904964640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f590010f270 0x7f5900118960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.507+0000 7f5904964640 1 --2- 172.21.15.129:0/2301826015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5900109090 0x7f590011f1c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:50.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.507+0000 7f5904964640 1 -- 172.21.15.129:0/2301826015 >> 172.21.15.129:0/2301826015 conn(0x7f590006d580 msgr2=0x7f590010a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:50.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.507+0000 7f5904964640 1 -- 172.21.15.129:0/2301826015 shutdown_connections 2024-05-20T18:05:50.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:05:50.507+0000 7f5904964640 1 -- 172.21.15.129:0/2301826015 wait complete. 2024-05-20T18:05:50.586 INFO:tasks.cephfs.filesystem:scrub status for tag:eb18d02a-ae41-4971-aca3-c840c2bf68d0 - {'path': '/', 'tag': 'eb18d02a-ae41-4971-aca3-c840c2bf68d0', 'options': 'recursive,force'} 2024-05-20T18:05:51.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:51 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:05:51.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:51 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:51.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:51 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:05:51.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:51 smithi154 ceph-mon[36482]: pgmap v223: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 952 MiB/s wr, 2.03k op/s 2024-05-20T18:05:51.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:51 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:05:51.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:51 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:51.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:51 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:05:51.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:51 smithi129 ceph-mon[32029]: pgmap v223: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 952 MiB/s wr, 2.03k op/s 2024-05-20T18:05:51.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:51 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:05:51.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:51 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:51.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:51 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:05:51.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:51 smithi204 ceph-mon[37076]: pgmap v223: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 952 MiB/s wr, 2.03k op/s 2024-05-20T18:05:53.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:53 smithi154 ceph-mon[36482]: pgmap v224: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 719 MiB/s wr, 1.52k op/s 2024-05-20T18:05:53.663 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:53 smithi129 ceph-mon[32029]: pgmap v224: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 719 MiB/s wr, 1.52k op/s 2024-05-20T18:05:53.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:53 smithi204 ceph-mon[37076]: pgmap v224: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 719 MiB/s wr, 1.52k op/s 2024-05-20T18:05:54.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:54 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:54.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:54 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:54.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:54 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:05:55.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:55 smithi154 ceph-mon[36482]: pgmap v225: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 735 MiB/s wr, 1.63k op/s 2024-05-20T18:05:55.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:55 smithi129 ceph-mon[32029]: pgmap v225: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 735 MiB/s wr, 1.63k op/s 2024-05-20T18:05:55.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:55 smithi204 ceph-mon[37076]: pgmap v225: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 735 MiB/s wr, 1.63k op/s 2024-05-20T18:05:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.734+0000 7fd44d145640 1 -- 172.21.15.129:0/4052356865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd44810d8b0 msgr2=0x7fd44810dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.734+0000 7fd44d145640 1 --2- 172.21.15.129:0/4052356865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd44810d8b0 0x7fd44810dce0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd440009be0 tx=0x7fd44002a710 comp rx=0 tx=0).stop 2024-05-20T18:05:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.736+0000 7fd44d145640 1 -- 172.21.15.129:0/4052356865 shutdown_connections 2024-05-20T18:05:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.736+0000 7fd44d145640 1 --2- 172.21.15.129:0/4052356865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44810e370 0x7fd448079e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.736+0000 7fd44d145640 1 --2- 172.21.15.129:0/4052356865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd44810d8b0 0x7fd44810dce0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.736+0000 7fd44d145640 1 --2- 172.21.15.129:0/4052356865 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd448109320 0x7fd448109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.736+0000 7fd44d145640 1 -- 172.21.15.129:0/4052356865 >> 172.21.15.129:0/4052356865 conn(0x7fd44806d780 msgr2=0x7fd44806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.737+0000 7fd44d145640 1 -- 172.21.15.129:0/4052356865 shutdown_connections 2024-05-20T18:05:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.741+0000 7fd44d145640 1 -- 172.21.15.129:0/4052356865 wait complete. 2024-05-20T18:05:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.743+0000 7fd44d145640 1 Processor -- start 2024-05-20T18:05:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.746+0000 7fd44d145640 1 -- start start 2024-05-20T18:05:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.746+0000 7fd44d145640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd448109320 0x7fd4480840b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.747+0000 7fd44d145640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd44810d8b0 0x7fd448084610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.747+0000 7fd44d145640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44810e370 0x7fd44807f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.747+0000 7fd44d145640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd44807d730 con 0x7fd448109320 2024-05-20T18:05:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.747+0000 7fd44d145640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd44807d5b0 con 0x7fd44810e370 2024-05-20T18:05:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.747+0000 7fd44d145640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd44807d8b0 con 0x7fd44810d8b0 2024-05-20T18:05:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.747+0000 7fd44759e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44810e370 0x7fd44807f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.747+0000 7fd44759e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44810e370 0x7fd44807f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:44264/0 (socket says 172.21.15.129:44264) 2024-05-20T18:05:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.747+0000 7fd44759e640 1 -- 172.21.15.129:0/1155119893 learned_addr learned my addr 172.21.15.129:0/1155119893 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.748+0000 7fd446d9d640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd448109320 0x7fd4480840b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.748+0000 7fd44759e640 1 -- 172.21.15.129:0/1155119893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd44810d8b0 msgr2=0x7fd448084610 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-20T18:05:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.748+0000 7fd44759e640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd44810d8b0 0x7fd448084610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.748+0000 7fd44759e640 1 -- 172.21.15.129:0/1155119893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd448109320 msgr2=0x7fd4480840b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.748+0000 7fd44759e640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd448109320 0x7fd4480840b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.748+0000 7fd44759e640 1 -- 172.21.15.129:0/1155119893 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44807fa60 con 0x7fd44810e370 2024-05-20T18:05:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.749+0000 7fd44759e640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44810e370 0x7fd44807f260 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd43c00b3f0 tx=0x7fd43c00b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.750+0000 7fd42ffff640 1 -- 172.21.15.129:0/1155119893 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c015020 con 0x7fd44810e370 2024-05-20T18:05:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.750+0000 7fd42ffff640 1 -- 172.21.15.129:0/1155119893 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd43c011d10 con 0x7fd44810e370 2024-05-20T18:05:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.750+0000 7fd42ffff640 1 -- 172.21.15.129:0/1155119893 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c00f2e0 con 0x7fd44810e370 2024-05-20T18:05:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.750+0000 7fd44d145640 1 -- 172.21.15.129:0/1155119893 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44807fd50 con 0x7fd44810e370 2024-05-20T18:05:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.750+0000 7fd44d145640 1 -- 172.21.15.129:0/1155119893 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4481bd7e0 con 0x7fd44810e370 2024-05-20T18:05:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.759+0000 7fd42ffff640 1 -- 172.21.15.129:0/1155119893 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fd43c018020 con 0x7fd44810e370 2024-05-20T18:05:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.763+0000 7fd453ff4740 1 Processor -- start 2024-05-20T18:05:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.763+0000 7fd453ff4740 1 -- start start 2024-05-20T18:05:55.762 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.763+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6dfd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.763+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6dfd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.763+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a3d4d0 0x55aac6dfdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.763+0000 7fd453ff4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55aac6d62340 con 0x55aac6a3d4d0 2024-05-20T18:05:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.763+0000 7fd453ff4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aac6d0a290 con 0x55aac6a352f0 2024-05-20T18:05:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.763+0000 7fd453ff4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55aac6d5e300 con 0x55aac6a5e1b0 2024-05-20T18:05:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.764+0000 7fd44659c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6dfd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.764+0000 7fd44659c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6dfd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:58988/0 (socket says 172.21.15.129:58988) 2024-05-20T18:05:55.763 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.764+0000 7fd44659c640 1 -- 172.21.15.129:0/1335566500 learned_addr learned my addr 172.21.15.129:0/1335566500 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:55.765 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.766+0000 7fd42ffff640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4300a2eb0 0x7fd4300a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.766 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.766+0000 7fd44759e640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4300a2eb0 0x7fd4300a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.766 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.767+0000 7fd446d9d640 1 --2- 172.21.15.129:0/1335566500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6dfd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.767+0000 7fd44659c640 1 --2- 172.21.15.129:0/1335566500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a3d4d0 0x55aac6dfdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.767 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.767+0000 7fd42ffff640 1 -- 172.21.15.129:0/1155119893 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fd43c0cfc50 con 0x7fd44810e370 2024-05-20T18:05:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.768+0000 7fd44659c640 1 -- 172.21.15.129:0/1335566500 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 msgr2=0x55aac6dfd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.768+0000 7fd44659c640 1 --2- 172.21.15.129:0/1335566500 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6dfd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.769+0000 7fd44659c640 1 -- 172.21.15.129:0/1335566500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 msgr2=0x55aac6dfd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.769+0000 7fd44659c640 1 --2- 172.21.15.129:0/1335566500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6dfd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.768 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.769+0000 7fd44659c640 1 -- 172.21.15.129:0/1335566500 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aac6d2e8e0 con 0x55aac6a3d4d0 2024-05-20T18:05:55.770 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.771+0000 7fd44659c640 1 --2- 172.21.15.129:0/1335566500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a3d4d0 0x55aac6dfdb00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd440004af0 tx=0x7fd440000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9542e84fe23a7bc6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:55.772 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.774+0000 7fd42d7fa640 1 -- 172.21.15.129:0/1335566500 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd440045020 con 0x55aac6a3d4d0 2024-05-20T18:05:55.772 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.774+0000 7fd42d7fa640 1 -- 172.21.15.129:0/1335566500 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd440003e50 con 0x55aac6a3d4d0 2024-05-20T18:05:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.776+0000 7fd453ff4740 1 -- 172.21.15.129:0/1335566500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a3d4d0 msgr2=0x55aac6dfdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:55.774 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.776+0000 7fd453ff4740 1 --2- 172.21.15.129:0/1335566500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a3d4d0 0x55aac6dfdb00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd440004af0 tx=0x7fd440000f40 comp rx=0 tx=0).stop 2024-05-20T18:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.776+0000 7fd453ff4740 1 -- 172.21.15.129:0/1335566500 shutdown_connections 2024-05-20T18:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.776+0000 7fd453ff4740 1 --2- 172.21.15.129:0/1335566500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6dfd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.776+0000 7fd453ff4740 1 --2- 172.21.15.129:0/1335566500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a3d4d0 0x55aac6dfdb00 secure :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd440004af0 tx=0x7fd440000f40 comp rx=0 tx=0).stop 2024-05-20T18:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.776+0000 7fd453ff4740 1 --2- 172.21.15.129:0/1335566500 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6dfd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.776+0000 7fd453ff4740 1 -- 172.21.15.129:0/1335566500 >> 172.21.15.129:0/1335566500 conn(0x55aac6a52300 msgr2=0x55aac6df9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.777+0000 7fd44759e640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4300a2eb0 0x7fd4300a53c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd43c013000 tx=0x7fd43c010000 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:55.788 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.790+0000 7fd453ff4740 1 -- 172.21.15.129:0/1335566500 shutdown_connections 2024-05-20T18:05:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.790+0000 7fd453ff4740 1 -- 172.21.15.129:0/1335566500 wait complete. 2024-05-20T18:05:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.791+0000 7fd453ff4740 1 -- start start 2024-05-20T18:05:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.792+0000 7fd453ff4740 1 Processor -- start 2024-05-20T18:05:55.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.792+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6e07740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.792+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6e07ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.793+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a1cd80 0x55aac6e0d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.793+0000 7fd453ff4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55aac6e0a1f0 con 0x55aac6a1cd80 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.793+0000 7fd453ff4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aac6d1d790 con 0x55aac6a352f0 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.793+0000 7fd453ff4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55aac6e0a370 con 0x55aac6a5e1b0 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.793+0000 7fd453ff4740 10 client.0 _ll_register_callbacks cb 0x55aac6d58a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.793+0000 7fd44659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6e07740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.793+0000 7fd446d9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6e07ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.793+0000 7fd446d9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6e07ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:58996/0 (socket says 172.21.15.129:58996) 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.793+0000 7fd446d9d640 1 -- 172.21.15.129:0/4223924969 learned_addr learned my addr 172.21.15.129:0/4223924969 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.794+0000 7fd446d9d640 1 -- 172.21.15.129:0/4223924969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 msgr2=0x55aac6e07740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:55.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.794+0000 7fd446d9d640 1 --2- 172.21.15.129:0/4223924969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6e07740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.794+0000 7fd446d9d640 1 -- 172.21.15.129:0/4223924969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a1cd80 msgr2=0x55aac6e0d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:55.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.794+0000 7fd446d9d640 1 --2- 172.21.15.129:0/4223924969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a1cd80 0x55aac6e0d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.794+0000 7fd446d9d640 1 -- 172.21.15.129:0/4223924969 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aac6a76210 con 0x55aac6a5e1b0 2024-05-20T18:05:55.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.795+0000 7fd446d9d640 1 --2- 172.21.15.129:0/4223924969 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6e07ca0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd438013710 tx=0x7fd438013bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15a3a6d67fae555a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.795+0000 7fd42aff5640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.796+0000 7fd42aff5640 1 -- 172.21.15.129:0/4223924969 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd438011dd0 con 0x55aac6a5e1b0 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.796+0000 7fd453ff4740 1 -- 172.21.15.129:0/4223924969 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd440003e50 con 0x55aac6a5e1b0 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.796+0000 7fd42aff5640 1 -- 172.21.15.129:0/4223924969 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fd438004020 con 0x55aac6a5e1b0 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.796+0000 7fd42aff5640 1 -- 172.21.15.129:0/4223924969 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd438004220 con 0x55aac6a5e1b0 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.797+0000 7fd453ff4740 10 client.26263 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.797+0000 7fd453ff4740 1 -- 172.21.15.129:0/4223924969 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55aac6b42620 con 0x55aac6a5e1b0 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.797+0000 7fd42aff5640 1 -- 172.21.15.129:0/4223924969 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd440003e50 con 0x55aac6a5e1b0 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.797+0000 7fd42aff5640 1 -- 172.21.15.129:0/4223924969 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fd43801a030 con 0x55aac6a5e1b0 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.797+0000 7fd453ff4740 10 client.26263 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.797+0000 7fd453ff4740 10 client.26263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:55.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.797+0000 7fd453ff4740 20 client.26263 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:55.800 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.802+0000 7fd453ff4740 1 --2- 172.21.15.129:0/4223924969 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aac69f21c0 0x55aac6e19fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.802+0000 7fd453ff4740 4 client.26263 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:05:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.802+0000 7fd453ff4740 1 -- 172.21.15.129:0/4223924969 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aac6d6fa70 con 0x55aac69f21c0 2024-05-20T18:05:55.806 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.806+0000 7fd44659c640 1 --2- 172.21.15.129:0/4223924969 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aac69f21c0 0x55aac6e19fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.814+0000 7fd44659c640 1 --2- 172.21.15.129:0/4223924969 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aac69f21c0 0x55aac6e19fe0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ac03d4dfaf1f5d2 server_cookie=d57321039d2fc0f2 in_seq=0 out_seq=0 2024-05-20T18:05:55.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.814+0000 7fd42aff5640 10 client.26263 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.822+0000 7fd42aff5640 1 -- 172.21.15.129:0/4223924969 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55aac6d6fa70 con 0x55aac69f21c0 2024-05-20T18:05:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.822+0000 7fd42aff5640 10 client.26263 handle_command_reply: tid=0 2024-05-20T18:05:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.822+0000 7fd453ff4740 1 client.26263 shutdown 2024-05-20T18:05:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.822+0000 7fd453ff4740 1 -- 172.21.15.129:0/4223924969 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 msgr2=0x55aac6e07ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.822+0000 7fd453ff4740 1 --2- 172.21.15.129:0/4223924969 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6e07ca0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd438013710 tx=0x7fd438013bd0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.823+0000 7fd453ff4740 1 -- 172.21.15.129:0/4223924969 shutdown_connections 2024-05-20T18:05:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.823+0000 7fd453ff4740 1 --2- 172.21.15.129:0/4223924969 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aac69f21c0 0x55aac6e19fe0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.823+0000 7fd453ff4740 1 --2- 172.21.15.129:0/4223924969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a1cd80 0x55aac6e0d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.823+0000 7fd453ff4740 1 --2- 172.21.15.129:0/4223924969 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6e07ca0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.823+0000 7fd453ff4740 1 --2- 172.21.15.129:0/4223924969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6e07740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.823+0000 7fd453ff4740 1 -- 172.21.15.129:0/4223924969 >> 172.21.15.129:0/4223924969 conn(0x55aac6a52300 msgr2=0x55aac6dfeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.823+0000 7fd42aff5640 0 client.26263 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.823+0000 7fd453ff4740 1 -- 172.21.15.129:0/4223924969 shutdown_connections 2024-05-20T18:05:55.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.823+0000 7fd453ff4740 1 -- 172.21.15.129:0/4223924969 wait complete. 2024-05-20T18:05:55.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.825+0000 7fd453ff4740 20 client.26263 trim_cache size 0 max 16384 2024-05-20T18:05:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.883+0000 7fd453ff4740 1 Processor -- start 2024-05-20T18:05:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.883+0000 7fd453ff4740 1 -- start start 2024-05-20T18:05:55.882 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.884+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a1cd80 0x55aac6e07aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.882 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.884+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6dfd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.882 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.885+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6e00e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.885+0000 7fd44659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a1cd80 0x55aac6e07aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.885+0000 7fd44659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a1cd80 0x55aac6e07aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38070/0 (socket says 172.21.15.129:38070) 2024-05-20T18:05:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.885+0000 7fd44659c640 1 -- 172.21.15.129:0/132342696 learned_addr learned my addr 172.21.15.129:0/132342696 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.885+0000 7fd453ff4740 1 -- 172.21.15.129:0/132342696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd43400c8d0 con 0x55aac6a1cd80 2024-05-20T18:05:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.885+0000 7fd453ff4740 1 -- 172.21.15.129:0/132342696 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd43400f150 con 0x55aac6a352f0 2024-05-20T18:05:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.885+0000 7fd453ff4740 1 -- 172.21.15.129:0/132342696 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd43400cf40 con 0x55aac6a5e1b0 2024-05-20T18:05:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.885+0000 7fd44659c640 1 --2- 172.21.15.129:0/132342696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6e00e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.885+0000 7fd446d9d640 1 --2- 172.21.15.129:0/132342696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6dfd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.889 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.891+0000 7fd446d9d640 1 -- 172.21.15.129:0/132342696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 msgr2=0x55aac6e00e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:55.889 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.891+0000 7fd446d9d640 1 --2- 172.21.15.129:0/132342696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6e00e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.891+0000 7fd446d9d640 1 -- 172.21.15.129:0/132342696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a1cd80 msgr2=0x55aac6e07aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.891+0000 7fd446d9d640 1 --2- 172.21.15.129:0/132342696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a1cd80 0x55aac6e07aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.891+0000 7fd446d9d640 1 -- 172.21.15.129:0/132342696 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aac6e07fe0 con 0x55aac6a352f0 2024-05-20T18:05:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.893+0000 7fd446d9d640 1 --2- 172.21.15.129:0/132342696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6dfd6a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd438000c00 tx=0x7fd438009d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75b4631bd4ddbb13 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:55.910 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.912+0000 7fd42cff9640 1 -- 172.21.15.129:0/132342696 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd438005440 con 0x55aac6a352f0 2024-05-20T18:05:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.912+0000 7fd42cff9640 1 -- 172.21.15.129:0/132342696 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4380055e0 con 0x55aac6a352f0 2024-05-20T18:05:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.914+0000 7fd453ff4740 1 -- 172.21.15.129:0/132342696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 msgr2=0x55aac6dfd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.914+0000 7fd453ff4740 1 --2- 172.21.15.129:0/132342696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6dfd6a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd438000c00 tx=0x7fd438009d30 comp rx=0 tx=0).stop 2024-05-20T18:05:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.915+0000 7fd453ff4740 1 -- 172.21.15.129:0/132342696 shutdown_connections 2024-05-20T18:05:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.915+0000 7fd453ff4740 1 --2- 172.21.15.129:0/132342696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6a352f0 0x55aac6dfd6a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.915+0000 7fd453ff4740 1 --2- 172.21.15.129:0/132342696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a5e1b0 0x55aac6e00e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.915+0000 7fd453ff4740 1 --2- 172.21.15.129:0/132342696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a1cd80 0x55aac6e07aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.915+0000 7fd453ff4740 1 -- 172.21.15.129:0/132342696 >> 172.21.15.129:0/132342696 conn(0x55aac6a52300 msgr2=0x55aac6dfcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:55.921 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.923+0000 7fd453ff4740 1 -- 172.21.15.129:0/132342696 shutdown_connections 2024-05-20T18:05:55.922 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.923+0000 7fd453ff4740 1 -- 172.21.15.129:0/132342696 wait complete. 2024-05-20T18:05:55.922 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.924+0000 7fd453ff4740 1 -- start start 2024-05-20T18:05:55.922 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.924+0000 7fd453ff4740 1 Processor -- start 2024-05-20T18:05:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.925+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a1cd80 0x55aac6e14140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.925+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a5e1b0 0x55aac6e0c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.925+0000 7fd453ff4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6e0cbc0 0x55aac6e0cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.925+0000 7fd453ff4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd43400d610 con 0x55aac6a5e1b0 2024-05-20T18:05:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.925+0000 7fd453ff4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd43400ea80 con 0x55aac6e0cbc0 2024-05-20T18:05:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.925+0000 7fd453ff4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd43400dce0 con 0x55aac6a1cd80 2024-05-20T18:05:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.925+0000 7fd453ff4740 10 client.0 _ll_register_callbacks cb 0x7fd438009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:05:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.926+0000 7fd44659c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a1cd80 0x55aac6e14140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.926+0000 7fd44659c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a1cd80 0x55aac6e14140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:59012/0 (socket says 172.21.15.129:59012) 2024-05-20T18:05:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.926+0000 7fd44659c640 1 -- 172.21.15.129:0/3641346325 learned_addr learned my addr 172.21.15.129:0/3641346325 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:05:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.926+0000 7fd44659c640 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6e0cbc0 0x55aac6e0cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.926+0000 7fd44659c640 1 -- 172.21.15.129:0/3641346325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6e0cbc0 msgr2=0x55aac6e0cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.926+0000 7fd44659c640 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6e0cbc0 0x55aac6e0cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.926+0000 7fd44659c640 1 -- 172.21.15.129:0/3641346325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a5e1b0 msgr2=0x55aac6e0c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:05:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.926+0000 7fd44659c640 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a5e1b0 0x55aac6e0c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.927+0000 7fd44659c640 1 -- 172.21.15.129:0/3641346325 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aac6e802f0 con 0x55aac6a1cd80 2024-05-20T18:05:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.927+0000 7fd44659c640 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a1cd80 0x55aac6e14140 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd44002ac30 tx=0x7fd440000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a7179b716182d73 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:05:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.927+0000 7fd42aff5640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:05:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.928+0000 7fd42aff5640 1 -- 172.21.15.129:0/3641346325 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd440045030 con 0x55aac6a1cd80 2024-05-20T18:05:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.929+0000 7fd453ff4740 1 -- 172.21.15.129:0/3641346325 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55aac6d69dc0 con 0x55aac6a1cd80 2024-05-20T18:05:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.929+0000 7fd42aff5640 1 -- 172.21.15.129:0/3641346325 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fd4400307b0 con 0x55aac6a1cd80 2024-05-20T18:05:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.930+0000 7fd42aff5640 1 -- 172.21.15.129:0/3641346325 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd440003970 con 0x55aac6a1cd80 2024-05-20T18:05:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.930+0000 7fd453ff4740 10 client.26269 fetch_fsmap learned FSMap version 24 2024-05-20T18:05:55.929 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.930+0000 7fd453ff4740 1 -- 172.21.15.129:0/3641346325 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55aac6e814c0 con 0x55aac6a1cd80 2024-05-20T18:05:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.942+0000 7fd42aff5640 1 -- 172.21.15.129:0/3641346325 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55aac6d69dc0 con 0x55aac6a1cd80 2024-05-20T18:05:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.944+0000 7fd42aff5640 1 -- 172.21.15.129:0/3641346325 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 24) ==== 8322+0+0 (secure 0 0 0) 0x7fd44004a040 con 0x55aac6a1cd80 2024-05-20T18:05:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.945+0000 7fd453ff4740 10 client.26269 fetch_fsmap finished waiting for FSMap version 24 2024-05-20T18:05:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.945+0000 7fd453ff4740 10 client.26269 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:05:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.945+0000 7fd453ff4740 20 client.26269 populate_metadata read hostname 'smithi129' 2024-05-20T18:05:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.945+0000 7fd453ff4740 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aac6e81ad0 0x55aac6e83f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:05:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.945+0000 7fd453ff4740 4 client.26269 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:05:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.945+0000 7fd453ff4740 1 -- 172.21.15.129:0/3641346325 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aac6ce78e0 con 0x55aac6e81ad0 2024-05-20T18:05:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.947+0000 7fd446d9d640 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aac6e81ad0 0x55aac6e83f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:05:55.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.963+0000 7fd42aff5640 10 client.26269 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:55.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:55.963+0000 7fd446d9d640 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aac6e81ad0 0x55aac6e83f60 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c3e3408fe9c0439 server_cookie=3724d9d04c1d63a2 in_seq=0 out_seq=0 2024-05-20T18:05:56.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:56 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:05:56.670 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:56 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:05:56.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:56 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:05:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.110+0000 7fd42aff5640 1 -- 172.21.15.129:0/3641346325 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aac6ce78e0 con 0x55aac6e81ad0 2024-05-20T18:05:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.110+0000 7fd42aff5640 10 client.26269 handle_command_reply: tid=0 2024-05-20T18:05:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.110+0000 7fd453ff4740 1 client.26269 shutdown 2024-05-20T18:05:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.111+0000 7fd453ff4740 1 -- 172.21.15.129:0/3641346325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a1cd80 msgr2=0x55aac6e14140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.111+0000 7fd453ff4740 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a1cd80 0x55aac6e14140 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd44002ac30 tx=0x7fd440000f40 comp rx=0 tx=0).stop 2024-05-20T18:05:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.111+0000 7fd453ff4740 1 -- 172.21.15.129:0/3641346325 shutdown_connections 2024-05-20T18:05:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.111+0000 7fd453ff4740 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aac6e81ad0 0x55aac6e83f60 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.111+0000 7fd453ff4740 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aac6e0cbc0 0x55aac6e0cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.111+0000 7fd453ff4740 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aac6a5e1b0 0x55aac6e0c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.111+0000 7fd453ff4740 1 --2- 172.21.15.129:0/3641346325 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aac6a1cd80 0x55aac6e14140 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.111+0000 7fd453ff4740 1 -- 172.21.15.129:0/3641346325 >> 172.21.15.129:0/3641346325 conn(0x55aac6a52300 msgr2=0x55aac6dff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.111+0000 7fd42aff5640 0 client.26269 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:05:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.111+0000 7fd453ff4740 1 -- 172.21.15.129:0/3641346325 shutdown_connections 2024-05-20T18:05:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.111+0000 7fd453ff4740 1 -- 172.21.15.129:0/3641346325 wait complete. 2024-05-20T18:05:57.111 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.113+0000 7fd453ff4740 20 client.26269 trim_cache size 0 max 16384 2024-05-20T18:05:57.111 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:05:57.111 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:05:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:05:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:05:57.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.115+0000 7fd44d145640 1 -- 172.21.15.129:0/1155119893 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4300a2eb0 msgr2=0x7fd4300a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.115+0000 7fd44d145640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4300a2eb0 0x7fd4300a53c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd43c013000 tx=0x7fd43c010000 comp rx=0 tx=0).stop 2024-05-20T18:05:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.116+0000 7fd44d145640 1 -- 172.21.15.129:0/1155119893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44810e370 msgr2=0x7fd44807f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:05:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.116+0000 7fd44d145640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44810e370 0x7fd44807f260 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd43c00b3f0 tx=0x7fd43c00b8b0 comp rx=0 tx=0).stop 2024-05-20T18:05:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.116+0000 7fd44d145640 1 -- 172.21.15.129:0/1155119893 shutdown_connections 2024-05-20T18:05:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.116+0000 7fd44d145640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4300a2eb0 0x7fd4300a53c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.116+0000 7fd44d145640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44810e370 0x7fd44807f260 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.116+0000 7fd44d145640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd44810d8b0 0x7fd448084610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.116+0000 7fd44d145640 1 --2- 172.21.15.129:0/1155119893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd448109320 0x7fd4480840b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:05:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.116+0000 7fd44d145640 1 -- 172.21.15.129:0/1155119893 >> 172.21.15.129:0/1155119893 conn(0x7fd44806d780 msgr2=0x7fd44810c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:05:57.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.117+0000 7fd44d145640 1 -- 172.21.15.129:0/1155119893 shutdown_connections 2024-05-20T18:05:57.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:05:57.117+0000 7fd44d145640 1 -- 172.21.15.129:0/1155119893 wait complete. 2024-05-20T18:05:57.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:57 smithi154 ceph-mon[36482]: pgmap v226: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 903 MiB/s wr, 2.18k op/s 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2744-gd2b507c7/rpm/el8/BUILD/ceph-19.0.0-2744-gd2b507c7/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7facd1645700 time 2024-05-20T18:05:57.169124+0000 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2744-gd2b507c7/rpm/el8/BUILD/ceph-19.0.0-2744-gd2b507c7/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7face3a6bafe] 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7face3a6bd29] 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 3: (MDLog::_replay_thread()+0x97a) [0x559bffdfa0de] 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 4: (MDLog::ReplayThread::entry()+0x11) [0x559bffe04ae7] 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 5: (Thread::entry_wrapper()+0x43) [0x7face3a3a6f1] 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 6: (Thread::_entry_func(void*)+0xd) [0x7face3a3a70d] 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7face25df1ca] 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 8: clone() 2024-05-20T18:05:57.643 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.168+0000 7facd1645700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2744-gd2b507c7/rpm/el8/BUILD/ceph-19.0.0-2744-gd2b507c7/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7facd1645700 time 2024-05-20T18:05:57.169124+0000 2024-05-20T18:05:57.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2744-gd2b507c7/rpm/el8/BUILD/ceph-19.0.0-2744-gd2b507c7/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-20T18:05:57.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:05:57.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7face3a6bafe] 2024-05-20T18:05:57.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7face3a6bd29] 2024-05-20T18:05:57.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 3: (MDLog::_replay_thread()+0x97a) [0x559bffdfa0de] 2024-05-20T18:05:57.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 4: (MDLog::ReplayThread::entry()+0x11) [0x559bffe04ae7] 2024-05-20T18:05:57.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 5: (Thread::entry_wrapper()+0x43) [0x7face3a3a6f1] 2024-05-20T18:05:57.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 6: (Thread::_entry_func(void*)+0xd) [0x7face3a3a70d] 2024-05-20T18:05:57.645 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7face25df1ca] 2024-05-20T18:05:57.645 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 8: clone() 2024-05-20T18:05:57.645 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.645 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: *** Caught signal (Aborted) ** 2024-05-20T18:05:57.645 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: in thread 7facd1645700 thread_name:md_log_replay 2024-05-20T18:05:57.645 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:05:57.645 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7face25e9d20] 2024-05-20T18:05:57.645 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2: gsignal() 2024-05-20T18:05:57.645 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 3: abort() 2024-05-20T18:05:57.645 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7face3a6bc45] 2024-05-20T18:05:57.645 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7face3a6bd29] 2024-05-20T18:05:57.646 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 6: (MDLog::_replay_thread()+0x97a) [0x559bffdfa0de] 2024-05-20T18:05:57.646 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 7: (MDLog::ReplayThread::entry()+0x11) [0x559bffe04ae7] 2024-05-20T18:05:57.646 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 8: (Thread::entry_wrapper()+0x43) [0x7face3a3a6f1] 2024-05-20T18:05:57.646 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 9: (Thread::_entry_func(void*)+0xd) [0x7face3a3a70d] 2024-05-20T18:05:57.646 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7face25df1ca] 2024-05-20T18:05:57.646 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 11: clone() 2024-05-20T18:05:57.646 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.169+0000 7facd1645700 -1 *** Caught signal (Aborted) ** 2024-05-20T18:05:57.646 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: in thread 7facd1645700 thread_name:md_log_replay 2024-05-20T18:05:57.646 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.646 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:05:57.646 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7face25e9d20] 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2: gsignal() 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 3: abort() 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7face3a6bc45] 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7face3a6bd29] 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 6: (MDLog::_replay_thread()+0x97a) [0x559bffdfa0de] 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 7: (MDLog::ReplayThread::entry()+0x11) [0x559bffe04ae7] 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 8: (Thread::entry_wrapper()+0x43) [0x7face3a3a6f1] 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 9: (Thread::_entry_func(void*)+0xd) [0x7face3a3a70d] 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7face25df1ca] 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 11: clone() 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.647 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: -1> 2024-05-20T18:05:57.168+0000 7facd1645700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2744-gd2b507c7/rpm/el8/BUILD/ceph-19.0.0-2744-gd2b507c7/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7facd1645700 time 2024-05-20T18:05:57.169124+0000 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2744-gd2b507c7/rpm/el8/BUILD/ceph-19.0.0-2744-gd2b507c7/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7face3a6bafe] 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7face3a6bd29] 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 3: (MDLog::_replay_thread()+0x97a) [0x559bffdfa0de] 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 4: (MDLog::ReplayThread::entry()+0x11) [0x559bffe04ae7] 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 5: (Thread::entry_wrapper()+0x43) [0x7face3a3a6f1] 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 6: (Thread::_entry_func(void*)+0xd) [0x7face3a3a70d] 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7face25df1ca] 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 8: clone() 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.648 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 0> 2024-05-20T18:05:57.169+0000 7facd1645700 -1 *** Caught signal (Aborted) ** 2024-05-20T18:05:57.649 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: in thread 7facd1645700 thread_name:md_log_replay 2024-05-20T18:05:57.649 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.649 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:05:57.649 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7face25e9d20] 2024-05-20T18:05:57.649 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2: gsignal() 2024-05-20T18:05:57.649 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 3: abort() 2024-05-20T18:05:57.649 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7face3a6bc45] 2024-05-20T18:05:57.649 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7face3a6bd29] 2024-05-20T18:05:57.649 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 6: (MDLog::_replay_thread()+0x97a) [0x559bffdfa0de] 2024-05-20T18:05:57.649 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 7: (MDLog::ReplayThread::entry()+0x11) [0x559bffe04ae7] 2024-05-20T18:05:57.650 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 8: (Thread::entry_wrapper()+0x43) [0x7face3a3a6f1] 2024-05-20T18:05:57.650 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 9: (Thread::_entry_func(void*)+0xd) [0x7face3a3a70d] 2024-05-20T18:05:57.650 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7face25df1ca] 2024-05-20T18:05:57.650 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 11: clone() 2024-05-20T18:05:57.650 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-20T18:05:57.650 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.650 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: -1> 2024-05-20T18:05:57.168+0000 7facd1645700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2744-gd2b507c7/rpm/el8/BUILD/ceph-19.0.0-2744-gd2b507c7/src/mds/MDLog.cc: In function 'void MDLog::_replay_thread()' thread 7facd1645700 time 2024-05-20T18:05:57.169124+0000 2024-05-20T18:05:57.650 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2744-gd2b507c7/rpm/el8/BUILD/ceph-19.0.0-2744-gd2b507c7/src/mds/MDLog.cc: 1417: FAILED ceph_assert(r) 2024-05-20T18:05:57.650 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.650 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7face3a6bafe] 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7face3a6bd29] 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 3: (MDLog::_replay_thread()+0x97a) [0x559bffdfa0de] 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 4: (MDLog::ReplayThread::entry()+0x11) [0x559bffe04ae7] 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 5: (Thread::entry_wrapper()+0x43) [0x7face3a3a6f1] 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 6: (Thread::_entry_func(void*)+0xd) [0x7face3a3a70d] 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7face25df1ca] 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 8: clone() 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 0> 2024-05-20T18:05:57.169+0000 7facd1645700 -1 *** Caught signal (Aborted) ** 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: in thread 7facd1645700 thread_name:md_log_replay 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.651 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7face25e9d20] 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2: gsignal() 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 3: abort() 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7face3a6bc45] 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7face3a6bd29] 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 6: (MDLog::_replay_thread()+0x97a) [0x559bffdfa0de] 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 7: (MDLog::ReplayThread::entry()+0x11) [0x559bffe04ae7] 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 8: (Thread::entry_wrapper()+0x43) [0x7face3a3a6f1] 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 9: (Thread::_entry_func(void*)+0xd) [0x7face3a3a70d] 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7face25df1ca] 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 11: clone() 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-20T18:05:57.652 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[69723]: 2024-05-20T18:05:57.667 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:57 smithi129 ceph-mon[32029]: pgmap v226: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 903 MiB/s wr, 2.18k op/s 2024-05-20T18:05:57.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:57 smithi204 ceph-mon[37076]: pgmap v226: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 903 MiB/s wr, 2.18k op/s 2024-05-20T18:05:58.392 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:05:57 smithi154 podman[70953]: 2024-05-20 18:05:57.906838194 +0000 UTC m=+0.463133880 container died c0c469bc41bf45e7f69bf0dbaa64d117d9ddfe94cf0da6ae36e8e5fc4ea0bc08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-20T18:05:59.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:05:59 smithi154 ceph-mon[36482]: pgmap v227: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 704 MiB/s wr, 1.81k op/s 2024-05-20T18:05:59.673 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:05:59 smithi129 ceph-mon[32029]: pgmap v227: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 704 MiB/s wr, 1.81k op/s 2024-05-20T18:05:59.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:05:59 smithi204 ceph-mon[37076]: pgmap v227: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 704 MiB/s wr, 1.81k op/s 2024-05-20T18:06:01.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:01 smithi154 ceph-mon[36482]: pgmap v228: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 930 MiB/s wr, 2.28k op/s 2024-05-20T18:06:01.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:01 smithi129 ceph-mon[32029]: pgmap v228: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 930 MiB/s wr, 2.28k op/s 2024-05-20T18:06:01.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:01 smithi204 ceph-mon[37076]: pgmap v228: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 930 MiB/s wr, 2.28k op/s 2024-05-20T18:06:03.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:03 smithi154 ceph-mon[36482]: pgmap v229: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 711 MiB/s wr, 1.78k op/s 2024-05-20T18:06:03.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:03 smithi129 ceph-mon[32029]: pgmap v229: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 711 MiB/s wr, 1.78k op/s 2024-05-20T18:06:03.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:03 smithi204 ceph-mon[37076]: pgmap v229: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 711 MiB/s wr, 1.78k op/s 2024-05-20T18:06:05.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:05 smithi154 ceph-mon[36482]: pgmap v230: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 736 MiB/s wr, 1.87k op/s 2024-05-20T18:06:05.664 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:05 smithi129 ceph-mon[32029]: pgmap v230: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 736 MiB/s wr, 1.87k op/s 2024-05-20T18:06:05.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:05 smithi204 ceph-mon[37076]: pgmap v230: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 736 MiB/s wr, 1.87k op/s 2024-05-20T18:06:06.142 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:05 smithi154 podman[70953]: 2024-05-20 18:06:05.752553493 +0000 UTC m=+8.308849159 container remove c0c469bc41bf45e7f69bf0dbaa64d117d9ddfe94cf0da6ae36e8e5fc4ea0bc08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, ceph=True) 2024-05-20T18:06:06.142 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:05 smithi154 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-05-20T18:06:06.642 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:06 smithi154 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-05-20T18:06:06.642 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:06 smithi154 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.h.service: Consumed 51.569s CPU time. 2024-05-20T18:06:07.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:07 smithi154 ceph-mon[36482]: pgmap v231: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 934 MiB/s wr, 2.26k op/s 2024-05-20T18:06:07.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:07 smithi129 ceph-mon[32029]: pgmap v231: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 934 MiB/s wr, 2.26k op/s 2024-05-20T18:06:07.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:07 smithi204 ceph-mon[37076]: pgmap v231: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 934 MiB/s wr, 2.26k op/s 2024-05-20T18:06:08.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:08 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:06:08.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:08 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:06:08.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:08 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:06:09.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:09 smithi154 ceph-mon[36482]: pgmap v232: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 783 MiB/s wr, 1.80k op/s 2024-05-20T18:06:09.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:09 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:06:09.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:09 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:06:09.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:09 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:06:09.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:09 smithi129 ceph-mon[32029]: pgmap v232: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 783 MiB/s wr, 1.80k op/s 2024-05-20T18:06:09.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:09 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:06:09.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:09 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:06:09.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:09 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:06:09.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:09 smithi204 ceph-mon[37076]: pgmap v232: 97 pgs: 97 active+clean; 60 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 783 MiB/s wr, 1.80k op/s 2024-05-20T18:06:09.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:09 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:06:09.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:09 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:06:09.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:09 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:06:10.308 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.310+0000 7fa5f7656640 1 -- 172.21.15.129:0/3028600155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5f0101540 msgr2=0x7fa5f010f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.310+0000 7fa5f7656640 1 --2- 172.21.15.129:0/3028600155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5f0101540 0x7fa5f010f370 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009900 tx=0x7fa5e002a3f0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.310+0000 7fa5f7656640 1 -- 172.21.15.129:0/3028600155 shutdown_connections 2024-05-20T18:06:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.310+0000 7fa5f7656640 1 --2- 172.21.15.129:0/3028600155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5f010f8b0 0x7fa5f0111cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.310+0000 7fa5f7656640 1 --2- 172.21.15.129:0/3028600155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5f0101540 0x7fa5f010f370 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.310+0000 7fa5f7656640 1 --2- 172.21.15.129:0/3028600155 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5f0100bf0 0x7fa5f0101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.310+0000 7fa5f7656640 1 -- 172.21.15.129:0/3028600155 >> 172.21.15.129:0/3028600155 conn(0x7fa5f0076860 msgr2=0x7fa5f0076ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.311+0000 7fa5f7656640 1 -- 172.21.15.129:0/3028600155 shutdown_connections 2024-05-20T18:06:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.311+0000 7fa5f7656640 1 -- 172.21.15.129:0/3028600155 wait complete. 2024-05-20T18:06:10.310 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.312+0000 7fa5f7656640 1 Processor -- start 2024-05-20T18:06:10.310 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.312+0000 7fa5f7656640 1 -- start start 2024-05-20T18:06:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.312+0000 7fa5f7656640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5f0100bf0 0x7fa5f01a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.312+0000 7fa5f7656640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5f0101540 0x7fa5f01a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.312+0000 7fa5f7656640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5f010f8b0 0x7fa5f019ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f7656640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f0115280 con 0x7fa5f010f8b0 2024-05-20T18:06:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f7656640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f0115100 con 0x7fa5f0100bf0 2024-05-20T18:06:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f7656640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f0115400 con 0x7fa5f0101540 2024-05-20T18:06:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f5bf3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5f010f8b0 0x7fa5f019ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f53f2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5f0100bf0 0x7fa5f01a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f5bf3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5f010f8b0 0x7fa5f019ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34970/0 (socket says 172.21.15.129:34970) 2024-05-20T18:06:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f5bf3640 1 -- 172.21.15.129:0/1255467775 learned_addr learned my addr 172.21.15.129:0/1255467775 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5f0101540 0x7fa5f01a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f5bf3640 1 -- 172.21.15.129:0/1255467775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5f0101540 msgr2=0x7fa5f01a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f5bf3640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5f0101540 0x7fa5f01a1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f5bf3640 1 -- 172.21.15.129:0/1255467775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5f0100bf0 msgr2=0x7fa5f01a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f5bf3640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5f0100bf0 0x7fa5f01a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.313+0000 7fa5f5bf3640 1 -- 172.21.15.129:0/1255467775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f019c100 con 0x7fa5f010f8b0 2024-05-20T18:06:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.314+0000 7fa5f5bf3640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5f010f8b0 0x7fa5f019ba60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800eaa0 tx=0x7fa5e800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.315+0000 7fa5e67fc640 1 -- 172.21.15.129:0/1255467775 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e800cdb0 con 0x7fa5f010f8b0 2024-05-20T18:06:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.315+0000 7fa5e67fc640 1 -- 172.21.15.129:0/1255467775 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5e8005020 con 0x7fa5f010f8b0 2024-05-20T18:06:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.315+0000 7fa5e67fc640 1 -- 172.21.15.129:0/1255467775 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e80116b0 con 0x7fa5f010f8b0 2024-05-20T18:06:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.315+0000 7fa5f7656640 1 -- 172.21.15.129:0/1255467775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f019c3f0 con 0x7fa5f010f8b0 2024-05-20T18:06:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.315+0000 7fa5f7656640 1 -- 172.21.15.129:0/1255467775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f01a8680 con 0x7fa5f010f8b0 2024-05-20T18:06:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.320+0000 7fa5fe5d5740 1 Processor -- start 2024-05-20T18:06:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.320+0000 7fa5e67fc640 1 -- 172.21.15.129:0/1255467775 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa5e8005560 con 0x7fa5f010f8b0 2024-05-20T18:06:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.320+0000 7fa5fe5d5740 1 -- start start 2024-05-20T18:06:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.320+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5621482451b0 0x5621485e4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.320+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56214821c2f0 0x5621485e45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.320+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x5621485e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.320+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562148549340 con 0x5621482244d0 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.320+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5621484f1290 con 0x56214821c2f0 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x562148545300 con 0x5621482451b0 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f53f2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56214821c2f0 0x5621485e45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f53f2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56214821c2f0 0x5621485e45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38140/0 (socket says 172.21.15.129:38140) 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f53f2640 1 -- 172.21.15.129:0/3641036076 learned_addr learned my addr 172.21.15.129:0/3641036076 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/3641036076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x5621485e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/3641036076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5621482451b0 0x5621485e4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f4bf1640 1 -- 172.21.15.129:0/3641036076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5621482451b0 msgr2=0x5621485e4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/3641036076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5621482451b0 0x5621485e4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f4bf1640 1 -- 172.21.15.129:0/3641036076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56214821c2f0 msgr2=0x5621485e45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/3641036076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56214821c2f0 0x5621485e45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.320 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f4bf1640 1 -- 172.21.15.129:0/3641036076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621485158e0 con 0x5621482244d0 2024-05-20T18:06:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5e67fc640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5dc0a2e40 0x7fa5dc0a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f53f2640 1 --2- 172.21.15.129:0/3641036076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56214821c2f0 0x5621485e45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:06:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/3641036076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x5621485e4b00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa5e002a910 tx=0x7fa5e0031420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae7081e7ff15828 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5e67fc640 1 -- 172.21.15.129:0/1255467775 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fa5e80cf560 con 0x7fa5f010f8b0 2024-05-20T18:06:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.321+0000 7fa5f53f2640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5dc0a2e40 0x7fa5dc0a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.322+0000 7fa5db7fe640 1 -- 172.21.15.129:0/3641036076 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0035040 con 0x5621482244d0 2024-05-20T18:06:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.322+0000 7fa5db7fe640 1 -- 172.21.15.129:0/3641036076 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5e003f070 con 0x5621482244d0 2024-05-20T18:06:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.322+0000 7fa5db7fe640 1 -- 172.21.15.129:0/3641036076 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e00038e0 con 0x5621482244d0 2024-05-20T18:06:10.323 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.323+0000 7fa5f53f2640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5dc0a2e40 0x7fa5dc0a5350 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x5621485e5430 tx=0x7fa5ec00a2d0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:10.323 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.323+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/3641036076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 msgr2=0x5621485e4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:10.323 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.323+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/3641036076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x5621485e4b00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa5e002a910 tx=0x7fa5e0031420 comp rx=0 tx=0).stop 2024-05-20T18:06:10.323 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.323+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/3641036076 shutdown_connections 2024-05-20T18:06:10.323 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.323+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/3641036076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56214821c2f0 0x5621485e45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.323 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.323+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/3641036076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x5621485e4b00 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.323+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/3641036076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5621482451b0 0x5621485e4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.323+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/3641036076 >> 172.21.15.129:0/3641036076 conn(0x562148239300 msgr2=0x5621485e0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.323+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/3641036076 shutdown_connections 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.323+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/3641036076 wait complete. 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.324+0000 7fa5fe5d5740 1 -- start start 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.324+0000 7fa5fe5d5740 1 Processor -- start 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.324+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x5621485f0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.324+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x5621485f0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.324+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621482451b0 0x5621485ee5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5621485f1150 con 0x5621482244d0 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562148504790 con 0x5621482451b0 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5621485f12d0 con 0x56214821c2f0 2024-05-20T18:06:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5fe5d5740 10 client.0 _ll_register_callbacks cb 0x56214853fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5f4bf1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x5621485f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5f53f2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x5621485f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5f4bf1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x5621485f0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34982/0 (socket says 172.21.15.129:34982) 2024-05-20T18:06:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5f4bf1640 1 -- 172.21.15.129:0/659917868 learned_addr learned my addr 172.21.15.129:0/659917868 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5f5bf3640 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621482451b0 0x5621485ee5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5f53f2640 1 -- 172.21.15.129:0/659917868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621482451b0 msgr2=0x5621485ee5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5f53f2640 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621482451b0 0x5621485ee5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5f53f2640 1 -- 172.21.15.129:0/659917868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 msgr2=0x5621485f0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5f53f2640 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x5621485f0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5f53f2640 1 -- 172.21.15.129:0/659917868 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56214825d210 con 0x56214821c2f0 2024-05-20T18:06:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.325+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x5621485f0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:06:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.326+0000 7fa5f53f2640 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x5621485f0120 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5621485e1b70 tx=0x7fa5ec004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2022bde5116f1b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.326+0000 7fa5d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:06:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.326+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/659917868 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec01cca0 con 0x56214821c2f0 2024-05-20T18:06:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.326+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/659917868 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fa5ec016030 con 0x56214821c2f0 2024-05-20T18:06:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.326+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/659917868 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5e003f070 con 0x56214821c2f0 2024-05-20T18:06:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.326+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/659917868 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa5ec01ce40 con 0x56214821c2f0 2024-05-20T18:06:10.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.326+0000 7fa5fe5d5740 10 client.26278 fetch_fsmap learned FSMap version 25 2024-05-20T18:06:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.326+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/659917868 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562148329620 con 0x56214821c2f0 2024-05-20T18:06:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.327+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/659917868 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5e003f070 con 0x56214821c2f0 2024-05-20T18:06:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.327+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/659917868 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 25) ==== 7717+0+0 (secure 0 0 0) 0x7fa5ec0047a0 con 0x56214821c2f0 2024-05-20T18:06:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.327+0000 7fa5fe5d5740 10 client.26278 fetch_fsmap finished waiting for FSMap version 25 2024-05-20T18:06:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.327+0000 7fa5fe5d5740 10 client.26278 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.327+0000 7fa5fe5d5740 20 client.26278 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.327+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562148203d80 0x562148600a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.327+0000 7fa5fe5d5740 4 client.26278 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.327+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/659917868 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562148556a70 con 0x562148203d80 2024-05-20T18:06:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.327+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562148203d80 0x562148600a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.328+0000 7fa5d8ff9640 10 client.26278 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.328+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562148203d80 0x562148600a10 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4146e584262991d server_cookie=4ce5ddc1daedb58e in_seq=0 out_seq=0 2024-05-20T18:06:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.329+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/659917868 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562148556a70 con 0x562148203d80 2024-05-20T18:06:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.329+0000 7fa5d8ff9640 10 client.26278 handle_command_reply: tid=0 2024-05-20T18:06:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.329+0000 7fa5fe5d5740 1 client.26278 shutdown 2024-05-20T18:06:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.329+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/659917868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 msgr2=0x5621485f0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.329+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x5621485f0120 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5621485e1b70 tx=0x7fa5ec004730 comp rx=0 tx=0).stop 2024-05-20T18:06:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.330+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/659917868 shutdown_connections 2024-05-20T18:06:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.330+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562148203d80 0x562148600a10 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.330+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621482451b0 0x5621485ee5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.330+0000 7fa5d8ff9640 0 client.26278 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.330+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x5621485f0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.330+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/659917868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x5621485f0120 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.330+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/659917868 >> 172.21.15.129:0/659917868 conn(0x562148239300 msgr2=0x5621485e5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.330+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/659917868 shutdown_connections 2024-05-20T18:06:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.330+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/659917868 wait complete. 2024-05-20T18:06:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.330+0000 7fa5fe5d5740 20 client.26278 trim_cache size 0 max 16384 2024-05-20T18:06:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.345+0000 7fa5fe5d5740 1 Processor -- start 2024-05-20T18:06:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.345+0000 7fa5fe5d5740 1 -- start start 2024-05-20T18:06:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562148203d80 0x5621485db730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x562148606210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x562148606750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f01ad680 con 0x5621482244d0 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f01aa750 con 0x562148203d80 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f01ac8e0 con 0x56214821c2f0 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5f4bf1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x562148606210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5f53f2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562148203d80 0x5621485db730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5f5bf3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x562148606750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5f4bf1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x562148606210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:35132/0 (socket says 172.21.15.129:35132) 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5f4bf1640 1 -- 172.21.15.129:0/1538341729 learned_addr learned my addr 172.21.15.129:0/1538341729 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5f53f2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562148203d80 0x5621485db730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:38160/0 (socket says 172.21.15.129:38160) 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5f4bf1640 1 -- 172.21.15.129:0/1538341729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562148203d80 msgr2=0x5621485db730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/1538341729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562148203d80 0x5621485db730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5f4bf1640 1 -- 172.21.15.129:0/1538341729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 msgr2=0x562148606750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/1538341729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x562148606750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.346+0000 7fa5f4bf1640 1 -- 172.21.15.129:0/1538341729 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621485e1400 con 0x56214821c2f0 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.347+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/1538341729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x562148606210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa5e002a910 tx=0x7fa5e0005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6c381821fcc625c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.347+0000 7fa5daffd640 1 -- 172.21.15.129:0/1538341729 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0035040 con 0x56214821c2f0 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.347+0000 7fa5daffd640 1 -- 172.21.15.129:0/1538341729 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5e003f040 con 0x56214821c2f0 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.347+0000 7fa5daffd640 1 -- 172.21.15.129:0/1538341729 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0031c60 con 0x56214821c2f0 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.348+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1538341729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 msgr2=0x562148606210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.348+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/1538341729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x562148606210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa5e002a910 tx=0x7fa5e0005f00 comp rx=0 tx=0).stop 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.348+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1538341729 shutdown_connections 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.348+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/1538341729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5621482244d0 0x562148606750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.348+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/1538341729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x562148606210 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.348+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/1538341729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562148203d80 0x5621485db730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.348+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1538341729 >> 172.21.15.129:0/1538341729 conn(0x562148239300 msgr2=0x5621485ece90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.348+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1538341729 shutdown_connections 2024-05-20T18:06:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.348+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1538341729 wait complete. 2024-05-20T18:06:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.348+0000 7fa5fe5d5740 1 -- start start 2024-05-20T18:06:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5fe5d5740 1 Processor -- start 2024-05-20T18:06:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562148203d80 0x5621485f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x5621485f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5fe5d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621482244d0 0x5621485f1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f01ac210 con 0x562148203d80 2024-05-20T18:06:10.351 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f01aada0 con 0x5621482244d0 2024-05-20T18:06:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5fe5d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f01ab470 con 0x56214821c2f0 2024-05-20T18:06:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5f53f2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562148203d80 0x5621485f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5f53f2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562148203d80 0x5621485f14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35006/0 (socket says 172.21.15.129:35006) 2024-05-20T18:06:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5fe5d5740 10 client.0 _ll_register_callbacks cb 0x7fa5ec00a6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5f53f2640 1 -- 172.21.15.129:0/1777080326 learned_addr learned my addr 172.21.15.129:0/1777080326 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5f5bf3640 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621482244d0 0x5621485f1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x5621485f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5f53f2640 1 -- 172.21.15.129:0/1777080326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 msgr2=0x5621485f1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5f53f2640 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x5621485f1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5f53f2640 1 -- 172.21.15.129:0/1777080326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621482244d0 msgr2=0x5621485f1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.349+0000 7fa5f53f2640 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621482244d0 0x5621485f1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.350+0000 7fa5f53f2640 1 -- 172.21.15.129:0/1777080326 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562148666ee0 con 0x562148203d80 2024-05-20T18:06:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.350+0000 7fa5f53f2640 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562148203d80 0x5621485f14f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec004d30 tx=0x7fa5ec01cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc548f8c3614477 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.350+0000 7fa5d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.350+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/1777080326 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec004020 con 0x562148203d80 2024-05-20T18:06:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.350+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/1777080326 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8543+0+0 (secure 0 0 0) 0x7fa5ec0041c0 con 0x562148203d80 2024-05-20T18:06:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.350+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1777080326 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5e003f040 con 0x562148203d80 2024-05-20T18:06:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.350+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/1777080326 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa5ec0043c0 con 0x562148203d80 2024-05-20T18:06:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.350+0000 7fa5fe5d5740 10 client.16524 fetch_fsmap learned FSMap version 25 2024-05-20T18:06:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.350+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1777080326 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562148668150 con 0x562148203d80 2024-05-20T18:06:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.351+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/1777080326 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5e003f040 con 0x562148203d80 2024-05-20T18:06:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.351+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/1777080326 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 25) ==== 7717+0+0 (secure 0 0 0) 0x7fa5ec003960 con 0x562148203d80 2024-05-20T18:06:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.351+0000 7fa5fe5d5740 10 client.16524 fetch_fsmap finished waiting for FSMap version 25 2024-05-20T18:06:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.351+0000 7fa5fe5d5740 10 client.16524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.351+0000 7fa5fe5d5740 20 client.16524 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.351+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5621482451b0 0x56214866a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.351+0000 7fa5fe5d5740 4 client.16524 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:06:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.351+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1777080326 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562148550dc0 con 0x5621482451b0 2024-05-20T18:06:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.351+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5621482451b0 0x56214866a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.352+0000 7fa5d8ff9640 10 client.16524 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:10.354 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.352+0000 7fa5f4bf1640 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5621482451b0 0x56214866a8b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c13a6c06402da26d server_cookie=a10f69ecf1ab5642 in_seq=0 out_seq=0 2024-05-20T18:06:10.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.368+0000 7fa5d8ff9640 1 -- 172.21.15.129:0/1777080326 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562148550dc0 con 0x5621482451b0 2024-05-20T18:06:10.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.368+0000 7fa5d8ff9640 10 client.16524 handle_command_reply: tid=0 2024-05-20T18:06:10.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.368+0000 7fa5fe5d5740 1 client.16524 shutdown 2024-05-20T18:06:10.368 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.369+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1777080326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562148203d80 msgr2=0x5621485f14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:10.368 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.369+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562148203d80 0x5621485f14f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec004d30 tx=0x7fa5ec01cd20 comp rx=0 tx=0).stop 2024-05-20T18:06:10.368 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.369+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1777080326 shutdown_connections 2024-05-20T18:06:10.368 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.369+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5621482451b0 0x56214866a8b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.368 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.369+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621482244d0 0x5621485f1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.368 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.369+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56214821c2f0 0x5621485f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.368 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.369+0000 7fa5fe5d5740 1 --2- 172.21.15.129:0/1777080326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562148203d80 0x5621485f14f0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.368 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.369+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1777080326 >> 172.21.15.129:0/1777080326 conn(0x562148239300 msgr2=0x5621485e5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:10.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5d8ff9640 0 client.16524 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:10.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1777080326 shutdown_connections 2024-05-20T18:06:10.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5fe5d5740 1 -- 172.21.15.129:0/1777080326 wait complete. 2024-05-20T18:06:10.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5fe5d5740 20 client.16524 trim_cache size 0 max 16384 2024-05-20T18:06:10.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:06:10.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:06:10.369 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:06:10.370 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:06:10.370 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5f7656640 1 -- 172.21.15.129:0/1255467775 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5dc0a2e40 msgr2=0x7fa5dc0a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:10.370 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5f7656640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5dc0a2e40 0x7fa5dc0a5350 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x5621485e5430 tx=0x7fa5ec00a2d0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.370 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5f7656640 1 -- 172.21.15.129:0/1255467775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5f010f8b0 msgr2=0x7fa5f019ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:10.371 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5f7656640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5f010f8b0 0x7fa5f019ba60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800eaa0 tx=0x7fa5e800ef60 comp rx=0 tx=0).stop 2024-05-20T18:06:10.371 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5f7656640 1 -- 172.21.15.129:0/1255467775 shutdown_connections 2024-05-20T18:06:10.371 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5f7656640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5dc0a2e40 0x7fa5dc0a5350 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.371 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5f7656640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5f010f8b0 0x7fa5f019ba60 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.371 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5f7656640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5f0101540 0x7fa5f01a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.371 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5f7656640 1 --2- 172.21.15.129:0/1255467775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5f0100bf0 0x7fa5f01a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:10.371 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.370+0000 7fa5f7656640 1 -- 172.21.15.129:0/1255467775 >> 172.21.15.129:0/1255467775 conn(0x7fa5f0076860 msgr2=0x7fa5f0110c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:10.371 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.371+0000 7fa5f7656640 1 -- 172.21.15.129:0/1255467775 shutdown_connections 2024-05-20T18:06:10.371 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:10.371+0000 7fa5f7656640 1 -- 172.21.15.129:0/1255467775 wait complete. 2024-05-20T18:06:10.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:10 smithi129 ceph-mon[32029]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-05-20T18:06:10.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:10 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 9 up:standby 2024-05-20T18:06:10.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:10 smithi204 ceph-mon[37076]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-05-20T18:06:10.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:10 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 9 up:standby 2024-05-20T18:06:10.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:10 smithi154 ceph-mon[36482]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-05-20T18:06:10.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:10 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 9 up:standby 2024-05-20T18:06:11.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:11 smithi129 ceph-mon[32029]: pgmap v233: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 922 MiB/s wr, 2.88k op/s 2024-05-20T18:06:11.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:11 smithi129 ceph-mon[32029]: osdmap e97: 12 total, 12 up, 12 in 2024-05-20T18:06:11.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:11 smithi204 ceph-mon[37076]: pgmap v233: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 922 MiB/s wr, 2.88k op/s 2024-05-20T18:06:11.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:11 smithi204 ceph-mon[37076]: osdmap e97: 12 total, 12 up, 12 in 2024-05-20T18:06:11.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:11 smithi154 ceph-mon[36482]: pgmap v233: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 922 MiB/s wr, 2.88k op/s 2024-05-20T18:06:11.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:11 smithi154 ceph-mon[36482]: osdmap e97: 12 total, 12 up, 12 in 2024-05-20T18:06:13.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:13 smithi129 ceph-mon[32029]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 766 MiB/s wr, 2.90k op/s 2024-05-20T18:06:13.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:13 smithi204 ceph-mon[37076]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 766 MiB/s wr, 2.90k op/s 2024-05-20T18:06:13.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:13 smithi154 ceph-mon[36482]: pgmap v235: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 766 MiB/s wr, 2.90k op/s 2024-05-20T18:06:15.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:15 smithi129 ceph-mon[32029]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 710 MiB/s wr, 2.91k op/s 2024-05-20T18:06:15.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:15 smithi204 ceph-mon[37076]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 710 MiB/s wr, 2.91k op/s 2024-05-20T18:06:15.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:15 smithi154 ceph-mon[36482]: pgmap v236: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 710 MiB/s wr, 2.91k op/s 2024-05-20T18:06:16.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:16 smithi154 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-05-20T18:06:16.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:16 smithi154 systemd[1]: Stopped Ceph mds.h for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T18:06:16.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:16 smithi154 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.h.service: Consumed 51.569s CPU time. 2024-05-20T18:06:16.644 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:16 smithi154 systemd[1]: Starting Ceph mds.h for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:06:16.929 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:16 smithi154 podman[71171]: 2024-05-20 18:06:16.549668167 +0000 UTC m=+0.017834317 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:06:16.929 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:16 smithi154 podman[71171]: 2024-05-20 18:06:16.66220283 +0000 UTC m=+0.130368983 container create 69763b3db3d969b5e3ed082bb228c583f7cba641d860e1d5399349caa0cb14c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-20T18:06:17.365 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:16 smithi154 podman[71171]: 2024-05-20 18:06:16.932000349 +0000 UTC m=+0.400166481 container init 69763b3db3d969b5e3ed082bb228c583f7cba641d860e1d5399349caa0cb14c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-20T18:06:17.365 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:16 smithi154 podman[71171]: 2024-05-20 18:06:16.935156003 +0000 UTC m=+0.403322135 container start 69763b3db3d969b5e3ed082bb228c583f7cba641d860e1d5399349caa0cb14c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c) 2024-05-20T18:06:17.365 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:17 smithi154 ceph-mds[71191]: -- 172.21.15.154:0/849708041 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b66da9a40 con 0x559b66e21a80 2024-05-20T18:06:17.366 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:17 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[71186]: starting mds.h at 2024-05-20T18:06:17.366 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:17 smithi154 bash[71171]: 69763b3db3d969b5e3ed082bb228c583f7cba641d860e1d5399349caa0cb14c7 2024-05-20T18:06:17.366 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:06:17 smithi154 systemd[1]: Started Ceph mds.h for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T18:06:17.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:17 smithi154 ceph-mon[36482]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 435 MiB/s wr, 2.59k op/s 2024-05-20T18:06:17.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:17 smithi204 ceph-mon[37076]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 435 MiB/s wr, 2.59k op/s 2024-05-20T18:06:17.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:17 smithi129 ceph-mon[32029]: pgmap v237: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 435 MiB/s wr, 2.59k op/s 2024-05-20T18:06:18.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:18 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.154:6836/2724397229,v1:172.21.15.154:6837/2724397229] up:boot 2024-05-20T18:06:18.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:18 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T18:06:18.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:18 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T18:06:18.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:18 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.154:6836/2724397229,v1:172.21.15.154:6837/2724397229] up:boot 2024-05-20T18:06:18.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:18 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T18:06:18.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:18 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T18:06:18.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:18 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.154:6836/2724397229,v1:172.21.15.154:6837/2724397229] up:boot 2024-05-20T18:06:18.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:18 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T18:06:18.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:18 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-20T18:06:19.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:19 smithi204 ceph-mon[37076]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 338 MiB/s wr, 2.33k op/s 2024-05-20T18:06:19.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:19 smithi154 ceph-mon[36482]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 338 MiB/s wr, 2.33k op/s 2024-05-20T18:06:19.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:19 smithi129 ceph-mon[32029]: pgmap v238: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 338 MiB/s wr, 2.33k op/s 2024-05-20T18:06:20.588 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:06:20.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:06:20.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.913+0000 7f961d5b2640 1 -- 172.21.15.129:0/385947990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f961009b910 msgr2=0x7f96100954a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:20.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.913+0000 7f961d5b2640 1 --2- 172.21.15.129:0/385947990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f961009b910 0x7f96100954a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f960400b9f0 tx=0x7f960402c0d0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.918+0000 7f961d5b2640 1 -- 172.21.15.129:0/385947990 shutdown_connections 2024-05-20T18:06:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.918+0000 7f961d5b2640 1 --2- 172.21.15.129:0/385947990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96100959e0 0x7f9610095df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.918+0000 7f961d5b2640 1 --2- 172.21.15.129:0/385947990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96100a0270 0x7f96100a2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.918+0000 7f961d5b2640 1 --2- 172.21.15.129:0/385947990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f961009b910 0x7f96100954a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.918+0000 7f961d5b2640 1 -- 172.21.15.129:0/385947990 >> 172.21.15.129:0/385947990 conn(0x7f961000afd0 msgr2=0x7f961000b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:20.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.921+0000 7f961d5b2640 1 -- 172.21.15.129:0/385947990 shutdown_connections 2024-05-20T18:06:20.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.921+0000 7f961d5b2640 1 -- 172.21.15.129:0/385947990 wait complete. 2024-05-20T18:06:20.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.922+0000 7f961d5b2640 1 Processor -- start 2024-05-20T18:06:20.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.925+0000 7f961d5b2640 1 -- start start 2024-05-20T18:06:20.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.925+0000 7f961d5b2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96100959e0 0x7f96101301a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:20.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.925+0000 7f961d5b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f961009b910 0x7f9610130700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:20.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.925+0000 7f961d5b2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96100a0270 0x7f9610135eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:20.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.925+0000 7f961d5b2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f96100a4be0 con 0x7f96100a0270 2024-05-20T18:06:20.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.926+0000 7f961d5b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96100a4a60 con 0x7f961009b910 2024-05-20T18:06:20.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.926+0000 7f961d5b2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f96100a4d60 con 0x7f96100959e0 2024-05-20T18:06:20.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.926+0000 7f9617fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96100959e0 0x7f96101301a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:20.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.926+0000 7f9617fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96100959e0 0x7f96101301a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:44528/0 (socket says 172.21.15.129:44528) 2024-05-20T18:06:20.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.926+0000 7f9617fff640 1 -- 172.21.15.129:0/1899046164 learned_addr learned my addr 172.21.15.129:0/1899046164 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:20.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.926+0000 7f96177fe640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f961009b910 0x7f9610130700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:20.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.927+0000 7f9617fff640 1 -- 172.21.15.129:0/1899046164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f961009b910 msgr2=0x7f9610130700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:20.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.927+0000 7f9617fff640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f961009b910 0x7f9610130700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.927+0000 7f9617fff640 1 -- 172.21.15.129:0/1899046164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96100a0270 msgr2=0x7f9610135eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:20.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.927+0000 7f9617fff640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96100a0270 0x7f9610135eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.927+0000 7f9617fff640 1 -- 172.21.15.129:0/1899046164 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96101366b0 con 0x7f96100959e0 2024-05-20T18:06:20.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.928+0000 7f9617fff640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96100959e0 0x7f96101301a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f960400b670 tx=0x7f96040045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:20.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.930+0000 7f96157fa640 1 -- 172.21.15.129:0/1899046164 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9604039070 con 0x7f96100959e0 2024-05-20T18:06:20.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.930+0000 7f96157fa640 1 -- 172.21.15.129:0/1899046164 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9604005510 con 0x7f96100959e0 2024-05-20T18:06:20.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.930+0000 7f96157fa640 1 -- 172.21.15.129:0/1899046164 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96040035f0 con 0x7f96100959e0 2024-05-20T18:06:20.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.930+0000 7f961d5b2640 1 -- 172.21.15.129:0/1899046164 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9610136940 con 0x7f96100959e0 2024-05-20T18:06:20.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.931+0000 7f961d5b2640 1 -- 172.21.15.129:0/1899046164 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f961012b580 con 0x7f96100959e0 2024-05-20T18:06:20.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.938+0000 7f9625799740 1 Processor -- start 2024-05-20T18:06:20.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.940+0000 7f96157fa640 1 -- 172.21.15.129:0/1899046164 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9604044060 con 0x7f96100959e0 2024-05-20T18:06:20.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.940+0000 7f9625799740 1 -- start start 2024-05-20T18:06:20.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.941+0000 7f9625799740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c505902390 0x55c505ca0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:20.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.941+0000 7f9625799740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505ca1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:20.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.942+0000 7f9625799740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c5058e16b0 0x55c505ca19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:20.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.942+0000 7f9625799740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c505c06d00 con 0x55c5058d94d0 2024-05-20T18:06:20.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.942+0000 7f9625799740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c505bae4c0 con 0x55c505902390 2024-05-20T18:06:20.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.942+0000 7f9625799740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c505c02cc0 con 0x55c5058e16b0 2024-05-20T18:06:20.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.943+0000 7f96157fa640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96080a2f00 0x7f96080a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:20.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.943+0000 7f96157fa640 1 -- 172.21.15.129:0/1899046164 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f96040ee4e0 con 0x7f96100959e0 2024-05-20T18:06:20.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.946+0000 7f9617fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505ca1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:20.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.946+0000 7f961cdb1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c505902390 0x55c505ca0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:20.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.946+0000 7f9617fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505ca1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53236/0 (socket says 172.21.15.129:53236) 2024-05-20T18:06:20.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.946+0000 7f9617fff640 1 -- 172.21.15.129:0/846200407 learned_addr learned my addr 172.21.15.129:0/846200407 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:20.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.946+0000 7f961cdb1640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96080a2f00 0x7f96080a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:20.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.947+0000 7f9617fff640 1 -- 172.21.15.129:0/846200407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c5058e16b0 msgr2=0x55c505ca19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:20.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.947+0000 7f9617fff640 1 --2- 172.21.15.129:0/846200407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c5058e16b0 0x55c505ca19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.947+0000 7f9617fff640 1 -- 172.21.15.129:0/846200407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c505902390 msgr2=0x55c505ca0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:20.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.947+0000 7f9617fff640 1 --2- 172.21.15.129:0/846200407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c505902390 0x55c505ca0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.947+0000 7f9617fff640 1 -- 172.21.15.129:0/846200407 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5059e47d0 con 0x55c5058d94d0 2024-05-20T18:06:20.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.949+0000 7f9617fff640 1 --2- 172.21.15.129:0/846200407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505ca1470 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f96040c31b0 tx=0x7f9604005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a296befa2350dce server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:20.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.950+0000 7f96027fc640 1 -- 172.21.15.129:0/846200407 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96040bee20 con 0x55c5058d94d0 2024-05-20T18:06:20.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.950+0000 7f96027fc640 1 -- 172.21.15.129:0/846200407 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96040c36e0 con 0x55c5058d94d0 2024-05-20T18:06:20.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.950+0000 7f96027fc640 1 -- 172.21.15.129:0/846200407 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96040046c0 con 0x55c5058d94d0 2024-05-20T18:06:20.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.952+0000 7f9625799740 1 -- 172.21.15.129:0/846200407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 msgr2=0x55c505ca1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:20.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.952+0000 7f9625799740 1 --2- 172.21.15.129:0/846200407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505ca1470 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f96040c31b0 tx=0x7f9604005f90 comp rx=0 tx=0).stop 2024-05-20T18:06:20.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.952+0000 7f9625799740 1 -- 172.21.15.129:0/846200407 shutdown_connections 2024-05-20T18:06:20.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.952+0000 7f9625799740 1 --2- 172.21.15.129:0/846200407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c5058e16b0 0x55c505ca19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.952+0000 7f9625799740 1 --2- 172.21.15.129:0/846200407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505ca1470 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.952+0000 7f9625799740 1 --2- 172.21.15.129:0/846200407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c505902390 0x55c505ca0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.952+0000 7f9625799740 1 -- 172.21.15.129:0/846200407 >> 172.21.15.129:0/846200407 conn(0x55c5058f64e0 msgr2=0x55c505c9d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:20.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.955+0000 7f961cdb1640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96080a2f00 0x7f96080a5410 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f96180678c0 tx=0x7f9618067a50 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:20.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.955+0000 7f9625799740 1 -- 172.21.15.129:0/846200407 shutdown_connections 2024-05-20T18:06:20.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.959+0000 7f9625799740 1 -- 172.21.15.129:0/846200407 wait complete. 2024-05-20T18:06:20.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.960+0000 7f9625799740 1 -- start start 2024-05-20T18:06:20.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.960+0000 7f9625799740 1 Processor -- start 2024-05-20T18:06:20.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.961+0000 7f9625799740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505cacf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:20.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.961+0000 7f9625799740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058e16b0 0x55c505cad4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:20.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.961+0000 7f9625799740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cab490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:20.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.962+0000 7f9625799740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c505cae0a0 con 0x55c5058d94d0 2024-05-20T18:06:20.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.962+0000 7f9625799740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c505bc1b90 con 0x55c5058e16b0 2024-05-20T18:06:20.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.962+0000 7f9625799740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c505cae220 con 0x55c505902390 2024-05-20T18:06:20.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.962+0000 7f9625799740 10 client.0 _ll_register_callbacks cb 0x55c505bfdda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:20.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.962+0000 7f9617fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505cacf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:20.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.962+0000 7f9617fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505cacf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53252/0 (socket says 172.21.15.129:53252) 2024-05-20T18:06:20.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.962+0000 7f9617fff640 1 -- 172.21.15.129:0/847776391 learned_addr learned my addr 172.21.15.129:0/847776391 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:20.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.963+0000 7f961cdb1640 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:20.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.963+0000 7f96177fe640 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058e16b0 0x55c505cad4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:20.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.967+0000 7f9617fff640 1 -- 172.21.15.129:0/847776391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 msgr2=0x55c505cab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:20.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.967+0000 7f9617fff640 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cab490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.967+0000 7f9617fff640 1 -- 172.21.15.129:0/847776391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058e16b0 msgr2=0x55c505cad4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:20.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.967+0000 7f9617fff640 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058e16b0 0x55c505cad4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:20.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.967+0000 7f9617fff640 1 -- 172.21.15.129:0/847776391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c50591a3f0 con 0x55c5058d94d0 2024-05-20T18:06:20.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.968+0000 7f9617fff640 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505cacf80 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f96040ba820 tx=0x7f96040baca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7a8b89c4d4b747 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:20.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.969+0000 7f96177fe640 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058e16b0 0x55c505cad4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:20.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.970+0000 7f95f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:20.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.971+0000 7f95f77fe640 1 -- 172.21.15.129:0/847776391 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96040bee20 con 0x55c5058d94d0 2024-05-20T18:06:20.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.971+0000 7f95f77fe640 1 -- 172.21.15.129:0/847776391 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f96040d9350 con 0x55c5058d94d0 2024-05-20T18:06:20.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.971+0000 7f9625799740 1 -- 172.21.15.129:0/847776391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96040c36e0 con 0x55c5058d94d0 2024-05-20T18:06:20.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.978+0000 7f95f77fe640 1 -- 172.21.15.129:0/847776391 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f96040b7190 con 0x55c5058d94d0 2024-05-20T18:06:20.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.979+0000 7f9625799740 10 client.16539 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:20.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.979+0000 7f9625799740 1 -- 172.21.15.129:0/847776391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c5058da0f0 con 0x55c5058d94d0 2024-05-20T18:06:20.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.984+0000 7f95f77fe640 1 -- 172.21.15.129:0/847776391 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96040c36e0 con 0x55c5058d94d0 2024-05-20T18:06:20.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.984+0000 7f95f77fe640 1 -- 172.21.15.129:0/847776391 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f96040c3250 con 0x55c5058d94d0 2024-05-20T18:06:20.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.986+0000 7f9625799740 10 client.16539 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:20.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.986+0000 7f9625799740 10 client.16539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:20.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.986+0000 7f9625799740 20 client.16539 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:20.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.986+0000 7f9625799740 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c5058c0f60 0x55c505cbdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:20.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.987+0000 7f9625799740 4 client.16539 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:20.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.987+0000 7f9625799740 1 -- 172.21.15.129:0/847776391 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c505c14430 con 0x55c5058c0f60 2024-05-20T18:06:20.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.988+0000 7f96177fe640 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c5058c0f60 0x55c505cbdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:20.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.996+0000 7f96177fe640 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c5058c0f60 0x55c505cbdab0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6e8999565d6ae86 server_cookie=5dc8fd88f5da5bf in_seq=0 out_seq=0 2024-05-20T18:06:20.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.997+0000 7f95f77fe640 10 client.16539 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:20.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.998+0000 7f95f77fe640 1 -- 172.21.15.129:0/847776391 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c505c14430 con 0x55c5058c0f60 2024-05-20T18:06:20.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:20.998+0000 7f95f77fe640 10 client.16539 handle_command_reply: tid=0 2024-05-20T18:06:20.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.001+0000 7f9625799740 1 client.16539 shutdown 2024-05-20T18:06:21.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.002+0000 7f9625799740 1 -- 172.21.15.129:0/847776391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 msgr2=0x55c505cacf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.002+0000 7f9625799740 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505cacf80 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f96040ba820 tx=0x7f96040baca0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.003+0000 7f9625799740 1 -- 172.21.15.129:0/847776391 shutdown_connections 2024-05-20T18:06:21.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.003+0000 7f9625799740 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c5058c0f60 0x55c505cbdab0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.003+0000 7f9625799740 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058e16b0 0x55c505cad4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.003+0000 7f9625799740 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.003+0000 7f9625799740 1 --2- 172.21.15.129:0/847776391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058d94d0 0x55c505cacf80 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.003+0000 7f9625799740 1 -- 172.21.15.129:0/847776391 >> 172.21.15.129:0/847776391 conn(0x55c5058f64e0 msgr2=0x55c505ca2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.003+0000 7f95f77fe640 0 client.16539 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:21.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.003+0000 7f9625799740 1 -- 172.21.15.129:0/847776391 shutdown_connections 2024-05-20T18:06:21.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.004+0000 7f9625799740 1 -- 172.21.15.129:0/847776391 wait complete. 2024-05-20T18:06:21.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.006+0000 7f9625799740 20 client.16539 trim_cache size 0 max 16384 2024-05-20T18:06:21.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.049+0000 7f9625799740 1 Processor -- start 2024-05-20T18:06:21.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.050+0000 7f9625799740 1 -- start start 2024-05-20T18:06:21.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.050+0000 7f9625799740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058d94d0 0x55c505ca3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.051+0000 7f9625799740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058e16b0 0x55c505cc0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.051+0000 7f9625799740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cc1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.051+0000 7f9625799740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc00c970 con 0x55c5058e16b0 2024-05-20T18:06:21.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.051+0000 7f9625799740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc00f1d0 con 0x55c5058d94d0 2024-05-20T18:06:21.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.051+0000 7f9625799740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc00cfc0 con 0x55c505902390 2024-05-20T18:06:21.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.051+0000 7f96177fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058e16b0 0x55c505cc0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.052+0000 7f961cdb1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cc1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.052+0000 7f96177fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058e16b0 0x55c505cc0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53254/0 (socket says 172.21.15.129:53254) 2024-05-20T18:06:21.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.052+0000 7f961cdb1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cc1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:44562/0 (socket says 172.21.15.129:44562) 2024-05-20T18:06:21.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.052+0000 7f96177fe640 1 -- 172.21.15.129:0/2829497998 learned_addr learned my addr 172.21.15.129:0/2829497998 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:21.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.052+0000 7f9617fff640 1 --2- 172.21.15.129:0/2829497998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058d94d0 0x55c505ca3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.052+0000 7f96177fe640 1 -- 172.21.15.129:0/2829497998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 msgr2=0x55c505cc1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:21.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.052+0000 7f96177fe640 1 --2- 172.21.15.129:0/2829497998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cc1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.052+0000 7f96177fe640 1 -- 172.21.15.129:0/2829497998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058d94d0 msgr2=0x55c505ca3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:21.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.052+0000 7f96177fe640 1 --2- 172.21.15.129:0/2829497998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058d94d0 0x55c505ca3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.052+0000 7f96177fe640 1 -- 172.21.15.129:0/2829497998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c505c9e830 con 0x55c5058e16b0 2024-05-20T18:06:21.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.053+0000 7f96177fe640 1 --2- 172.21.15.129:0/2829497998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058e16b0 0x55c505cc0ae0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9610008c60 tx=0x7f960c009460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d229381cd7a47772 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:21.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.054+0000 7f9601ffb640 1 -- 172.21.15.129:0/2829497998 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960c0140d0 con 0x55c5058e16b0 2024-05-20T18:06:21.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.054+0000 7f9601ffb640 1 -- 172.21.15.129:0/2829497998 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f960c01f020 con 0x55c5058e16b0 2024-05-20T18:06:21.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.055+0000 7f9625799740 1 -- 172.21.15.129:0/2829497998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058e16b0 msgr2=0x55c505cc0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.055+0000 7f9625799740 1 --2- 172.21.15.129:0/2829497998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058e16b0 0x55c505cc0ae0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9610008c60 tx=0x7f960c009460 comp rx=0 tx=0).stop 2024-05-20T18:06:21.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.056+0000 7f9625799740 1 -- 172.21.15.129:0/2829497998 shutdown_connections 2024-05-20T18:06:21.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.056+0000 7f9625799740 1 --2- 172.21.15.129:0/2829497998 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cc1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.056+0000 7f9625799740 1 --2- 172.21.15.129:0/2829497998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058e16b0 0x55c505cc0ae0 secure :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9610008c60 tx=0x7f960c009460 comp rx=0 tx=0).stop 2024-05-20T18:06:21.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.056+0000 7f9625799740 1 --2- 172.21.15.129:0/2829497998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058d94d0 0x55c505ca3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.056+0000 7f9625799740 1 -- 172.21.15.129:0/2829497998 >> 172.21.15.129:0/2829497998 conn(0x55c5058f64e0 msgr2=0x55c505ca2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.059+0000 7f9625799740 1 -- 172.21.15.129:0/2829497998 shutdown_connections 2024-05-20T18:06:21.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.062+0000 7f9625799740 1 -- 172.21.15.129:0/2829497998 wait complete. 2024-05-20T18:06:21.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.063+0000 7f9625799740 1 -- start start 2024-05-20T18:06:21.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.063+0000 7f9625799740 1 Processor -- start 2024-05-20T18:06:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.064+0000 7f9625799740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058d94d0 0x55c505cc0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.064+0000 7f9625799740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cc0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.064+0000 7f9625799740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058963a0 0x55c505d1ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.064+0000 7f9625799740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc00d690 con 0x55c5058963a0 2024-05-20T18:06:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.064+0000 7f9625799740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc00eb00 con 0x55c5058d94d0 2024-05-20T18:06:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.064+0000 7f9625799740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc00dd60 con 0x55c505902390 2024-05-20T18:06:21.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.064+0000 7f9625799740 10 client.0 _ll_register_callbacks cb 0x7f9618076970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:21.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.065+0000 7f961cdb1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058963a0 0x55c505d1ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.066+0000 7f961cdb1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058963a0 0x55c505d1ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53258/0 (socket says 172.21.15.129:53258) 2024-05-20T18:06:21.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.066+0000 7f961cdb1640 1 -- 172.21.15.129:0/1993622572 learned_addr learned my addr 172.21.15.129:0/1993622572 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:21.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.066+0000 7f961cdb1640 1 -- 172.21.15.129:0/1993622572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 msgr2=0x55c505cc0690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:06:21.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.066+0000 7f961cdb1640 1 --2- 172.21.15.129:0/1993622572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cc0690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.066+0000 7f961cdb1640 1 -- 172.21.15.129:0/1993622572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058d94d0 msgr2=0x55c505cc0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:21.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.066+0000 7f961cdb1640 1 --2- 172.21.15.129:0/1993622572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058d94d0 0x55c505cc0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.066+0000 7f961cdb1640 1 -- 172.21.15.129:0/1993622572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c505d1d2b0 con 0x55c5058963a0 2024-05-20T18:06:21.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.067+0000 7f961cdb1640 1 --2- 172.21.15.129:0/1993622572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058963a0 0x55c505d1ccf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f96180504d0 tx=0x7f96180500b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe44e26cc01ad3 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:21.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.067+0000 7f95f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:21.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.067+0000 7f95f77fe640 1 -- 172.21.15.129:0/1993622572 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9618073cf0 con 0x55c5058963a0 2024-05-20T18:06:21.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.068+0000 7f9625799740 1 -- 172.21.15.129:0/1993622572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c505c0e780 con 0x55c5058963a0 2024-05-20T18:06:21.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.068+0000 7f95f77fe640 1 -- 172.21.15.129:0/1993622572 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f9618085430 con 0x55c5058963a0 2024-05-20T18:06:21.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.070+0000 7f95f77fe640 1 -- 172.21.15.129:0/1993622572 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9618085650 con 0x55c5058963a0 2024-05-20T18:06:21.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.070+0000 7f9625799740 10 client.16545 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:21.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.070+0000 7f9625799740 1 -- 172.21.15.129:0/1993622572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c505d258d0 con 0x55c5058963a0 2024-05-20T18:06:21.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.071+0000 7f95f77fe640 1 -- 172.21.15.129:0/1993622572 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c505c0e780 con 0x55c5058963a0 2024-05-20T18:06:21.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.071+0000 7f95f77fe640 1 -- 172.21.15.129:0/1993622572 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f9618097940 con 0x55c5058963a0 2024-05-20T18:06:21.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.072+0000 7f9625799740 10 client.16545 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:21.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.072+0000 7f9625799740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:21.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.072+0000 7f9625799740 20 client.16545 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:21.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.072+0000 7f9625799740 1 --2- 172.21.15.129:0/1993622572 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c505d26000 0x55c505d284b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.072+0000 7f9625799740 4 client.16545 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:06:21.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.072+0000 7f9625799740 1 -- 172.21.15.129:0/1993622572 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c505b8b6f0 con 0x55c505d26000 2024-05-20T18:06:21.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.074+0000 7f9617fff640 1 --2- 172.21.15.129:0/1993622572 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c505d26000 0x55c505d284b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.086+0000 7f9617fff640 1 --2- 172.21.15.129:0/1993622572 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c505d26000 0x55c505d284b0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16df98338784b280 server_cookie=61289010d773c22c in_seq=0 out_seq=0 2024-05-20T18:06:21.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.086+0000 7f95f77fe640 10 client.16545 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:21.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.087+0000 7f95f77fe640 1 -- 172.21.15.129:0/1993622572 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c505b8b6f0 con 0x55c505d26000 2024-05-20T18:06:21.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.087+0000 7f95f77fe640 10 client.16545 handle_command_reply: tid=0 2024-05-20T18:06:21.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.087+0000 7f9625799740 1 client.16545 shutdown 2024-05-20T18:06:21.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.089+0000 7f9625799740 1 -- 172.21.15.129:0/1993622572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058963a0 msgr2=0x55c505d1ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.089+0000 7f9625799740 1 --2- 172.21.15.129:0/1993622572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058963a0 0x55c505d1ccf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f96180504d0 tx=0x7f96180500b0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.090+0000 7f9625799740 1 -- 172.21.15.129:0/1993622572 shutdown_connections 2024-05-20T18:06:21.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.090+0000 7f9625799740 1 --2- 172.21.15.129:0/1993622572 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c505d26000 0x55c505d284b0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.090+0000 7f9625799740 1 --2- 172.21.15.129:0/1993622572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5058963a0 0x55c505d1ccf0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.090+0000 7f9625799740 1 --2- 172.21.15.129:0/1993622572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c505902390 0x55c505cc0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.090+0000 7f9625799740 1 --2- 172.21.15.129:0/1993622572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c5058d94d0 0x55c505cc0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.090+0000 7f9625799740 1 -- 172.21.15.129:0/1993622572 >> 172.21.15.129:0/1993622572 conn(0x55c5058f64e0 msgr2=0x55c505ca2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.090+0000 7f95f77fe640 0 client.16545 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:21.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.090+0000 7f9625799740 1 -- 172.21.15.129:0/1993622572 shutdown_connections 2024-05-20T18:06:21.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.090+0000 7f9625799740 1 -- 172.21.15.129:0/1993622572 wait complete. 2024-05-20T18:06:21.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.091+0000 7f9625799740 20 client.16545 trim_cache size 0 max 16384 2024-05-20T18:06:21.093 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:06:21.093 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:06:21.093 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:06:21.093 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:06:21.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.096+0000 7f961d5b2640 1 -- 172.21.15.129:0/1899046164 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96080a2f00 msgr2=0x7f96080a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.096+0000 7f961d5b2640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96080a2f00 0x7f96080a5410 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f96180678c0 tx=0x7f9618067a50 comp rx=0 tx=0).stop 2024-05-20T18:06:21.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.096+0000 7f961d5b2640 1 -- 172.21.15.129:0/1899046164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96100959e0 msgr2=0x7f96101301a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.096+0000 7f961d5b2640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96100959e0 0x7f96101301a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f960400b670 tx=0x7f96040045e0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.098+0000 7f961d5b2640 1 -- 172.21.15.129:0/1899046164 shutdown_connections 2024-05-20T18:06:21.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.098+0000 7f961d5b2640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96080a2f00 0x7f96080a5410 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.098+0000 7f961d5b2640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f961009b910 0x7f9610130700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.098+0000 7f961d5b2640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96100a0270 0x7f9610135eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.098+0000 7f961d5b2640 1 --2- 172.21.15.129:0/1899046164 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f96100959e0 0x7f96101301a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.098+0000 7f961d5b2640 1 -- 172.21.15.129:0/1899046164 >> 172.21.15.129:0/1899046164 conn(0x7f961000afd0 msgr2=0x7f961009c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.099+0000 7f961d5b2640 1 -- 172.21.15.129:0/1899046164 shutdown_connections 2024-05-20T18:06:21.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.099+0000 7f961d5b2640 1 -- 172.21.15.129:0/1899046164 wait complete. 2024-05-20T18:06:21.131 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:06:21.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:06:21.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.330+0000 7fc83af45640 1 -- 172.21.15.129:0/109772538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc834074310 msgr2=0x7fc834074720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.330+0000 7fc83af45640 1 --2- 172.21.15.129:0/109772538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc834074310 0x7fc834074720 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc828009900 tx=0x7fc82802a4a0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.330+0000 7fc83af45640 1 -- 172.21.15.129:0/109772538 shutdown_connections 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.330+0000 7fc83af45640 1 --2- 172.21.15.129:0/109772538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83410f300 0x7fc834111720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.330+0000 7fc83af45640 1 --2- 172.21.15.129:0/109772538 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc83410c9a0 0x7fc83410edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.330+0000 7fc83af45640 1 --2- 172.21.15.129:0/109772538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc834074310 0x7fc834074720 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.330+0000 7fc83af45640 1 -- 172.21.15.129:0/109772538 >> 172.21.15.129:0/109772538 conn(0x7fc8340780f0 msgr2=0x7fc834078530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.330+0000 7fc83af45640 1 -- 172.21.15.129:0/109772538 shutdown_connections 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.330+0000 7fc83af45640 1 -- 172.21.15.129:0/109772538 wait complete. 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.331+0000 7fc83af45640 1 Processor -- start 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.331+0000 7fc83af45640 1 -- start start 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.332+0000 7fc83af45640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc834074310 0x7fc834102610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.332+0000 7fc83af45640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83410c9a0 0x7fc834102b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.332+0000 7fc83af45640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc83410f300 0x7fc834103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.332+0000 7fc83af45640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc834115110 con 0x7fc83410f300 2024-05-20T18:06:21.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.332+0000 7fc83af45640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc834114f90 con 0x7fc83410c9a0 2024-05-20T18:06:21.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.332+0000 7fc83af45640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc834115290 con 0x7fc834074310 2024-05-20T18:06:21.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.332+0000 7fc8394e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc83410f300 0x7fc834103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.332+0000 7fc8394e2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc83410f300 0x7fc834103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53286/0 (socket says 172.21.15.129:53286) 2024-05-20T18:06:21.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.332+0000 7fc8394e2640 1 -- 172.21.15.129:0/928670469 learned_addr learned my addr 172.21.15.129:0/928670469 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:21.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.332+0000 7fc838ce1640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc834074310 0x7fc834102610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.333+0000 7fc8394e2640 1 -- 172.21.15.129:0/928670469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc834074310 msgr2=0x7fc834102610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.333+0000 7fc8394e2640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc834074310 0x7fc834102610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.333+0000 7fc8394e2640 1 -- 172.21.15.129:0/928670469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83410c9a0 msgr2=0x7fc834102b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.333+0000 7fc8394e2640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83410c9a0 0x7fc834102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.333+0000 7fc8394e2640 1 -- 172.21.15.129:0/928670469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8341ac1a0 con 0x7fc83410f300 2024-05-20T18:06:21.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.333+0000 7fc8394e2640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc83410f300 0x7fc834103090 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00b970 tx=0x7fc82c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:21.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.333+0000 7fc831ffb640 1 -- 172.21.15.129:0/928670469 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c019070 con 0x7fc83410f300 2024-05-20T18:06:21.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.334+0000 7fc83af45640 1 -- 172.21.15.129:0/928670469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8341ac430 con 0x7fc83410f300 2024-05-20T18:06:21.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.334+0000 7fc831ffb640 1 -- 172.21.15.129:0/928670469 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc82c00cb40 con 0x7fc83410f300 2024-05-20T18:06:21.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.334+0000 7fc831ffb640 1 -- 172.21.15.129:0/928670469 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c007650 con 0x7fc83410f300 2024-05-20T18:06:21.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.334+0000 7fc83af45640 1 -- 172.21.15.129:0/928670469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8341ac980 con 0x7fc83410f300 2024-05-20T18:06:21.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.336+0000 7fc83af45640 1 -- 172.21.15.129:0/928670469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8340755e0 con 0x7fc83410f300 2024-05-20T18:06:21.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.338+0000 7fc831ffb640 1 -- 172.21.15.129:0/928670469 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc82c01c070 con 0x7fc83410f300 2024-05-20T18:06:21.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.340+0000 7fc831ffb640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8240a2e00 0x7fc8240a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.340+0000 7fc831ffb640 1 -- 172.21.15.129:0/928670469 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7fc82c0cea10 con 0x7fc83410f300 2024-05-20T18:06:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.341+0000 7fc838ce1640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8240a2e00 0x7fc8240a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.342+0000 7fc838ce1640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8240a2e00 0x7fc8240a5310 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc82802aa90 tx=0x7fc828000bc0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.346+0000 7fc831ffb640 1 -- 172.21.15.129:0/928670469 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc82c09b6a0 con 0x7fc83410f300 2024-05-20T18:06:21.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:21 smithi129 ceph-mon[32029]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 75 MiB/s wr, 849 op/s 2024-05-20T18:06:21.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:21 smithi129 ceph-mon[32029]: osdmap e98: 12 total, 12 up, 12 in 2024-05-20T18:06:21.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.637+0000 7fc83af45640 1 -- 172.21.15.129:0/928670469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8341acc10 con 0x7fc83410f300 2024-05-20T18:06:21.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.637+0000 7fc831ffb640 1 -- 172.21.15.129:0/928670469 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11829 (secure 0 0 0) 0x7fc82c09afa0 con 0x7fc83410f300 2024-05-20T18:06:21.636 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:06:21.636 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":26255,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/2724397229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":2724397229},{"type":"v1","addr":"172.21.15.154:6837","nonce":2724397229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:06:10.247052+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:06:21.636 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.639+0000 7fc83af45640 1 -- 172.21.15.129:0/928670469 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8240a2e00 msgr2=0x7fc8240a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.639+0000 7fc83af45640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8240a2e00 0x7fc8240a5310 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc82802aa90 tx=0x7fc828000bc0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.639+0000 7fc83af45640 1 -- 172.21.15.129:0/928670469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc83410f300 msgr2=0x7fc834103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.639+0000 7fc83af45640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc83410f300 0x7fc834103090 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00b970 tx=0x7fc82c00be30 comp rx=0 tx=0).stop 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.640+0000 7fc83af45640 1 -- 172.21.15.129:0/928670469 shutdown_connections 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.640+0000 7fc83af45640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8240a2e00 0x7fc8240a5310 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.640+0000 7fc83af45640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83410c9a0 0x7fc834102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.640+0000 7fc83af45640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc83410f300 0x7fc834103090 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.640+0000 7fc83af45640 1 --2- 172.21.15.129:0/928670469 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc834074310 0x7fc834102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.640+0000 7fc83af45640 1 -- 172.21.15.129:0/928670469 >> 172.21.15.129:0/928670469 conn(0x7fc8340780f0 msgr2=0x7fc8341012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.640+0000 7fc83af45640 1 -- 172.21.15.129:0/928670469 shutdown_connections 2024-05-20T18:06:21.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.640+0000 7fc83af45640 1 -- 172.21.15.129:0/928670469 wait complete. 2024-05-20T18:06:21.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:21 smithi154 ceph-mon[36482]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 75 MiB/s wr, 849 op/s 2024-05-20T18:06:21.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:21 smithi154 ceph-mon[36482]: osdmap e98: 12 total, 12 up, 12 in 2024-05-20T18:06:21.651 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:06:21.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:21 smithi204 ceph-mon[37076]: pgmap v239: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 75 MiB/s wr, 849 op/s 2024-05-20T18:06:21.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:21 smithi204 ceph-mon[37076]: osdmap e98: 12 total, 12 up, 12 in 2024-05-20T18:06:21.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.777+0000 7f1ad092f640 1 -- 172.21.15.129:0/3992489885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1acc1079d0 msgr2=0x7f1acc101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.777+0000 7f1ad092f640 1 --2- 172.21.15.129:0/3992489885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1acc1079d0 0x7f1acc101560 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009900 tx=0x7f1abc02a410 comp rx=0 tx=0).stop 2024-05-20T18:06:21.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.777+0000 7f1ad092f640 1 -- 172.21.15.129:0/3992489885 shutdown_connections 2024-05-20T18:06:21.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.777+0000 7f1ad092f640 1 --2- 172.21.15.129:0/3992489885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1acc101aa0 0x7f1acc101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.777+0000 7f1ad092f640 1 --2- 172.21.15.129:0/3992489885 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1acc10c330 0x7f1acc10e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.777+0000 7f1ad092f640 1 --2- 172.21.15.129:0/3992489885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1acc1079d0 0x7f1acc101560 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.778+0000 7f1ad092f640 1 -- 172.21.15.129:0/3992489885 >> 172.21.15.129:0/3992489885 conn(0x7f1acc0770b0 msgr2=0x7f1acc0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.778+0000 7f1ad092f640 1 -- 172.21.15.129:0/3992489885 shutdown_connections 2024-05-20T18:06:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.778+0000 7f1ad092f640 1 -- 172.21.15.129:0/3992489885 wait complete. 2024-05-20T18:06:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.778+0000 7f1ad092f640 1 Processor -- start 2024-05-20T18:06:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1ad092f640 1 -- start start 2024-05-20T18:06:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1ad092f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1acc101aa0 0x7f1acc19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1ad092f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1acc1079d0 0x7f1acc19aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1ad092f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1acc10c330 0x7f1acc1a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1ad092f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc111ce0 con 0x7f1acc10c330 2024-05-20T18:06:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1ad092f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc111b60 con 0x7f1acc1079d0 2024-05-20T18:06:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1ad092f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc111e60 con 0x7f1acc101aa0 2024-05-20T18:06:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1acad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1acc10c330 0x7f1acc1a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1aca59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1acc101aa0 0x7f1acc19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1ac9d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1acc1079d0 0x7f1acc19aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1aca59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1acc101aa0 0x7f1acc19a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:44584/0 (socket says 172.21.15.129:44584) 2024-05-20T18:06:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1acad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1acc10c330 0x7f1acc1a5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53304/0 (socket says 172.21.15.129:53304) 2024-05-20T18:06:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1aca59c640 1 -- 172.21.15.129:0/2784019800 learned_addr learned my addr 172.21.15.129:0/2784019800 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1ac9d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1acc1079d0 0x7f1acc19aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:35668/0 (socket says 172.21.15.129:35668) 2024-05-20T18:06:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1aca59c640 1 -- 172.21.15.129:0/2784019800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1acc1079d0 msgr2=0x7f1acc19aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1aca59c640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1acc1079d0 0x7f1acc19aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1aca59c640 1 -- 172.21.15.129:0/2784019800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1acc10c330 msgr2=0x7f1acc1a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1aca59c640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1acc10c330 0x7f1acc1a5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.779+0000 7f1aca59c640 1 -- 172.21.15.129:0/2784019800 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1acc19b360 con 0x7f1acc101aa0 2024-05-20T18:06:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.780+0000 7f1aca59c640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1acc101aa0 0x7f1acc19a5a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1abc005940 tx=0x7f1abc004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.780+0000 7f1abb7fe640 1 -- 172.21.15.129:0/2784019800 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc03f030 con 0x7f1acc101aa0 2024-05-20T18:06:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.780+0000 7f1abb7fe640 1 -- 172.21.15.129:0/2784019800 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1abc003ad0 con 0x7f1acc101aa0 2024-05-20T18:06:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.780+0000 7f1ad092f640 1 -- 172.21.15.129:0/2784019800 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1acc1a5bc0 con 0x7f1acc101aa0 2024-05-20T18:06:21.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.780+0000 7f1abb7fe640 1 -- 172.21.15.129:0/2784019800 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc035070 con 0x7f1acc101aa0 2024-05-20T18:06:21.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.780+0000 7f1ad092f640 1 -- 172.21.15.129:0/2784019800 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1acc1a60b0 con 0x7f1acc101aa0 2024-05-20T18:06:21.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.783+0000 7f1abb7fe640 1 -- 172.21.15.129:0/2784019800 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f1abc042030 con 0x7f1acc101aa0 2024-05-20T18:06:21.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.783+0000 7f1abb7fe640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ab40a2d00 0x7f1ab40a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.784+0000 7f1abb7fe640 1 -- 172.21.15.129:0/2784019800 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f1abc0efb60 con 0x7f1acc101aa0 2024-05-20T18:06:21.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.784+0000 7f1ac9d9b640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ab40a2d00 0x7f1ab40a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.784+0000 7f1ad77d2740 1 Processor -- start 2024-05-20T18:06:21.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.784+0000 7f1ac9d9b640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ab40a2d00 0x7f1ab40a5210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1acc10d840 tx=0x7f1ac000b390 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:21.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.784+0000 7f1ad77d2740 1 -- start start 2024-05-20T18:06:21.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b599d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1d24d0 0x55714b59a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b59a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55714b4ffc40 con 0x55714b1da6b0 2024-05-20T18:06:21.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55714b4a7580 con 0x55714b1fb390 2024-05-20T18:06:21.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55714b4fbc00 con 0x55714b1d24d0 2024-05-20T18:06:21.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1aca59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b599d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1acad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b59a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1aca59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b599d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:35672/0 (socket says 172.21.15.129:35672) 2024-05-20T18:06:21.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1ac9d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1d24d0 0x55714b59a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1aca59c640 1 -- 172.21.15.129:0/3596906496 learned_addr learned my addr 172.21.15.129:0/3596906496 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:21.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1aca59c640 1 -- 172.21.15.129:0/3596906496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1d24d0 msgr2=0x55714b59a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:21.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1aca59c640 1 --2- 172.21.15.129:0/3596906496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1d24d0 0x55714b59a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1aca59c640 1 -- 172.21.15.129:0/3596906496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 msgr2=0x55714b59a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:21.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1aca59c640 1 --2- 172.21.15.129:0/3596906496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b59a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1aca59c640 1 -- 172.21.15.129:0/3596906496 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55714b2133f0 con 0x55714b1fb390 2024-05-20T18:06:21.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.785+0000 7f1aca59c640 1 --2- 172.21.15.129:0/3596906496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b599d20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0c4780 tx=0x7f1abc0bba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24290497f135c454 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:21.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.786+0000 7f1ab8ff9640 1 -- 172.21.15.129:0/3596906496 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc049040 con 0x55714b1fb390 2024-05-20T18:06:21.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.786+0000 7f1ab8ff9640 1 -- 172.21.15.129:0/3596906496 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1abc0c4cb0 con 0x55714b1fb390 2024-05-20T18:06:21.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.786+0000 7f1ab8ff9640 1 -- 172.21.15.129:0/3596906496 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc0b8760 con 0x55714b1fb390 2024-05-20T18:06:21.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.787+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3596906496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 msgr2=0x55714b599d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.787+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/3596906496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b599d20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0c4780 tx=0x7f1abc0bba50 comp rx=0 tx=0).stop 2024-05-20T18:06:21.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.787+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3596906496 shutdown_connections 2024-05-20T18:06:21.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.787+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/3596906496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b59a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.787+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/3596906496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1d24d0 0x55714b59a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.787+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/3596906496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b599d20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.787+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3596906496 >> 172.21.15.129:0/3596906496 conn(0x55714b1ef4e0 msgr2=0x55714b5964d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.787+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3596906496 shutdown_connections 2024-05-20T18:06:21.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.787+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3596906496 wait complete. 2024-05-20T18:06:21.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.788+0000 7f1ad77d2740 1 -- start start 2024-05-20T18:06:21.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.788+0000 7f1ad77d2740 1 Processor -- start 2024-05-20T18:06:21.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.788+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1d24d0 0x55714b5a4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.788+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b5a4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.788+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b5aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.788+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55714b5a6ef0 con 0x55714b1da6b0 2024-05-20T18:06:21.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.788+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55714b4bac40 con 0x55714b1fb390 2024-05-20T18:06:21.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.788+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55714b5a7070 con 0x55714b1d24d0 2024-05-20T18:06:21.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.788+0000 7f1ad77d2740 10 client.0 _ll_register_callbacks cb 0x55714b212be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:21.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1ac9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b5a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1acad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b5aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1acad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b5aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:35676/0 (socket says 172.21.15.129:35676) 2024-05-20T18:06:21.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1ac9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b5a4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53314/0 (socket says 172.21.15.129:53314) 2024-05-20T18:06:21.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1aca59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1d24d0 0x55714b5a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1acad9d640 1 -- 172.21.15.129:0/3277598516 learned_addr learned my addr 172.21.15.129:0/3277598516 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:21.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1acad9d640 1 -- 172.21.15.129:0/3277598516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1d24d0 msgr2=0x55714b5a4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:21.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1acad9d640 1 --2- 172.21.15.129:0/3277598516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1d24d0 0x55714b5a4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1acad9d640 1 -- 172.21.15.129:0/3277598516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 msgr2=0x55714b5a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1acad9d640 1 --2- 172.21.15.129:0/3277598516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b5a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1acad9d640 1 -- 172.21.15.129:0/3277598516 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55714b2dd7d0 con 0x55714b1fb390 2024-05-20T18:06:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1acad9d640 1 --2- 172.21.15.129:0/3277598516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b5aa390 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4011d10 tx=0x7f1ac400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f43614ba444761a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1ab1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:06:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/3277598516 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac400fe60 con 0x55714b1fb390 2024-05-20T18:06:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/3277598516 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f1ac4004b20 con 0x55714b1fb390 2024-05-20T18:06:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3277598516 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1abc0c4cb0 con 0x55714b1fb390 2024-05-20T18:06:21.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.789+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/3277598516 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1ac4004d20 con 0x55714b1fb390 2024-05-20T18:06:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.790+0000 7f1ad77d2740 10 client.26279 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.790+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3277598516 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55714b5b36e0 con 0x55714b1fb390 2024-05-20T18:06:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.790+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/3277598516 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1abc0c4cb0 con 0x55714b1fb390 2024-05-20T18:06:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.790+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/3277598516 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f1ac401b470 con 0x55714b1fb390 2024-05-20T18:06:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.790+0000 7f1ad77d2740 10 client.26279 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.790+0000 7f1ad77d2740 10 client.26279 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.790+0000 7f1ad77d2740 20 client.26279 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.790+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/3277598516 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55714b1b9f60 0x55714b5b6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.790+0000 7f1ad77d2740 4 client.26279 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.790+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3277598516 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55714b50d370 con 0x55714b1b9f60 2024-05-20T18:06:21.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.790+0000 7f1aca59c640 1 --2- 172.21.15.129:0/3277598516 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55714b1b9f60 0x55714b5b6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.791+0000 7f1ab1ffb640 10 client.26279 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:21.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.791+0000 7f1aca59c640 1 --2- 172.21.15.129:0/3277598516 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55714b1b9f60 0x55714b5b6c00 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d76ca43add75179 server_cookie=432dbb58d217bc96 in_seq=0 out_seq=0 2024-05-20T18:06:21.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.791+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/3277598516 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55714b50d370 con 0x55714b1b9f60 2024-05-20T18:06:21.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.791+0000 7f1ab1ffb640 10 client.26279 handle_command_reply: tid=0 2024-05-20T18:06:21.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.791+0000 7f1ad77d2740 1 client.26279 shutdown 2024-05-20T18:06:21.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3277598516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 msgr2=0x55714b5aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/3277598516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b5aa390 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4011d10 tx=0x7f1ac400f790 comp rx=0 tx=0).stop 2024-05-20T18:06:21.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3277598516 shutdown_connections 2024-05-20T18:06:21.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/3277598516 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55714b1b9f60 0x55714b5b6c00 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/3277598516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1fb390 0x55714b5aa390 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ab1ffb640 0 client.26279 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:21.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/3277598516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b5a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/3277598516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1d24d0 0x55714b5a4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3277598516 >> 172.21.15.129:0/3277598516 conn(0x55714b1ef4e0 msgr2=0x55714b59ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3277598516 shutdown_connections 2024-05-20T18:06:21.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ad77d2740 1 -- 172.21.15.129:0/3277598516 wait complete. 2024-05-20T18:06:21.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.792+0000 7f1ad77d2740 20 client.26279 trim_cache size 0 max 16384 2024-05-20T18:06:21.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.820+0000 7f1ad77d2740 1 Processor -- start 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.820+0000 7f1ad77d2740 1 -- start start 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.820+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1b9f60 0x55714b5a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.820+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5a3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b5bc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1acab0 con 0x55714b1da6b0 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1af310 con 0x55714b1d24d0 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1ad100 con 0x55714b1b9f60 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1acad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b5bc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1ac9d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1aca59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1b9f60 0x55714b5a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1ac9d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5a3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:35684/0 (socket says 172.21.15.129:35684) 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1acad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b5bc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53320/0 (socket says 172.21.15.129:53320) 2024-05-20T18:06:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1ac9d9b640 1 -- 172.21.15.129:0/2425944119 learned_addr learned my addr 172.21.15.129:0/2425944119 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1aca59c640 1 -- 172.21.15.129:0/2425944119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 msgr2=0x55714b5a3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1aca59c640 1 --2- 172.21.15.129:0/2425944119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5a3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1aca59c640 1 -- 172.21.15.129:0/2425944119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 msgr2=0x55714b5bc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1aca59c640 1 --2- 172.21.15.129:0/2425944119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b5bc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1aca59c640 1 -- 172.21.15.129:0/2425944119 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55714b595c00 con 0x55714b1b9f60 2024-05-20T18:06:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1ac9d9b640 1 --2- 172.21.15.129:0/2425944119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5a3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:06:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1aca59c640 1 --2- 172.21.15.129:0/2425944119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1b9f60 0x55714b5a2ad0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0bc080 tx=0x7f1abc0bc270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d96a1f0a26a8943 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.821+0000 7f1ab3fff640 1 -- 172.21.15.129:0/2425944119 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc049040 con 0x55714b1b9f60 2024-05-20T18:06:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.822+0000 7f1ab3fff640 1 -- 172.21.15.129:0/2425944119 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1abc0c4cb0 con 0x55714b1b9f60 2024-05-20T18:06:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.822+0000 7f1ab3fff640 1 -- 172.21.15.129:0/2425944119 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc0ed660 con 0x55714b1b9f60 2024-05-20T18:06:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.823+0000 7f1ad77d2740 1 -- 172.21.15.129:0/2425944119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1b9f60 msgr2=0x55714b5a2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.823+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/2425944119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1b9f60 0x55714b5a2ad0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0bc080 tx=0x7f1abc0bc270 comp rx=0 tx=0).stop 2024-05-20T18:06:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.823+0000 7f1ad77d2740 1 -- 172.21.15.129:0/2425944119 shutdown_connections 2024-05-20T18:06:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.823+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/2425944119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5a3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.823+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/2425944119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1da6b0 0x55714b5bc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.823+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/2425944119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1b9f60 0x55714b5a2ad0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.823+0000 7f1ad77d2740 1 -- 172.21.15.129:0/2425944119 >> 172.21.15.129:0/2425944119 conn(0x55714b1ef4e0 msgr2=0x55714b59f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.823+0000 7f1ad77d2740 1 -- 172.21.15.129:0/2425944119 shutdown_connections 2024-05-20T18:06:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.823+0000 7f1ad77d2740 1 -- 172.21.15.129:0/2425944119 wait complete. 2024-05-20T18:06:21.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.823+0000 7f1ad77d2740 1 -- start start 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1ad77d2740 1 Processor -- start 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1b9f60 0x55714b5aba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5abf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1ad77d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1da6b0 0x55714b61cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1ad7d0 con 0x55714b1b9f60 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1aec40 con 0x55714b1d24d0 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1ad77d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1adea0 con 0x55714b1da6b0 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1ad77d2740 10 client.0 _ll_register_callbacks cb 0x7f1ac4009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1ac9d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1aca59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1b9f60 0x55714b5aba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1acad9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1da6b0 0x55714b61cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1ac9d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5abf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:35692/0 (socket says 172.21.15.129:35692) 2024-05-20T18:06:21.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.824+0000 7f1ac9d9b640 1 -- 172.21.15.129:0/232116811 learned_addr learned my addr 172.21.15.129:0/232116811 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ac9d9b640 1 -- 172.21.15.129:0/232116811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1da6b0 msgr2=0x55714b61cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ac9d9b640 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1da6b0 0x55714b61cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ac9d9b640 1 -- 172.21.15.129:0/232116811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1b9f60 msgr2=0x55714b5aba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ac9d9b640 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1b9f60 0x55714b5aba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ac9d9b640 1 -- 172.21.15.129:0/232116811 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55714b61d4a0 con 0x55714b1d24d0 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1acad9d640 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1da6b0 0x55714b61cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1aca59c640 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1b9f60 0x55714b5aba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ac9d9b640 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5abf90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1acc06b480 tx=0x7f1ac0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f2856b1a6078da9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ab1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/232116811 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac002e060 con 0x55714b1d24d0 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/232116811 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f1ac001ecb0 con 0x55714b1d24d0 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ad77d2740 1 -- 172.21.15.129:0/232116811 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1abc0c4cb0 con 0x55714b1d24d0 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/232116811 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1ac00313f0 con 0x55714b1d24d0 2024-05-20T18:06:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ad77d2740 10 client.26285 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.825+0000 7f1ad77d2740 1 -- 172.21.15.129:0/232116811 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55714b61e690 con 0x55714b1d24d0 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.826+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/232116811 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1abc0c4cb0 con 0x55714b1d24d0 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.826+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/232116811 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f1ac0028070 con 0x55714b1d24d0 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.826+0000 7f1ad77d2740 10 client.26285 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.826+0000 7f1ad77d2740 10 client.26285 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.826+0000 7f1ad77d2740 20 client.26285 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.826+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55714b1fb390 0x55714b620eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.826+0000 7f1ad77d2740 4 client.26285 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.826+0000 7f1ad77d2740 1 -- 172.21.15.129:0/232116811 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55714b5076c0 con 0x55714b1fb390 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.826+0000 7f1aca59c640 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55714b1fb390 0x55714b620eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.826+0000 7f1ab1ffb640 10 client.26285 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.826+0000 7f1aca59c640 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55714b1fb390 0x55714b620eb0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7c76a44eb0974c5 server_cookie=fc18d04c11c73057 in_seq=0 out_seq=0 2024-05-20T18:06:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ab1ffb640 1 -- 172.21.15.129:0/232116811 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55714b5076c0 con 0x55714b1fb390 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ab1ffb640 10 client.26285 handle_command_reply: tid=0 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 1 client.26285 shutdown 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 1 -- 172.21.15.129:0/232116811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 msgr2=0x55714b5abf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5abf90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1acc06b480 tx=0x7f1ac0004730 comp rx=0 tx=0).stop 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 1 -- 172.21.15.129:0/232116811 shutdown_connections 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55714b1fb390 0x55714b620eb0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55714b1d24d0 0x55714b5abf90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ab1ffb640 0 client.26285 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55714b1da6b0 0x55714b61cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 1 --2- 172.21.15.129:0/232116811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55714b1b9f60 0x55714b5aba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 1 -- 172.21.15.129:0/232116811 >> 172.21.15.129:0/232116811 conn(0x55714b1ef4e0 msgr2=0x55714b5a2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 1 -- 172.21.15.129:0/232116811 shutdown_connections 2024-05-20T18:06:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 1 -- 172.21.15.129:0/232116811 wait complete. 2024-05-20T18:06:21.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.827+0000 7f1ad77d2740 20 client.26285 trim_cache size 0 max 16384 2024-05-20T18:06:21.831 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:06:21.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 -- 172.21.15.129:0/2784019800 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ab40a2d00 msgr2=0x7f1ab40a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ab40a2d00 0x7f1ab40a5210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1acc10d840 tx=0x7f1ac000b390 comp rx=0 tx=0).stop 2024-05-20T18:06:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 -- 172.21.15.129:0/2784019800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1acc101aa0 msgr2=0x7f1acc19a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1acc101aa0 0x7f1acc19a5a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1abc005940 tx=0x7f1abc004270 comp rx=0 tx=0).stop 2024-05-20T18:06:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 -- 172.21.15.129:0/2784019800 shutdown_connections 2024-05-20T18:06:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f1ab40a2d00 0x7f1ab40a5210 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1acc1079d0 0x7f1acc19aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1acc10c330 0x7f1acc1a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 --2- 172.21.15.129:0/2784019800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1acc101aa0 0x7f1acc19a5a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 -- 172.21.15.129:0/2784019800 >> 172.21.15.129:0/2784019800 conn(0x7f1acc0770b0 msgr2=0x7f1acc103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 -- 172.21.15.129:0/2784019800 shutdown_connections 2024-05-20T18:06:21.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:21.828+0000 7f1ad092f640 1 -- 172.21.15.129:0/2784019800 wait complete. 2024-05-20T18:06:22.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:22 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/928670469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:06:22.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:22 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/928670469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:06:22.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:22 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/928670469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:06:22.846 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:06:22.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:06:22.846 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:06:22.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.969+0000 7f963cd02640 1 -- 172.21.15.129:0/3042752500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9638101790 msgr2=0x7f963810f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:22.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.969+0000 7f963cd02640 1 --2- 172.21.15.129:0/3042752500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9638101790 0x7f963810f640 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f9630009ca0 tx=0x7f963002a8c0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.969+0000 7f963cd02640 1 -- 172.21.15.129:0/3042752500 shutdown_connections 2024-05-20T18:06:22.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.969+0000 7f963cd02640 1 --2- 172.21.15.129:0/3042752500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9638101790 0x7f963810f640 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.969+0000 7f963cd02640 1 --2- 172.21.15.129:0/3042752500 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9638108ff0 0x7f9638101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.969+0000 7f963cd02640 1 --2- 172.21.15.129:0/3042752500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638108b50 0x7f9638100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.969+0000 7f963cd02640 1 -- 172.21.15.129:0/3042752500 >> 172.21.15.129:0/3042752500 conn(0x7f9638076830 msgr2=0x7f9638076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:22.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.969+0000 7f963cd02640 1 -- 172.21.15.129:0/3042752500 shutdown_connections 2024-05-20T18:06:22.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.969+0000 7f963cd02640 1 -- 172.21.15.129:0/3042752500 wait complete. 2024-05-20T18:06:22.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f963cd02640 1 Processor -- start 2024-05-20T18:06:22.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f963cd02640 1 -- start start 2024-05-20T18:06:22.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f963cd02640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9638101790 0x7f96381997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f963cd02640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638108b50 0x7f9638199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f963cd02640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9638108ff0 0x7f963819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f963cd02640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9638112e60 con 0x7f9638108ff0 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f963cd02640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9638112ce0 con 0x7f9638108b50 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f963cd02640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9638112fe0 con 0x7f9638101790 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f9636d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9638108ff0 0x7f963819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f9636d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9638108ff0 0x7f963819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53358/0 (socket says 172.21.15.129:53358) 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f9636d9d640 1 -- 172.21.15.129:0/69917108 learned_addr learned my addr 172.21.15.129:0/69917108 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f962ffff640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638108b50 0x7f9638199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f963659c640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9638101790 0x7f96381997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f9636d9d640 1 -- 172.21.15.129:0/69917108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9638101790 msgr2=0x7f96381997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:22.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f9636d9d640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9638101790 0x7f96381997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f9636d9d640 1 -- 172.21.15.129:0/69917108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638108b50 msgr2=0x7f9638199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:22.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f9636d9d640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638108b50 0x7f9638199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f9636d9d640 1 -- 172.21.15.129:0/69917108 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963819e640 con 0x7f9638108ff0 2024-05-20T18:06:22.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.970+0000 7f962ffff640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638108b50 0x7f9638199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:22.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.971+0000 7f9636d9d640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9638108ff0 0x7f963819e000 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f96300319f0 tx=0x7f9630031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:22.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.971+0000 7f962f7fe640 1 -- 172.21.15.129:0/69917108 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630003c30 con 0x7f9638108ff0 2024-05-20T18:06:22.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.971+0000 7f963cd02640 1 -- 172.21.15.129:0/69917108 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f963819e8d0 con 0x7f9638108ff0 2024-05-20T18:06:22.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.971+0000 7f962f7fe640 1 -- 172.21.15.129:0/69917108 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9630003dd0 con 0x7f9638108ff0 2024-05-20T18:06:22.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.971+0000 7f962f7fe640 1 -- 172.21.15.129:0/69917108 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963002fcb0 con 0x7f9638108ff0 2024-05-20T18:06:22.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.971+0000 7f963cd02640 1 -- 172.21.15.129:0/69917108 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f963819ee20 con 0x7f9638108ff0 2024-05-20T18:06:22.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.974+0000 7f9643bab740 1 Processor -- start 2024-05-20T18:06:22.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.974+0000 7f962f7fe640 1 -- 172.21.15.129:0/69917108 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9630004050 con 0x7f9638108ff0 2024-05-20T18:06:22.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.974+0000 7f9643bab740 1 -- start start 2024-05-20T18:06:22.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.974+0000 7f9643bab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b8009fb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.974+0000 7f9643bab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b8009fbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.974+0000 7f9643bab740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80063b730 0x55b8009fc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.974+0000 7f9643bab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b800960990 con 0x55b80065c410 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.974+0000 7f9643bab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b8009085b0 con 0x55b800633550 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.974+0000 7f9643bab740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b80095c950 con 0x55b80063b730 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f962ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b8009fb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f963659c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b8009fbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f962ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b8009fb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53372/0 (socket says 172.21.15.129:53372) 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f962ffff640 1 -- 172.21.15.129:0/3209911968 learned_addr learned my addr 172.21.15.129:0/3209911968 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f962f7fe640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96240a2d00 0x7f96240a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f962ffff640 1 --2- 172.21.15.129:0/3209911968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80063b730 0x55b8009fc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f963659c640 1 -- 172.21.15.129:0/3209911968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80063b730 msgr2=0x55b8009fc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f963659c640 1 --2- 172.21.15.129:0/3209911968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80063b730 0x55b8009fc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f9636d9d640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96240a2d00 0x7f96240a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f963659c640 1 -- 172.21.15.129:0/3209911968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 msgr2=0x55b8009fb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f963659c640 1 --2- 172.21.15.129:0/3209911968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b8009fb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f963659c640 1 -- 172.21.15.129:0/3209911968 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b80073c7e0 con 0x55b800633550 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f962ffff640 1 --2- 172.21.15.129:0/3209911968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b8009fb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f962f7fe640 1 -- 172.21.15.129:0/69917108 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f96300f0620 con 0x7f9638108ff0 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f963659c640 1 --2- 172.21.15.129:0/3209911968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b8009fbb60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9628009b70 tx=0x7f96280117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7da5e33eb53eb657 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f9636d9d640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96240a2d00 0x7f96240a5210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f96300edef0 tx=0x7f963002d730 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f962cff9640 1 -- 172.21.15.129:0/3209911968 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628009e60 con 0x55b800633550 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f962cff9640 1 -- 172.21.15.129:0/3209911968 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9628012930 con 0x55b800633550 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.975+0000 7f962cff9640 1 -- 172.21.15.129:0/3209911968 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962800fa70 con 0x55b800633550 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.976+0000 7f9643bab740 1 -- 172.21.15.129:0/3209911968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 msgr2=0x55b8009fbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:22.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.976+0000 7f9643bab740 1 --2- 172.21.15.129:0/3209911968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b8009fbb60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9628009b70 tx=0x7f96280117b0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.976+0000 7f9643bab740 1 -- 172.21.15.129:0/3209911968 shutdown_connections 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.976+0000 7f9643bab740 1 --2- 172.21.15.129:0/3209911968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b8009fbb60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.976+0000 7f9643bab740 1 --2- 172.21.15.129:0/3209911968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80063b730 0x55b8009fc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.976+0000 7f9643bab740 1 --2- 172.21.15.129:0/3209911968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b8009fb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.976+0000 7f9643bab740 1 -- 172.21.15.129:0/3209911968 >> 172.21.15.129:0/3209911968 conn(0x55b800650560 msgr2=0x55b8009f7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.976+0000 7f9643bab740 1 -- 172.21.15.129:0/3209911968 shutdown_connections 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.976+0000 7f9643bab740 1 -- 172.21.15.129:0/3209911968 wait complete. 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f9643bab740 1 -- start start 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f9643bab740 1 Processor -- start 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f9643bab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b800a07660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f9643bab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b800a07ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:22.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f9643bab740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80065c410 0x55b800a05b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f9643bab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b800a08970 con 0x55b80063b730 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f9643bab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b80091bb40 con 0x55b800633550 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f9643bab740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b800a08af0 con 0x55b80065c410 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f9643bab740 10 client.0 _ll_register_callbacks cb 0x55b8009100f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f963659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b800a07ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f963659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b800a07ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53386/0 (socket says 172.21.15.129:53386) 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f963659c640 1 -- 172.21.15.129:0/1618108770 learned_addr learned my addr 172.21.15.129:0/1618108770 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f962ffff640 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b800a07660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.977+0000 7f962ffff640 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80065c410 0x55b800a05b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f963659c640 1 -- 172.21.15.129:0/1618108770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80065c410 msgr2=0x55b800a05b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f963659c640 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80065c410 0x55b800a05b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f963659c640 1 -- 172.21.15.129:0/1618108770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 msgr2=0x55b800a07660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f963659c640 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b800a07660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f963659c640 1 -- 172.21.15.129:0/1618108770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b80079cf70 con 0x55b80063b730 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f963659c640 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b800a07ba0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f962801c040 tx=0x7f9628012700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7891c44b72123c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f961dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f961dffb640 1 -- 172.21.15.129:0/1618108770 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628028070 con 0x55b80063b730 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f961dffb640 1 -- 172.21.15.129:0/1618108770 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f9628005040 con 0x55b80063b730 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f9643bab740 1 -- 172.21.15.129:0/1618108770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9628012930 con 0x55b80063b730 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f961dffb640 1 -- 172.21.15.129:0/1618108770 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9628034050 con 0x55b80063b730 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f9643bab740 10 client.16581 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f9643bab740 1 -- 172.21.15.129:0/1618108770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b800634170 con 0x55b80063b730 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f961dffb640 1 -- 172.21.15.129:0/1618108770 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9628012930 con 0x55b80063b730 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f961dffb640 1 -- 172.21.15.129:0/1618108770 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f96280046c0 con 0x55b80063b730 2024-05-20T18:06:22.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f9643bab740 10 client.16581 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.978+0000 7f9643bab740 10 client.16581 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.979+0000 7f9643bab740 20 client.16581 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.979+0000 7f9643bab740 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b80061b1f0 0x55b800a17fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.979+0000 7f9643bab740 4 client.16581 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.979+0000 7f9643bab740 1 -- 172.21.15.129:0/1618108770 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b80096e0c0 con 0x55b80061b1f0 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.979+0000 7f962ffff640 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b80061b1f0 0x55b800a17fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.979+0000 7f961dffb640 10 client.16581 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.979+0000 7f962ffff640 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b80061b1f0 0x55b800a17fc0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ae5ffa878518aa3 server_cookie=461a09a516778733 in_seq=0 out_seq=0 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f961dffb640 1 -- 172.21.15.129:0/1618108770 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b80096e0c0 con 0x55b80061b1f0 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f961dffb640 10 client.16581 handle_command_reply: tid=0 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 1 client.16581 shutdown 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 1 -- 172.21.15.129:0/1618108770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 msgr2=0x55b800a07ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:22.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b800a07ba0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f962801c040 tx=0x7f9628012700 comp rx=0 tx=0).stop 2024-05-20T18:06:22.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 1 -- 172.21.15.129:0/1618108770 shutdown_connections 2024-05-20T18:06:22.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b80061b1f0 0x55b800a17fc0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80065c410 0x55b800a05b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f961dffb640 0 client.16581 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:22.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b800a07ba0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 1 --2- 172.21.15.129:0/1618108770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b800a07660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:22.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 1 -- 172.21.15.129:0/1618108770 >> 172.21.15.129:0/1618108770 conn(0x55b800650560 msgr2=0x55b8009fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:22.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 1 -- 172.21.15.129:0/1618108770 shutdown_connections 2024-05-20T18:06:22.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 1 -- 172.21.15.129:0/1618108770 wait complete. 2024-05-20T18:06:22.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.980+0000 7f9643bab740 20 client.16581 trim_cache size 0 max 16384 2024-05-20T18:06:23.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.998+0000 7f9643bab740 1 Processor -- start 2024-05-20T18:06:23.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.998+0000 7f9643bab740 1 -- start start 2024-05-20T18:06:23.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.998+0000 7f9643bab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b8009fd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.998+0000 7f9643bab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b8009ff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.998+0000 7f9643bab740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80065c410 0x55b8009ff810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.998+0000 7f9643bab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f96381acae0 con 0x55b80063b730 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.998+0000 7f9643bab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96381af340 con 0x55b800633550 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.998+0000 7f9643bab740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f96381ad130 con 0x55b80065c410 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f963659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b8009ff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f963659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b8009ff2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53392/0 (socket says 172.21.15.129:53392) 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f963659c640 1 -- 172.21.15.129:0/2032390760 learned_addr learned my addr 172.21.15.129:0/2032390760 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f962ffff640 1 --2- 172.21.15.129:0/2032390760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80065c410 0x55b8009ff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f962ffff640 1 --2- 172.21.15.129:0/2032390760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b8009fd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f963659c640 1 -- 172.21.15.129:0/2032390760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80065c410 msgr2=0x55b8009ff810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f963659c640 1 --2- 172.21.15.129:0/2032390760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80065c410 0x55b8009ff810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f963659c640 1 -- 172.21.15.129:0/2032390760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 msgr2=0x55b8009fd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f963659c640 1 --2- 172.21.15.129:0/2032390760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b8009fd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f963659c640 1 -- 172.21.15.129:0/2032390760 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b800a04080 con 0x55b80063b730 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f963659c640 1 --2- 172.21.15.129:0/2032390760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b8009ff2d0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f9628004a90 tx=0x7f9628005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df56e0890406b9d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f961ffff640 1 -- 172.21.15.129:0/2032390760 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628028040 con 0x55b80063b730 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f961ffff640 1 -- 172.21.15.129:0/2032390760 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f962800fbe0 con 0x55b80063b730 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:22.999+0000 7f961ffff640 1 -- 172.21.15.129:0/2032390760 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628004290 con 0x55b80063b730 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.000+0000 7f9643bab740 1 -- 172.21.15.129:0/2032390760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 msgr2=0x55b8009ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.000+0000 7f9643bab740 1 --2- 172.21.15.129:0/2032390760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b8009ff2d0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f9628004a90 tx=0x7f9628005fd0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.000+0000 7f9643bab740 1 -- 172.21.15.129:0/2032390760 shutdown_connections 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.000+0000 7f9643bab740 1 --2- 172.21.15.129:0/2032390760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80065c410 0x55b8009ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.000+0000 7f9643bab740 1 --2- 172.21.15.129:0/2032390760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80063b730 0x55b8009ff2d0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.000+0000 7f9643bab740 1 --2- 172.21.15.129:0/2032390760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b8009fd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.000+0000 7f9643bab740 1 -- 172.21.15.129:0/2032390760 >> 172.21.15.129:0/2032390760 conn(0x55b800650560 msgr2=0x55b8009fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.000+0000 7f9643bab740 1 -- 172.21.15.129:0/2032390760 shutdown_connections 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.000+0000 7f9643bab740 1 -- 172.21.15.129:0/2032390760 wait complete. 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.000+0000 7f9643bab740 1 -- start start 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f9643bab740 1 Processor -- start 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f9643bab740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b800a18b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f9643bab740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80063b730 0x55b800a19070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f9643bab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b800a195b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f9643bab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f96381ad800 con 0x55b80065c410 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f9643bab740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96381aec70 con 0x55b800633550 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f9643bab740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f96381aded0 con 0x55b80063b730 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f9643bab740 10 client.0 _ll_register_callbacks cb 0x7f962000d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f962ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b800a195b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f962ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b800a195b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53400/0 (socket says 172.21.15.129:53400) 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f962ffff640 1 -- 172.21.15.129:0/1030696003 learned_addr learned my addr 172.21.15.129:0/1030696003 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f963659c640 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80063b730 0x55b800a19070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f962ffff640 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b800a18b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f962ffff640 1 -- 172.21.15.129:0/1030696003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80063b730 msgr2=0x55b800a19070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f962ffff640 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80063b730 0x55b800a19070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f962ffff640 1 -- 172.21.15.129:0/1030696003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 msgr2=0x55b800a18b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f962ffff640 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b800a18b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f962ffff640 1 -- 172.21.15.129:0/1030696003 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b800a7f0f0 con 0x55b80065c410 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f962ffff640 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b800a195b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9620004650 tx=0x7f9620024040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf5035535897e9c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f961dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.001+0000 7f961dffb640 1 -- 172.21.15.129:0/1030696003 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9620031060 con 0x55b80065c410 2024-05-20T18:06:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f961dffb640 1 -- 172.21.15.129:0/1030696003 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f9620003cc0 con 0x55b80065c410 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f9643bab740 1 -- 172.21.15.129:0/1030696003 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f962800fbe0 con 0x55b80065c410 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f961dffb640 1 -- 172.21.15.129:0/1030696003 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9620003ec0 con 0x55b80065c410 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f9643bab740 10 client.16587 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f9643bab740 1 -- 172.21.15.129:0/1030696003 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b800a802c0 con 0x55b80065c410 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f961dffb640 1 -- 172.21.15.129:0/1030696003 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f962800fbe0 con 0x55b80065c410 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f961dffb640 1 -- 172.21.15.129:0/1030696003 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f96200123f0 con 0x55b80065c410 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f9643bab740 10 client.16587 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f9643bab740 10 client.16587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f9643bab740 20 client.16587 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f9643bab740 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b8005f0630 0x55b800a82a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f9643bab740 4 client.16587 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f9643bab740 1 -- 172.21.15.129:0/1030696003 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b800968410 con 0x55b8005f0630 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.002+0000 7f963659c640 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b8005f0630 0x55b800a82a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.003+0000 7f961dffb640 10 client.16587 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:23.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.003+0000 7f963659c640 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b8005f0630 0x55b800a82a00 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce233afc9d61a66c server_cookie=e5e781a8ec48b3d0 in_seq=0 out_seq=0 2024-05-20T18:06:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.004+0000 7f961dffb640 1 -- 172.21.15.129:0/1030696003 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b800968410 con 0x55b8005f0630 2024-05-20T18:06:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.004+0000 7f961dffb640 10 client.16587 handle_command_reply: tid=0 2024-05-20T18:06:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.004+0000 7f9643bab740 1 client.16587 shutdown 2024-05-20T18:06:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.004+0000 7f9643bab740 1 -- 172.21.15.129:0/1030696003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 msgr2=0x55b800a195b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.004+0000 7f9643bab740 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b800a195b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9620004650 tx=0x7f9620024040 comp rx=0 tx=0).stop 2024-05-20T18:06:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.004+0000 7f9643bab740 1 -- 172.21.15.129:0/1030696003 shutdown_connections 2024-05-20T18:06:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f9643bab740 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b8005f0630 0x55b800a82a00 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f9643bab740 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b80065c410 0x55b800a195b0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f9643bab740 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b80063b730 0x55b800a19070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f9643bab740 1 --2- 172.21.15.129:0/1030696003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b800633550 0x55b800a18b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f9643bab740 1 -- 172.21.15.129:0/1030696003 >> 172.21.15.129:0/1030696003 conn(0x55b800650560 msgr2=0x55b8009fb490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f961dffb640 0 client.16587 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:23.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f9643bab740 1 -- 172.21.15.129:0/1030696003 shutdown_connections 2024-05-20T18:06:23.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f9643bab740 1 -- 172.21.15.129:0/1030696003 wait complete. 2024-05-20T18:06:23.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f9643bab740 20 client.16587 trim_cache size 0 max 16384 2024-05-20T18:06:23.009 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:06:23.009 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:06:23.009 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "3d9609cb-978c-470f-a00b-1b929c6f5307", 2024-05-20T18:06:23.009 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:06:23.010 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:06:23.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 -- 172.21.15.129:0/69917108 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96240a2d00 msgr2=0x7f96240a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96240a2d00 0x7f96240a5210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f96300edef0 tx=0x7f963002d730 comp rx=0 tx=0).stop 2024-05-20T18:06:23.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 -- 172.21.15.129:0/69917108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9638108ff0 msgr2=0x7f963819e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9638108ff0 0x7f963819e000 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f96300319f0 tx=0x7f9630031a20 comp rx=0 tx=0).stop 2024-05-20T18:06:23.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 -- 172.21.15.129:0/69917108 shutdown_connections 2024-05-20T18:06:23.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f96240a2d00 0x7f96240a5210 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638108b50 0x7f9638199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9638108ff0 0x7f963819e000 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 --2- 172.21.15.129:0/69917108 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9638101790 0x7f96381997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 -- 172.21.15.129:0/69917108 >> 172.21.15.129:0/69917108 conn(0x7f9638076830 msgr2=0x7f96380fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 -- 172.21.15.129:0/69917108 shutdown_connections 2024-05-20T18:06:23.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.005+0000 7f963cd02640 1 -- 172.21.15.129:0/69917108 wait complete. 2024-05-20T18:06:23.015 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:06:23.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:06:23.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.132+0000 7f5623daa640 1 -- 172.21.15.129:0/2940360949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f561c1079d0 msgr2=0x7f561c101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.132+0000 7f5623daa640 1 --2- 172.21.15.129:0/2940360949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f561c1079d0 0x7f561c101560 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5618009900 tx=0x7f561802a410 comp rx=0 tx=0).stop 2024-05-20T18:06:23.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.132+0000 7f5623daa640 1 -- 172.21.15.129:0/2940360949 shutdown_connections 2024-05-20T18:06:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.132+0000 7f5623daa640 1 --2- 172.21.15.129:0/2940360949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f561c10c330 0x7f561c10e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.132+0000 7f5623daa640 1 --2- 172.21.15.129:0/2940360949 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f561c101aa0 0x7f561c101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.132+0000 7f5623daa640 1 --2- 172.21.15.129:0/2940360949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f561c1079d0 0x7f561c101560 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.132+0000 7f5623daa640 1 -- 172.21.15.129:0/2940360949 >> 172.21.15.129:0/2940360949 conn(0x7f561c0770b0 msgr2=0x7f561c0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.132+0000 7f5623daa640 1 -- 172.21.15.129:0/2940360949 shutdown_connections 2024-05-20T18:06:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.132+0000 7f5623daa640 1 -- 172.21.15.129:0/2940360949 wait complete. 2024-05-20T18:06:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5623daa640 1 Processor -- start 2024-05-20T18:06:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5623daa640 1 -- start start 2024-05-20T18:06:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5623daa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f561c101aa0 0x7f561c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5623daa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f561c1079d0 0x7f561c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5623daa640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f561c10c330 0x7f561c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5623daa640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f561c111ce0 con 0x7f561c101aa0 2024-05-20T18:06:23.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5623daa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f561c111b60 con 0x7f561c1079d0 2024-05-20T18:06:23.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5623daa640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f561c111e60 con 0x7f561c10c330 2024-05-20T18:06:23.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5621b46640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f561c101aa0 0x7f561c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5621b46640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f561c101aa0 0x7f561c198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53422/0 (socket says 172.21.15.129:53422) 2024-05-20T18:06:23.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5621b46640 1 -- 172.21.15.129:0/2467788380 learned_addr learned my addr 172.21.15.129:0/2467788380 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5621345640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f561c1079d0 0x7f561c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.133+0000 7f5622347640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f561c10c330 0x7f561c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f5621b46640 1 -- 172.21.15.129:0/2467788380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f561c10c330 msgr2=0x7f561c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f5621b46640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f561c10c330 0x7f561c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f5621b46640 1 -- 172.21.15.129:0/2467788380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f561c1079d0 msgr2=0x7f561c198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f5621b46640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f561c1079d0 0x7f561c198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f5621b46640 1 -- 172.21.15.129:0/2467788380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c19d580 con 0x7f561c101aa0 2024-05-20T18:06:23.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f5621345640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f561c1079d0 0x7f561c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:23.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f5621b46640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f561c101aa0 0x7f561c198720 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5618005940 tx=0x7f5618004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f560effd640 1 -- 172.21.15.129:0/2467788380 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561803f030 con 0x7f561c101aa0 2024-05-20T18:06:23.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f560effd640 1 -- 172.21.15.129:0/2467788380 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5618003ad0 con 0x7f561c101aa0 2024-05-20T18:06:23.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f5623daa640 1 -- 172.21.15.129:0/2467788380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561c19d810 con 0x7f561c101aa0 2024-05-20T18:06:23.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f560effd640 1 -- 172.21.15.129:0/2467788380 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618035070 con 0x7f561c101aa0 2024-05-20T18:06:23.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.134+0000 7f5623daa640 1 -- 172.21.15.129:0/2467788380 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561c19dd00 con 0x7f561c101aa0 2024-05-20T18:06:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.137+0000 7f560effd640 1 -- 172.21.15.129:0/2467788380 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5618042030 con 0x7f561c101aa0 2024-05-20T18:06:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.137+0000 7f562ac4b740 1 Processor -- start 2024-05-20T18:06:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.137+0000 7f562ac4b740 1 -- start start 2024-05-20T18:06:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.137+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d81390 0x55f22811fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.137+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f228120270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.137+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d606b0 0x55f2281207b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.137+0000 7f562ac4b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f228085d00 con 0x55f227d584d0 2024-05-20T18:06:23.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.137+0000 7f562ac4b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f22802d4c0 con 0x55f227d81390 2024-05-20T18:06:23.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.137+0000 7f562ac4b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f228081cc0 con 0x55f227d606b0 2024-05-20T18:06:23.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5621b46640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f228120270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5621b46640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f228120270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53438/0 (socket says 172.21.15.129:53438) 2024-05-20T18:06:23.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5621b46640 1 -- 172.21.15.129:0/4214381858 learned_addr learned my addr 172.21.15.129:0/4214381858 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5621345640 1 --2- 172.21.15.129:0/4214381858 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d606b0 0x55f2281207b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5622347640 1 --2- 172.21.15.129:0/4214381858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d81390 0x55f22811fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f560effd640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56080a2d00 0x7f56080a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5622347640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56080a2d00 0x7f56080a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f560effd640 1 -- 172.21.15.129:0/2467788380 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f56180efcc0 con 0x7f561c101aa0 2024-05-20T18:06:23.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5622347640 1 -- 172.21.15.129:0/4214381858 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d606b0 msgr2=0x55f2281207b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5622347640 1 --2- 172.21.15.129:0/4214381858 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d606b0 0x55f2281207b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5622347640 1 -- 172.21.15.129:0/4214381858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 msgr2=0x55f228120270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5622347640 1 --2- 172.21.15.129:0/4214381858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f228120270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5622347640 1 -- 172.21.15.129:0/4214381858 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f227e637d0 con 0x55f227d81390 2024-05-20T18:06:23.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5621b46640 1 --2- 172.21.15.129:0/4214381858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f228120270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:06:23.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5622347640 1 --2- 172.21.15.129:0/4214381858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d81390 0x55f22811fd30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5614005fd0 tx=0x7f5614012de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d4cb8dd1ac8e35d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5622347640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56080a2d00 0x7f56080a5210 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f561401b000 tx=0x7f561401f000 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.138+0000 7f5607fff640 1 -- 172.21.15.129:0/4214381858 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614003b70 con 0x55f227d81390 2024-05-20T18:06:23.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.139+0000 7f5607fff640 1 -- 172.21.15.129:0/4214381858 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5614003d10 con 0x55f227d81390 2024-05-20T18:06:23.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.139+0000 7f5607fff640 1 -- 172.21.15.129:0/4214381858 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614004600 con 0x55f227d81390 2024-05-20T18:06:23.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.139+0000 7f562ac4b740 1 -- 172.21.15.129:0/4214381858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d81390 msgr2=0x55f22811fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.139+0000 7f562ac4b740 1 --2- 172.21.15.129:0/4214381858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d81390 0x55f22811fd30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5614005fd0 tx=0x7f5614012de0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.139+0000 7f562ac4b740 1 -- 172.21.15.129:0/4214381858 shutdown_connections 2024-05-20T18:06:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.139+0000 7f562ac4b740 1 --2- 172.21.15.129:0/4214381858 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d606b0 0x55f2281207b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.139+0000 7f562ac4b740 1 --2- 172.21.15.129:0/4214381858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f228120270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.139+0000 7f562ac4b740 1 --2- 172.21.15.129:0/4214381858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d81390 0x55f22811fd30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.139+0000 7f562ac4b740 1 -- 172.21.15.129:0/4214381858 >> 172.21.15.129:0/4214381858 conn(0x55f227d754e0 msgr2=0x55f22811c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.140+0000 7f562ac4b740 1 -- 172.21.15.129:0/4214381858 shutdown_connections 2024-05-20T18:06:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.140+0000 7f562ac4b740 1 -- 172.21.15.129:0/4214381858 wait complete. 2024-05-20T18:06:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.140+0000 7f562ac4b740 1 -- start start 2024-05-20T18:06:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.140+0000 7f562ac4b740 1 Processor -- start 2024-05-20T18:06:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.140+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f22812be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d606b0 0x55f22812c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d81390 0x55f22812a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f562ac4b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f22812cfc0 con 0x55f227d584d0 2024-05-20T18:06:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f562ac4b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f228040b90 con 0x55f227d81390 2024-05-20T18:06:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f562ac4b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f22812d140 con 0x55f227d606b0 2024-05-20T18:06:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f562ac4b740 10 client.0 _ll_register_callbacks cb 0x55f22807cda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5621b46640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f22812be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5621b46640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f22812be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53448/0 (socket says 172.21.15.129:53448) 2024-05-20T18:06:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5621b46640 1 -- 172.21.15.129:0/1659956718 learned_addr learned my addr 172.21.15.129:0/1659956718 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5622347640 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d81390 0x55f22812a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5621345640 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d606b0 0x55f22812c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5621b46640 1 -- 172.21.15.129:0/1659956718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d606b0 msgr2=0x55f22812c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5621b46640 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d606b0 0x55f22812c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5621b46640 1 -- 172.21.15.129:0/1659956718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d81390 msgr2=0x55f22812a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5621b46640 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d81390 0x55f22812a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5621b46640 1 -- 172.21.15.129:0/1659956718 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f227d993f0 con 0x55f227d584d0 2024-05-20T18:06:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5621345640 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d606b0 0x55f22812c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:06:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f5621b46640 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f22812be70 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f56180ed480 tx=0x7f56180bc000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f9ebbdc9e1cc2 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f56057fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f56057fa640 1 -- 172.21.15.129:0/1659956718 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56180f1070 con 0x55f227d584d0 2024-05-20T18:06:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f562ac4b740 1 -- 172.21.15.129:0/1659956718 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5614003d10 con 0x55f227d584d0 2024-05-20T18:06:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f56057fa640 1 -- 172.21.15.129:0/1659956718 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f56180ed650 con 0x55f227d584d0 2024-05-20T18:06:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f56057fa640 1 -- 172.21.15.129:0/1659956718 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5618049070 con 0x55f227d584d0 2024-05-20T18:06:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f562ac4b740 10 client.16599 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.141+0000 7f562ac4b740 1 -- 172.21.15.129:0/1659956718 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f227d590f0 con 0x55f227d584d0 2024-05-20T18:06:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.142+0000 7f56057fa640 1 -- 172.21.15.129:0/1659956718 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5614003d10 con 0x55f227d584d0 2024-05-20T18:06:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.142+0000 7f56057fa640 1 -- 172.21.15.129:0/1659956718 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f56180b8810 con 0x55f227d584d0 2024-05-20T18:06:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.142+0000 7f562ac4b740 10 client.16599 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.142+0000 7f562ac4b740 10 client.16599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.142+0000 7f562ac4b740 20 client.16599 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.142+0000 7f562ac4b740 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f227d3ff60 0x55f22813cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.142+0000 7f562ac4b740 4 client.16599 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.142+0000 7f562ac4b740 1 -- 172.21.15.129:0/1659956718 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f228093430 con 0x55f227d3ff60 2024-05-20T18:06:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.142+0000 7f5621345640 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f227d3ff60 0x55f22813cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.143+0000 7f56057fa640 10 client.16599 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.143+0000 7f5621345640 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f227d3ff60 0x55f22813cb00 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa98ec65b1305aac server_cookie=7dd3339b727746c2 in_seq=0 out_seq=0 2024-05-20T18:06:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.143+0000 7f56057fa640 1 -- 172.21.15.129:0/1659956718 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f228093430 con 0x55f227d3ff60 2024-05-20T18:06:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.143+0000 7f56057fa640 10 client.16599 handle_command_reply: tid=0 2024-05-20T18:06:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.143+0000 7f562ac4b740 1 client.16599 shutdown 2024-05-20T18:06:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f562ac4b740 1 -- 172.21.15.129:0/1659956718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 msgr2=0x55f22812be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f562ac4b740 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f22812be70 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f56180ed480 tx=0x7f56180bc000 comp rx=0 tx=0).stop 2024-05-20T18:06:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f562ac4b740 1 -- 172.21.15.129:0/1659956718 shutdown_connections 2024-05-20T18:06:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f562ac4b740 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f227d3ff60 0x55f22813cb00 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f562ac4b740 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d81390 0x55f22812a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f562ac4b740 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d606b0 0x55f22812c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f562ac4b740 1 --2- 172.21.15.129:0/1659956718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d584d0 0x55f22812be70 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f56057fa640 0 client.16599 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f562ac4b740 1 -- 172.21.15.129:0/1659956718 >> 172.21.15.129:0/1659956718 conn(0x55f227d754e0 msgr2=0x55f228121a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f562ac4b740 1 -- 172.21.15.129:0/1659956718 shutdown_connections 2024-05-20T18:06:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f562ac4b740 1 -- 172.21.15.129:0/1659956718 wait complete. 2024-05-20T18:06:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.144+0000 7f562ac4b740 20 client.16599 trim_cache size 0 max 16384 2024-05-20T18:06:23.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.161+0000 7f562ac4b740 1 Processor -- start 2024-05-20T18:06:23.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.161+0000 7f562ac4b740 1 -- start start 2024-05-20T18:06:23.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d584d0 0x55f2281292c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d606b0 0x55f22812c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d81390 0x55f2281422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f562ac4b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1aebc0 con 0x55f227d606b0 2024-05-20T18:06:23.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f562ac4b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1b1420 con 0x55f227d584d0 2024-05-20T18:06:23.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f562ac4b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1af210 con 0x55f227d81390 2024-05-20T18:06:23.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5621345640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d606b0 0x55f22812c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5621345640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d606b0 0x55f22812c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53460/0 (socket says 172.21.15.129:53460) 2024-05-20T18:06:23.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5621345640 1 -- 172.21.15.129:0/4039680179 learned_addr learned my addr 172.21.15.129:0/4039680179 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5621b46640 1 --2- 172.21.15.129:0/4039680179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d584d0 0x55f2281292c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5621345640 1 -- 172.21.15.129:0/4039680179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d81390 msgr2=0x55f2281422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5622347640 1 --2- 172.21.15.129:0/4039680179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d81390 0x55f2281422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5621345640 1 --2- 172.21.15.129:0/4039680179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d81390 0x55f2281422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5621345640 1 -- 172.21.15.129:0/4039680179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d584d0 msgr2=0x55f2281292c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5621345640 1 --2- 172.21.15.129:0/4039680179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d584d0 0x55f2281292c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5621345640 1 -- 172.21.15.129:0/4039680179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2281288d0 con 0x55f227d606b0 2024-05-20T18:06:23.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5622347640 1 --2- 172.21.15.129:0/4039680179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d81390 0x55f2281422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:06:23.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5621b46640 1 --2- 172.21.15.129:0/4039680179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d584d0 0x55f2281292c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:06:23.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f5621345640 1 --2- 172.21.15.129:0/4039680179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d606b0 0x55f22812c140 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f561c074f80 tx=0x7f561000b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4301e2cd55cd817 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f56077fe640 1 -- 172.21.15.129:0/4039680179 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561001f020 con 0x55f227d606b0 2024-05-20T18:06:23.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f56077fe640 1 -- 172.21.15.129:0/4039680179 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56100042f0 con 0x55f227d606b0 2024-05-20T18:06:23.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.162+0000 7f56077fe640 1 -- 172.21.15.129:0/4039680179 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5610003b50 con 0x55f227d606b0 2024-05-20T18:06:23.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.163+0000 7f562ac4b740 1 -- 172.21.15.129:0/4039680179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d606b0 msgr2=0x55f22812c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.163+0000 7f562ac4b740 1 --2- 172.21.15.129:0/4039680179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d606b0 0x55f22812c140 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f561c074f80 tx=0x7f561000b5a0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.163+0000 7f562ac4b740 1 -- 172.21.15.129:0/4039680179 shutdown_connections 2024-05-20T18:06:23.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.163+0000 7f562ac4b740 1 --2- 172.21.15.129:0/4039680179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d81390 0x55f2281422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.163+0000 7f562ac4b740 1 --2- 172.21.15.129:0/4039680179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d606b0 0x55f22812c140 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.163+0000 7f562ac4b740 1 --2- 172.21.15.129:0/4039680179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d584d0 0x55f2281292c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.163+0000 7f562ac4b740 1 -- 172.21.15.129:0/4039680179 >> 172.21.15.129:0/4039680179 conn(0x55f227d754e0 msgr2=0x55f22812ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.163+0000 7f562ac4b740 1 -- 172.21.15.129:0/4039680179 shutdown_connections 2024-05-20T18:06:23.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.163+0000 7f562ac4b740 1 -- 172.21.15.129:0/4039680179 wait complete. 2024-05-20T18:06:23.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.163+0000 7f562ac4b740 1 -- start start 2024-05-20T18:06:23.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f562ac4b740 1 Processor -- start 2024-05-20T18:06:23.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d584d0 0x55f228132360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d606b0 0x55f2281328a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f562ac4b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d81390 0x55f228132de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f562ac4b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1af8e0 con 0x55f227d81390 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f562ac4b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1b0d50 con 0x55f227d606b0 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f562ac4b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f561c1affb0 con 0x55f227d584d0 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f562ac4b740 10 client.0 _ll_register_callbacks cb 0x7f5618005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f5622347640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d81390 0x55f228132de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f5622347640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d81390 0x55f228132de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53476/0 (socket says 172.21.15.129:53476) 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f5622347640 1 -- 172.21.15.129:0/420398301 learned_addr learned my addr 172.21.15.129:0/420398301 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f5621345640 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d606b0 0x55f2281328a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.164+0000 7f5621b46640 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d584d0 0x55f228132360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f5622347640 1 -- 172.21.15.129:0/420398301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d584d0 msgr2=0x55f228132360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f5622347640 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d584d0 0x55f228132360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f5622347640 1 -- 172.21.15.129:0/420398301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d606b0 msgr2=0x55f2281328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f5622347640 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d606b0 0x55f2281328a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f5622347640 1 -- 172.21.15.129:0/420398301 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2281a3e70 con 0x55f227d81390 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f5621b46640 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d584d0 0x55f228132360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f5622347640 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d81390 0x55f228132de0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5614005fd0 tx=0x7f5614004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591b1a49204a97bf server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f56057fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f56057fa640 1 -- 172.21.15.129:0/420398301 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561401ea20 con 0x55f227d81390 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f56057fa640 1 -- 172.21.15.129:0/420398301 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f561401ebc0 con 0x55f227d81390 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f562ac4b740 1 -- 172.21.15.129:0/420398301 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56100042f0 con 0x55f227d81390 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f56057fa640 1 -- 172.21.15.129:0/420398301 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5614012510 con 0x55f227d81390 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f562ac4b740 10 client.16605 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f562ac4b740 1 -- 172.21.15.129:0/420398301 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f2281a4e00 con 0x55f227d81390 2024-05-20T18:06:23.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f56057fa640 1 -- 172.21.15.129:0/420398301 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56100042f0 con 0x55f227d81390 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f56057fa640 1 -- 172.21.15.129:0/420398301 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f5614027030 con 0x55f227d81390 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f562ac4b740 10 client.16605 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f562ac4b740 10 client.16605 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.165+0000 7f562ac4b740 20 client.16605 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f562ac4b740 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f227d153a0 0x55f2281a7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f562ac4b740 4 client.16605 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f562ac4b740 1 -- 172.21.15.129:0/420398301 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f22808d780 con 0x55f227d153a0 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f5621b46640 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f227d153a0 0x55f2281a7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f56057fa640 10 client.16605 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f5621b46640 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f227d153a0 0x55f2281a7480 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c17391840efd21d server_cookie=93b9dd7d5013a1da in_seq=0 out_seq=0 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f56057fa640 1 -- 172.21.15.129:0/420398301 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f22808d780 con 0x55f227d153a0 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f56057fa640 10 client.16605 handle_command_reply: tid=0 2024-05-20T18:06:23.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f562ac4b740 1 client.16605 shutdown 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f562ac4b740 1 -- 172.21.15.129:0/420398301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d81390 msgr2=0x55f228132de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f562ac4b740 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d81390 0x55f228132de0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5614005fd0 tx=0x7f5614004280 comp rx=0 tx=0).stop 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f562ac4b740 1 -- 172.21.15.129:0/420398301 shutdown_connections 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f562ac4b740 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f227d153a0 0x55f2281a7480 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f562ac4b740 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f227d606b0 0x55f2281328a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f56057fa640 0 client.16605 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.166+0000 7f562ac4b740 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f227d81390 0x55f228132de0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f562ac4b740 1 --2- 172.21.15.129:0/420398301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f227d584d0 0x55f228132360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f562ac4b740 1 -- 172.21.15.129:0/420398301 >> 172.21.15.129:0/420398301 conn(0x55f227d754e0 msgr2=0x55f22812b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f562ac4b740 1 -- 172.21.15.129:0/420398301 shutdown_connections 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f562ac4b740 1 -- 172.21.15.129:0/420398301 wait complete. 2024-05-20T18:06:23.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f562ac4b740 20 client.16605 trim_cache size 0 max 16384 2024-05-20T18:06:23.188 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:06:23.188 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:06:23.188 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:06:23.188 INFO:teuthology.orchestra.run.smithi129.stdout: "3d9609cb-978c-470f-a00b-1b929c6f5307": { 2024-05-20T18:06:23.188 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:06:23.188 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "3d9609cb-978c-470f-a00b-1b929c6f5307", 2024-05-20T18:06:23.188 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:06:23.188 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:06:23.188 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:06:23.188 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 -- 172.21.15.129:0/2467788380 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56080a2d00 msgr2=0x7f56080a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56080a2d00 0x7f56080a5210 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f561401b000 tx=0x7f561401f000 comp rx=0 tx=0).stop 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 -- 172.21.15.129:0/2467788380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f561c101aa0 msgr2=0x7f561c198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f561c101aa0 0x7f561c198720 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5618005940 tx=0x7f5618004270 comp rx=0 tx=0).stop 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 -- 172.21.15.129:0/2467788380 shutdown_connections 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56080a2d00 0x7f56080a5210 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f561c1079d0 0x7f561c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f561c10c330 0x7f561c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 --2- 172.21.15.129:0/2467788380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f561c101aa0 0x7f561c198720 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 -- 172.21.15.129:0/2467788380 >> 172.21.15.129:0/2467788380 conn(0x7f561c0770b0 msgr2=0x7f561c103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 -- 172.21.15.129:0/2467788380 shutdown_connections 2024-05-20T18:06:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:23.167+0000 7f5623daa640 1 -- 172.21.15.129:0/2467788380 wait complete. 2024-05-20T18:06:23.191 INFO:tasks.cephfs.filesystem:scrub status for tag:3d9609cb-978c-470f-a00b-1b929c6f5307 - {'path': '/', 'tag': '3d9609cb-978c-470f-a00b-1b929c6f5307', 'options': 'recursive,force'} 2024-05-20T18:06:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.469+0000 7f41369e3640 1 -- 172.21.15.129:0/668184078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130108b00 msgr2=0x7f41301026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.469+0000 7f41369e3640 1 --2- 172.21.15.129:0/668184078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130108b00 0x7f41301026a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4120009900 tx=0x7f412002a410 comp rx=0 tx=0).stop 2024-05-20T18:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.469+0000 7f41369e3640 1 -- 172.21.15.129:0/668184078 shutdown_connections 2024-05-20T18:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.469+0000 7f41369e3640 1 --2- 172.21.15.129:0/668184078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4130108fa0 0x7f4130102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.469+0000 7f41369e3640 1 --2- 172.21.15.129:0/668184078 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4130103250 0x7f413010f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.469+0000 7f41369e3640 1 --2- 172.21.15.129:0/668184078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130108b00 0x7f41301026a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.469+0000 7f41369e3640 1 -- 172.21.15.129:0/668184078 >> 172.21.15.129:0/668184078 conn(0x7f41300fe1e0 msgr2=0x7f4130100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.469+0000 7f41369e3640 1 -- 172.21.15.129:0/668184078 shutdown_connections 2024-05-20T18:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.469+0000 7f41369e3640 1 -- 172.21.15.129:0/668184078 wait complete. 2024-05-20T18:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f41369e3640 1 Processor -- start 2024-05-20T18:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f41369e3640 1 -- start start 2024-05-20T18:06:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f41369e3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103250 0x7f413019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f41369e3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4130108b00 0x7f413019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f41369e3640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4130108fa0 0x7f41301996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f41369e3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4130112e40 con 0x7f4130103250 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f41369e3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4130112cc0 con 0x7f4130108b00 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f41369e3640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4130112fc0 con 0x7f4130108fa0 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f412ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103250 0x7f413019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f412ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103250 0x7f413019f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53482/0 (socket says 172.21.15.129:53482) 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f412ffff640 1 -- 172.21.15.129:0/1594148658 learned_addr learned my addr 172.21.15.129:0/1594148658 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f412f7fe640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4130108b00 0x7f413019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.470+0000 7f4134f80640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4130108fa0 0x7f41301996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f412ffff640 1 -- 172.21.15.129:0/1594148658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4130108fa0 msgr2=0x7f41301996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f412ffff640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4130108fa0 0x7f41301996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f412ffff640 1 -- 172.21.15.129:0/1594148658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4130108b00 msgr2=0x7f413019fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f412ffff640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4130108b00 0x7f413019fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f412ffff640 1 -- 172.21.15.129:0/1594148658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4130199dd0 con 0x7f4130103250 2024-05-20T18:06:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f412f7fe640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4130108b00 0x7f413019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f4134f80640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4130108fa0 0x7f41301996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:06:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f412ffff640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103250 0x7f413019f520 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4120005940 tx=0x7f41200040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f412d7fa640 1 -- 172.21.15.129:0/1594148658 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412003f030 con 0x7f4130103250 2024-05-20T18:06:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f41369e3640 1 -- 172.21.15.129:0/1594148658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f413019a000 con 0x7f4130103250 2024-05-20T18:06:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f412d7fa640 1 -- 172.21.15.129:0/1594148658 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41200044c0 con 0x7f4130103250 2024-05-20T18:06:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f412d7fa640 1 -- 172.21.15.129:0/1594148658 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120035070 con 0x7f4130103250 2024-05-20T18:06:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.471+0000 7f41369e3640 1 -- 172.21.15.129:0/1594148658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41301a62c0 con 0x7f4130103250 2024-05-20T18:06:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.474+0000 7f412d7fa640 1 -- 172.21.15.129:0/1594148658 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f4120042030 con 0x7f4130103250 2024-05-20T18:06:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.474+0000 7f413d967740 1 Processor -- start 2024-05-20T18:06:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.474+0000 7f413d967740 1 -- start start 2024-05-20T18:06:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.474+0000 7f413d967740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452265d1b0 0x5645229fc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.474+0000 7f413d967740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x5645229fc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.474+0000 7f413d967740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x5645229fcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.474+0000 7f413d967740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564522961340 con 0x56452263c4d0 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.474+0000 7f413d967740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564522909290 con 0x5645226342f0 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.474+0000 7f413d967740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56452295d300 con 0x56452265d1b0 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x5645229fcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x5645229fcb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53486/0 (socket says 172.21.15.129:53486) 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412f7fe640 1 -- 172.21.15.129:0/4094870733 learned_addr learned my addr 172.21.15.129:0/4094870733 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412ffff640 1 --2- 172.21.15.129:0/4094870733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x5645229fc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f4134f80640 1 --2- 172.21.15.129:0/4094870733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452265d1b0 0x5645229fc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412f7fe640 1 -- 172.21.15.129:0/4094870733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452265d1b0 msgr2=0x5645229fc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412d7fa640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41180a2cf0 0x7f41180a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412f7fe640 1 --2- 172.21.15.129:0/4094870733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452265d1b0 0x5645229fc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412f7fe640 1 -- 172.21.15.129:0/4094870733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 msgr2=0x5645229fc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412f7fe640 1 --2- 172.21.15.129:0/4094870733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x5645229fc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412f7fe640 1 -- 172.21.15.129:0/4094870733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56452292d8e0 con 0x56452263c4d0 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412ffff640 1 --2- 172.21.15.129:0/4094870733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x5645229fc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f4134f80640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41180a2cf0 0x7f41180a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412d7fa640 1 -- 172.21.15.129:0/1594148658 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f41200ef9f0 con 0x7f4130103250 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f412f7fe640 1 --2- 172.21.15.129:0/4094870733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x5645229fcb00 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4124009bb0 tx=0x7f41240117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb78c7623f1bb7d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f41167fc640 1 -- 172.21.15.129:0/4094870733 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124004050 con 0x56452263c4d0 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f41167fc640 1 -- 172.21.15.129:0/4094870733 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4124016070 con 0x56452263c4d0 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.475+0000 7f41167fc640 1 -- 172.21.15.129:0/4094870733 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124004ad0 con 0x56452263c4d0 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.476+0000 7f413d967740 1 -- 172.21.15.129:0/4094870733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 msgr2=0x5645229fcb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.476+0000 7f4134f80640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41180a2cf0 0x7f41180a5200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x5645229f91f0 tx=0x7f4128004a40 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.476+0000 7f413d967740 1 --2- 172.21.15.129:0/4094870733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x5645229fcb00 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4124009bb0 tx=0x7f41240117b0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.476+0000 7f413d967740 1 -- 172.21.15.129:0/4094870733 shutdown_connections 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.476+0000 7f413d967740 1 --2- 172.21.15.129:0/4094870733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x5645229fc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.476+0000 7f413d967740 1 --2- 172.21.15.129:0/4094870733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x5645229fcb00 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.476+0000 7f413d967740 1 --2- 172.21.15.129:0/4094870733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452265d1b0 0x5645229fc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.476+0000 7f413d967740 1 -- 172.21.15.129:0/4094870733 >> 172.21.15.129:0/4094870733 conn(0x564522651300 msgr2=0x5645229f8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.476+0000 7f413d967740 1 -- 172.21.15.129:0/4094870733 shutdown_connections 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.476+0000 7f413d967740 1 -- 172.21.15.129:0/4094870733 wait complete. 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f413d967740 1 -- start start 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f413d967740 1 Processor -- start 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f413d967740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645226342f0 0x564522a08220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f413d967740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452263c4d0 0x564522a08760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f413d967740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56452265d1b0 0x564522a06630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f413d967740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564522a091b0 con 0x5645226342f0 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f413d967740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56452291c790 con 0x56452265d1b0 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f413d967740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564522a09330 con 0x56452263c4d0 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f413d967740 10 client.0 _ll_register_callbacks cb 0x564522957a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f412ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645226342f0 0x564522a08220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f412ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645226342f0 0x564522a08220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53488/0 (socket says 172.21.15.129:53488) 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f412ffff640 1 -- 172.21.15.129:0/3292211828 learned_addr learned my addr 172.21.15.129:0/3292211828 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f4134f80640 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56452265d1b0 0x564522a06630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.477+0000 7f412f7fe640 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452263c4d0 0x564522a08760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f412ffff640 1 -- 172.21.15.129:0/3292211828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452263c4d0 msgr2=0x564522a08760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f412ffff640 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452263c4d0 0x564522a08760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f412ffff640 1 -- 172.21.15.129:0/3292211828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56452265d1b0 msgr2=0x564522a06630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f412ffff640 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56452265d1b0 0x564522a06630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f412ffff640 1 -- 172.21.15.129:0/3292211828 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564522675210 con 0x5645226342f0 2024-05-20T18:06:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f412ffff640 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645226342f0 0x564522a08220 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4120000c00 tx=0x7f41200bbd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26eecffb75afac1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f410f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f410f7fe640 1 -- 172.21.15.129:0/3292211828 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120049050 con 0x5645226342f0 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f410f7fe640 1 -- 172.21.15.129:0/3292211828 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f41200b8540 con 0x5645226342f0 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f413d967740 1 -- 172.21.15.129:0/3292211828 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4124016070 con 0x5645226342f0 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f410f7fe640 1 -- 172.21.15.129:0/3292211828 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f412003b460 con 0x5645226342f0 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f413d967740 10 client.16617 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f413d967740 1 -- 172.21.15.129:0/3292211828 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564522741620 con 0x5645226342f0 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f410f7fe640 1 -- 172.21.15.129:0/3292211828 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4124016070 con 0x5645226342f0 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f410f7fe640 1 -- 172.21.15.129:0/3292211828 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f41200bc130 con 0x5645226342f0 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f413d967740 10 client.16617 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f413d967740 10 client.16617 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f413d967740 20 client.16617 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:23.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f413d967740 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56452261bd80 0x564522a189f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f413d967740 4 client.16617 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.478+0000 7f413d967740 1 -- 172.21.15.129:0/3292211828 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56452296ea70 con 0x56452261bd80 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.479+0000 7f412f7fe640 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56452261bd80 0x564522a189f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.479+0000 7f412f7fe640 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56452261bd80 0x564522a189f0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=170fbc63b7f5c5d server_cookie=5a1ed11cd1ebac6c in_seq=0 out_seq=0 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.479+0000 7f410f7fe640 10 client.16617 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f410f7fe640 1 -- 172.21.15.129:0/3292211828 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56452296ea70 con 0x56452261bd80 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f410f7fe640 10 client.16617 handle_command_reply: tid=0 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f413d967740 1 client.16617 shutdown 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f413d967740 1 -- 172.21.15.129:0/3292211828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645226342f0 msgr2=0x564522a08220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f413d967740 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645226342f0 0x564522a08220 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4120000c00 tx=0x7f41200bbd60 comp rx=0 tx=0).stop 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f413d967740 1 -- 172.21.15.129:0/3292211828 shutdown_connections 2024-05-20T18:06:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f413d967740 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56452261bd80 0x564522a189f0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f413d967740 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56452265d1b0 0x564522a06630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f410f7fe640 0 client.16617 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f413d967740 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452263c4d0 0x564522a08760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f413d967740 1 --2- 172.21.15.129:0/3292211828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5645226342f0 0x564522a08220 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f413d967740 1 -- 172.21.15.129:0/3292211828 >> 172.21.15.129:0/3292211828 conn(0x564522651300 msgr2=0x5645229fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f413d967740 1 -- 172.21.15.129:0/3292211828 shutdown_connections 2024-05-20T18:06:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.480+0000 7f413d967740 1 -- 172.21.15.129:0/3292211828 wait complete. 2024-05-20T18:06:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.481+0000 7f413d967740 20 client.16617 trim_cache size 0 max 16384 2024-05-20T18:06:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.495+0000 7f413d967740 1 Processor -- start 2024-05-20T18:06:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.495+0000 7f413d967740 1 -- start start 2024-05-20T18:06:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f413d967740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452261bd80 0x5645229fe3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f413d967740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x564522a1e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f413d967740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a1e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f413d967740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41301acb70 con 0x56452263c4d0 2024-05-20T18:06:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f413d967740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41301af3d0 con 0x5645226342f0 2024-05-20T18:06:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f413d967740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f41301ad1c0 con 0x56452261bd80 2024-05-20T18:06:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f4134f80640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a1e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f4134f80640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a1e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53496/0 (socket says 172.21.15.129:53496) 2024-05-20T18:06:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f4134f80640 1 -- 172.21.15.129:0/4167308323 learned_addr learned my addr 172.21.15.129:0/4167308323 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f412f7fe640 1 --2- 172.21.15.129:0/4167308323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x564522a1e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f412ffff640 1 --2- 172.21.15.129:0/4167308323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452261bd80 0x5645229fe3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f4134f80640 1 -- 172.21.15.129:0/4167308323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452261bd80 msgr2=0x5645229fe3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f4134f80640 1 --2- 172.21.15.129:0/4167308323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452261bd80 0x5645229fe3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f4134f80640 1 -- 172.21.15.129:0/4167308323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 msgr2=0x564522a1e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f4134f80640 1 --2- 172.21.15.129:0/4167308323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x564522a1e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f4134f80640 1 -- 172.21.15.129:0/4167308323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564522a07180 con 0x56452263c4d0 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f412f7fe640 1 --2- 172.21.15.129:0/4167308323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x564522a1e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f4134f80640 1 --2- 172.21.15.129:0/4167308323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a1e730 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f412800f7c0 tx=0x7f4128015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734dda8c2b588efc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.496+0000 7f4115ffb640 1 -- 172.21.15.129:0/4167308323 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412802b070 con 0x56452263c4d0 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.497+0000 7f4115ffb640 1 -- 172.21.15.129:0/4167308323 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f412801ae00 con 0x56452263c4d0 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.497+0000 7f4115ffb640 1 -- 172.21.15.129:0/4167308323 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128026d50 con 0x56452263c4d0 2024-05-20T18:06:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.497+0000 7f413d967740 1 -- 172.21.15.129:0/4167308323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 msgr2=0x564522a1e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.497+0000 7f413d967740 1 --2- 172.21.15.129:0/4167308323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a1e730 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f412800f7c0 tx=0x7f4128015000 comp rx=0 tx=0).stop 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.497+0000 7f413d967740 1 -- 172.21.15.129:0/4167308323 shutdown_connections 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.497+0000 7f413d967740 1 --2- 172.21.15.129:0/4167308323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x564522a1e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.497+0000 7f413d967740 1 --2- 172.21.15.129:0/4167308323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a1e730 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.497+0000 7f413d967740 1 --2- 172.21.15.129:0/4167308323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452261bd80 0x5645229fe3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.497+0000 7f413d967740 1 -- 172.21.15.129:0/4167308323 >> 172.21.15.129:0/4167308323 conn(0x564522651300 msgr2=0x5645229fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.497+0000 7f413d967740 1 -- 172.21.15.129:0/4167308323 shutdown_connections 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.497+0000 7f413d967740 1 -- 172.21.15.129:0/4167308323 wait complete. 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f413d967740 1 -- start start 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f413d967740 1 Processor -- start 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f413d967740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452261bd80 0x564522a0e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f413d967740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x564522a0bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f413d967740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a0c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f413d967740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41301ad890 con 0x56452263c4d0 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f413d967740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41301aed00 con 0x5645226342f0 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f413d967740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f41301adf60 con 0x56452261bd80 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f413d967740 10 client.0 _ll_register_callbacks cb 0x7f4120005e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f4134f80640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a0c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f4134f80640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a0c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53510/0 (socket says 172.21.15.129:53510) 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f4134f80640 1 -- 172.21.15.129:0/3487331553 learned_addr learned my addr 172.21.15.129:0/3487331553 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f412ffff640 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452261bd80 0x564522a0e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.498+0000 7f412f7fe640 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x564522a0bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f4134f80640 1 -- 172.21.15.129:0/3487331553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452261bd80 msgr2=0x564522a0e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f4134f80640 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452261bd80 0x564522a0e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f4134f80640 1 -- 172.21.15.129:0/3487331553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 msgr2=0x564522a0bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f4134f80640 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x564522a0bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f4134f80640 1 -- 172.21.15.129:0/3487331553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564522a0c5a0 con 0x56452263c4d0 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f4134f80640 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a0c060 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4128026970 tx=0x7f412800ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af864cd8f62802f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f410f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f410f7fe640 1 -- 172.21.15.129:0/3487331553 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412802b070 con 0x56452263c4d0 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f410f7fe640 1 -- 172.21.15.129:0/3487331553 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f4128004ab0 con 0x56452263c4d0 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f413d967740 1 -- 172.21.15.129:0/3487331553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f412801ae00 con 0x56452263c4d0 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f410f7fe640 1 -- 172.21.15.129:0/3487331553 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4128004cd0 con 0x56452263c4d0 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f413d967740 10 client.16623 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f413d967740 1 -- 172.21.15.129:0/3487331553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564522a80920 con 0x56452263c4d0 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f410f7fe640 1 -- 172.21.15.129:0/3487331553 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f412801ae00 con 0x56452263c4d0 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f410f7fe640 1 -- 172.21.15.129:0/3487331553 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f4128025070 con 0x56452263c4d0 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f413d967740 10 client.16623 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.499+0000 7f413d967740 10 client.16623 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.500+0000 7f413d967740 20 client.16623 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.500+0000 7f413d967740 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56452265d1b0 0x564522a83000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.500+0000 7f413d967740 4 client.16623 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:06:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.500+0000 7f413d967740 1 -- 172.21.15.129:0/3487331553 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564522968dc0 con 0x56452265d1b0 2024-05-20T18:06:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.500+0000 7f412ffff640 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56452265d1b0 0x564522a83000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.500+0000 7f410f7fe640 10 client.16623 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.500+0000 7f412ffff640 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56452265d1b0 0x564522a83000 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0f16358ab9fc3c5 server_cookie=b9e185da22b24c85 in_seq=0 out_seq=0 2024-05-20T18:06:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.515+0000 7f410f7fe640 1 -- 172.21.15.129:0/3487331553 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564522968dc0 con 0x56452265d1b0 2024-05-20T18:06:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.515+0000 7f410f7fe640 10 client.16623 handle_command_reply: tid=0 2024-05-20T18:06:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.515+0000 7f413d967740 1 client.16623 shutdown 2024-05-20T18:06:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.515+0000 7f413d967740 1 -- 172.21.15.129:0/3487331553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 msgr2=0x564522a0c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f413d967740 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a0c060 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4128026970 tx=0x7f412800ff70 comp rx=0 tx=0).stop 2024-05-20T18:06:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f413d967740 1 -- 172.21.15.129:0/3487331553 shutdown_connections 2024-05-20T18:06:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f413d967740 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56452265d1b0 0x564522a83000 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f413d967740 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5645226342f0 0x564522a0bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f413d967740 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56452263c4d0 0x564522a0c060 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f410f7fe640 0 client.16623 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f413d967740 1 --2- 172.21.15.129:0/3487331553 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56452261bd80 0x564522a0e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f413d967740 1 -- 172.21.15.129:0/3487331553 >> 172.21.15.129:0/3487331553 conn(0x564522651300 msgr2=0x564522a07c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f413d967740 1 -- 172.21.15.129:0/3487331553 shutdown_connections 2024-05-20T18:06:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f413d967740 1 -- 172.21.15.129:0/3487331553 wait complete. 2024-05-20T18:06:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f413d967740 20 client.16623 trim_cache size 0 max 16384 2024-05-20T18:06:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:06:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:06:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:06:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:06:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f41369e3640 1 -- 172.21.15.129:0/1594148658 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41180a2cf0 msgr2=0x7f41180a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f41369e3640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41180a2cf0 0x7f41180a5200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x5645229f91f0 tx=0x7f4128004a40 comp rx=0 tx=0).stop 2024-05-20T18:06:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f41369e3640 1 -- 172.21.15.129:0/1594148658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103250 msgr2=0x7f413019f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f41369e3640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103250 0x7f413019f520 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4120005940 tx=0x7f41200040c0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f41369e3640 1 -- 172.21.15.129:0/1594148658 shutdown_connections 2024-05-20T18:06:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f41369e3640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f41180a2cf0 0x7f41180a5200 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f41369e3640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4130108b00 0x7f413019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f41369e3640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4130108fa0 0x7f41301996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f41369e3640 1 --2- 172.21.15.129:0/1594148658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103250 0x7f413019f520 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.516+0000 7f41369e3640 1 -- 172.21.15.129:0/1594148658 >> 172.21.15.129:0/1594148658 conn(0x7f41300fe1e0 msgr2=0x7f41300fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.517+0000 7f41369e3640 1 -- 172.21.15.129:0/1594148658 shutdown_connections 2024-05-20T18:06:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:23.517+0000 7f41369e3640 1 -- 172.21.15.129:0/1594148658 wait complete. 2024-05-20T18:06:23.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:23 smithi154 ceph-mon[36482]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 58 MiB/s wr, 688 op/s 2024-05-20T18:06:23.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:23 smithi129 ceph-mon[32029]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 58 MiB/s wr, 688 op/s 2024-05-20T18:06:23.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:23 smithi204 ceph-mon[37076]: pgmap v241: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 58 MiB/s wr, 688 op/s 2024-05-20T18:06:24.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:24 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:06:24.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:24 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:24.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:24 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:06:24.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:24 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:24.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:24 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:06:24.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:24 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:24.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:24 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:06:24.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:24 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:24.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:24 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:06:24.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:24 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:24.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:24 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:06:24.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:24 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:25.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:25 smithi154 ceph-mon[36482]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 55 MiB/s wr, 577 op/s 2024-05-20T18:06:25.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:25 smithi129 ceph-mon[32029]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 55 MiB/s wr, 577 op/s 2024-05-20T18:06:25.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:25 smithi204 ceph-mon[37076]: pgmap v242: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 55 MiB/s wr, 577 op/s 2024-05-20T18:06:26.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:26 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:06:26.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:26 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:06:26.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:26 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:06:27.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:27 smithi154 ceph-mon[36482]: pgmap v243: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 MiB/s wr, 721 op/s 2024-05-20T18:06:27.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:27 smithi129 ceph-mon[32029]: pgmap v243: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 MiB/s wr, 721 op/s 2024-05-20T18:06:27.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:27 smithi204 ceph-mon[37076]: pgmap v243: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 MiB/s wr, 721 op/s 2024-05-20T18:06:29.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:29 smithi154 ceph-mon[36482]: pgmap v244: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 102 MiB/s wr, 723 op/s 2024-05-20T18:06:29.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:29 smithi129 ceph-mon[32029]: pgmap v244: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 102 MiB/s wr, 723 op/s 2024-05-20T18:06:29.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:29 smithi204 ceph-mon[37076]: pgmap v244: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 102 MiB/s wr, 723 op/s 2024-05-20T18:06:31.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:31 smithi154 ceph-mon[36482]: pgmap v245: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 102 MiB/s wr, 723 op/s 2024-05-20T18:06:31.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:31 smithi129 ceph-mon[32029]: pgmap v245: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 102 MiB/s wr, 723 op/s 2024-05-20T18:06:31.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:31 smithi204 ceph-mon[37076]: pgmap v245: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 102 MiB/s wr, 723 op/s 2024-05-20T18:06:33.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:33 smithi154 ceph-mon[36482]: pgmap v246: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 87 MiB/s wr, 613 op/s 2024-05-20T18:06:33.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:33 smithi129 ceph-mon[32029]: pgmap v246: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 87 MiB/s wr, 613 op/s 2024-05-20T18:06:33.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:33 smithi204 ceph-mon[37076]: pgmap v246: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 87 MiB/s wr, 613 op/s 2024-05-20T18:06:35.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:35 smithi154 ceph-mon[36482]: pgmap v247: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 MiB/s wr, 549 op/s 2024-05-20T18:06:35.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:35 smithi129 ceph-mon[32029]: pgmap v247: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 MiB/s wr, 549 op/s 2024-05-20T18:06:35.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:35 smithi204 ceph-mon[37076]: pgmap v247: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 MiB/s wr, 549 op/s 2024-05-20T18:06:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.625+0000 7f77152a8640 1 -- 172.21.15.129:0/2870434931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7710102170 msgr2=0x7f771010e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.625+0000 7f77152a8640 1 --2- 172.21.15.129:0/2870434931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7710102170 0x7f771010e4a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f7708009ca0 tx=0x7f770802a8c0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.625+0000 7f77152a8640 1 -- 172.21.15.129:0/2870434931 shutdown_connections 2024-05-20T18:06:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.625+0000 7f77152a8640 1 --2- 172.21.15.129:0/2870434931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7710107eb0 0x7f7710101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.625+0000 7f77152a8640 1 --2- 172.21.15.129:0/2870434931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7710102170 0x7f771010e4a0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.625+0000 7f77152a8640 1 --2- 172.21.15.129:0/2870434931 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7710107a10 0x7f77101015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.625+0000 7f77152a8640 1 -- 172.21.15.129:0/2870434931 >> 172.21.15.129:0/2870434931 conn(0x7f7710077160 msgr2=0x7f77100775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.625+0000 7f77152a8640 1 -- 172.21.15.129:0/2870434931 shutdown_connections 2024-05-20T18:06:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.625+0000 7f77152a8640 1 -- 172.21.15.129:0/2870434931 wait complete. 2024-05-20T18:06:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.626+0000 7f77152a8640 1 Processor -- start 2024-05-20T18:06:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.626+0000 7f77152a8640 1 -- start start 2024-05-20T18:06:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f77152a8640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7710102170 0x7f771019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f77152a8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7710107a10 0x7f771019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f77152a8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7710107eb0 0x7f771019b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f77152a8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7710111df0 con 0x7f7710107a10 2024-05-20T18:06:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f77152a8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7710111c70 con 0x7f7710107eb0 2024-05-20T18:06:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f77152a8640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7710111f70 con 0x7f7710102170 2024-05-20T18:06:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7710107a10 0x7f771019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7710107a10 0x7f771019ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52332/0 (socket says 172.21.15.129:52332) 2024-05-20T18:06:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770e7fc640 1 -- 172.21.15.129:0/1636294200 learned_addr learned my addr 172.21.15.129:0/1636294200 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770f7fe640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7710107eb0 0x7f771019b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770effd640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7710102170 0x7f771019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770e7fc640 1 -- 172.21.15.129:0/1636294200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7710102170 msgr2=0x7f771019a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770e7fc640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7710102170 0x7f771019a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770e7fc640 1 -- 172.21.15.129:0/1636294200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7710107eb0 msgr2=0x7f771019b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770e7fc640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7710107eb0 0x7f771019b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770e7fc640 1 -- 172.21.15.129:0/1636294200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77101a5910 con 0x7f7710107a10 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770f7fe640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7710107eb0 0x7f771019b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f770e7fc640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7710107a10 0x7f771019ab10 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f770400d740 tx=0x7f770400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f76f7fff640 1 -- 172.21.15.129:0/1636294200 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704017070 con 0x7f7710107a10 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f76f7fff640 1 -- 172.21.15.129:0/1636294200 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770400de50 con 0x7f7710107a10 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f77152a8640 1 -- 172.21.15.129:0/1636294200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77101a5c00 con 0x7f7710107a10 2024-05-20T18:06:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f76f7fff640 1 -- 172.21.15.129:0/1636294200 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704003e00 con 0x7f7710107a10 2024-05-20T18:06:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.627+0000 7f77152a8640 1 -- 172.21.15.129:0/1636294200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77101a61a0 con 0x7f7710107a10 2024-05-20T18:06:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f76f7fff640 1 -- 172.21.15.129:0/1636294200 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7704003fa0 con 0x7f7710107a10 2024-05-20T18:06:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f771c22c740 1 Processor -- start 2024-05-20T18:06:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f771c22c740 1 -- start start 2024-05-20T18:06:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f771c22c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c31490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.635 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f771c22c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e298652f0 0x558e29c319d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f771c22c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2986d4d0 0x558e29c31f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f771c22c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558e29b92340 con 0x558e2986d4d0 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f771c22c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558e29b3a290 con 0x558e298652f0 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f771c22c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558e29b8e300 con 0x558e2988e1b0 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f770e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2986d4d0 0x558e29c31f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f770e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2986d4d0 0x558e29c31f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52348/0 (socket says 172.21.15.129:52348) 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f770e7fc640 1 -- 172.21.15.129:0/1791514767 learned_addr learned my addr 172.21.15.129:0/1791514767 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.630+0000 7f770effd640 1 --2- 172.21.15.129:0/1791514767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e298652f0 0x558e29c319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f76f7fff640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f76f80a2d70 0x7f76f80a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f770f7fe640 1 --2- 172.21.15.129:0/1791514767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c31490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f770f7fe640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f76f80a2d70 0x7f76f80a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.636 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f76f7fff640 1 -- 172.21.15.129:0/1636294200 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f77040cee60 con 0x7f7710107a10 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f770e7fc640 1 -- 172.21.15.129:0/1791514767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 msgr2=0x558e29c31490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f770e7fc640 1 --2- 172.21.15.129:0/1791514767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c31490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f770e7fc640 1 -- 172.21.15.129:0/1791514767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e298652f0 msgr2=0x558e29c319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f770e7fc640 1 --2- 172.21.15.129:0/1791514767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e298652f0 0x558e29c319d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f770e7fc640 1 -- 172.21.15.129:0/1791514767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e29b5e8e0 con 0x558e2986d4d0 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f770e7fc640 1 --2- 172.21.15.129:0/1791514767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2986d4d0 0x558e29c31f10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f77040977a0 tx=0x7f77040038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423c08025d74a11d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f76f57fa640 1 -- 172.21.15.129:0/1791514767 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704020030 con 0x558e2986d4d0 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f770f7fe640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f76f80a2d70 0x7f76f80a5280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f771006a450 tx=0x7f770802a8c0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f76f57fa640 1 -- 172.21.15.129:0/1791514767 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7704009e50 con 0x558e2986d4d0 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.631+0000 7f76f57fa640 1 -- 172.21.15.129:0/1791514767 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77040a3fb0 con 0x558e2986d4d0 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.632+0000 7f771c22c740 1 -- 172.21.15.129:0/1791514767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2986d4d0 msgr2=0x558e29c31f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.632+0000 7f771c22c740 1 --2- 172.21.15.129:0/1791514767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2986d4d0 0x558e29c31f10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f77040977a0 tx=0x7f77040038a0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.632+0000 7f771c22c740 1 -- 172.21.15.129:0/1791514767 shutdown_connections 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.632+0000 7f771c22c740 1 --2- 172.21.15.129:0/1791514767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e298652f0 0x558e29c319d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.632+0000 7f771c22c740 1 --2- 172.21.15.129:0/1791514767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2986d4d0 0x558e29c31f10 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.632+0000 7f771c22c740 1 --2- 172.21.15.129:0/1791514767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c31490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.632+0000 7f771c22c740 1 -- 172.21.15.129:0/1791514767 >> 172.21.15.129:0/1791514767 conn(0x558e29882300 msgr2=0x558e29c2dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.632+0000 7f771c22c740 1 -- 172.21.15.129:0/1791514767 shutdown_connections 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.632+0000 7f771c22c740 1 -- 172.21.15.129:0/1791514767 wait complete. 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f771c22c740 1 -- start start 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f771c22c740 1 Processor -- start 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f771c22c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c3d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f771c22c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c3db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f771c22c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c3ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f771c22c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558e29c3e560 con 0x558e298652f0 2024-05-20T18:06:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f771c22c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558e29b4d790 con 0x558e2986d4d0 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f771c22c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558e29c3e6e0 con 0x558e2988e1b0 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f771c22c740 10 client.0 _ll_register_callbacks cb 0x558e29b88a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f770effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c3d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f770effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c3d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52356/0 (socket says 172.21.15.129:52356) 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f770effd640 1 -- 172.21.15.129:0/959834202 learned_addr learned my addr 172.21.15.129:0/959834202 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f770e7fc640 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c3db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f770f7fe640 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c3ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f770effd640 1 -- 172.21.15.129:0/959834202 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 msgr2=0x558e29c3ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f770effd640 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c3ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f770effd640 1 -- 172.21.15.129:0/959834202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 msgr2=0x558e29c3db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f770effd640 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c3db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.633+0000 7f770effd640 1 -- 172.21.15.129:0/959834202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e298a6210 con 0x558e298652f0 2024-05-20T18:06:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f770effd640 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c3d5f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f770000bd90 tx=0x7f7700012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ce718e87ce67f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f76f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f76f2ff5640 1 -- 172.21.15.129:0/959834202 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77000107e0 con 0x558e298652f0 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f76f2ff5640 1 -- 172.21.15.129:0/959834202 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f7700010980 con 0x558e298652f0 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f771c22c740 1 -- 172.21.15.129:0/959834202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7704009e50 con 0x558e298652f0 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f76f2ff5640 1 -- 172.21.15.129:0/959834202 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7700010b80 con 0x558e298652f0 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f771c22c740 10 client.16635 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f771c22c740 1 -- 172.21.15.129:0/959834202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558e29972620 con 0x558e298652f0 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f76f2ff5640 1 -- 172.21.15.129:0/959834202 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7704009e50 con 0x558e298652f0 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f76f2ff5640 1 -- 172.21.15.129:0/959834202 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f7700021030 con 0x558e298652f0 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f771c22c740 10 client.16635 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f771c22c740 10 client.16635 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f771c22c740 20 client.16635 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f771c22c740 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558e2984cd80 0x558e29c4de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f771c22c740 4 client.16635 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.634+0000 7f771c22c740 1 -- 172.21.15.129:0/959834202 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e29b9fa70 con 0x558e2984cd80 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.635+0000 7f770e7fc640 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558e2984cd80 0x558e29c4de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.635+0000 7f76f2ff5640 10 client.16635 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.635+0000 7f770e7fc640 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558e2984cd80 0x558e29c4de60 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e72d8ca4a2269451 server_cookie=8ba1833d02da8167 in_seq=0 out_seq=0 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f76f2ff5640 1 -- 172.21.15.129:0/959834202 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558e29b9fa70 con 0x558e2984cd80 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f76f2ff5640 10 client.16635 handle_command_reply: tid=0 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 1 client.16635 shutdown 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 1 -- 172.21.15.129:0/959834202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 msgr2=0x558e29c3d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c3d5f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f770000bd90 tx=0x7f7700012820 comp rx=0 tx=0).stop 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 1 -- 172.21.15.129:0/959834202 shutdown_connections 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558e2984cd80 0x558e29c4de60 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c3db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c3ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 1 --2- 172.21.15.129:0/959834202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c3d5f0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f76f2ff5640 0 client.16635 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:36.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 1 -- 172.21.15.129:0/959834202 >> 172.21.15.129:0/959834202 conn(0x558e29882300 msgr2=0x558e29c331b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:36.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 1 -- 172.21.15.129:0/959834202 shutdown_connections 2024-05-20T18:06:36.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 1 -- 172.21.15.129:0/959834202 wait complete. 2024-05-20T18:06:36.642 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.636+0000 7f771c22c740 20 client.16635 trim_cache size 0 max 16384 2024-05-20T18:06:36.653 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f771c22c740 1 Processor -- start 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f771c22c740 1 -- start start 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f771c22c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c2dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f771c22c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c536b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f771c22c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c53bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f771c22c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f77101aaab0 con 0x558e298652f0 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f771c22c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f77101ad310 con 0x558e2986d4d0 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f771c22c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f77101ab100 con 0x558e2988e1b0 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f770effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c2dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f770effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c2dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52362/0 (socket says 172.21.15.129:52362) 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f770effd640 1 -- 172.21.15.129:0/671763765 learned_addr learned my addr 172.21.15.129:0/671763765 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f770e7fc640 1 --2- 172.21.15.129:0/671763765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.654 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f770f7fe640 1 --2- 172.21.15.129:0/671763765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c53bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f770effd640 1 -- 172.21.15.129:0/671763765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 msgr2=0x558e29c53bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f770effd640 1 --2- 172.21.15.129:0/671763765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c53bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.651+0000 7f770effd640 1 -- 172.21.15.129:0/671763765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 msgr2=0x558e29c536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.652+0000 7f770effd640 1 --2- 172.21.15.129:0/671763765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.652+0000 7f770effd640 1 -- 172.21.15.129:0/671763765 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e29c2e180 con 0x558e298652f0 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.652+0000 7f770e7fc640 1 --2- 172.21.15.129:0/671763765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.652+0000 7f770effd640 1 --2- 172.21.15.129:0/671763765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c2dc40 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f770001a430 tx=0x7f7700005650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102abe1a0ba293c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.652+0000 7f76f4ff9640 1 -- 172.21.15.129:0/671763765 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770001a6a0 con 0x558e298652f0 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.652+0000 7f76f4ff9640 1 -- 172.21.15.129:0/671763765 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770001a840 con 0x558e298652f0 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.652+0000 7f76f4ff9640 1 -- 172.21.15.129:0/671763765 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770001b950 con 0x558e298652f0 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 -- 172.21.15.129:0/671763765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 msgr2=0x558e29c2dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 --2- 172.21.15.129:0/671763765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c2dc40 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f770001a430 tx=0x7f7700005650 comp rx=0 tx=0).stop 2024-05-20T18:06:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 -- 172.21.15.129:0/671763765 shutdown_connections 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 --2- 172.21.15.129:0/671763765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 --2- 172.21.15.129:0/671763765 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e2988e1b0 0x558e29c53bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 --2- 172.21.15.129:0/671763765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e298652f0 0x558e29c2dc40 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 -- 172.21.15.129:0/671763765 >> 172.21.15.129:0/671763765 conn(0x558e29882300 msgr2=0x558e29c2d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 -- 172.21.15.129:0/671763765 shutdown_connections 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 -- 172.21.15.129:0/671763765 wait complete. 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 -- start start 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 Processor -- start 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e298652f0 0x558e29c433d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c43910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2988e1b0 0x558e29cb4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f77101ab7d0 con 0x558e2988e1b0 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f77101acc40 con 0x558e2986d4d0 2024-05-20T18:06:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f77101abea0 con 0x558e298652f0 2024-05-20T18:06:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f771c22c740 10 client.0 _ll_register_callbacks cb 0x7f770000b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f770f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2988e1b0 0x558e29cb4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f770f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2988e1b0 0x558e29cb4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52372/0 (socket says 172.21.15.129:52372) 2024-05-20T18:06:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.653+0000 7f770f7fe640 1 -- 172.21.15.129:0/673766503 learned_addr learned my addr 172.21.15.129:0/673766503 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f770effd640 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e298652f0 0x558e29c433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f770e7fc640 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c43910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f770f7fe640 1 -- 172.21.15.129:0/673766503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e298652f0 msgr2=0x558e29c433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f770f7fe640 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e298652f0 0x558e29c433d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f770f7fe640 1 -- 172.21.15.129:0/673766503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 msgr2=0x558e29c43910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f770f7fe640 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c43910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f770f7fe640 1 -- 172.21.15.129:0/673766503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e29cb4790 con 0x558e2988e1b0 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f770f7fe640 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2988e1b0 0x558e29cb4250 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7708009ca0 tx=0x7f77080095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343ac76c2d0aa217 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f76f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f76f2ff5640 1 -- 172.21.15.129:0/673766503 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708043070 con 0x558e2988e1b0 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f76f2ff5640 1 -- 172.21.15.129:0/673766503 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f7708004450 con 0x558e2988e1b0 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f771c22c740 1 -- 172.21.15.129:0/673766503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f770001a840 con 0x558e2988e1b0 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f76f2ff5640 1 -- 172.21.15.129:0/673766503 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7708004650 con 0x558e2988e1b0 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f771c22c740 10 client.16641 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f771c22c740 1 -- 172.21.15.129:0/673766503 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558e29cb5950 con 0x558e2988e1b0 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.654+0000 7f76f2ff5640 1 -- 172.21.15.129:0/673766503 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770001a840 con 0x558e2988e1b0 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.655+0000 7f76f2ff5640 1 -- 172.21.15.129:0/673766503 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f770805a020 con 0x558e2988e1b0 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.655+0000 7f771c22c740 10 client.16641 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.655+0000 7f771c22c740 10 client.16641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.655+0000 7f771c22c740 20 client.16641 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.655+0000 7f771c22c740 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558e298221c0 0x558e29cb8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.655+0000 7f771c22c740 4 client.16641 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:06:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.655+0000 7f771c22c740 1 -- 172.21.15.129:0/673766503 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558e29b99dc0 con 0x558e298221c0 2024-05-20T18:06:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.655+0000 7f770effd640 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558e298221c0 0x558e29cb8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.655+0000 7f76f2ff5640 10 client.16641 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.655+0000 7f770effd640 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558e298221c0 0x558e29cb8140 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cac5b239c75c519a server_cookie=8cb4979095c51d24 in_seq=0 out_seq=0 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f76f2ff5640 1 -- 172.21.15.129:0/673766503 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558e29b99dc0 con 0x558e298221c0 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f76f2ff5640 10 client.16641 handle_command_reply: tid=0 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f771c22c740 1 client.16641 shutdown 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f771c22c740 1 -- 172.21.15.129:0/673766503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2988e1b0 msgr2=0x558e29cb4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f771c22c740 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2988e1b0 0x558e29cb4250 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7708009ca0 tx=0x7f77080095c0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f771c22c740 1 -- 172.21.15.129:0/673766503 shutdown_connections 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f771c22c740 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558e298221c0 0x558e29cb8140 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f771c22c740 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e2986d4d0 0x558e29c43910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f771c22c740 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558e2988e1b0 0x558e29cb4250 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f771c22c740 1 --2- 172.21.15.129:0/673766503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558e298652f0 0x558e29c433d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f76f2ff5640 0 client.16641 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f771c22c740 1 -- 172.21.15.129:0/673766503 >> 172.21.15.129:0/673766503 conn(0x558e29882300 msgr2=0x558e29c313d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.667+0000 7f771c22c740 1 -- 172.21.15.129:0/673766503 shutdown_connections 2024-05-20T18:06:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f771c22c740 1 -- 172.21.15.129:0/673766503 wait complete. 2024-05-20T18:06:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f771c22c740 20 client.16641 trim_cache size 0 max 16384 2024-05-20T18:06:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:06:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:06:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:06:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:06:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 -- 172.21.15.129:0/1636294200 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f76f80a2d70 msgr2=0x7f76f80a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f76f80a2d70 0x7f76f80a5280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f771006a450 tx=0x7f770802a8c0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 -- 172.21.15.129:0/1636294200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7710107a10 msgr2=0x7f771019ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7710107a10 0x7f771019ab10 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f770400d740 tx=0x7f770400dc00 comp rx=0 tx=0).stop 2024-05-20T18:06:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 -- 172.21.15.129:0/1636294200 shutdown_connections 2024-05-20T18:06:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f76f80a2d70 0x7f76f80a5280 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7710107eb0 0x7f771019b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7710107a10 0x7f771019ab10 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 --2- 172.21.15.129:0/1636294200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7710102170 0x7f771019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 -- 172.21.15.129:0/1636294200 >> 172.21.15.129:0/1636294200 conn(0x7f7710077160 msgr2=0x7f77100ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 -- 172.21.15.129:0/1636294200 shutdown_connections 2024-05-20T18:06:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:36.668+0000 7f77152a8640 1 -- 172.21.15.129:0/1636294200 wait complete. 2024-05-20T18:06:37.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:37 smithi154 ceph-mon[36482]: pgmap v248: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 64 MiB/s wr, 448 op/s 2024-05-20T18:06:37.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:37 smithi129 ceph-mon[32029]: pgmap v248: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 64 MiB/s wr, 448 op/s 2024-05-20T18:06:37.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:37 smithi204 ceph-mon[37076]: pgmap v248: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 64 MiB/s wr, 448 op/s 2024-05-20T18:06:39.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:39 smithi154 ceph-mon[36482]: pgmap v249: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 108 KiB/s wr, 4 op/s 2024-05-20T18:06:39.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:39 smithi129 ceph-mon[32029]: pgmap v249: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 108 KiB/s wr, 4 op/s 2024-05-20T18:06:39.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:39 smithi204 ceph-mon[37076]: pgmap v249: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 108 KiB/s wr, 4 op/s 2024-05-20T18:06:41.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:41 smithi154 ceph-mon[36482]: pgmap v250: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 89 KiB/s wr, 6 op/s 2024-05-20T18:06:41.665 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:41 smithi129 ceph-mon[32029]: pgmap v250: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 89 KiB/s wr, 6 op/s 2024-05-20T18:06:41.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:41 smithi204 ceph-mon[37076]: pgmap v250: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 89 KiB/s wr, 6 op/s 2024-05-20T18:06:43.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:43 smithi154 ceph-mon[36482]: pgmap v251: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.9 MiB/s wr, 27 op/s 2024-05-20T18:06:43.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:43 smithi129 ceph-mon[32029]: pgmap v251: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.9 MiB/s wr, 27 op/s 2024-05-20T18:06:43.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:43 smithi204 ceph-mon[37076]: pgmap v251: 97 pgs: 97 active+clean; 60 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.9 MiB/s wr, 27 op/s 2024-05-20T18:06:45.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:45 smithi154 ceph-mon[36482]: pgmap v252: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 11 MiB/s wr, 87 op/s 2024-05-20T18:06:45.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:45 smithi129 ceph-mon[32029]: pgmap v252: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 11 MiB/s wr, 87 op/s 2024-05-20T18:06:45.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:45 smithi204 ceph-mon[37076]: pgmap v252: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 11 MiB/s wr, 87 op/s 2024-05-20T18:06:47.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:47 smithi154 ceph-mon[36482]: pgmap v253: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 MiB/s wr, 243 op/s 2024-05-20T18:06:47.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:47 smithi129 ceph-mon[32029]: pgmap v253: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 MiB/s wr, 243 op/s 2024-05-20T18:06:47.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:47 smithi204 ceph-mon[37076]: pgmap v253: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 MiB/s wr, 243 op/s 2024-05-20T18:06:49.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:49 smithi154 ceph-mon[36482]: pgmap v254: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 32 MiB/s wr, 242 op/s 2024-05-20T18:06:49.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:49 smithi129 ceph-mon[32029]: pgmap v254: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 32 MiB/s wr, 242 op/s 2024-05-20T18:06:49.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:49 smithi204 ceph-mon[37076]: pgmap v254: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 32 MiB/s wr, 242 op/s 2024-05-20T18:06:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.779+0000 7fa33157e640 1 -- 172.21.15.129:0/2641333858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa32c107eb0 msgr2=0x7fa32c101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.779+0000 7fa33157e640 1 --2- 172.21.15.129:0/2641333858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa32c107eb0 0x7fa32c101b00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa314009900 tx=0x7fa31402a410 comp rx=0 tx=0).stop 2024-05-20T18:06:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.779+0000 7fa33157e640 1 -- 172.21.15.129:0/2641333858 shutdown_connections 2024-05-20T18:06:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.779+0000 7fa33157e640 1 --2- 172.21.15.129:0/2641333858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa32c102170 0x7fa32c10e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.779+0000 7fa33157e640 1 --2- 172.21.15.129:0/2641333858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa32c107eb0 0x7fa32c101b00 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.779+0000 7fa33157e640 1 --2- 172.21.15.129:0/2641333858 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa32c107a10 0x7fa32c1015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.779+0000 7fa33157e640 1 -- 172.21.15.129:0/2641333858 >> 172.21.15.129:0/2641333858 conn(0x7fa32c077160 msgr2=0x7fa32c0775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.779+0000 7fa33157e640 1 -- 172.21.15.129:0/2641333858 shutdown_connections 2024-05-20T18:06:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.779+0000 7fa33157e640 1 -- 172.21.15.129:0/2641333858 wait complete. 2024-05-20T18:06:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.780+0000 7fa33157e640 1 Processor -- start 2024-05-20T18:06:49.779 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.780+0000 7fa33157e640 1 -- start start 2024-05-20T18:06:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.780+0000 7fa33157e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa32c102170 0x7fa32c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.780+0000 7fa33157e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa32c107a10 0x7fa32c198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.780+0000 7fa33157e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa32c107eb0 0x7fa32c19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.780+0000 7fa33157e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c111d80 con 0x7fa32c107eb0 2024-05-20T18:06:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.780+0000 7fa33157e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c111c00 con 0x7fa32c107a10 2024-05-20T18:06:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.780+0000 7fa33157e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c111f00 con 0x7fa32c102170 2024-05-20T18:06:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa32b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa32c107eb0 0x7fa32c19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa32b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa32c107eb0 0x7fa32c19cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52898/0 (socket says 172.21.15.129:52898) 2024-05-20T18:06:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa32b7fe640 1 -- 172.21.15.129:0/1142298478 learned_addr learned my addr 172.21.15.129:0/1142298478 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa32affd640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa32c102170 0x7fa32c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa3227fc640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa32c107a10 0x7fa32c198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa32b7fe640 1 -- 172.21.15.129:0/1142298478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa32c102170 msgr2=0x7fa32c1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa32b7fe640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa32c102170 0x7fa32c1987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa32b7fe640 1 -- 172.21.15.129:0/1142298478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa32c107a10 msgr2=0x7fa32c198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa32b7fe640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa32c107a10 0x7fa32c198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa32b7fe640 1 -- 172.21.15.129:0/1142298478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32c19d600 con 0x7fa32c107eb0 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa3227fc640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa32c107a10 0x7fa32c198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa32b7fe640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa32c107eb0 0x7fa32c19cfc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa32400eaa0 tx=0x7fa32400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa328ff9640 1 -- 172.21.15.129:0/1142298478 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32400cdb0 con 0x7fa32c107eb0 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa328ff9640 1 -- 172.21.15.129:0/1142298478 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa324005020 con 0x7fa32c107eb0 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa33157e640 1 -- 172.21.15.129:0/1142298478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa32c19d8f0 con 0x7fa32c107eb0 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa328ff9640 1 -- 172.21.15.129:0/1142298478 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3240116b0 con 0x7fa32c107eb0 2024-05-20T18:06:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.781+0000 7fa33157e640 1 -- 172.21.15.129:0/1142298478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa32c19ddc0 con 0x7fa32c107eb0 2024-05-20T18:06:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.784+0000 7fa328ff9640 1 -- 172.21.15.129:0/1142298478 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa324005560 con 0x7fa32c107eb0 2024-05-20T18:06:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.784+0000 7fa338421740 1 Processor -- start 2024-05-20T18:06:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.784+0000 7fa338421740 1 -- start start 2024-05-20T18:06:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa338421740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af5d1b0 0x55a27b2fbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa338421740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af342f0 0x55a27b2fc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa338421740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b2fc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa338421740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a27b261340 con 0x55a27af3c4d0 2024-05-20T18:06:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa338421740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a27b209290 con 0x55a27af5d1b0 2024-05-20T18:06:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa338421740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a27b25d300 con 0x55a27af342f0 2024-05-20T18:06:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa3227fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b2fc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa3227fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b2fc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52914/0 (socket says 172.21.15.129:52914) 2024-05-20T18:06:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa3227fc640 1 -- 172.21.15.129:0/1004009811 learned_addr learned my addr 172.21.15.129:0/1004009811 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:49.792 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa3227fc640 1 --2- 172.21.15.129:0/1004009811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af5d1b0 0x55a27b2fbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa328ff9640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa3180a2cf0 0x7fa3180a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa32affd640 1 --2- 172.21.15.129:0/1004009811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af342f0 0x55a27b2fc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.785+0000 7fa32b7fe640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa3180a2cf0 0x7fa3180a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.786+0000 7fa3227fc640 1 -- 172.21.15.129:0/1004009811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af342f0 msgr2=0x55a27b2fc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.786+0000 7fa3227fc640 1 --2- 172.21.15.129:0/1004009811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af342f0 0x55a27b2fc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.786+0000 7fa3227fc640 1 -- 172.21.15.129:0/1004009811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af5d1b0 msgr2=0x55a27b2fbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.786+0000 7fa3227fc640 1 --2- 172.21.15.129:0/1004009811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af5d1b0 0x55a27b2fbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.786+0000 7fa3227fc640 1 -- 172.21.15.129:0/1004009811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a27b22d8e0 con 0x55a27af3c4d0 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.786+0000 7fa328ff9640 1 -- 172.21.15.129:0/1142298478 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7fa3240cf6e0 con 0x7fa32c107eb0 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.786+0000 7fa3227fc640 1 --2- 172.21.15.129:0/1004009811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b2fc9d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa3140096c0 tx=0x7fa3140056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a63759865f2334 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.786+0000 7fa32b7fe640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa3180a2cf0 0x7fa3180a5200 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa324098020 tx=0x7fa32406aaf0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.786+0000 7fa3217fa640 1 -- 172.21.15.129:0/1004009811 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa314035040 con 0x55a27af3c4d0 2024-05-20T18:06:49.793 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.786+0000 7fa3217fa640 1 -- 172.21.15.129:0/1004009811 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3140035f0 con 0x55a27af3c4d0 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.786+0000 7fa3217fa640 1 -- 172.21.15.129:0/1004009811 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31403d6c0 con 0x55a27af3c4d0 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.787+0000 7fa338421740 1 -- 172.21.15.129:0/1004009811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 msgr2=0x55a27b2fc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.787+0000 7fa338421740 1 --2- 172.21.15.129:0/1004009811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b2fc9d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa3140096c0 tx=0x7fa3140056d0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.787+0000 7fa338421740 1 -- 172.21.15.129:0/1004009811 shutdown_connections 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.787+0000 7fa338421740 1 --2- 172.21.15.129:0/1004009811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b2fc9d0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.787+0000 7fa338421740 1 --2- 172.21.15.129:0/1004009811 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af342f0 0x55a27b2fc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.787+0000 7fa338421740 1 --2- 172.21.15.129:0/1004009811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af5d1b0 0x55a27b2fbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.787+0000 7fa338421740 1 -- 172.21.15.129:0/1004009811 >> 172.21.15.129:0/1004009811 conn(0x55a27af51300 msgr2=0x55a27b2f8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.787+0000 7fa338421740 1 -- 172.21.15.129:0/1004009811 shutdown_connections 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.787+0000 7fa338421740 1 -- 172.21.15.129:0/1004009811 wait complete. 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.787+0000 7fa338421740 1 -- start start 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.787+0000 7fa338421740 1 Processor -- start 2024-05-20T18:06:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa338421740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b30a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa338421740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af3c4d0 0x55a27b30a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa338421740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b3087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa338421740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a27b30b410 con 0x55a27af5d1b0 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa338421740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a27b21c790 con 0x55a27af342f0 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa338421740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a27b30b590 con 0x55a27af3c4d0 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa338421740 10 client.0 _ll_register_callbacks cb 0x55a27b257a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa3227fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b3087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa3227fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b3087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52924/0 (socket says 172.21.15.129:52924) 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa3227fc640 1 -- 172.21.15.129:0/1942957798 learned_addr learned my addr 172.21.15.129:0/1942957798 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa32affd640 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af3c4d0 0x55a27b30a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa3227fc640 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b30a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa3227fc640 1 -- 172.21.15.129:0/1942957798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af3c4d0 msgr2=0x55a27b30a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa3227fc640 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af3c4d0 0x55a27b30a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa3227fc640 1 -- 172.21.15.129:0/1942957798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 msgr2=0x55a27b30a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa3227fc640 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b30a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa3227fc640 1 -- 172.21.15.129:0/1942957798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a27af75210 con 0x55a27af5d1b0 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa3227fc640 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b3087a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa31402a450 tx=0x7fa314032ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=559f597ecadd3a23 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.788+0000 7fa3127fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa3127fc640 1 -- 172.21.15.129:0/1942957798 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa314000e60 con 0x55a27af5d1b0 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa3127fc640 1 -- 172.21.15.129:0/1942957798 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7fa314003ab0 con 0x55a27af5d1b0 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa338421740 1 -- 172.21.15.129:0/1942957798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3140035f0 con 0x55a27af5d1b0 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa3127fc640 1 -- 172.21.15.129:0/1942957798 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa314035070 con 0x55a27af5d1b0 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa338421740 10 client.16653 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa338421740 1 -- 172.21.15.129:0/1942957798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a27b041620 con 0x55a27af5d1b0 2024-05-20T18:06:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa3127fc640 1 -- 172.21.15.129:0/1942957798 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3140035f0 con 0x55a27af5d1b0 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa3127fc640 1 -- 172.21.15.129:0/1942957798 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fa31403a040 con 0x55a27af5d1b0 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa338421740 10 client.16653 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa338421740 10 client.16653 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa338421740 20 client.16653 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa338421740 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a27af1bd80 0x55a27b31aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa338421740 4 client.16653 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa338421740 1 -- 172.21.15.129:0/1942957798 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a27b26ea70 con 0x55a27af1bd80 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.789+0000 7fa32affd640 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a27af1bd80 0x55a27b31aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.790+0000 7fa3127fc640 10 client.16653 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.790+0000 7fa32affd640 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a27af1bd80 0x55a27b31aa00 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76012300313f20ef server_cookie=e009464e81ac3622 in_seq=0 out_seq=0 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa3127fc640 1 -- 172.21.15.129:0/1942957798 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a27b26ea70 con 0x55a27af1bd80 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa3127fc640 10 client.16653 handle_command_reply: tid=0 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 1 client.16653 shutdown 2024-05-20T18:06:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 1 -- 172.21.15.129:0/1942957798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 msgr2=0x55a27b3087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b3087a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa31402a450 tx=0x7fa314032ed0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 1 -- 172.21.15.129:0/1942957798 shutdown_connections 2024-05-20T18:06:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a27af1bd80 0x55a27b31aa00 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b3087a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af3c4d0 0x55a27b30a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa3127fc640 0 client.16653 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 1 --2- 172.21.15.129:0/1942957798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b30a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 1 -- 172.21.15.129:0/1942957798 >> 172.21.15.129:0/1942957798 conn(0x55a27af51300 msgr2=0x55a27b2fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 1 -- 172.21.15.129:0/1942957798 shutdown_connections 2024-05-20T18:06:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 1 -- 172.21.15.129:0/1942957798 wait complete. 2024-05-20T18:06:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.791+0000 7fa338421740 20 client.16653 trim_cache size 0 max 16384 2024-05-20T18:06:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.806+0000 7fa338421740 1 Processor -- start 2024-05-20T18:06:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.806+0000 7fa338421740 1 -- start start 2024-05-20T18:06:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa338421740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b307d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa338421740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b3020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa338421740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af5d1b0 0x55a27b302600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa338421740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c1aa9f0 con 0x55a27af3c4d0 2024-05-20T18:06:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa338421740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c1ad250 con 0x55a27af342f0 2024-05-20T18:06:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa338421740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c1ab040 con 0x55a27af5d1b0 2024-05-20T18:06:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa32affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b3020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa32affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b3020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52926/0 (socket says 172.21.15.129:52926) 2024-05-20T18:06:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa32affd640 1 -- 172.21.15.129:0/3082735011 learned_addr learned my addr 172.21.15.129:0/3082735011 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa3227fc640 1 --2- 172.21.15.129:0/3082735011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b307d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa3227fc640 1 --2- 172.21.15.129:0/3082735011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af5d1b0 0x55a27b302600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa32affd640 1 -- 172.21.15.129:0/3082735011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af5d1b0 msgr2=0x55a27b302600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa32affd640 1 --2- 172.21.15.129:0/3082735011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af5d1b0 0x55a27b302600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa32affd640 1 -- 172.21.15.129:0/3082735011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 msgr2=0x55a27b307d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa32affd640 1 --2- 172.21.15.129:0/3082735011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b307d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa32affd640 1 -- 172.21.15.129:0/3082735011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a27b30a0a0 con 0x55a27af3c4d0 2024-05-20T18:06:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.807+0000 7fa3227fc640 1 --2- 172.21.15.129:0/3082735011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af5d1b0 0x55a27b302600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:06:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.808+0000 7fa32affd640 1 --2- 172.21.15.129:0/3082735011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b3020c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa31c005fd0 tx=0x7fa31c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841bd7dc6f844ca5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.808+0000 7fa320ff9640 1 -- 172.21.15.129:0/3082735011 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c005020 con 0x55a27af3c4d0 2024-05-20T18:06:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.808+0000 7fa320ff9640 1 -- 172.21.15.129:0/3082735011 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa31c024030 con 0x55a27af3c4d0 2024-05-20T18:06:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.808+0000 7fa320ff9640 1 -- 172.21.15.129:0/3082735011 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c020160 con 0x55a27af3c4d0 2024-05-20T18:06:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.809+0000 7fa338421740 1 -- 172.21.15.129:0/3082735011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 msgr2=0x55a27b3020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.809+0000 7fa338421740 1 --2- 172.21.15.129:0/3082735011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b3020c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa31c005fd0 tx=0x7fa31c00b390 comp rx=0 tx=0).stop 2024-05-20T18:06:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.809+0000 7fa338421740 1 -- 172.21.15.129:0/3082735011 shutdown_connections 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.809+0000 7fa338421740 1 --2- 172.21.15.129:0/3082735011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af5d1b0 0x55a27b302600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.809+0000 7fa338421740 1 --2- 172.21.15.129:0/3082735011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af3c4d0 0x55a27b3020c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.809+0000 7fa338421740 1 --2- 172.21.15.129:0/3082735011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b307d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.809+0000 7fa338421740 1 -- 172.21.15.129:0/3082735011 >> 172.21.15.129:0/3082735011 conn(0x55a27af51300 msgr2=0x55a27b2fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.809+0000 7fa338421740 1 -- 172.21.15.129:0/3082735011 shutdown_connections 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.809+0000 7fa338421740 1 -- 172.21.15.129:0/3082735011 wait complete. 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.809+0000 7fa338421740 1 -- start start 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.809+0000 7fa338421740 1 Processor -- start 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa338421740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b31dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa338421740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af3c4d0 0x55a27b31b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa338421740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b31b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa338421740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c1ab710 con 0x55a27af5d1b0 2024-05-20T18:06:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa338421740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c1acb80 con 0x55a27af342f0 2024-05-20T18:06:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa338421740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c1abde0 con 0x55a27af3c4d0 2024-05-20T18:06:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa338421740 10 client.0 _ll_register_callbacks cb 0x7fa314032d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3227fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b31b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3227fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b31b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52942/0 (socket says 172.21.15.129:52942) 2024-05-20T18:06:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3227fc640 1 -- 172.21.15.129:0/442688759 learned_addr learned my addr 172.21.15.129:0/442688759 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa32affd640 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af3c4d0 0x55a27b31b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3227fc640 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b31dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3227fc640 1 -- 172.21.15.129:0/442688759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af3c4d0 msgr2=0x55a27b31b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3227fc640 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af3c4d0 0x55a27b31b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3227fc640 1 -- 172.21.15.129:0/442688759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 msgr2=0x55a27b31dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3227fc640 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b31dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3227fc640 1 -- 172.21.15.129:0/442688759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a27b31ba80 con 0x55a27af5d1b0 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3227fc640 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b31b540 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x55a27b31a860 tx=0x7fa314031610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181438cf4eeaf4d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3127fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3127fc640 1 -- 172.21.15.129:0/442688759 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa314005d50 con 0x55a27af5d1b0 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3127fc640 1 -- 172.21.15.129:0/442688759 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7fa314051070 con 0x55a27af5d1b0 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa338421740 1 -- 172.21.15.129:0/442688759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa31c024030 con 0x55a27af5d1b0 2024-05-20T18:06:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.810+0000 7fa3127fc640 1 -- 172.21.15.129:0/442688759 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa31404c580 con 0x55a27af5d1b0 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.811+0000 7fa338421740 10 client.16659 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.811+0000 7fa338421740 1 -- 172.21.15.129:0/442688759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a27b382670 con 0x55a27af5d1b0 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.811+0000 7fa3127fc640 1 -- 172.21.15.129:0/442688759 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa31c024030 con 0x55a27af5d1b0 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.811+0000 7fa3127fc640 1 -- 172.21.15.129:0/442688759 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fa314049550 con 0x55a27af5d1b0 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.811+0000 7fa338421740 10 client.16659 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.811+0000 7fa338421740 10 client.16659 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.811+0000 7fa338421740 20 client.16659 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.811+0000 7fa338421740 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a27aef11c0 0x55a27b384ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.811+0000 7fa338421740 4 client.16659 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.811+0000 7fa338421740 1 -- 172.21.15.129:0/442688759 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a27b268dc0 con 0x55a27aef11c0 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.811+0000 7fa32affd640 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a27aef11c0 0x55a27b384ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.812+0000 7fa3127fc640 10 client.16659 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.812+0000 7fa32affd640 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a27aef11c0 0x55a27b384ca0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97d3a4673050e3b3 server_cookie=d2717db86827292f in_seq=0 out_seq=0 2024-05-20T18:06:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa3127fc640 1 -- 172.21.15.129:0/442688759 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a27b268dc0 con 0x55a27aef11c0 2024-05-20T18:06:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa3127fc640 10 client.16659 handle_command_reply: tid=0 2024-05-20T18:06:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 1 client.16659 shutdown 2024-05-20T18:06:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 1 -- 172.21.15.129:0/442688759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 msgr2=0x55a27b31b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b31b540 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x55a27b31a860 tx=0x7fa314031610 comp rx=0 tx=0).stop 2024-05-20T18:06:49.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 1 -- 172.21.15.129:0/442688759 shutdown_connections 2024-05-20T18:06:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a27aef11c0 0x55a27b384ca0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a27af5d1b0 0x55a27b31b540 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a27af3c4d0 0x55a27b31b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa3127fc640 0 client.16659 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 1 --2- 172.21.15.129:0/442688759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a27af342f0 0x55a27b31dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 1 -- 172.21.15.129:0/442688759 >> 172.21.15.129:0/442688759 conn(0x55a27af51300 msgr2=0x55a27b2fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 1 -- 172.21.15.129:0/442688759 shutdown_connections 2024-05-20T18:06:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 1 -- 172.21.15.129:0/442688759 wait complete. 2024-05-20T18:06:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.822+0000 7fa338421740 20 client.16659 trim_cache size 0 max 16384 2024-05-20T18:06:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:06:49.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:06:49.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:06:49.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:06:49.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 -- 172.21.15.129:0/1142298478 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa3180a2cf0 msgr2=0x7fa3180a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa3180a2cf0 0x7fa3180a5200 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa324098020 tx=0x7fa32406aaf0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 -- 172.21.15.129:0/1142298478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa32c107eb0 msgr2=0x7fa32c19cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa32c107eb0 0x7fa32c19cfc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa32400eaa0 tx=0x7fa32400ef60 comp rx=0 tx=0).stop 2024-05-20T18:06:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 -- 172.21.15.129:0/1142298478 shutdown_connections 2024-05-20T18:06:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa3180a2cf0 0x7fa3180a5200 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa32c107a10 0x7fa32c198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa32c107eb0 0x7fa32c19cfc0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 --2- 172.21.15.129:0/1142298478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa32c102170 0x7fa32c1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 -- 172.21.15.129:0/1142298478 >> 172.21.15.129:0/1142298478 conn(0x7fa32c077160 msgr2=0x7fa32c1030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 -- 172.21.15.129:0/1142298478 shutdown_connections 2024-05-20T18:06:49.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:06:49.823+0000 7fa33157e640 1 -- 172.21.15.129:0/1142298478 wait complete. 2024-05-20T18:06:51.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:51 smithi154 ceph-mon[36482]: pgmap v255: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 32 MiB/s wr, 242 op/s 2024-05-20T18:06:51.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:51 smithi129 ceph-mon[32029]: pgmap v255: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 32 MiB/s wr, 242 op/s 2024-05-20T18:06:51.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:51 smithi204 ceph-mon[37076]: pgmap v255: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 32 MiB/s wr, 242 op/s 2024-05-20T18:06:53.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:06:53.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:06:53.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.322+0000 7f64ad91a640 1 -- 172.21.15.129:0/3067711003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64a8108f70 msgr2=0x7f64a8102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.322+0000 7f64ad91a640 1 --2- 172.21.15.129:0/3067711003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64a8108f70 0x7f64a8102bb0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f649c009900 tx=0x7f649c02a410 comp rx=0 tx=0).stop 2024-05-20T18:06:53.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.322+0000 7f64ad91a640 1 -- 172.21.15.129:0/3067711003 shutdown_connections 2024-05-20T18:06:53.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.322+0000 7f64ad91a640 1 --2- 172.21.15.129:0/3067711003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8103220 0x7f64a810f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.322+0000 7f64ad91a640 1 --2- 172.21.15.129:0/3067711003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64a8108f70 0x7f64a8102bb0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.322+0000 7f64ad91a640 1 --2- 172.21.15.129:0/3067711003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64a8108ad0 0x7f64a8102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.322+0000 7f64ad91a640 1 -- 172.21.15.129:0/3067711003 >> 172.21.15.129:0/3067711003 conn(0x7f64a80fe190 msgr2=0x7f64a81005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.322+0000 7f64ad91a640 1 -- 172.21.15.129:0/3067711003 shutdown_connections 2024-05-20T18:06:53.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.322+0000 7f64ad91a640 1 -- 172.21.15.129:0/3067711003 wait complete. 2024-05-20T18:06:53.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64ad91a640 1 Processor -- start 2024-05-20T18:06:53.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64ad91a640 1 -- start start 2024-05-20T18:06:53.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64ad91a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64a8103220 0x7f64a819f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64ad91a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8108ad0 0x7f64a819fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64ad91a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64a8108f70 0x7f64a81996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64ad91a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f64a8112e10 con 0x7f64a8103220 2024-05-20T18:06:53.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64ad91a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f64a8112c90 con 0x7f64a8108f70 2024-05-20T18:06:53.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64ad91a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f64a8112f90 con 0x7f64a8108ad0 2024-05-20T18:06:53.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64a6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64a8103220 0x7f64a819f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64a6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64a8103220 0x7f64a819f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52948/0 (socket says 172.21.15.129:52948) 2024-05-20T18:06:53.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64a6ffd640 1 -- 172.21.15.129:0/621549651 learned_addr learned my addr 172.21.15.129:0/621549651 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:53.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64a67fc640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8108ad0 0x7f64a819fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.323+0000 7f64a77fe640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64a8108f70 0x7f64a81996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.324+0000 7f64a6ffd640 1 -- 172.21.15.129:0/621549651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8108ad0 msgr2=0x7f64a819fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.324+0000 7f64a6ffd640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8108ad0 0x7f64a819fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.324+0000 7f64a6ffd640 1 -- 172.21.15.129:0/621549651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64a8108f70 msgr2=0x7f64a81996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.324+0000 7f64a6ffd640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64a8108f70 0x7f64a81996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.324+0000 7f64a6ffd640 1 -- 172.21.15.129:0/621549651 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a8199e00 con 0x7f64a8103220 2024-05-20T18:06:53.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.324+0000 7f64a6ffd640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64a8103220 0x7f64a819f550 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f649800d770 tx=0x7f649800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.326+0000 7f64ac918640 1 -- 172.21.15.129:0/621549651 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6498017070 con 0x7f64a8103220 2024-05-20T18:06:53.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.326+0000 7f64ac918640 1 -- 172.21.15.129:0/621549651 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6498009c30 con 0x7f64a8103220 2024-05-20T18:06:53.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.326+0000 7f64ac918640 1 -- 172.21.15.129:0/621549651 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649800f620 con 0x7f64a8103220 2024-05-20T18:06:53.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.326+0000 7f64ad91a640 1 -- 172.21.15.129:0/621549651 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a819a090 con 0x7f64a8103220 2024-05-20T18:06:53.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.326+0000 7f64ad91a640 1 -- 172.21.15.129:0/621549651 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a81a62f0 con 0x7f64a8103220 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.328+0000 7f64b47c3740 1 Processor -- start 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64ac918640 1 -- 172.21.15.129:0/621549651 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f649801a070 con 0x7f64a8103220 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64b47c3740 1 -- start start 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cde2390 0x55724d180d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdb94d0 0x55724d181270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cdc16b0 0x55724d1817b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64b47c3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55724d0e6d00 con 0x55724cde2390 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64b47c3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55724d08e4c0 con 0x55724cdb94d0 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64b47c3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55724d0e2cc0 con 0x55724cdc16b0 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cde2390 0x55724d180d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cde2390 0x55724d180d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52960/0 (socket says 172.21.15.129:52960) 2024-05-20T18:06:53.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64a77fe640 1 -- 172.21.15.129:0/1251188674 learned_addr learned my addr 172.21.15.129:0/1251188674 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64a6ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdb94d0 0x55724d181270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64a67fc640 1 --2- 172.21.15.129:0/1251188674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cdc16b0 0x55724d1817b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64a77fe640 1 -- 172.21.15.129:0/1251188674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cdc16b0 msgr2=0x55724d1817b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64a77fe640 1 --2- 172.21.15.129:0/1251188674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cdc16b0 0x55724d1817b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64a77fe640 1 -- 172.21.15.129:0/1251188674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdb94d0 msgr2=0x55724d181270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64a77fe640 1 --2- 172.21.15.129:0/1251188674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdb94d0 0x55724d181270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64a77fe640 1 -- 172.21.15.129:0/1251188674 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55724cec47d0 con 0x55724cde2390 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.329+0000 7f64ac918640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f64900a2d00 0x7f64900a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.330+0000 7f64a77fe640 1 --2- 172.21.15.129:0/1251188674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cde2390 0x55724d180d30 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f64a0010370 tx=0x7f64a0010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a374d8ba0ab0889f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.330+0000 7f64a6ffd640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f64900a2d00 0x7f64900a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.330+0000 7f648dffb640 1 -- 172.21.15.129:0/1251188674 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0016070 con 0x55724cde2390 2024-05-20T18:06:53.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.330+0000 7f64ac918640 1 -- 172.21.15.129:0/621549651 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f64980cee90 con 0x7f64a8103220 2024-05-20T18:06:53.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.330+0000 7f648dffb640 1 -- 172.21.15.129:0/1251188674 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64a0004d20 con 0x55724cde2390 2024-05-20T18:06:53.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.330+0000 7f648dffb640 1 -- 172.21.15.129:0/1251188674 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a00054c0 con 0x55724cde2390 2024-05-20T18:06:53.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64b47c3740 1 -- 172.21.15.129:0/1251188674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cde2390 msgr2=0x55724d180d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64a6ffd640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f64900a2d00 0x7f64900a5210 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f64980cc650 tx=0x7f64980d29e0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64b47c3740 1 --2- 172.21.15.129:0/1251188674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cde2390 0x55724d180d30 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f64a0010370 tx=0x7f64a0010830 comp rx=0 tx=0).stop 2024-05-20T18:06:53.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64b47c3740 1 -- 172.21.15.129:0/1251188674 shutdown_connections 2024-05-20T18:06:53.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64b47c3740 1 --2- 172.21.15.129:0/1251188674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdb94d0 0x55724d181270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64b47c3740 1 --2- 172.21.15.129:0/1251188674 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cdc16b0 0x55724d1817b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64b47c3740 1 --2- 172.21.15.129:0/1251188674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cde2390 0x55724d180d30 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64b47c3740 1 -- 172.21.15.129:0/1251188674 >> 172.21.15.129:0/1251188674 conn(0x55724cdd64e0 msgr2=0x55724d17d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64b47c3740 1 -- 172.21.15.129:0/1251188674 shutdown_connections 2024-05-20T18:06:53.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64b47c3740 1 -- 172.21.15.129:0/1251188674 wait complete. 2024-05-20T18:06:53.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64b47c3740 1 -- start start 2024-05-20T18:06:53.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.331+0000 7f64b47c3740 1 Processor -- start 2024-05-20T18:06:53.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cdb94d0 0x55724d18cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdc16b0 0x55724d18d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cde2390 0x55724d18b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64b47c3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55724d18dfc0 con 0x55724cdc16b0 2024-05-20T18:06:53.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64b47c3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55724d0a1b90 con 0x55724cde2390 2024-05-20T18:06:53.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64b47c3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55724d18e140 con 0x55724cdb94d0 2024-05-20T18:06:53.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64b47c3740 10 client.0 _ll_register_callbacks cb 0x55724d0ddda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:53.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64a67fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdc16b0 0x55724d18d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64a67fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdc16b0 0x55724d18d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52964/0 (socket says 172.21.15.129:52964) 2024-05-20T18:06:53.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64a67fc640 1 -- 172.21.15.129:0/3984246037 learned_addr learned my addr 172.21.15.129:0/3984246037 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:53.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64a77fe640 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cdb94d0 0x55724d18cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64a77fe640 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cde2390 0x55724d18b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64a67fc640 1 -- 172.21.15.129:0/3984246037 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cdb94d0 msgr2=0x55724d18cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64a67fc640 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cdb94d0 0x55724d18cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64a67fc640 1 -- 172.21.15.129:0/3984246037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cde2390 msgr2=0x55724d18b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64a67fc640 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cde2390 0x55724d18b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64a67fc640 1 -- 172.21.15.129:0/3984246037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55724cdfa3f0 con 0x55724cdc16b0 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f64a67fc640 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdc16b0 0x55724d18d480 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f649c009900 tx=0x7f649c0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b16b2e5124395e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f6486ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.332+0000 7f6486ffd640 1 -- 172.21.15.129:0/3984246037 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c035040 con 0x55724cdc16b0 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f6486ffd640 1 -- 172.21.15.129:0/3984246037 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f649c004040 con 0x55724cdc16b0 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f64b47c3740 1 -- 172.21.15.129:0/3984246037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64a0004d20 con 0x55724cdc16b0 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f6486ffd640 1 -- 172.21.15.129:0/3984246037 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f649c004240 con 0x55724cdc16b0 2024-05-20T18:06:53.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f64b47c3740 10 client.16671 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f64b47c3740 1 -- 172.21.15.129:0/3984246037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55724cdba0f0 con 0x55724cdc16b0 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f6486ffd640 1 -- 172.21.15.129:0/3984246037 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64a0004d20 con 0x55724cdc16b0 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f6486ffd640 1 -- 172.21.15.129:0/3984246037 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f649c004a20 con 0x55724cdc16b0 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f64b47c3740 10 client.16671 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f64b47c3740 10 client.16671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f64b47c3740 20 client.16671 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55724cda0f60 0x55724d19dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f64b47c3740 4 client.16671 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.333+0000 7f64b47c3740 1 -- 172.21.15.129:0/3984246037 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55724d0f4430 con 0x55724cda0f60 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.334+0000 7f64a77fe640 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55724cda0f60 0x55724d19dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.334+0000 7f6486ffd640 10 client.16671 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.334+0000 7f64a77fe640 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55724cda0f60 0x55724d19dbe0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93b972af194d16a8 server_cookie=b428805a4d662fa6 in_seq=0 out_seq=0 2024-05-20T18:06:53.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f6486ffd640 1 -- 172.21.15.129:0/3984246037 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55724d0f4430 con 0x55724cda0f60 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f6486ffd640 10 client.16671 handle_command_reply: tid=0 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 1 client.16671 shutdown 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 1 -- 172.21.15.129:0/3984246037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdc16b0 msgr2=0x55724d18d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdc16b0 0x55724d18d480 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f649c009900 tx=0x7f649c0317e0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 1 -- 172.21.15.129:0/3984246037 shutdown_connections 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55724cda0f60 0x55724d19dbe0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cde2390 0x55724d18b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f6486ffd640 0 client.16671 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdc16b0 0x55724d18d480 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3984246037 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cdb94d0 0x55724d18cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 1 -- 172.21.15.129:0/3984246037 >> 172.21.15.129:0/3984246037 conn(0x55724cdd64e0 msgr2=0x55724d182a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 1 -- 172.21.15.129:0/3984246037 shutdown_connections 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 1 -- 172.21.15.129:0/3984246037 wait complete. 2024-05-20T18:06:53.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.335+0000 7f64b47c3740 20 client.16671 trim_cache size 0 max 16384 2024-05-20T18:06:53.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.353+0000 7f64b47c3740 1 Processor -- start 2024-05-20T18:06:53.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.353+0000 7f64b47c3740 1 -- start start 2024-05-20T18:06:53.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.353+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d18b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.353+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdc16b0 0x55724d1a33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.353+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cde2390 0x55724d1a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.353+0000 7f64b47c3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81acc10 con 0x55724cdb94d0 2024-05-20T18:06:53.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.353+0000 7f64b47c3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81af470 con 0x55724cdc16b0 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.353+0000 7f64b47c3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81ad260 con 0x55724cde2390 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d18b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d18b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52980/0 (socket says 172.21.15.129:52980) 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f64a77fe640 1 -- 172.21.15.129:0/1007702390 learned_addr learned my addr 172.21.15.129:0/1007702390 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f64a67fc640 1 --2- 172.21.15.129:0/1007702390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdc16b0 0x55724d1a33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f64a77fe640 1 -- 172.21.15.129:0/1007702390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cde2390 msgr2=0x55724d1a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f64a77fe640 1 --2- 172.21.15.129:0/1007702390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cde2390 0x55724d1a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f64a77fe640 1 -- 172.21.15.129:0/1007702390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdc16b0 msgr2=0x55724d1a33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f64a77fe640 1 --2- 172.21.15.129:0/1007702390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdc16b0 0x55724d1a33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f64a77fe640 1 -- 172.21.15.129:0/1007702390 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55724d1898a0 con 0x55724cdb94d0 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f64a67fc640 1 --2- 172.21.15.129:0/1007702390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdc16b0 0x55724d1a33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f64a77fe640 1 --2- 172.21.15.129:0/1007702390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d18b2f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f64a0011010 tx=0x7f64a0010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1516ba3eba6f41 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f648d7fa640 1 -- 172.21.15.129:0/1007702390 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0016070 con 0x55724cdb94d0 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f648d7fa640 1 -- 172.21.15.129:0/1007702390 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64a0004ce0 con 0x55724cdb94d0 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.354+0000 7f648d7fa640 1 -- 172.21.15.129:0/1007702390 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0023040 con 0x55724cdb94d0 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.355+0000 7f64b47c3740 1 -- 172.21.15.129:0/1007702390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 msgr2=0x55724d18b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.355+0000 7f64b47c3740 1 --2- 172.21.15.129:0/1007702390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d18b2f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f64a0011010 tx=0x7f64a0010830 comp rx=0 tx=0).stop 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.355+0000 7f64b47c3740 1 -- 172.21.15.129:0/1007702390 shutdown_connections 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.355+0000 7f64b47c3740 1 --2- 172.21.15.129:0/1007702390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdc16b0 0x55724d1a33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.355+0000 7f64b47c3740 1 --2- 172.21.15.129:0/1007702390 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cde2390 0x55724d1a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.355+0000 7f64b47c3740 1 --2- 172.21.15.129:0/1007702390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d18b2f0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.355+0000 7f64b47c3740 1 -- 172.21.15.129:0/1007702390 >> 172.21.15.129:0/1007702390 conn(0x55724cdd64e0 msgr2=0x55724d18abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.355+0000 7f64b47c3740 1 -- 172.21.15.129:0/1007702390 shutdown_connections 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.355+0000 7f64b47c3740 1 -- 172.21.15.129:0/1007702390 wait complete. 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.355+0000 7f64b47c3740 1 -- start start 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.355+0000 7f64b47c3740 1 Processor -- start 2024-05-20T18:06:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d190850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdc16b0 0x55724d190db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64b47c3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cde2390 0x55724d1fdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64b47c3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81ad930 con 0x55724cdb94d0 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64b47c3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81aeda0 con 0x55724cdc16b0 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64b47c3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81ae000 con 0x55724cde2390 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64b47c3740 10 client.0 _ll_register_callbacks cb 0x7f649c005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d190850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d190850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52994/0 (socket says 172.21.15.129:52994) 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64a77fe640 1 -- 172.21.15.129:0/3256877588 learned_addr learned my addr 172.21.15.129:0/3256877588 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64a77fe640 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cde2390 0x55724d1fdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64a67fc640 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdc16b0 0x55724d190db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64a77fe640 1 -- 172.21.15.129:0/3256877588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cde2390 msgr2=0x55724d1fdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64a77fe640 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cde2390 0x55724d1fdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64a77fe640 1 -- 172.21.15.129:0/3256877588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdc16b0 msgr2=0x55724d190db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64a77fe640 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdc16b0 0x55724d190db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64a77fe640 1 -- 172.21.15.129:0/3256877588 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55724d1fe550 con 0x55724cdb94d0 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f64a77fe640 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d190850 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009f40 tx=0x7f64a0009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a5ebae3591d8c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f6486ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f6486ffd640 1 -- 172.21.15.129:0/3256877588 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0016030 con 0x55724cdb94d0 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.356+0000 7f6486ffd640 1 -- 172.21.15.129:0/3256877588 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f64a002b640 con 0x55724cdb94d0 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f64b47c3740 1 -- 172.21.15.129:0/3256877588 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64a0004ce0 con 0x55724cdb94d0 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f6486ffd640 1 -- 172.21.15.129:0/3256877588 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f64a002b840 con 0x55724cdb94d0 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f64b47c3740 10 client.16677 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f64b47c3740 1 -- 172.21.15.129:0/3256877588 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55724d205800 con 0x55724cdb94d0 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f6486ffd640 1 -- 172.21.15.129:0/3256877588 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64a0004ce0 con 0x55724cdb94d0 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f6486ffd640 1 -- 172.21.15.129:0/3256877588 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f64a003e070 con 0x55724cdb94d0 2024-05-20T18:06:53.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f64b47c3740 10 client.16677 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:53.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f64b47c3740 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:53.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f64b47c3740 20 client.16677 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:53.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55724cd763a0 0x55724d207cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f64b47c3740 4 client.16677 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:06:53.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.357+0000 7f64b47c3740 1 -- 172.21.15.129:0/3256877588 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55724d0ee780 con 0x55724cd763a0 2024-05-20T18:06:53.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64a67fc640 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55724cd763a0 0x55724d207cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f6486ffd640 10 client.16677 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64a67fc640 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55724cd763a0 0x55724d207cf0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f815fb1a75dcf5c5 server_cookie=6842b38ec52b85e1 in_seq=0 out_seq=0 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f6486ffd640 1 -- 172.21.15.129:0/3256877588 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55724d0ee780 con 0x55724cd763a0 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f6486ffd640 10 client.16677 handle_command_reply: tid=0 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64b47c3740 1 client.16677 shutdown 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64b47c3740 1 -- 172.21.15.129:0/3256877588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 msgr2=0x55724d190850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d190850 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009f40 tx=0x7f64a0009660 comp rx=0 tx=0).stop 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64b47c3740 1 -- 172.21.15.129:0/3256877588 shutdown_connections 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55724cd763a0 0x55724d207cf0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55724cdc16b0 0x55724d190db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55724cde2390 0x55724d1fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f6486ffd640 0 client.16677 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:53.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64b47c3740 1 --2- 172.21.15.129:0/3256877588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55724cdb94d0 0x55724d190850 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64b47c3740 1 -- 172.21.15.129:0/3256877588 >> 172.21.15.129:0/3256877588 conn(0x55724cdd64e0 msgr2=0x55724d17bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64b47c3740 1 -- 172.21.15.129:0/3256877588 shutdown_connections 2024-05-20T18:06:53.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.358+0000 7f64b47c3740 1 -- 172.21.15.129:0/3256877588 wait complete. 2024-05-20T18:06:53.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64b47c3740 20 client.16677 trim_cache size 0 max 16384 2024-05-20T18:06:53.366 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:06:53.366 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:06:53.366 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:06:53.366 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:06:53.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 -- 172.21.15.129:0/621549651 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f64900a2d00 msgr2=0x7f64900a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f64900a2d00 0x7f64900a5210 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f64980cc650 tx=0x7f64980d29e0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 -- 172.21.15.129:0/621549651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64a8103220 msgr2=0x7f64a819f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64a8103220 0x7f64a819f550 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f649800d770 tx=0x7f649800dc30 comp rx=0 tx=0).stop 2024-05-20T18:06:53.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 -- 172.21.15.129:0/621549651 shutdown_connections 2024-05-20T18:06:53.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f64900a2d00 0x7f64900a5210 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64a8108f70 0x7f64a81996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f64a8108ad0 0x7f64a819fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 --2- 172.21.15.129:0/621549651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64a8103220 0x7f64a819f550 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 -- 172.21.15.129:0/621549651 >> 172.21.15.129:0/621549651 conn(0x7f64a80fe190 msgr2=0x7f64a810e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 -- 172.21.15.129:0/621549651 shutdown_connections 2024-05-20T18:06:53.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.359+0000 7f64ad91a640 1 -- 172.21.15.129:0/621549651 wait complete. 2024-05-20T18:06:53.369 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:06:53.369 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:06:53.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.483+0000 7ff2fd27a640 1 -- 172.21.15.129:0/690089033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f8101aa0 msgr2=0x7ff2f8101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.483+0000 7ff2fd27a640 1 --2- 172.21.15.129:0/690089033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f8101aa0 0x7ff2f8101eb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009900 tx=0x7ff2ec02a410 comp rx=0 tx=0).stop 2024-05-20T18:06:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.483+0000 7ff2fd27a640 1 -- 172.21.15.129:0/690089033 shutdown_connections 2024-05-20T18:06:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.483+0000 7ff2fd27a640 1 --2- 172.21.15.129:0/690089033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2f810c330 0x7ff2f810e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.483+0000 7ff2fd27a640 1 --2- 172.21.15.129:0/690089033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f8101aa0 0x7ff2f8101eb0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.483+0000 7ff2fd27a640 1 --2- 172.21.15.129:0/690089033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2f81079d0 0x7ff2f8101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.483+0000 7ff2fd27a640 1 -- 172.21.15.129:0/690089033 >> 172.21.15.129:0/690089033 conn(0x7ff2f80770c0 msgr2=0x7ff2f8077500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.483+0000 7ff2fd27a640 1 -- 172.21.15.129:0/690089033 shutdown_connections 2024-05-20T18:06:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.483+0000 7ff2fd27a640 1 -- 172.21.15.129:0/690089033 wait complete. 2024-05-20T18:06:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2fd27a640 1 Processor -- start 2024-05-20T18:06:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2fd27a640 1 -- start start 2024-05-20T18:06:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2fd27a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f8101aa0 0x7ff2f8199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2fd27a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2f81079d0 0x7ff2f8199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2fd27a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2f810c330 0x7ff2f819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2fd27a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f8111ce0 con 0x7ff2f8101aa0 2024-05-20T18:06:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2fd27a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f8111b60 con 0x7ff2f81079d0 2024-05-20T18:06:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2fd27a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f8111e60 con 0x7ff2f810c330 2024-05-20T18:06:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2f6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f8101aa0 0x7ff2f8199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2f6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f8101aa0 0x7ff2f8199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53004/0 (socket says 172.21.15.129:53004) 2024-05-20T18:06:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2f6ffd640 1 -- 172.21.15.129:0/3466706058 learned_addr learned my addr 172.21.15.129:0/3466706058 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2f67fc640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2f81079d0 0x7ff2f8199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.484+0000 7ff2f77fe640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2f810c330 0x7ff2f819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2f6ffd640 1 -- 172.21.15.129:0/3466706058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2f810c330 msgr2=0x7ff2f819df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2f6ffd640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2f810c330 0x7ff2f819df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2f6ffd640 1 -- 172.21.15.129:0/3466706058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2f81079d0 msgr2=0x7ff2f8199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2f6ffd640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2f81079d0 0x7ff2f8199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2f6ffd640 1 -- 172.21.15.129:0/3466706058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f819e5d0 con 0x7ff2f8101aa0 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2f77fe640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2f810c330 0x7ff2f819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2f6ffd640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f8101aa0 0x7ff2f8199770 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8005fd0 tx=0x7ff2e800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2dffff640 1 -- 172.21.15.129:0/3466706058 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8017070 con 0x7ff2f8101aa0 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2fd27a640 1 -- 172.21.15.129:0/3466706058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f819e8c0 con 0x7ff2f8101aa0 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2dffff640 1 -- 172.21.15.129:0/3466706058 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2e8009c30 con 0x7ff2f8101aa0 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2dffff640 1 -- 172.21.15.129:0/3466706058 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8010460 con 0x7ff2f8101aa0 2024-05-20T18:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.485+0000 7ff2fd27a640 1 -- 172.21.15.129:0/3466706058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f81a6240 con 0x7ff2f8101aa0 2024-05-20T18:06:53.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.486+0000 7ff2fd27a640 1 -- 172.21.15.129:0/3466706058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2f8102c80 con 0x7ff2f8101aa0 2024-05-20T18:06:53.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.487+0000 7ff2dffff640 1 -- 172.21.15.129:0/3466706058 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ff2e8003e70 con 0x7ff2f8101aa0 2024-05-20T18:06:53.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.488+0000 7ff2dffff640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2e00a2d00 0x7ff2e00a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.488+0000 7ff2f67fc640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2e00a2d00 0x7ff2e00a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.488+0000 7ff2dffff640 1 -- 172.21.15.129:0/3466706058 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7ff2e80cf930 con 0x7ff2f8101aa0 2024-05-20T18:06:53.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.489+0000 7ff2f67fc640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2e00a2d00 0x7ff2e00a5210 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec005940 tx=0x7ff2ec0058d0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.492+0000 7ff2dffff640 1 -- 172.21.15.129:0/3466706058 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff2e8020030 con 0x7ff2f8101aa0 2024-05-20T18:06:53.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:53 smithi154 ceph-mon[36482]: pgmap v256: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 32 MiB/s wr, 238 op/s 2024-05-20T18:06:53.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:53 smithi129 ceph-mon[32029]: pgmap v256: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 32 MiB/s wr, 238 op/s 2024-05-20T18:06:53.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:53 smithi204 ceph-mon[37076]: pgmap v256: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 32 MiB/s wr, 238 op/s 2024-05-20T18:06:53.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.710+0000 7ff2fd27a640 1 -- 172.21.15.129:0/3466706058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2f8069f50 con 0x7ff2f8101aa0 2024-05-20T18:06:53.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.711+0000 7ff2dffff640 1 -- 172.21.15.129:0/3466706058 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11829 (secure 0 0 0) 0x7ff2e809c510 con 0x7ff2f8101aa0 2024-05-20T18:06:53.709 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:06:53.709 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":26255,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/2724397229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":2724397229},{"type":"v1","addr":"172.21.15.154:6837","nonce":2724397229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:06:10.247052+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:06:53.709 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-05-20T18:06:53.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 -- 172.21.15.129:0/3466706058 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2e00a2d00 msgr2=0x7ff2e00a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2e00a2d00 0x7ff2e00a5210 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec005940 tx=0x7ff2ec0058d0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 -- 172.21.15.129:0/3466706058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f8101aa0 msgr2=0x7ff2f8199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f8101aa0 0x7ff2f8199770 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8005fd0 tx=0x7ff2e800dde0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 -- 172.21.15.129:0/3466706058 shutdown_connections 2024-05-20T18:06:53.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff2e00a2d00 0x7ff2e00a5210 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2f81079d0 0x7ff2f8199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff2f810c330 0x7ff2f819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 --2- 172.21.15.129:0/3466706058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f8101aa0 0x7ff2f8199770 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 -- 172.21.15.129:0/3466706058 >> 172.21.15.129:0/3466706058 conn(0x7ff2f80770c0 msgr2=0x7ff2f8068e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 -- 172.21.15.129:0/3466706058 shutdown_connections 2024-05-20T18:06:53.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.713+0000 7ff2fd27a640 1 -- 172.21.15.129:0/3466706058 wait complete. 2024-05-20T18:06:53.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:06:53.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.840+0000 7f83594a8640 1 -- 172.21.15.129:0/3559493253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835410c330 msgr2=0x7f835410e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.840+0000 7f83594a8640 1 --2- 172.21.15.129:0/3559493253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835410c330 0x7f835410e750 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f834c009ca0 tx=0x7f834c02a8c0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.840+0000 7f83594a8640 1 -- 172.21.15.129:0/3559493253 shutdown_connections 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.840+0000 7f83594a8640 1 --2- 172.21.15.129:0/3559493253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8354101aa0 0x7f8354101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.840+0000 7f83594a8640 1 --2- 172.21.15.129:0/3559493253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835410c330 0x7f835410e750 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.840+0000 7f83594a8640 1 --2- 172.21.15.129:0/3559493253 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f83541079d0 0x7f8354101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.840+0000 7f83594a8640 1 -- 172.21.15.129:0/3559493253 >> 172.21.15.129:0/3559493253 conn(0x7f83540770b0 msgr2=0x7f83540774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.840+0000 7f83594a8640 1 -- 172.21.15.129:0/3559493253 shutdown_connections 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.840+0000 7f83594a8640 1 -- 172.21.15.129:0/3559493253 wait complete. 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f83594a8640 1 Processor -- start 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f83594a8640 1 -- start start 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f83594a8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8354101aa0 0x7f83541986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f83594a8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83541079d0 0x7f8354198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f83594a8640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f835410c330 0x7f835419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f83594a8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8354111d00 con 0x7f8354101aa0 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f83594a8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8354111b80 con 0x7f83541079d0 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f83594a8640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8354111e80 con 0x7f835410c330 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f8352ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8354101aa0 0x7f83541986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f8352ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8354101aa0 0x7f83541986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53028/0 (socket says 172.21.15.129:53028) 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f8352ffd640 1 -- 172.21.15.129:0/2184626436 learned_addr learned my addr 172.21.15.129:0/2184626436 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f83527fc640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83541079d0 0x7f8354198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.841+0000 7f83537fe640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f835410c330 0x7f835419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.842+0000 7f8352ffd640 1 -- 172.21.15.129:0/2184626436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f835410c330 msgr2=0x7f835419cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.842+0000 7f8352ffd640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f835410c330 0x7f835419cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.842+0000 7f8352ffd640 1 -- 172.21.15.129:0/2184626436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83541079d0 msgr2=0x7f8354198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.842+0000 7f8352ffd640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83541079d0 0x7f8354198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.842+0000 7f8352ffd640 1 -- 172.21.15.129:0/2184626436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835419d540 con 0x7f8354101aa0 2024-05-20T18:06:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.842+0000 7f8352ffd640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8354101aa0 0x7f83541986e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8344007920 tx=0x7f8344007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.842+0000 7f834bfff640 1 -- 172.21.15.129:0/2184626436 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834400e690 con 0x7f8354101aa0 2024-05-20T18:06:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.842+0000 7f834bfff640 1 -- 172.21.15.129:0/2184626436 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f834400ed10 con 0x7f8354101aa0 2024-05-20T18:06:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.842+0000 7f83594a8640 1 -- 172.21.15.129:0/2184626436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835419d830 con 0x7f8354101aa0 2024-05-20T18:06:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.842+0000 7f834bfff640 1 -- 172.21.15.129:0/2184626436 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834400ad20 con 0x7f8354101aa0 2024-05-20T18:06:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.842+0000 7f83594a8640 1 -- 172.21.15.129:0/2184626436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f835419dd30 con 0x7f8354101aa0 2024-05-20T18:06:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f834bfff640 1 -- 172.21.15.129:0/2184626436 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f834400e830 con 0x7f8354101aa0 2024-05-20T18:06:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f836042b740 1 Processor -- start 2024-05-20T18:06:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f836042b740 1 -- start start 2024-05-20T18:06:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f836042b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa36390 0x55ec9fdd4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f836042b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdd5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f836042b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa156b0 0x55ec9fdd57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f836042b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ec9fd3ac40 con 0x55ec9fa156b0 2024-05-20T18:06:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f836042b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ec9fce2580 con 0x55ec9fa36390 2024-05-20T18:06:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f836042b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ec9fd36c00 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f83527fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa156b0 0x55ec9fdd57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f83527fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa156b0 0x55ec9fdd57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53040/0 (socket says 172.21.15.129:53040) 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.845+0000 7f83527fc640 1 -- 172.21.15.129:0/2785003378 learned_addr learned my addr 172.21.15.129:0/2785003378 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f834bfff640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f83400a7110 0x7f83400a9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83537fe640 1 --2- 172.21.15.129:0/2785003378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa36390 0x55ec9fdd4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f8352ffd640 1 --2- 172.21.15.129:0/2785003378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdd5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83537fe640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f83400a7110 0x7f83400a9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83527fc640 1 -- 172.21.15.129:0/2785003378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa0d4d0 msgr2=0x55ec9fdd5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83527fc640 1 --2- 172.21.15.129:0/2785003378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdd5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83527fc640 1 -- 172.21.15.129:0/2785003378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa36390 msgr2=0x55ec9fdd4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83527fc640 1 --2- 172.21.15.129:0/2785003378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa36390 0x55ec9fdd4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83527fc640 1 -- 172.21.15.129:0/2785003378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec9fa4e3f0 con 0x55ec9fa156b0 2024-05-20T18:06:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f8352ffd640 1 --2- 172.21.15.129:0/2785003378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdd5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f834bfff640 1 -- 172.21.15.129:0/2184626436 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f8344008230 con 0x7f8354101aa0 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83527fc640 1 --2- 172.21.15.129:0/2785003378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa156b0 0x55ec9fdd57a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f833c009c20 tx=0x7f833c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e7219faf048bd80 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83497fa640 1 -- 172.21.15.129:0/2785003378 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c004220 con 0x55ec9fa156b0 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83537fe640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f83400a7110 0x7f83400a9620 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f834c004af0 tx=0x7f834c02a8c0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83497fa640 1 -- 172.21.15.129:0/2785003378 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c0043c0 con 0x55ec9fa156b0 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.846+0000 7f83497fa640 1 -- 172.21.15.129:0/2785003378 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c0197a0 con 0x55ec9fa156b0 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.847+0000 7f836042b740 1 -- 172.21.15.129:0/2785003378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa156b0 msgr2=0x55ec9fdd57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.847+0000 7f836042b740 1 --2- 172.21.15.129:0/2785003378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa156b0 0x55ec9fdd57a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f833c009c20 tx=0x7f833c011810 comp rx=0 tx=0).stop 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.847+0000 7f836042b740 1 -- 172.21.15.129:0/2785003378 shutdown_connections 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.847+0000 7f836042b740 1 --2- 172.21.15.129:0/2785003378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa156b0 0x55ec9fdd57a0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.847+0000 7f836042b740 1 --2- 172.21.15.129:0/2785003378 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdd5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.847+0000 7f836042b740 1 --2- 172.21.15.129:0/2785003378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa36390 0x55ec9fdd4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.847+0000 7f836042b740 1 -- 172.21.15.129:0/2785003378 >> 172.21.15.129:0/2785003378 conn(0x55ec9fa2a4e0 msgr2=0x55ec9fdd14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.847+0000 7f836042b740 1 -- 172.21.15.129:0/2785003378 shutdown_connections 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.847+0000 7f836042b740 1 -- 172.21.15.129:0/2785003378 wait complete. 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f836042b740 1 -- start start 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f836042b740 1 Processor -- start 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f836042b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fde0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f836042b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa156b0 0x55ec9fde1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f836042b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa36390 0x55ec9fddf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f836042b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ec9fde1fb0 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f836042b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ec9fcf5c40 con 0x55ec9fa156b0 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f836042b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ec9fde2130 con 0x55ec9fa36390 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f836042b740 10 client.0 _ll_register_callbacks cb 0x55ec9fa4dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f8352ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fde0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f8352ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fde0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53042/0 (socket says 172.21.15.129:53042) 2024-05-20T18:06:53.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f8352ffd640 1 -- 172.21.15.129:0/1461753578 learned_addr learned my addr 172.21.15.129:0/1461753578 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f83527fc640 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa156b0 0x55ec9fde1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f8352ffd640 1 -- 172.21.15.129:0/1461753578 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa36390 msgr2=0x55ec9fddf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f83537fe640 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa36390 0x55ec9fddf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f8352ffd640 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa36390 0x55ec9fddf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f8352ffd640 1 -- 172.21.15.129:0/1461753578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa156b0 msgr2=0x55ec9fde1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f8352ffd640 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa156b0 0x55ec9fde1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.848+0000 7f8352ffd640 1 -- 172.21.15.129:0/1461753578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec9fb187d0 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f8352ffd640 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fde0de0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8344099b30 tx=0x7f834400af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbbc51b8425c43a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f833a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f833a7fc640 1 -- 172.21.15.129:0/1461753578 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834409cca0 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f833a7fc640 1 -- 172.21.15.129:0/1461753578 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f83440997d0 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f836042b740 1 -- 172.21.15.129:0/1461753578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f833c0043c0 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f833a7fc640 1 -- 172.21.15.129:0/1461753578 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f834409cf30 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f836042b740 10 client.16695 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f836042b740 1 -- 172.21.15.129:0/1461753578 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ec9fdee7f0 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f833a7fc640 1 -- 172.21.15.129:0/1461753578 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c0043c0 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f833a7fc640 1 -- 172.21.15.129:0/1461753578 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f8344014070 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f836042b740 10 client.16695 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f836042b740 10 client.16695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f836042b740 20 client.16695 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f836042b740 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec9f9f4f60 0x55ec9fdf1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f836042b740 4 client.16695 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.849+0000 7f836042b740 1 -- 172.21.15.129:0/1461753578 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec9fd48370 con 0x55ec9f9f4f60 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.850+0000 7f83527fc640 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec9f9f4f60 0x55ec9fdf1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.850+0000 7f833a7fc640 10 client.16695 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.850+0000 7f83527fc640 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec9f9f4f60 0x55ec9fdf1cb0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64e6e831d392228d server_cookie=e49e97b0bd7ae729 in_seq=0 out_seq=0 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f833a7fc640 1 -- 172.21.15.129:0/1461753578 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ec9fd48370 con 0x55ec9f9f4f60 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f833a7fc640 10 client.16695 handle_command_reply: tid=0 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 1 client.16695 shutdown 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 1 -- 172.21.15.129:0/1461753578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 msgr2=0x55ec9fde0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fde0de0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8344099b30 tx=0x7f834400af90 comp rx=0 tx=0).stop 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 1 -- 172.21.15.129:0/1461753578 shutdown_connections 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec9f9f4f60 0x55ec9fdf1cb0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa156b0 0x55ec9fde1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa36390 0x55ec9fddf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 1 --2- 172.21.15.129:0/1461753578 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fde0de0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f833a7fc640 0 client.16695 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 1 -- 172.21.15.129:0/1461753578 >> 172.21.15.129:0/1461753578 conn(0x55ec9fa2a4e0 msgr2=0x55ec9fdd6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 1 -- 172.21.15.129:0/1461753578 shutdown_connections 2024-05-20T18:06:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 1 -- 172.21.15.129:0/1461753578 wait complete. 2024-05-20T18:06:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.851+0000 7f836042b740 20 client.16695 trim_cache size 0 max 16384 2024-05-20T18:06:53.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.864+0000 7f836042b740 1 Processor -- start 2024-05-20T18:06:53.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.864+0000 7f836042b740 1 -- start start 2024-05-20T18:06:53.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f836042b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdde2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f836042b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa156b0 0x55ec9fdd0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f836042b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa36390 0x55ec9fdd1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f836042b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f83541aa9a0 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f836042b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f83541ad200 con 0x55ec9fa156b0 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f836042b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f83541aaff0 con 0x55ec9fa36390 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f8352ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdde2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f8352ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdde2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53054/0 (socket says 172.21.15.129:53054) 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f8352ffd640 1 -- 172.21.15.129:0/2019392272 learned_addr learned my addr 172.21.15.129:0/2019392272 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f8352ffd640 1 -- 172.21.15.129:0/2019392272 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa36390 msgr2=0x55ec9fdd1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f8352ffd640 1 --2- 172.21.15.129:0/2019392272 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa36390 0x55ec9fdd1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f83527fc640 1 --2- 172.21.15.129:0/2019392272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa156b0 0x55ec9fdd0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f8352ffd640 1 -- 172.21.15.129:0/2019392272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa156b0 msgr2=0x55ec9fdd0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f8352ffd640 1 --2- 172.21.15.129:0/2019392272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa156b0 0x55ec9fdd0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f8352ffd640 1 -- 172.21.15.129:0/2019392272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec9fdf2070 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.865+0000 7f8352ffd640 1 --2- 172.21.15.129:0/2019392272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdde2d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f83440078f0 tx=0x7f834409b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9710713114af31a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f8348ff9640 1 -- 172.21.15.129:0/2019392272 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834409cac0 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f8348ff9640 1 -- 172.21.15.129:0/2019392272 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f834409d860 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f8348ff9640 1 -- 172.21.15.129:0/2019392272 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8344099970 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f836042b740 1 -- 172.21.15.129:0/2019392272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 msgr2=0x55ec9fdde2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f836042b740 1 --2- 172.21.15.129:0/2019392272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdde2d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f83440078f0 tx=0x7f834409b670 comp rx=0 tx=0).stop 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f836042b740 1 -- 172.21.15.129:0/2019392272 shutdown_connections 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f836042b740 1 --2- 172.21.15.129:0/2019392272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa156b0 0x55ec9fdd0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f836042b740 1 --2- 172.21.15.129:0/2019392272 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa36390 0x55ec9fdd1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f836042b740 1 --2- 172.21.15.129:0/2019392272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdde2d0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f836042b740 1 -- 172.21.15.129:0/2019392272 >> 172.21.15.129:0/2019392272 conn(0x55ec9fa2a4e0 msgr2=0x55ec9fdddbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f836042b740 1 -- 172.21.15.129:0/2019392272 shutdown_connections 2024-05-20T18:06:53.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.866+0000 7f836042b740 1 -- 172.21.15.129:0/2019392272 wait complete. 2024-05-20T18:06:53.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f836042b740 1 -- start start 2024-05-20T18:06:53.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f836042b740 1 Processor -- start 2024-05-20T18:06:53.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f836042b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdeb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f836042b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa156b0 0x55ec9fde8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f836042b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa36390 0x55ec9fde9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f836042b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f83541ab6c0 con 0x55ec9fa36390 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f836042b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f83541acb30 con 0x55ec9fa0d4d0 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f836042b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f83541abd90 con 0x55ec9fa156b0 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f836042b740 10 client.0 _ll_register_callbacks cb 0x7f83440058a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f83537fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa36390 0x55ec9fde9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f83537fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa36390 0x55ec9fde9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53056/0 (socket says 172.21.15.129:53056) 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f83537fe640 1 -- 172.21.15.129:0/1646404753 learned_addr learned my addr 172.21.15.129:0/1646404753 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f8352ffd640 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdeb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.867+0000 7f83527fc640 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa156b0 0x55ec9fde8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f83537fe640 1 -- 172.21.15.129:0/1646404753 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa156b0 msgr2=0x55ec9fde8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f83537fe640 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa156b0 0x55ec9fde8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f83537fe640 1 -- 172.21.15.129:0/1646404753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa0d4d0 msgr2=0x55ec9fdeb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f83537fe640 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdeb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f83537fe640 1 -- 172.21.15.129:0/1646404753 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec9fde9670 con 0x55ec9fa36390 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f83537fe640 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa36390 0x55ec9fde9100 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f834c035000 tx=0x7f834c02f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14512142e3793cdc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f833a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f833a7fc640 1 -- 172.21.15.129:0/1646404753 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c004440 con 0x55ec9fa36390 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f833a7fc640 1 -- 172.21.15.129:0/1646404753 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f834c0045e0 con 0x55ec9fa36390 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f836042b740 1 -- 172.21.15.129:0/1646404753 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f834409d860 con 0x55ec9fa36390 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f833a7fc640 1 -- 172.21.15.129:0/1646404753 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f834c0047e0 con 0x55ec9fa36390 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f836042b740 10 client.16701 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f836042b740 1 -- 172.21.15.129:0/1646404753 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ec9fe59da0 con 0x55ec9fa36390 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f833a7fc640 1 -- 172.21.15.129:0/1646404753 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f834409d860 con 0x55ec9fa36390 2024-05-20T18:06:53.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f833a7fc640 1 -- 172.21.15.129:0/1646404753 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f834c058020 con 0x55ec9fa36390 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f836042b740 10 client.16701 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f836042b740 10 client.16701 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.868+0000 7f836042b740 20 client.16701 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.869+0000 7f836042b740 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec9f9ca3a0 0x55ec9fe5c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.869+0000 7f836042b740 4 client.16701 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.869+0000 7f836042b740 1 -- 172.21.15.129:0/1646404753 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ec9fd426c0 con 0x55ec9f9ca3a0 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.869+0000 7f8352ffd640 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec9f9ca3a0 0x55ec9fe5c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.869+0000 7f833a7fc640 10 client.16701 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.869+0000 7f8352ffd640 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec9f9ca3a0 0x55ec9fe5c5b0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b8dd5bc6b6198ae server_cookie=929f3179d8581d35 in_seq=0 out_seq=0 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.869+0000 7f833a7fc640 1 -- 172.21.15.129:0/1646404753 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ec9fd426c0 con 0x55ec9f9ca3a0 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.869+0000 7f833a7fc640 10 client.16701 handle_command_reply: tid=0 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 1 client.16701 shutdown 2024-05-20T18:06:53.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 1 -- 172.21.15.129:0/1646404753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa36390 msgr2=0x55ec9fde9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa36390 0x55ec9fde9100 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f834c035000 tx=0x7f834c02f330 comp rx=0 tx=0).stop 2024-05-20T18:06:53.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 1 -- 172.21.15.129:0/1646404753 shutdown_connections 2024-05-20T18:06:53.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ec9f9ca3a0 0x55ec9fe5c5b0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ec9fa36390 0x55ec9fde9100 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f833a7fc640 0 client.16701 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:53.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ec9fa156b0 0x55ec9fde8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 1 --2- 172.21.15.129:0/1646404753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec9fa0d4d0 0x55ec9fdeb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 1 -- 172.21.15.129:0/1646404753 >> 172.21.15.129:0/1646404753 conn(0x55ec9fa2a4e0 msgr2=0x55ec9fdde350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 1 -- 172.21.15.129:0/1646404753 shutdown_connections 2024-05-20T18:06:53.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 1 -- 172.21.15.129:0/1646404753 wait complete. 2024-05-20T18:06:53.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f836042b740 20 client.16701 trim_cache size 0 max 16384 2024-05-20T18:06:53.875 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:06:53.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f83594a8640 1 -- 172.21.15.129:0/2184626436 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f83400a7110 msgr2=0x7f83400a9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f83594a8640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f83400a7110 0x7f83400a9620 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f834c004af0 tx=0x7f834c02a8c0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f83594a8640 1 -- 172.21.15.129:0/2184626436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8354101aa0 msgr2=0x7f83541986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:53.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f83594a8640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8354101aa0 0x7f83541986e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8344007920 tx=0x7f8344007de0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f83594a8640 1 -- 172.21.15.129:0/2184626436 shutdown_connections 2024-05-20T18:06:53.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f83594a8640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f83400a7110 0x7f83400a9620 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f83594a8640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83541079d0 0x7f8354198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f83594a8640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f835410c330 0x7f835419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f83594a8640 1 --2- 172.21.15.129:0/2184626436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8354101aa0 0x7f83541986e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:53.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.870+0000 7f83594a8640 1 -- 172.21.15.129:0/2184626436 >> 172.21.15.129:0/2184626436 conn(0x7f83540770b0 msgr2=0x7f8354103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:53.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.871+0000 7f83594a8640 1 -- 172.21.15.129:0/2184626436 shutdown_connections 2024-05-20T18:06:53.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:53.871+0000 7f83594a8640 1 -- 172.21.15.129:0/2184626436 wait complete. 2024-05-20T18:06:54.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:54 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3466706058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:06:54.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:54 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3466706058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:06:54.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:54 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3466706058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:06:54.880 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:06:54.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:06:54.881 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:06:55.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.012+0000 7f09e6bbc640 1 -- 172.21.15.129:0/1598276874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09e010a6e0 msgr2=0x7f09e010cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.012+0000 7f09e6bbc640 1 --2- 172.21.15.129:0/1598276874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09e010a6e0 0x7f09e010cb00 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009900 tx=0x7f09d402a410 comp rx=0 tx=0).stop 2024-05-20T18:06:55.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.012+0000 7f09e6bbc640 1 -- 172.21.15.129:0/1598276874 shutdown_connections 2024-05-20T18:06:55.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.012+0000 7f09e6bbc640 1 --2- 172.21.15.129:0/1598276874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09e010d040 0x7f09e010f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.012+0000 7f09e6bbc640 1 --2- 172.21.15.129:0/1598276874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09e010a6e0 0x7f09e010cb00 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.012+0000 7f09e6bbc640 1 --2- 172.21.15.129:0/1598276874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e0103c80 0x7f09e0104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.012+0000 7f09e6bbc640 1 -- 172.21.15.129:0/1598276874 >> 172.21.15.129:0/1598276874 conn(0x7f09e00770c0 msgr2=0x7f09e0077500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.012+0000 7f09e6bbc640 1 -- 172.21.15.129:0/1598276874 shutdown_connections 2024-05-20T18:06:55.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.012+0000 7f09e6bbc640 1 -- 172.21.15.129:0/1598276874 wait complete. 2024-05-20T18:06:55.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e6bbc640 1 Processor -- start 2024-05-20T18:06:55.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e6bbc640 1 -- start start 2024-05-20T18:06:55.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e6bbc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09e0103c80 0x7f09e019f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e6bbc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09e010a6e0 0x7f09e019fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e6bbc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e010d040 0x7f09e0199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e6bbc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f09e0112de0 con 0x7f09e010a6e0 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e6bbc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09e0112c60 con 0x7f09e010d040 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e6bbc640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f09e0112f60 con 0x7f09e0103c80 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09dffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09e010a6e0 0x7f09e019fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09dffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09e010a6e0 0x7f09e019fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53064/0 (socket says 172.21.15.129:53064) 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09dffff640 1 -- 172.21.15.129:0/3773485290 learned_addr learned my addr 172.21.15.129:0/3773485290 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e5159640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e010d040 0x7f09e0199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e4958640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09e0103c80 0x7f09e019f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09dffff640 1 -- 172.21.15.129:0/3773485290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09e0103c80 msgr2=0x7f09e019f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09dffff640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09e0103c80 0x7f09e019f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09dffff640 1 -- 172.21.15.129:0/3773485290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e010d040 msgr2=0x7f09e0199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09dffff640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e010d040 0x7f09e0199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09dffff640 1 -- 172.21.15.129:0/3773485290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e0199d90 con 0x7f09e010a6e0 2024-05-20T18:06:55.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e4958640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09e0103c80 0x7f09e019f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:55.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.013+0000 7f09e5159640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e010d040 0x7f09e0199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:55.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.014+0000 7f09dffff640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09e010a6e0 0x7f09e019fa20 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f09d4005940 tx=0x7f09d4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.014+0000 7f09ddffb640 1 -- 172.21.15.129:0/3773485290 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d403f030 con 0x7f09e010a6e0 2024-05-20T18:06:55.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.014+0000 7f09ddffb640 1 -- 172.21.15.129:0/3773485290 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09d4003ad0 con 0x7f09e010a6e0 2024-05-20T18:06:55.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.014+0000 7f09ddffb640 1 -- 172.21.15.129:0/3773485290 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4035070 con 0x7f09e010a6e0 2024-05-20T18:06:55.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.014+0000 7f09e6bbc640 1 -- 172.21.15.129:0/3773485290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e0199fc0 con 0x7f09e010a6e0 2024-05-20T18:06:55.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.014+0000 7f09e6bbc640 1 -- 172.21.15.129:0/3773485290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e019a4b0 con 0x7f09e010a6e0 2024-05-20T18:06:55.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.016+0000 7f09ddffb640 1 -- 172.21.15.129:0/3773485290 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f09d4042030 con 0x7f09e010a6e0 2024-05-20T18:06:55.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09eda61740 1 Processor -- start 2024-05-20T18:06:55.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09eda61740 1 -- start start 2024-05-20T18:06:55.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09eda61740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020bc3410 0x55e020f624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09eda61740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f629f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09eda61740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020ba2730 0x55e020f62f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09eda61740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e020ec7990 con 0x55e020b9a550 2024-05-20T18:06:55.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09eda61740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e020e6f5b0 con 0x55e020bc3410 2024-05-20T18:06:55.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09eda61740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e020ec3950 con 0x55e020ba2730 2024-05-20T18:06:55.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09e4958640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f629f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09e4958640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f629f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53080/0 (socket says 172.21.15.129:53080) 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09e4958640 1 -- 172.21.15.129:0/1601143296 learned_addr learned my addr 172.21.15.129:0/1601143296 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09dffff640 1 --2- 172.21.15.129:0/1601143296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020ba2730 0x55e020f62f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09e5159640 1 --2- 172.21.15.129:0/1601143296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020bc3410 0x55e020f624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09ddffb640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f09c80a2d00 0x7f09c80a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09e4958640 1 -- 172.21.15.129:0/1601143296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020ba2730 msgr2=0x55e020f62f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09e4958640 1 --2- 172.21.15.129:0/1601143296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020ba2730 0x55e020f62f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09e4958640 1 -- 172.21.15.129:0/1601143296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020bc3410 msgr2=0x55e020f624b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09e4958640 1 --2- 172.21.15.129:0/1601143296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020bc3410 0x55e020f624b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09e4958640 1 -- 172.21.15.129:0/1601143296 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e020ca37e0 con 0x55e020b9a550 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.017+0000 7f09e5159640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f09c80a2d00 0x7f09c80a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.018+0000 7f09ddffb640 1 -- 172.21.15.129:0/3773485290 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f09d40efcc0 con 0x7f09e010a6e0 2024-05-20T18:06:55.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.018+0000 7f09e4958640 1 --2- 172.21.15.129:0/1601143296 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f629f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f09d000bd40 tx=0x7f09d000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5f27ac02d08843 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.018+0000 7f09c6ffd640 1 -- 172.21.15.129:0/1601143296 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d0015070 con 0x55e020b9a550 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.018+0000 7f09c6ffd640 1 -- 172.21.15.129:0/1601143296 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09d000be30 con 0x55e020b9a550 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.018+0000 7f09c6ffd640 1 -- 172.21.15.129:0/1601143296 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d0011a30 con 0x55e020b9a550 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09eda61740 1 -- 172.21.15.129:0/1601143296 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 msgr2=0x55e020f629f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09eda61740 1 --2- 172.21.15.129:0/1601143296 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f629f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f09d000bd40 tx=0x7f09d000f730 comp rx=0 tx=0).stop 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09e5159640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f09c80a2d00 0x7f09c80a5210 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55e020f5f7c0 tx=0x7f09d8004a40 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09eda61740 1 -- 172.21.15.129:0/1601143296 shutdown_connections 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09eda61740 1 --2- 172.21.15.129:0/1601143296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020ba2730 0x55e020f62f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09eda61740 1 --2- 172.21.15.129:0/1601143296 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f629f0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09eda61740 1 --2- 172.21.15.129:0/1601143296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020bc3410 0x55e020f624b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09eda61740 1 -- 172.21.15.129:0/1601143296 >> 172.21.15.129:0/1601143296 conn(0x55e020bb7560 msgr2=0x55e020f5edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09eda61740 1 -- 172.21.15.129:0/1601143296 shutdown_connections 2024-05-20T18:06:55.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09eda61740 1 -- 172.21.15.129:0/1601143296 wait complete. 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09eda61740 1 -- start start 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.019+0000 7f09eda61740 1 Processor -- start 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09eda61740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f72b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09eda61740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020ba2730 0x55e020f73070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09eda61740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020bc3410 0x55e020f70fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09eda61740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e020f73d80 con 0x55e020b9a550 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09eda61740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e020e82b40 con 0x55e020ba2730 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09eda61740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e020f73f00 con 0x55e020bc3410 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09eda61740 10 client.0 _ll_register_callbacks cb 0x55e020e770f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09e4958640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f72b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09e4958640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f72b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53082/0 (socket says 172.21.15.129:53082) 2024-05-20T18:06:55.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09e4958640 1 -- 172.21.15.129:0/2893147064 learned_addr learned my addr 172.21.15.129:0/2893147064 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09dffff640 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020ba2730 0x55e020f73070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09e5159640 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020bc3410 0x55e020f70fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09e4958640 1 -- 172.21.15.129:0/2893147064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020bc3410 msgr2=0x55e020f70fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09e4958640 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020bc3410 0x55e020f70fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09e4958640 1 -- 172.21.15.129:0/2893147064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020ba2730 msgr2=0x55e020f73070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09e4958640 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020ba2730 0x55e020f73070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09e4958640 1 -- 172.21.15.129:0/2893147064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e020d03f70 con 0x55e020b9a550 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09e4958640 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f72b30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f09d00115b0 tx=0x7f09d001aef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381fa98fecec707e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.020+0000 7f09bffff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09bffff640 1 -- 172.21.15.129:0/2893147064 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d0028060 con 0x55e020b9a550 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09bffff640 1 -- 172.21.15.129:0/2893147064 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f09d0005250 con 0x55e020b9a550 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09eda61740 1 -- 172.21.15.129:0/2893147064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09d000be30 con 0x55e020b9a550 2024-05-20T18:06:55.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09bffff640 1 -- 172.21.15.129:0/2893147064 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f09d0015070 con 0x55e020b9a550 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09eda61740 10 client.16713 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09eda61740 1 -- 172.21.15.129:0/2893147064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e020b9b170 con 0x55e020b9a550 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09bffff640 1 -- 172.21.15.129:0/2893147064 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09d000be30 con 0x55e020b9a550 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09bffff640 1 -- 172.21.15.129:0/2893147064 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f09d0004920 con 0x55e020b9a550 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09eda61740 10 client.16713 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09eda61740 10 client.16713 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09eda61740 20 client.16713 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09eda61740 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e020b821f0 0x55e020f83420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09eda61740 4 client.16713 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.021+0000 7f09eda61740 1 -- 172.21.15.129:0/2893147064 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e020ed50c0 con 0x55e020b821f0 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.022+0000 7f09dffff640 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e020b821f0 0x55e020f83420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.022+0000 7f09bffff640 10 client.16713 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:55.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.022+0000 7f09dffff640 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e020b821f0 0x55e020f83420 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=561e337559115f66 server_cookie=9db19012f20fcbca in_seq=0 out_seq=0 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09bffff640 1 -- 172.21.15.129:0/2893147064 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e020ed50c0 con 0x55e020b821f0 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09bffff640 10 client.16713 handle_command_reply: tid=0 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 1 client.16713 shutdown 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 1 -- 172.21.15.129:0/2893147064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 msgr2=0x55e020f72b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f72b30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f09d00115b0 tx=0x7f09d001aef0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 1 -- 172.21.15.129:0/2893147064 shutdown_connections 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e020b821f0 0x55e020f83420 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020ba2730 0x55e020f73070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09bffff640 0 client.16713 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020bc3410 0x55e020f70fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 1 --2- 172.21.15.129:0/2893147064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f72b30 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 1 -- 172.21.15.129:0/2893147064 >> 172.21.15.129:0/2893147064 conn(0x55e020bb7560 msgr2=0x55e020f64340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 1 -- 172.21.15.129:0/2893147064 shutdown_connections 2024-05-20T18:06:55.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 1 -- 172.21.15.129:0/2893147064 wait complete. 2024-05-20T18:06:55.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.023+0000 7f09eda61740 20 client.16713 trim_cache size 0 max 16384 2024-05-20T18:06:55.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09eda61740 1 Processor -- start 2024-05-20T18:06:55.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09eda61740 1 -- start start 2024-05-20T18:06:55.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09eda61740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020bb7560 0x55e020f62a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09eda61740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f70890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09eda61740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020ba2730 0x55e020f70dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09eda61740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f09e01aa9c0 con 0x55e020b9a550 2024-05-20T18:06:55.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09eda61740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09e01ad220 con 0x55e020bb7560 2024-05-20T18:06:55.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09eda61740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f09e01ab010 con 0x55e020ba2730 2024-05-20T18:06:55.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09dffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f70890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09dffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f70890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53088/0 (socket says 172.21.15.129:53088) 2024-05-20T18:06:55.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09dffff640 1 -- 172.21.15.129:0/4021936036 learned_addr learned my addr 172.21.15.129:0/4021936036 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:55.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09e4958640 1 --2- 172.21.15.129:0/4021936036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020bb7560 0x55e020f62a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.041+0000 7f09e5159640 1 --2- 172.21.15.129:0/4021936036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020ba2730 0x55e020f70dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.042+0000 7f09dffff640 1 -- 172.21.15.129:0/4021936036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020ba2730 msgr2=0x55e020f70dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.042+0000 7f09dffff640 1 --2- 172.21.15.129:0/4021936036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020ba2730 0x55e020f70dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.042+0000 7f09dffff640 1 -- 172.21.15.129:0/4021936036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020bb7560 msgr2=0x55e020f62a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.042+0000 7f09dffff640 1 --2- 172.21.15.129:0/4021936036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020bb7560 0x55e020f62a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.042+0000 7f09dffff640 1 -- 172.21.15.129:0/4021936036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e020f7e1d0 con 0x55e020b9a550 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.042+0000 7f09e5159640 1 --2- 172.21.15.129:0/4021936036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020ba2730 0x55e020f70dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.042+0000 7f09dffff640 1 --2- 172.21.15.129:0/4021936036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f70890 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f09d40b8600 tx=0x7f09d40b8a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756fc65ea4d37385 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.042+0000 7f09c67fc640 1 -- 172.21.15.129:0/4021936036 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4049040 con 0x55e020b9a550 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.042+0000 7f09c67fc640 1 -- 172.21.15.129:0/4021936036 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09d40bc8a0 con 0x55e020b9a550 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.042+0000 7f09c67fc640 1 -- 172.21.15.129:0/4021936036 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d40f1040 con 0x55e020b9a550 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 -- 172.21.15.129:0/4021936036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 msgr2=0x55e020f70890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 --2- 172.21.15.129:0/4021936036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f70890 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f09d40b8600 tx=0x7f09d40b8a30 comp rx=0 tx=0).stop 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 -- 172.21.15.129:0/4021936036 shutdown_connections 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 --2- 172.21.15.129:0/4021936036 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020ba2730 0x55e020f70dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 --2- 172.21.15.129:0/4021936036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020b9a550 0x55e020f70890 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 --2- 172.21.15.129:0/4021936036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020bb7560 0x55e020f62a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 -- 172.21.15.129:0/4021936036 >> 172.21.15.129:0/4021936036 conn(0x55e020f73210 msgr2=0x55e020f64b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 -- 172.21.15.129:0/4021936036 shutdown_connections 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 -- 172.21.15.129:0/4021936036 wait complete. 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 -- start start 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 Processor -- start 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020b9a550 0x55e020f83fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.043+0000 7f09eda61740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020ba2730 0x55e020f84520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09eda61740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020bb7560 0x55e020f84a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09eda61740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f09e01ab6e0 con 0x55e020ba2730 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09eda61740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09e01acb50 con 0x55e020b9a550 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09eda61740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f09e01abdb0 con 0x55e020bb7560 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09eda61740 10 client.0 _ll_register_callbacks cb 0x7f09d40f36e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09dffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020ba2730 0x55e020f84520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09dffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020ba2730 0x55e020f84520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53094/0 (socket says 172.21.15.129:53094) 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09dffff640 1 -- 172.21.15.129:0/4285335669 learned_addr learned my addr 172.21.15.129:0/4285335669 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09e4958640 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020b9a550 0x55e020f83fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09e5159640 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020bb7560 0x55e020f84a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09dffff640 1 -- 172.21.15.129:0/4285335669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020bb7560 msgr2=0x55e020f84a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09dffff640 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020bb7560 0x55e020f84a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09dffff640 1 -- 172.21.15.129:0/4285335669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020b9a550 msgr2=0x55e020f83fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09dffff640 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020b9a550 0x55e020f83fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09dffff640 1 -- 172.21.15.129:0/4285335669 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e020fea550 con 0x55e020ba2730 2024-05-20T18:06:55.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09dffff640 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020ba2730 0x55e020f84520 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f09d4032c00 tx=0x7f09d40095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e5323da8fc091 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09bffff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:55.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09bffff640 1 -- 172.21.15.129:0/4285335669 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d40f1030 con 0x55e020ba2730 2024-05-20T18:06:55.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09bffff640 1 -- 172.21.15.129:0/4285335669 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7f09d403b460 con 0x55e020ba2730 2024-05-20T18:06:55.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09eda61740 1 -- 172.21.15.129:0/4285335669 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09d40bc8a0 con 0x55e020ba2730 2024-05-20T18:06:55.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09bffff640 1 -- 172.21.15.129:0/4285335669 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f09d40bbb00 con 0x55e020ba2730 2024-05-20T18:06:55.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09eda61740 10 client.16719 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:55.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.044+0000 7f09eda61740 1 -- 172.21.15.129:0/4285335669 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e020feb680 con 0x55e020ba2730 2024-05-20T18:06:55.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.045+0000 7f09bffff640 1 -- 172.21.15.129:0/4285335669 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09d40bc8a0 con 0x55e020ba2730 2024-05-20T18:06:55.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.045+0000 7f09bffff640 1 -- 172.21.15.129:0/4285335669 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f09d40035f0 con 0x55e020ba2730 2024-05-20T18:06:55.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.045+0000 7f09eda61740 10 client.16719 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:55.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.045+0000 7f09eda61740 10 client.16719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:55.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.045+0000 7f09eda61740 20 client.16719 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:55.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.045+0000 7f09eda61740 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e020bc3410 0x55e020fedde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.045+0000 7f09eda61740 4 client.16719 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:06:55.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.045+0000 7f09eda61740 1 -- 172.21.15.129:0/4285335669 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e020ecf410 con 0x55e020bc3410 2024-05-20T18:06:55.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.045+0000 7f09e4958640 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e020bc3410 0x55e020fedde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.046+0000 7f09bffff640 10 client.16719 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.046+0000 7f09e4958640 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e020bc3410 0x55e020fedde0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=972f09cdc59fa62b server_cookie=9f8f35ed68008e5d in_seq=0 out_seq=0 2024-05-20T18:06:55.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09bffff640 1 -- 172.21.15.129:0/4285335669 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e020ecf410 con 0x55e020bc3410 2024-05-20T18:06:55.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09bffff640 10 client.16719 handle_command_reply: tid=0 2024-05-20T18:06:55.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 1 client.16719 shutdown 2024-05-20T18:06:55.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 1 -- 172.21.15.129:0/4285335669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020ba2730 msgr2=0x55e020f84520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020ba2730 0x55e020f84520 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f09d4032c00 tx=0x7f09d40095d0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 1 -- 172.21.15.129:0/4285335669 shutdown_connections 2024-05-20T18:06:55.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e020bc3410 0x55e020fedde0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e020bb7560 0x55e020f84a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e020ba2730 0x55e020f84520 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09bffff640 0 client.16719 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:55.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 1 --2- 172.21.15.129:0/4285335669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e020b9a550 0x55e020f83fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 1 -- 172.21.15.129:0/4285335669 >> 172.21.15.129:0/4285335669 conn(0x55e020f73210 msgr2=0x55e020f64af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 1 -- 172.21.15.129:0/4285335669 shutdown_connections 2024-05-20T18:06:55.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 1 -- 172.21.15.129:0/4285335669 wait complete. 2024-05-20T18:06:55.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.047+0000 7f09eda61740 20 client.16719 trim_cache size 0 max 16384 2024-05-20T18:06:55.052 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:06:55.052 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:06:55.052 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "722c2a0f-29bf-494b-a7ae-b9ad5c480c48", 2024-05-20T18:06:55.053 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:06:55.053 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:06:55.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 -- 172.21.15.129:0/3773485290 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f09c80a2d00 msgr2=0x7f09c80a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f09c80a2d00 0x7f09c80a5210 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x55e020f5f7c0 tx=0x7f09d8004a40 comp rx=0 tx=0).stop 2024-05-20T18:06:55.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 -- 172.21.15.129:0/3773485290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09e010a6e0 msgr2=0x7f09e019fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09e010a6e0 0x7f09e019fa20 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f09d4005940 tx=0x7f09d4004270 comp rx=0 tx=0).stop 2024-05-20T18:06:55.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 -- 172.21.15.129:0/3773485290 shutdown_connections 2024-05-20T18:06:55.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f09c80a2d00 0x7f09c80a5210 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e010d040 0x7f09e0199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09e010a6e0 0x7f09e019fa20 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 --2- 172.21.15.129:0/3773485290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f09e0103c80 0x7f09e019f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 -- 172.21.15.129:0/3773485290 >> 172.21.15.129:0/3773485290 conn(0x7f09e00770c0 msgr2=0x7f09e0100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 -- 172.21.15.129:0/3773485290 shutdown_connections 2024-05-20T18:06:55.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.048+0000 7f09e6bbc640 1 -- 172.21.15.129:0/3773485290 wait complete. 2024-05-20T18:06:55.058 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:06:55.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:06:55.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.175+0000 7fd1684ca640 1 -- 172.21.15.129:0/756873879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd160102210 msgr2=0x7fd160102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.175+0000 7fd1684ca640 1 --2- 172.21.15.129:0/756873879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd160102210 0x7fd160102620 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd150009900 tx=0x7fd15002a410 comp rx=0 tx=0).stop 2024-05-20T18:06:55.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.175+0000 7fd1684ca640 1 -- 172.21.15.129:0/756873879 shutdown_connections 2024-05-20T18:06:55.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.175+0000 7fd1684ca640 1 --2- 172.21.15.129:0/756873879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd16010c350 0x7fd16010e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.175+0000 7fd1684ca640 1 --2- 172.21.15.129:0/756873879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd160102210 0x7fd160102620 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.175+0000 7fd1684ca640 1 --2- 172.21.15.129:0/756873879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1601079f0 0x7fd160101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.175+0000 7fd1684ca640 1 -- 172.21.15.129:0/756873879 >> 172.21.15.129:0/756873879 conn(0x7fd160076830 msgr2=0x7fd160076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.175+0000 7fd1684ca640 1 -- 172.21.15.129:0/756873879 shutdown_connections 2024-05-20T18:06:55.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.175+0000 7fd1684ca640 1 -- 172.21.15.129:0/756873879 wait complete. 2024-05-20T18:06:55.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.175+0000 7fd1684ca640 1 Processor -- start 2024-05-20T18:06:55.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd1684ca640 1 -- start start 2024-05-20T18:06:55.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd1684ca640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd160102210 0x7fd16019f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd1684ca640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1601079f0 0x7fd16019fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd1684ca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd16010c350 0x7fd1601996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd1684ca640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd160111d00 con 0x7fd16010c350 2024-05-20T18:06:55.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd1684ca640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd160111b80 con 0x7fd160102210 2024-05-20T18:06:55.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd1684ca640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd160111e80 con 0x7fd1601079f0 2024-05-20T18:06:55.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd166a67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd16010c350 0x7fd1601996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd166a67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd16010c350 0x7fd1601996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53110/0 (socket says 172.21.15.129:53110) 2024-05-20T18:06:55.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd166a67640 1 -- 172.21.15.129:0/960342387 learned_addr learned my addr 172.21.15.129:0/960342387 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:55.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd165a65640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1601079f0 0x7fd16019fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd166266640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd160102210 0x7fd16019f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd166a67640 1 -- 172.21.15.129:0/960342387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1601079f0 msgr2=0x7fd16019fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd166a67640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1601079f0 0x7fd16019fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd166a67640 1 -- 172.21.15.129:0/960342387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd160102210 msgr2=0x7fd16019f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd166a67640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd160102210 0x7fd16019f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd166a67640 1 -- 172.21.15.129:0/960342387 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd160199e00 con 0x7fd16010c350 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd165a65640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1601079f0 0x7fd16019fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.176+0000 7fd166a67640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd16010c350 0x7fd1601996d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd15800eaa0 tx=0x7fd15800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.177+0000 7fd1577fe640 1 -- 172.21.15.129:0/960342387 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15800cdb0 con 0x7fd16010c350 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.177+0000 7fd1577fe640 1 -- 172.21.15.129:0/960342387 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd158005020 con 0x7fd16010c350 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.177+0000 7fd1577fe640 1 -- 172.21.15.129:0/960342387 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1580116b0 con 0x7fd16010c350 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.177+0000 7fd1684ca640 1 -- 172.21.15.129:0/960342387 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16019a0f0 con 0x7fd16010c350 2024-05-20T18:06:55.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.177+0000 7fd1684ca640 1 -- 172.21.15.129:0/960342387 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1601a62f0 con 0x7fd16010c350 2024-05-20T18:06:55.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.179+0000 7fd1577fe640 1 -- 172.21.15.129:0/960342387 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fd158005560 con 0x7fd16010c350 2024-05-20T18:06:55.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd16f36b740 1 Processor -- start 2024-05-20T18:06:55.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd16f36b740 1 -- start start 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349167d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd1577fe640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd14c0a2d80 0x7fd14c0a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349168270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c3491687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd16f36b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c3490cdd00 con 0x55c348da04d0 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd16f36b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c3490754c0 con 0x55c348dc9390 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd16f36b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c3490c9cc0 con 0x55c348da86b0 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd166266640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd14c0a2d80 0x7fd14c0a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd165a65640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349168270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd165a65640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349168270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53122/0 (socket says 172.21.15.129:53122) 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd1577fe640 1 -- 172.21.15.129:0/960342387 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7fd1580cf6d0 con 0x7fd16010c350 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd165a65640 1 -- 172.21.15.129:0/3523514966 learned_addr learned my addr 172.21.15.129:0/3523514966 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd165a65640 1 --2- 172.21.15.129:0/3523514966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349167d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.180+0000 7fd166a67640 1 --2- 172.21.15.129:0/3523514966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c3491687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.181+0000 7fd165a65640 1 -- 172.21.15.129:0/3523514966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 msgr2=0x55c3491687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.181+0000 7fd165a65640 1 --2- 172.21.15.129:0/3523514966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c3491687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.181+0000 7fd165a65640 1 -- 172.21.15.129:0/3523514966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 msgr2=0x55c349167d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.181+0000 7fd165a65640 1 --2- 172.21.15.129:0/3523514966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349167d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.181+0000 7fd165a65640 1 -- 172.21.15.129:0/3523514966 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c348eab7d0 con 0x55c348da04d0 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.181+0000 7fd166a67640 1 --2- 172.21.15.129:0/3523514966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c3491687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.181+0000 7fd166266640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd14c0a2d80 0x7fd14c0a5290 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd15c005fd0 tx=0x7fd15c005850 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.181+0000 7fd165a65640 1 --2- 172.21.15.129:0/3523514966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349168270 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd150009230 tx=0x7fd1500056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6f7151f3377e6a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.181+0000 7fd154bfe640 1 -- 172.21.15.129:0/3523514966 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150035040 con 0x55c348da04d0 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.181+0000 7fd154bfe640 1 -- 172.21.15.129:0/3523514966 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1500318f0 con 0x55c348da04d0 2024-05-20T18:06:55.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.181+0000 7fd154bfe640 1 -- 172.21.15.129:0/3523514966 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150003db0 con 0x55c348da04d0 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.182+0000 7fd16f36b740 1 -- 172.21.15.129:0/3523514966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 msgr2=0x55c349168270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.182+0000 7fd16f36b740 1 --2- 172.21.15.129:0/3523514966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349168270 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd150009230 tx=0x7fd1500056d0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.182+0000 7fd16f36b740 1 -- 172.21.15.129:0/3523514966 shutdown_connections 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.182+0000 7fd16f36b740 1 --2- 172.21.15.129:0/3523514966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c3491687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.182+0000 7fd16f36b740 1 --2- 172.21.15.129:0/3523514966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349168270 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.182+0000 7fd16f36b740 1 --2- 172.21.15.129:0/3523514966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349167d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.182+0000 7fd16f36b740 1 -- 172.21.15.129:0/3523514966 >> 172.21.15.129:0/3523514966 conn(0x55c348dbd4e0 msgr2=0x55c3491644e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.182+0000 7fd16f36b740 1 -- 172.21.15.129:0/3523514966 shutdown_connections 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.182+0000 7fd16f36b740 1 -- 172.21.15.129:0/3523514966 wait complete. 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.182+0000 7fd16f36b740 1 -- start start 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.182+0000 7fd16f36b740 1 Processor -- start 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da04d0 0x55c349173e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da86b0 0x55c349174390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349172360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd16f36b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c349174fc0 con 0x55c348da86b0 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd16f36b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c349088b90 con 0x55c348dc9390 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd16f36b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c349175140 con 0x55c348da04d0 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd16f36b740 10 client.0 _ll_register_callbacks cb 0x55c3490c4da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd165a65640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da86b0 0x55c349174390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd165a65640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da86b0 0x55c349174390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53124/0 (socket says 172.21.15.129:53124) 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd165a65640 1 -- 172.21.15.129:0/1882560586 learned_addr learned my addr 172.21.15.129:0/1882560586 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd166a67640 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349172360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd166266640 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da04d0 0x55c349173e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd165a65640 1 -- 172.21.15.129:0/1882560586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da04d0 msgr2=0x55c349173e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd165a65640 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da04d0 0x55c349173e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd165a65640 1 -- 172.21.15.129:0/1882560586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 msgr2=0x55c349172360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd165a65640 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349172360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd165a65640 1 -- 172.21.15.129:0/1882560586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c348de13f0 con 0x55c348da86b0 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd165a65640 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da86b0 0x55c349174390 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd1500092a0 tx=0x7fd15003d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd734e42c49a98b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd149ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd149ffb640 1 -- 172.21.15.129:0/1882560586 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150035070 con 0x55c348da86b0 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd149ffb640 1 -- 172.21.15.129:0/1882560586 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7fd15003dab0 con 0x55c348da86b0 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.183+0000 7fd16f36b740 1 -- 172.21.15.129:0/1882560586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1500318f0 con 0x55c348da86b0 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd149ffb640 1 -- 172.21.15.129:0/1882560586 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd15003dcb0 con 0x55c348da86b0 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd16f36b740 10 client.16731 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd16f36b740 1 -- 172.21.15.129:0/1882560586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c348da10f0 con 0x55c348da86b0 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd149ffb640 1 -- 172.21.15.129:0/1882560586 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1500318f0 con 0x55c348da86b0 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd149ffb640 1 -- 172.21.15.129:0/1882560586 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fd15003a040 con 0x55c348da86b0 2024-05-20T18:06:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd16f36b740 10 client.16731 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd16f36b740 10 client.16731 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd16f36b740 20 client.16731 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd16f36b740 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c348d87f60 0x55c349184a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd16f36b740 4 client.16731 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd16f36b740 1 -- 172.21.15.129:0/1882560586 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3490db430 con 0x55c348d87f60 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.184+0000 7fd166266640 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c348d87f60 0x55c349184a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.185+0000 7fd149ffb640 10 client.16731 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.185+0000 7fd166266640 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c348d87f60 0x55c349184a20 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9c3e2e64fbf0a67 server_cookie=a07a03efd34b6387 in_seq=0 out_seq=0 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.185+0000 7fd149ffb640 1 -- 172.21.15.129:0/1882560586 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c3490db430 con 0x55c348d87f60 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.185+0000 7fd149ffb640 10 client.16731 handle_command_reply: tid=0 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 1 client.16731 shutdown 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 1 -- 172.21.15.129:0/1882560586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da86b0 msgr2=0x55c349174390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da86b0 0x55c349174390 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd1500092a0 tx=0x7fd15003d6b0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 1 -- 172.21.15.129:0/1882560586 shutdown_connections 2024-05-20T18:06:55.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c348d87f60 0x55c349184a20 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349172360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da86b0 0x55c349174390 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 1 --2- 172.21.15.129:0/1882560586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da04d0 0x55c349173e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd149ffb640 0 client.16731 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:55.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 1 -- 172.21.15.129:0/1882560586 >> 172.21.15.129:0/1882560586 conn(0x55c348dbd4e0 msgr2=0x55c349169a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 1 -- 172.21.15.129:0/1882560586 shutdown_connections 2024-05-20T18:06:55.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 1 -- 172.21.15.129:0/1882560586 wait complete. 2024-05-20T18:06:55.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.186+0000 7fd16f36b740 20 client.16731 trim_cache size 0 max 16384 2024-05-20T18:06:55.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.204+0000 7fd16f36b740 1 Processor -- start 2024-05-20T18:06:55.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.204+0000 7fd16f36b740 1 -- start start 2024-05-20T18:06:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.204+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349174190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.204+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c349171b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.204+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349187a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.204+0000 7fd16f36b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1601aca70 con 0x55c348da04d0 2024-05-20T18:06:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.204+0000 7fd16f36b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1601af2d0 con 0x55c348dc9390 2024-05-20T18:06:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.204+0000 7fd16f36b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1601ad0c0 con 0x55c348da86b0 2024-05-20T18:06:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.204+0000 7fd166266640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349174190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd166266640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349174190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53136/0 (socket says 172.21.15.129:53136) 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd166266640 1 -- 172.21.15.129:0/2820357651 learned_addr learned my addr 172.21.15.129:0/2820357651 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd165a65640 1 --2- 172.21.15.129:0/2820357651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c349171b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd166a67640 1 --2- 172.21.15.129:0/2820357651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349187a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd166266640 1 -- 172.21.15.129:0/2820357651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 msgr2=0x55c349171b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd166266640 1 --2- 172.21.15.129:0/2820357651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c349171b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd166266640 1 -- 172.21.15.129:0/2820357651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 msgr2=0x55c349187a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd166266640 1 --2- 172.21.15.129:0/2820357651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349187a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd166266640 1 -- 172.21.15.129:0/2820357651 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c34917f3a0 con 0x55c348da04d0 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd165a65640 1 --2- 172.21.15.129:0/2820357651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c349171b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd166266640 1 --2- 172.21.15.129:0/2820357651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349174190 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd16006a230 tx=0x7fd15c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b76f259ce6f739 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd14bfff640 1 -- 172.21.15.129:0/2820357651 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c026250 con 0x55c348da04d0 2024-05-20T18:06:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd14bfff640 1 -- 172.21.15.129:0/2820357651 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd15c0268d0 con 0x55c348da04d0 2024-05-20T18:06:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.205+0000 7fd14bfff640 1 -- 172.21.15.129:0/2820357651 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c01fd90 con 0x55c348da04d0 2024-05-20T18:06:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.206+0000 7fd16f36b740 1 -- 172.21.15.129:0/2820357651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 msgr2=0x55c349174190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.206+0000 7fd16f36b740 1 --2- 172.21.15.129:0/2820357651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349174190 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd16006a230 tx=0x7fd15c015000 comp rx=0 tx=0).stop 2024-05-20T18:06:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.206+0000 7fd16f36b740 1 -- 172.21.15.129:0/2820357651 shutdown_connections 2024-05-20T18:06:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.206+0000 7fd16f36b740 1 --2- 172.21.15.129:0/2820357651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348dc9390 0x55c349187a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.206+0000 7fd16f36b740 1 --2- 172.21.15.129:0/2820357651 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c349171b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.206+0000 7fd16f36b740 1 --2- 172.21.15.129:0/2820357651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348da04d0 0x55c349174190 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.206+0000 7fd16f36b740 1 -- 172.21.15.129:0/2820357651 >> 172.21.15.129:0/2820357651 conn(0x55c348dbd4e0 msgr2=0x55c349173a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.206+0000 7fd16f36b740 1 -- 172.21.15.129:0/2820357651 shutdown_connections 2024-05-20T18:06:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.206+0000 7fd16f36b740 1 -- 172.21.15.129:0/2820357651 wait complete. 2024-05-20T18:06:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.206+0000 7fd16f36b740 1 -- start start 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.206+0000 7fd16f36b740 1 Processor -- start 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348da04d0 0x55c349187cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c349185060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd16f36b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348dc9390 0x55c3491855a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd16f36b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1601ad790 con 0x55c348dc9390 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd16f36b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1601aec00 con 0x55c348da04d0 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd16f36b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd1601ade60 con 0x55c348da86b0 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd16f36b740 10 client.0 _ll_register_callbacks cb 0x7fd150032d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd166a67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348dc9390 0x55c3491855a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd166a67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348dc9390 0x55c3491855a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53146/0 (socket says 172.21.15.129:53146) 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd166a67640 1 -- 172.21.15.129:0/2743246149 learned_addr learned my addr 172.21.15.129:0/2743246149 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd165a65640 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c349185060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd166266640 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348da04d0 0x55c349187cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd166a67640 1 -- 172.21.15.129:0/2743246149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 msgr2=0x55c349185060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd166a67640 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c349185060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd166a67640 1 -- 172.21.15.129:0/2743246149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348da04d0 msgr2=0x55c349187cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd166a67640 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348da04d0 0x55c349187cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd166a67640 1 -- 172.21.15.129:0/2743246149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c349185ae0 con 0x55c348dc9390 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd166a67640 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348dc9390 0x55c3491855a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fd158097fb0 tx=0x7fd15809c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4026daa66d50aaf server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd149ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd149ffb640 1 -- 172.21.15.129:0/2743246149 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15801f040 con 0x55c348dc9390 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.207+0000 7fd149ffb640 1 -- 172.21.15.129:0/2743246149 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7fd1580cd000 con 0x55c348dc9390 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd16f36b740 1 -- 172.21.15.129:0/2743246149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd15c0268d0 con 0x55c348dc9390 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd149ffb640 1 -- 172.21.15.129:0/2743246149 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd15809bb50 con 0x55c348dc9390 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd16f36b740 10 client.16737 fetch_fsmap learned FSMap version 26 2024-05-20T18:06:55.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd16f36b740 1 -- 172.21.15.129:0/2743246149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c3491ec7c0 con 0x55c348dc9390 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd149ffb640 1 -- 172.21.15.129:0/2743246149 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd15c0268d0 con 0x55c348dc9390 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd149ffb640 1 -- 172.21.15.129:0/2743246149 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fd15800f040 con 0x55c348dc9390 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd16f36b740 10 client.16737 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd16f36b740 10 client.16737 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd16f36b740 20 client.16737 populate_metadata read hostname 'smithi129' 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd16f36b740 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c348d5d3a0 0x55c3491eef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd16f36b740 4 client.16737 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd16f36b740 1 -- 172.21.15.129:0/2743246149 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c3490d5780 con 0x55c348d5d3a0 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.208+0000 7fd166266640 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c348d5d3a0 0x55c3491eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd149ffb640 10 client.16737 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd166266640 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c348d5d3a0 0x55c3491eef80 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=912a8ac3bb508775 server_cookie=5e673d819e52b503 in_seq=0 out_seq=0 2024-05-20T18:06:55.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd149ffb640 1 -- 172.21.15.129:0/2743246149 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c3490d5780 con 0x55c348d5d3a0 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd149ffb640 10 client.16737 handle_command_reply: tid=0 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 1 client.16737 shutdown 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 1 -- 172.21.15.129:0/2743246149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348dc9390 msgr2=0x55c3491855a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348dc9390 0x55c3491855a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fd158097fb0 tx=0x7fd15809c390 comp rx=0 tx=0).stop 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 1 -- 172.21.15.129:0/2743246149 shutdown_connections 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55c348d5d3a0 0x55c3491eef80 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c348dc9390 0x55c3491855a0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd149ffb640 0 client.16737 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c348da86b0 0x55c349185060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 1 --2- 172.21.15.129:0/2743246149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c348da04d0 0x55c349187cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 1 -- 172.21.15.129:0/2743246149 >> 172.21.15.129:0/2743246149 conn(0x55c348dbd4e0 msgr2=0x55c349167ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 1 -- 172.21.15.129:0/2743246149 shutdown_connections 2024-05-20T18:06:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 1 -- 172.21.15.129:0/2743246149 wait complete. 2024-05-20T18:06:55.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.209+0000 7fd16f36b740 20 client.16737 trim_cache size 0 max 16384 2024-05-20T18:06:55.217 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:06:55.217 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:06:55.217 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:06:55.217 INFO:teuthology.orchestra.run.smithi129.stdout: "722c2a0f-29bf-494b-a7ae-b9ad5c480c48": { 2024-05-20T18:06:55.217 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:06:55.217 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "722c2a0f-29bf-494b-a7ae-b9ad5c480c48", 2024-05-20T18:06:55.217 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:06:55.217 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:06:55.218 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:06:55.218 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:06:55.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 -- 172.21.15.129:0/960342387 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd14c0a2d80 msgr2=0x7fd14c0a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd14c0a2d80 0x7fd14c0a5290 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd15c005fd0 tx=0x7fd15c005850 comp rx=0 tx=0).stop 2024-05-20T18:06:55.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 -- 172.21.15.129:0/960342387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd16010c350 msgr2=0x7fd1601996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:06:55.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd16010c350 0x7fd1601996d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd15800eaa0 tx=0x7fd15800ef60 comp rx=0 tx=0).stop 2024-05-20T18:06:55.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 -- 172.21.15.129:0/960342387 shutdown_connections 2024-05-20T18:06:55.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd14c0a2d80 0x7fd14c0a5290 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd16010c350 0x7fd1601996d0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd1601079f0 0x7fd16019fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 --2- 172.21.15.129:0/960342387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd160102210 0x7fd16019f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:06:55.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 -- 172.21.15.129:0/960342387 >> 172.21.15.129:0/960342387 conn(0x7fd160076830 msgr2=0x7fd160100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:06:55.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 -- 172.21.15.129:0/960342387 shutdown_connections 2024-05-20T18:06:55.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:06:55.210+0000 7fd1684ca640 1 -- 172.21.15.129:0/960342387 wait complete. 2024-05-20T18:06:55.220 INFO:tasks.cephfs.filesystem:scrub status for tag:722c2a0f-29bf-494b-a7ae-b9ad5c480c48 - {'path': '/', 'tag': '722c2a0f-29bf-494b-a7ae-b9ad5c480c48', 'options': 'recursive,force'} 2024-05-20T18:06:55.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:55 smithi154 ceph-mon[36482]: pgmap v257: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 29 MiB/s wr, 217 op/s 2024-05-20T18:06:55.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:55 smithi129 ceph-mon[32029]: pgmap v257: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 29 MiB/s wr, 217 op/s 2024-05-20T18:06:55.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:55 smithi204 ceph-mon[37076]: pgmap v257: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 29 MiB/s wr, 217 op/s 2024-05-20T18:06:56.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:56 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:06:56.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:56 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:56.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:56 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:06:56.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:56 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:56.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:56 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:06:56.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:56 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:06:56.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:56 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:56.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:56 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:06:56.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:56 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:56.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:56 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:06:56.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:56 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:06:56.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:56 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:56.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:56 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:06:56.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:56 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:06:56.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:56 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:06:57.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:57 smithi154 ceph-mon[36482]: pgmap v258: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 21 MiB/s wr, 160 op/s 2024-05-20T18:06:57.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:57 smithi129 ceph-mon[32029]: pgmap v258: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 21 MiB/s wr, 160 op/s 2024-05-20T18:06:57.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:57 smithi204 ceph-mon[37076]: pgmap v258: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 21 MiB/s wr, 160 op/s 2024-05-20T18:06:59.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:06:59 smithi154 ceph-mon[36482]: pgmap v259: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.2 KiB/s wr, 4 op/s 2024-05-20T18:06:59.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:06:59 smithi129 ceph-mon[32029]: pgmap v259: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.2 KiB/s wr, 4 op/s 2024-05-20T18:06:59.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:06:59 smithi204 ceph-mon[37076]: pgmap v259: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3.2 KiB/s wr, 4 op/s 2024-05-20T18:07:01.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:01 smithi154 ceph-mon[36482]: pgmap v260: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-05-20T18:07:01.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:01 smithi129 ceph-mon[32029]: pgmap v260: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-05-20T18:07:01.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:01 smithi204 ceph-mon[37076]: pgmap v260: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-05-20T18:07:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.933+0000 7fbc63c92640 1 -- 172.21.15.129:0/374279102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc5c107a40 msgr2=0x7fbc5c1015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.933+0000 7fbc63c92640 1 --2- 172.21.15.129:0/374279102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc5c107a40 0x7fbc5c1015d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009900 tx=0x7fbc5802a410 comp rx=0 tx=0).stop 2024-05-20T18:07:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.934+0000 7fbc63c92640 1 -- 172.21.15.129:0/374279102 shutdown_connections 2024-05-20T18:07:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.934+0000 7fbc63c92640 1 --2- 172.21.15.129:0/374279102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c10c3a0 0x7fbc5c10e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.934+0000 7fbc63c92640 1 --2- 172.21.15.129:0/374279102 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc5c101b10 0x7fbc5c101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.934+0000 7fbc63c92640 1 --2- 172.21.15.129:0/374279102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc5c107a40 0x7fbc5c1015d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.934+0000 7fbc63c92640 1 -- 172.21.15.129:0/374279102 >> 172.21.15.129:0/374279102 conn(0x7fbc5c0fd210 msgr2=0x7fbc5c0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.934+0000 7fbc63c92640 1 -- 172.21.15.129:0/374279102 shutdown_connections 2024-05-20T18:07:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.934+0000 7fbc63c92640 1 -- 172.21.15.129:0/374279102 wait complete. 2024-05-20T18:07:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.934+0000 7fbc63c92640 1 Processor -- start 2024-05-20T18:07:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.934+0000 7fbc63c92640 1 -- start start 2024-05-20T18:07:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc63c92640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc5c101b10 0x7fbc5c19f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc63c92640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc5c107a40 0x7fbc5c19fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc63c92640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c10c3a0 0x7fbc5c199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc63c92640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c111d50 con 0x7fbc5c107a40 2024-05-20T18:07:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc63c92640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c111bd0 con 0x7fbc5c10c3a0 2024-05-20T18:07:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc63c92640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c111ed0 con 0x7fbc5c101b10 2024-05-20T18:07:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc6122d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc5c107a40 0x7fbc5c19fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc6122d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc5c107a40 0x7fbc5c19fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35158/0 (socket says 172.21.15.129:35158) 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc6122d640 1 -- 172.21.15.129:0/1507474219 learned_addr learned my addr 172.21.15.129:0/1507474219 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc61a2e640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc5c101b10 0x7fbc5c19f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc6222f640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c10c3a0 0x7fbc5c199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc6122d640 1 -- 172.21.15.129:0/1507474219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc5c101b10 msgr2=0x7fbc5c19f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc6122d640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc5c101b10 0x7fbc5c19f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc6122d640 1 -- 172.21.15.129:0/1507474219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c10c3a0 msgr2=0x7fbc5c199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc6122d640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c10c3a0 0x7fbc5c199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc6122d640 1 -- 172.21.15.129:0/1507474219 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5c199ca0 con 0x7fbc5c107a40 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc6222f640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c10c3a0 0x7fbc5c199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc6122d640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc5c107a40 0x7fbc5c19fa20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00e9f0 tx=0x7fbc4c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.935+0000 7fbc52ffd640 1 -- 172.21.15.129:0/1507474219 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c00ccc0 con 0x7fbc5c107a40 2024-05-20T18:07:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.936+0000 7fbc63c92640 1 -- 172.21.15.129:0/1507474219 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc5c199f90 con 0x7fbc5c107a40 2024-05-20T18:07:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.936+0000 7fbc52ffd640 1 -- 172.21.15.129:0/1507474219 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc4c0035f0 con 0x7fbc5c107a40 2024-05-20T18:07:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.936+0000 7fbc52ffd640 1 -- 172.21.15.129:0/1507474219 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c00ce60 con 0x7fbc5c107a40 2024-05-20T18:07:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.936+0000 7fbc63c92640 1 -- 172.21.15.129:0/1507474219 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc5c19a490 con 0x7fbc5c107a40 2024-05-20T18:07:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.938+0000 7fbc52ffd640 1 -- 172.21.15.129:0/1507474219 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fbc4c0037a0 con 0x7fbc5c107a40 2024-05-20T18:07:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.939+0000 7fbc6ac1b740 1 Processor -- start 2024-05-20T18:07:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.939+0000 7fbc6ac1b740 1 -- start start 2024-05-20T18:07:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.939+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afd8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.939+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afd85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.939+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221afd8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.939+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56221af3d340 con 0x56221ac391b0 2024-05-20T18:07:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.939+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56221aee5290 con 0x56221ac102f0 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.939+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56221af39300 con 0x56221ac184d0 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.939+0000 7fbc6222f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afd8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc6222f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afd8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35172/0 (socket says 172.21.15.129:35172) 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc6222f640 1 -- 172.21.15.129:0/4216237353 learned_addr learned my addr 172.21.15.129:0/4216237353 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc52ffd640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbc480a2cf0 0x7fbc480a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc61a2e640 1 --2- 172.21.15.129:0/4216237353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afd85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc6122d640 1 --2- 172.21.15.129:0/4216237353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221afd8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc52ffd640 1 -- 172.21.15.129:0/1507474219 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7fbc4c0cfa30 con 0x7fbc5c107a40 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc6222f640 1 -- 172.21.15.129:0/4216237353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 msgr2=0x56221afd8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc6222f640 1 --2- 172.21.15.129:0/4216237353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221afd8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc6222f640 1 -- 172.21.15.129:0/4216237353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 msgr2=0x56221afd85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc6222f640 1 --2- 172.21.15.129:0/4216237353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afd85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc6222f640 1 -- 172.21.15.129:0/4216237353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56221af098e0 con 0x56221ac391b0 2024-05-20T18:07:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc6122d640 1 --2- 172.21.15.129:0/4216237353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221afd8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc6222f640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbc480a2cf0 0x7fbc480a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc6222f640 1 --2- 172.21.15.129:0/4216237353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afd8080 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbc54010420 tx=0x7fbc540108e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbe1124efb745e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.940+0000 7fbc47fff640 1 -- 172.21.15.129:0/4216237353 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54020070 con 0x56221ac391b0 2024-05-20T18:07:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.941+0000 7fbc47fff640 1 -- 172.21.15.129:0/4216237353 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc54011da0 con 0x56221ac391b0 2024-05-20T18:07:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.941+0000 7fbc47fff640 1 -- 172.21.15.129:0/4216237353 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54019550 con 0x56221ac391b0 2024-05-20T18:07:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.941+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/4216237353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 msgr2=0x56221afd8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.941+0000 7fbc6222f640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbc480a2cf0 0x7fbc480a5200 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbc5401d000 tx=0x7fbc5401a000 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.941+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/4216237353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afd8080 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbc54010420 tx=0x7fbc540108e0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.941+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/4216237353 shutdown_connections 2024-05-20T18:07:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.941+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/4216237353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afd85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.942+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/4216237353 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221afd8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.942+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/4216237353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afd8080 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.942+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/4216237353 >> 172.21.15.129:0/4216237353 conn(0x56221ac2d300 msgr2=0x56221afd4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.942+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/4216237353 shutdown_connections 2024-05-20T18:07:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.942+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/4216237353 wait complete. 2024-05-20T18:07:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.942+0000 7fbc6ac1b740 1 -- start start 2024-05-20T18:07:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.942+0000 7fbc6ac1b740 1 Processor -- start 2024-05-20T18:07:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac102f0 0x56221afe2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac184d0 0x56221afe2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac391b0 0x56221afe8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56221afe51b0 con 0x56221ac102f0 2024-05-20T18:07:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56221aef8790 con 0x56221ac184d0 2024-05-20T18:07:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56221afe5330 con 0x56221ac391b0 2024-05-20T18:07:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc6ac1b740 10 client.0 _ll_register_callbacks cb 0x56221af33a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc61a2e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac102f0 0x56221afe2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc61a2e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac102f0 0x56221afe2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35182/0 (socket says 172.21.15.129:35182) 2024-05-20T18:07:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc61a2e640 1 -- 172.21.15.129:0/1067374929 learned_addr learned my addr 172.21.15.129:0/1067374929 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc6122d640 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac184d0 0x56221afe2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc6222f640 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac391b0 0x56221afe8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc61a2e640 1 -- 172.21.15.129:0/1067374929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac391b0 msgr2=0x56221afe8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc61a2e640 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac391b0 0x56221afe8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc61a2e640 1 -- 172.21.15.129:0/1067374929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac184d0 msgr2=0x56221afe2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc61a2e640 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac184d0 0x56221afe2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc61a2e640 1 -- 172.21.15.129:0/1067374929 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56221ac51210 con 0x56221ac102f0 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc6122d640 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac184d0 0x56221afe2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc61a2e640 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac102f0 0x56221afe2770 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c102a40 tx=0x7fbc58031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8331545a2a6ec221 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc457fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc457fa640 1 -- 172.21.15.129:0/1067374929 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc58035040 con 0x56221ac102f0 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.943+0000 7fbc457fa640 1 -- 172.21.15.129:0/1067374929 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7fbc58004240 con 0x56221ac102f0 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1067374929 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc54011da0 con 0x56221ac102f0 2024-05-20T18:07:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc457fa640 1 -- 172.21.15.129:0/1067374929 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbc58004440 con 0x56221ac102f0 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc6ac1b740 10 client.16749 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1067374929 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56221ad1d620 con 0x56221ac102f0 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc457fa640 1 -- 172.21.15.129:0/1067374929 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc54011da0 con 0x56221ac102f0 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc457fa640 1 -- 172.21.15.129:0/1067374929 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fbc580035f0 con 0x56221ac102f0 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc6ac1b740 10 client.16749 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc6ac1b740 10 client.16749 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc6ac1b740 20 client.16749 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56221abf7d80 0x56221aff49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc6ac1b740 4 client.16749 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1067374929 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56221af4aa70 con 0x56221abf7d80 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.944+0000 7fbc6122d640 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56221abf7d80 0x56221aff49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.945+0000 7fbc457fa640 10 client.16749 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.945+0000 7fbc6122d640 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56221abf7d80 0x56221aff49f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f730c0ade6b48bfd server_cookie=85f61c8f29c3c34a in_seq=0 out_seq=0 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc457fa640 1 -- 172.21.15.129:0/1067374929 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56221af4aa70 con 0x56221abf7d80 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc457fa640 10 client.16749 handle_command_reply: tid=0 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 1 client.16749 shutdown 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1067374929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac102f0 msgr2=0x56221afe2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac102f0 0x56221afe2770 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c102a40 tx=0x7fbc58031920 comp rx=0 tx=0).stop 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1067374929 shutdown_connections 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56221abf7d80 0x56221aff49f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac184d0 0x56221afe2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac391b0 0x56221afe8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc457fa640 0 client.16749 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1067374929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac102f0 0x56221afe2770 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1067374929 >> 172.21.15.129:0/1067374929 conn(0x56221ac2d300 msgr2=0x56221afd7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1067374929 shutdown_connections 2024-05-20T18:07:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1067374929 wait complete. 2024-05-20T18:07:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.946+0000 7fbc6ac1b740 20 client.16749 trim_cache size 0 max 16384 2024-05-20T18:07:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.961+0000 7fbc6ac1b740 1 Processor -- start 2024-05-20T18:07:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.961+0000 7fbc6ac1b740 1 -- start start 2024-05-20T18:07:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.961+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afd8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.961+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221affa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.961+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221affa730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.961+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c1aca90 con 0x56221ac391b0 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.961+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c1af2f0 con 0x56221ac102f0 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.961+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c1ad0e0 con 0x56221ac184d0 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc6222f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221affa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc6222f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221affa730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35194/0 (socket says 172.21.15.129:35194) 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc6222f640 1 -- 172.21.15.129:0/2648158193 learned_addr learned my addr 172.21.15.129:0/2648158193 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc61a2e640 1 --2- 172.21.15.129:0/2648158193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afd8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc6122d640 1 --2- 172.21.15.129:0/2648158193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221affa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc6222f640 1 -- 172.21.15.129:0/2648158193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 msgr2=0x56221affa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc6222f640 1 --2- 172.21.15.129:0/2648158193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221affa1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc6222f640 1 -- 172.21.15.129:0/2648158193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 msgr2=0x56221afd8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc6222f640 1 --2- 172.21.15.129:0/2648158193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afd8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc6222f640 1 -- 172.21.15.129:0/2648158193 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56221afd5420 con 0x56221ac391b0 2024-05-20T18:07:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc6122d640 1 --2- 172.21.15.129:0/2648158193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221affa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc6222f640 1 --2- 172.21.15.129:0/2648158193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221affa730 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbc54010420 tx=0x7fbc54024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eeb9b352087afbf server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc477fe640 1 -- 172.21.15.129:0/2648158193 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54034070 con 0x56221ac391b0 2024-05-20T18:07:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc477fe640 1 -- 172.21.15.129:0/2648158193 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc54012a00 con 0x56221ac391b0 2024-05-20T18:07:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.962+0000 7fbc477fe640 1 -- 172.21.15.129:0/2648158193 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54020050 con 0x56221ac391b0 2024-05-20T18:07:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.963+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/2648158193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 msgr2=0x56221affa730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.963+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/2648158193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221affa730 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbc54010420 tx=0x7fbc54024000 comp rx=0 tx=0).stop 2024-05-20T18:07:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.963+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/2648158193 shutdown_connections 2024-05-20T18:07:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.963+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/2648158193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221affa730 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.963+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/2648158193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221affa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.963+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/2648158193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afd8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.963+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/2648158193 >> 172.21.15.129:0/2648158193 conn(0x56221ac2d300 msgr2=0x56221afd7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.963+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/2648158193 shutdown_connections 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.963+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/2648158193 wait complete. 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.963+0000 7fbc6ac1b740 1 -- start start 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6ac1b740 1 Processor -- start 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afeead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221afe9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6ac1b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afea450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c1ad7b0 con 0x56221ac391b0 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c1aec20 con 0x56221ac102f0 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6ac1b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c1ade80 con 0x56221ac184d0 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6ac1b740 10 client.0 _ll_register_callbacks cb 0x7fbc58009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6222f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6222f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afea450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35204/0 (socket says 172.21.15.129:35204) 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6222f640 1 -- 172.21.15.129:0/1150234699 learned_addr learned my addr 172.21.15.129:0/1150234699 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6122d640 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221afe9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc61a2e640 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afeead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6222f640 1 -- 172.21.15.129:0/1150234699 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 msgr2=0x56221afe9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6222f640 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221afe9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6222f640 1 -- 172.21.15.129:0/1150234699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 msgr2=0x56221afeead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6222f640 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afeead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc6222f640 1 -- 172.21.15.129:0/1150234699 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56221afeabe0 con 0x56221ac391b0 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.964+0000 7fbc61a2e640 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afeead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc6222f640 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afea450 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbc54014010 tx=0x7fbc54012f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc74b5cab2ecf52 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc457fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc457fa640 1 -- 172.21.15.129:0/1150234699 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5403f060 con 0x56221ac391b0 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc457fa640 1 -- 172.21.15.129:0/1150234699 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8587+0+0 (secure 0 0 0) 0x7fbc5401a3f0 con 0x56221ac391b0 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1150234699 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc54012a00 con 0x56221ac391b0 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc457fa640 1 -- 172.21.15.129:0/1150234699 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbc54020030 con 0x56221ac391b0 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc6ac1b740 10 client.16755 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1150234699 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56221b05cdf0 con 0x56221ac391b0 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc457fa640 1 -- 172.21.15.129:0/1150234699 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc54012a00 con 0x56221ac391b0 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc457fa640 1 -- 172.21.15.129:0/1150234699 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fbc54019030 con 0x56221ac391b0 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc6ac1b740 10 client.16755 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc6ac1b740 10 client.16755 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc6ac1b740 20 client.16755 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56221abcd1c0 0x56221b05f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc6ac1b740 4 client.16755 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:07:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.965+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1150234699 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56221af44dc0 con 0x56221abcd1c0 2024-05-20T18:07:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.966+0000 7fbc61a2e640 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56221abcd1c0 0x56221b05f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.966+0000 7fbc457fa640 10 client.16755 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.966+0000 7fbc61a2e640 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56221abcd1c0 0x56221b05f270 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=914c590911a3b82f server_cookie=9ed083531b7c1f16 in_seq=0 out_seq=0 2024-05-20T18:07:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.974+0000 7fbc457fa640 1 -- 172.21.15.129:0/1150234699 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56221af44dc0 con 0x56221abcd1c0 2024-05-20T18:07:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.974+0000 7fbc457fa640 10 client.16755 handle_command_reply: tid=0 2024-05-20T18:07:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.974+0000 7fbc6ac1b740 1 client.16755 shutdown 2024-05-20T18:07:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1150234699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 msgr2=0x56221afea450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afea450 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbc54014010 tx=0x7fbc54012f40 comp rx=0 tx=0).stop 2024-05-20T18:07:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1150234699 shutdown_connections 2024-05-20T18:07:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56221abcd1c0 0x56221b05f270 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56221ac391b0 0x56221afea450 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56221ac184d0 0x56221afe9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc457fa640 0 client.16755 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc6ac1b740 1 --2- 172.21.15.129:0/1150234699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56221ac102f0 0x56221afeead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1150234699 >> 172.21.15.129:0/1150234699 conn(0x56221ac2d300 msgr2=0x56221afe4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1150234699 shutdown_connections 2024-05-20T18:07:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc6ac1b740 1 -- 172.21.15.129:0/1150234699 wait complete. 2024-05-20T18:07:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc6ac1b740 20 client.16755 trim_cache size 0 max 16384 2024-05-20T18:07:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:07:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:07:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:07:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:07:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc63c92640 1 -- 172.21.15.129:0/1507474219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbc480a2cf0 msgr2=0x7fbc480a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc63c92640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbc480a2cf0 0x7fbc480a5200 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbc5401d000 tx=0x7fbc5401a000 comp rx=0 tx=0).stop 2024-05-20T18:07:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc63c92640 1 -- 172.21.15.129:0/1507474219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc5c107a40 msgr2=0x7fbc5c19fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc63c92640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc5c107a40 0x7fbc5c19fa20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00e9f0 tx=0x7fbc4c00eeb0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc63c92640 1 -- 172.21.15.129:0/1507474219 shutdown_connections 2024-05-20T18:07:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc63c92640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbc480a2cf0 0x7fbc480a5200 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc63c92640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c10c3a0 0x7fbc5c199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc63c92640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc5c107a40 0x7fbc5c19fa20 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc63c92640 1 --2- 172.21.15.129:0/1507474219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc5c101b10 0x7fbc5c19f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.975+0000 7fbc63c92640 1 -- 172.21.15.129:0/1507474219 >> 172.21.15.129:0/1507474219 conn(0x7fbc5c0fd210 msgr2=0x7fbc5c0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.976+0000 7fbc63c92640 1 -- 172.21.15.129:0/1507474219 shutdown_connections 2024-05-20T18:07:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:02.976+0000 7fbc63c92640 1 -- 172.21.15.129:0/1507474219 wait complete. 2024-05-20T18:07:03.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:03 smithi154 ceph-mon[36482]: pgmap v261: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-05-20T18:07:03.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:03 smithi129 ceph-mon[32029]: pgmap v261: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-05-20T18:07:03.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:03 smithi204 ceph-mon[37076]: pgmap v261: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-05-20T18:07:05.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:05 smithi154 ceph-mon[36482]: pgmap v262: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.2 MiB/s wr, 66 op/s 2024-05-20T18:07:05.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:05 smithi129 ceph-mon[32029]: pgmap v262: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.2 MiB/s wr, 66 op/s 2024-05-20T18:07:05.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:05 smithi204 ceph-mon[37076]: pgmap v262: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.2 MiB/s wr, 66 op/s 2024-05-20T18:07:06.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:06 smithi154 ceph-mon[36482]: osdmap e99: 12 total, 12 up, 12 in 2024-05-20T18:07:06.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:06 smithi129 ceph-mon[32029]: osdmap e99: 12 total, 12 up, 12 in 2024-05-20T18:07:06.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:06 smithi204 ceph-mon[37076]: osdmap e99: 12 total, 12 up, 12 in 2024-05-20T18:07:07.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:07 smithi154 ceph-mon[36482]: pgmap v264: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 34 MiB/s wr, 275 op/s 2024-05-20T18:07:07.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:07 smithi129 ceph-mon[32029]: pgmap v264: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 34 MiB/s wr, 275 op/s 2024-05-20T18:07:07.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:07 smithi204 ceph-mon[37076]: pgmap v264: 97 pgs: 97 active+clean; 61 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 34 MiB/s wr, 275 op/s 2024-05-20T18:07:09.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:09 smithi154 ceph-mon[36482]: pgmap v265: 97 pgs: 2 active+clean+snaptrim, 90 active+clean, 5 active+clean+snaptrim_wait; 59 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 MiB/s wr, 297 op/s 2024-05-20T18:07:09.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:09 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:07:09.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:09 smithi129 ceph-mon[32029]: pgmap v265: 97 pgs: 2 active+clean+snaptrim, 90 active+clean, 5 active+clean+snaptrim_wait; 59 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 MiB/s wr, 297 op/s 2024-05-20T18:07:09.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:09 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:07:09.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:09 smithi204 ceph-mon[37076]: pgmap v265: 97 pgs: 2 active+clean+snaptrim, 90 active+clean, 5 active+clean+snaptrim_wait; 59 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 MiB/s wr, 297 op/s 2024-05-20T18:07:09.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:09 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:07:10.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:10 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:07:10.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:10 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:07:10.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:10 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:07:10.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:10 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:07:10.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:10 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:07:10.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:10 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:07:10.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:10 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:07:10.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:10 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:07:10.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:10 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:07:12.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:11 smithi154 ceph-mon[36482]: pgmap v266: 97 pgs: 2 active+clean+snaptrim, 90 active+clean, 5 active+clean+snaptrim_wait; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 38 MiB/s wr, 302 op/s 2024-05-20T18:07:12.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:11 smithi129 ceph-mon[32029]: pgmap v266: 97 pgs: 2 active+clean+snaptrim, 90 active+clean, 5 active+clean+snaptrim_wait; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 38 MiB/s wr, 302 op/s 2024-05-20T18:07:12.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:11 smithi204 ceph-mon[37076]: pgmap v266: 97 pgs: 2 active+clean+snaptrim, 90 active+clean, 5 active+clean+snaptrim_wait; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 38 MiB/s wr, 302 op/s 2024-05-20T18:07:14.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:13 smithi154 ceph-mon[36482]: pgmap v267: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 38 MiB/s wr, 302 op/s 2024-05-20T18:07:14.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:13 smithi129 ceph-mon[32029]: pgmap v267: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 38 MiB/s wr, 302 op/s 2024-05-20T18:07:14.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:13 smithi204 ceph-mon[37076]: pgmap v267: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 38 MiB/s wr, 302 op/s 2024-05-20T18:07:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.086+0000 7f0732e0d640 1 -- 172.21.15.129:0/2166551184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c102190 msgr2=0x7f072c10e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.086+0000 7f0732e0d640 1 --2- 172.21.15.129:0/2166551184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c102190 0x7f072c10e4c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0724009ca0 tx=0x7f072402a8c0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.086+0000 7f0732e0d640 1 -- 172.21.15.129:0/2166551184 shutdown_connections 2024-05-20T18:07:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.086+0000 7f0732e0d640 1 --2- 172.21.15.129:0/2166551184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f072c107ed0 0x7f072c101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.086+0000 7f0732e0d640 1 --2- 172.21.15.129:0/2166551184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c102190 0x7f072c10e4c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.086+0000 7f0732e0d640 1 --2- 172.21.15.129:0/2166551184 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f072c107a30 0x7f072c1015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.086+0000 7f0732e0d640 1 -- 172.21.15.129:0/2166551184 >> 172.21.15.129:0/2166551184 conn(0x7f072c0fd210 msgr2=0x7f072c0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.087+0000 7f0732e0d640 1 -- 172.21.15.129:0/2166551184 shutdown_connections 2024-05-20T18:07:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.087+0000 7f0732e0d640 1 -- 172.21.15.129:0/2166551184 wait complete. 2024-05-20T18:07:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.087+0000 7f0732e0d640 1 Processor -- start 2024-05-20T18:07:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.087+0000 7f0732e0d640 1 -- start start 2024-05-20T18:07:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.087+0000 7f0732e0d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f072c102190 0x7f072c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.087+0000 7f0732e0d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f072c107a30 0x7f072c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.087+0000 7f0732e0d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c107ed0 0x7f072c19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.087+0000 7f0732e0d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f072c111dc0 con 0x7f072c107ed0 2024-05-20T18:07:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f0732e0d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f072c111c40 con 0x7f072c102190 2024-05-20T18:07:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f0732e0d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f072c111f40 con 0x7f072c107a30 2024-05-20T18:07:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f07313aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c107ed0 0x7f072c19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f07313aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c107ed0 0x7f072c19cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39102/0 (socket says 172.21.15.129:39102) 2024-05-20T18:07:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f07313aa640 1 -- 172.21.15.129:0/4042203093 learned_addr learned my addr 172.21.15.129:0/4042203093 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f0730ba9640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f072c102190 0x7f072c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f072bfff640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f072c107a30 0x7f072c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f07313aa640 1 -- 172.21.15.129:0/4042203093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f072c107a30 msgr2=0x7f072c198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f07313aa640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f072c107a30 0x7f072c198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f07313aa640 1 -- 172.21.15.129:0/4042203093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f072c102190 msgr2=0x7f072c1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f07313aa640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f072c102190 0x7f072c1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f07313aa640 1 -- 172.21.15.129:0/4042203093 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072c19d500 con 0x7f072c107ed0 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f0730ba9640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f072c102190 0x7f072c1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f07313aa640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c107ed0 0x7f072c19cec0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0724031a20 tx=0x7f0724031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f0729ffb640 1 -- 172.21.15.129:0/4042203093 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724003b70 con 0x7f072c107ed0 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f0729ffb640 1 -- 172.21.15.129:0/4042203093 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0724003d10 con 0x7f072c107ed0 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f0732e0d640 1 -- 172.21.15.129:0/4042203093 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072c19d790 con 0x7f072c107ed0 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.088+0000 7f0729ffb640 1 -- 172.21.15.129:0/4042203093 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072402fc40 con 0x7f072c107ed0 2024-05-20T18:07:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.089+0000 7f0732e0d640 1 -- 172.21.15.129:0/4042203093 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f072c19dce0 con 0x7f072c107ed0 2024-05-20T18:07:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0739d8c740 1 Processor -- start 2024-05-20T18:07:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0729ffb640 1 -- 172.21.15.129:0/4042203093 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f0724004050 con 0x7f072c107ed0 2024-05-20T18:07:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0739d8c740 1 -- start start 2024-05-20T18:07:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2e92f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2e93470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2e939b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0739d8c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2df8340 con 0x55d6d2acb2f0 2024-05-20T18:07:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0739d8c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2da0290 con 0x55d6d2af41b0 2024-05-20T18:07:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0739d8c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2df4300 con 0x55d6d2ad34d0 2024-05-20T18:07:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0730ba9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2e93470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0730ba9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2e93470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39118/0 (socket says 172.21.15.129:39118) 2024-05-20T18:07:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.092+0000 7f0730ba9640 1 -- 172.21.15.129:0/2841673589 learned_addr learned my addr 172.21.15.129:0/2841673589 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f0729ffb640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f071c0a2cf0 0x7f071c0a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f072bfff640 1 --2- 172.21.15.129:0/2841673589 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2e939b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f072bfff640 1 --2- 172.21.15.129:0/2841673589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2e92f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f07313aa640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f071c0a2cf0 0x7f071c0a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f0730ba9640 1 -- 172.21.15.129:0/2841673589 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 msgr2=0x55d6d2e939b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f0730ba9640 1 --2- 172.21.15.129:0/2841673589 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2e939b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f0730ba9640 1 -- 172.21.15.129:0/2841673589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 msgr2=0x55d6d2e92f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f0730ba9640 1 --2- 172.21.15.129:0/2841673589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2e92f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f0730ba9640 1 -- 172.21.15.129:0/2841673589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6d2dc48e0 con 0x55d6d2acb2f0 2024-05-20T18:07:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f072bfff640 1 --2- 172.21.15.129:0/2841673589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2e92f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f0729ffb640 1 -- 172.21.15.129:0/4042203093 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f07240f05c0 con 0x7f072c107ed0 2024-05-20T18:07:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f0730ba9640 1 --2- 172.21.15.129:0/2841673589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2e93470 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0720009cb0 tx=0x7f07200117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f160e047d8c9768b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f07313aa640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f071c0a2cf0 0x7f071c0a5200 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f07240ede90 tx=0x7f072408fcb0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f0716ffd640 1 -- 172.21.15.129:0/2841673589 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720012450 con 0x55d6d2acb2f0 2024-05-20T18:07:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f0716ffd640 1 -- 172.21.15.129:0/2841673589 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0720012ad0 con 0x55d6d2acb2f0 2024-05-20T18:07:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.093+0000 7f0716ffd640 1 -- 172.21.15.129:0/2841673589 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072000fbe0 con 0x55d6d2acb2f0 2024-05-20T18:07:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.094+0000 7f0739d8c740 1 -- 172.21.15.129:0/2841673589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 msgr2=0x55d6d2e93470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.094+0000 7f0739d8c740 1 --2- 172.21.15.129:0/2841673589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2e93470 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0720009cb0 tx=0x7f07200117b0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.094+0000 7f0739d8c740 1 -- 172.21.15.129:0/2841673589 shutdown_connections 2024-05-20T18:07:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.094+0000 7f0739d8c740 1 --2- 172.21.15.129:0/2841673589 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2e939b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.094+0000 7f0739d8c740 1 --2- 172.21.15.129:0/2841673589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2e93470 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.094+0000 7f0739d8c740 1 --2- 172.21.15.129:0/2841673589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2e92f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.094+0000 7f0739d8c740 1 -- 172.21.15.129:0/2841673589 >> 172.21.15.129:0/2841673589 conn(0x55d6d2ae8300 msgr2=0x55d6d2e8f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.094+0000 7f0739d8c740 1 -- 172.21.15.129:0/2841673589 shutdown_connections 2024-05-20T18:07:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.094+0000 7f0739d8c740 1 -- 172.21.15.129:0/2841673589 wait complete. 2024-05-20T18:07:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0739d8c740 1 -- start start 2024-05-20T18:07:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0739d8c740 1 Processor -- start 2024-05-20T18:07:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2acb2f0 0x55d6d2ea12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2ad34d0 0x55d6d2ea1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2e9f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0739d8c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2ea23c0 con 0x55d6d2ad34d0 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0739d8c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2db3790 con 0x55d6d2af41b0 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0739d8c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d6d2ea2540 con 0x55d6d2acb2f0 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0739d8c740 10 client.0 _ll_register_callbacks cb 0x55d6d2deea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0730ba9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2ad34d0 0x55d6d2ea1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0730ba9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2ad34d0 0x55d6d2ea1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39132/0 (socket says 172.21.15.129:39132) 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f0730ba9640 1 -- 172.21.15.129:0/3125408181 learned_addr learned my addr 172.21.15.129:0/3125408181 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.095+0000 7f072bfff640 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2acb2f0 0x55d6d2ea12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f072bfff640 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2e9f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f0730ba9640 1 -- 172.21.15.129:0/3125408181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2acb2f0 msgr2=0x55d6d2ea12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f0730ba9640 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2acb2f0 0x55d6d2ea12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f0730ba9640 1 -- 172.21.15.129:0/3125408181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 msgr2=0x55d6d2e9f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f0730ba9640 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2e9f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f0730ba9640 1 -- 172.21.15.129:0/3125408181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6d2b0c210 con 0x55d6d2ad34d0 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f072bfff640 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2e9f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f0730ba9640 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2ad34d0 0x55d6d2ea1810 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f072000f6e0 tx=0x7f0720009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435c59b7ba9c213c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f070bfff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f070bfff640 1 -- 172.21.15.129:0/3125408181 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720029020 con 0x55d6d2ad34d0 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f070bfff640 1 -- 172.21.15.129:0/3125408181 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f0720012660 con 0x55d6d2ad34d0 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f0739d8c740 1 -- 172.21.15.129:0/3125408181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0720012ad0 con 0x55d6d2ad34d0 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f070bfff640 1 -- 172.21.15.129:0/3125408181 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0720034050 con 0x55d6d2ad34d0 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f0739d8c740 10 client.16767 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f0739d8c740 1 -- 172.21.15.129:0/3125408181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d6d2bd8620 con 0x55d6d2ad34d0 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.096+0000 7f070bfff640 1 -- 172.21.15.129:0/3125408181 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0720012ad0 con 0x55d6d2ad34d0 2024-05-20T18:07:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.097+0000 7f070bfff640 1 -- 172.21.15.129:0/3125408181 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f07200046c0 con 0x55d6d2ad34d0 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.097+0000 7f0739d8c740 10 client.16767 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.097+0000 7f0739d8c740 10 client.16767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.097+0000 7f0739d8c740 20 client.16767 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.097+0000 7f0739d8c740 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6d2ab2d80 0x55d6d2eb1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.097+0000 7f0739d8c740 4 client.16767 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.097+0000 7f0739d8c740 1 -- 172.21.15.129:0/3125408181 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6d2e05a70 con 0x55d6d2ab2d80 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.097+0000 7f072bfff640 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6d2ab2d80 0x55d6d2eb1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.097+0000 7f072bfff640 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6d2ab2d80 0x55d6d2eb1a70 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a841e8d25d34832 server_cookie=722f0fcf16b33c34 in_seq=0 out_seq=0 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.097+0000 7f070bfff640 10 client.16767 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f070bfff640 1 -- 172.21.15.129:0/3125408181 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d6d2e05a70 con 0x55d6d2ab2d80 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f070bfff640 10 client.16767 handle_command_reply: tid=0 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f0739d8c740 1 client.16767 shutdown 2024-05-20T18:07:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f0739d8c740 1 -- 172.21.15.129:0/3125408181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2ad34d0 msgr2=0x55d6d2ea1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f0739d8c740 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2ad34d0 0x55d6d2ea1810 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f072000f6e0 tx=0x7f0720009e20 comp rx=0 tx=0).stop 2024-05-20T18:07:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f0739d8c740 1 -- 172.21.15.129:0/3125408181 shutdown_connections 2024-05-20T18:07:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f0739d8c740 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6d2ab2d80 0x55d6d2eb1a70 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f0739d8c740 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2e9f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f0739d8c740 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2ad34d0 0x55d6d2ea1810 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f070bfff640 0 client.16767 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f0739d8c740 1 --2- 172.21.15.129:0/3125408181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2acb2f0 0x55d6d2ea12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.098+0000 7f0739d8c740 1 -- 172.21.15.129:0/3125408181 >> 172.21.15.129:0/3125408181 conn(0x55d6d2ae8300 msgr2=0x55d6d2e94d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.099+0000 7f0739d8c740 1 -- 172.21.15.129:0/3125408181 shutdown_connections 2024-05-20T18:07:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.099+0000 7f0739d8c740 1 -- 172.21.15.129:0/3125408181 wait complete. 2024-05-20T18:07:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.099+0000 7f0739d8c740 20 client.16767 trim_cache size 0 max 16384 2024-05-20T18:07:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f0739d8c740 1 Processor -- start 2024-05-20T18:07:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f0739d8c740 1 -- start start 2024-05-20T18:07:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2acb2f0 0x55d6d2e95440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2e990c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2af41b0 0x55d6d2e99600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f0739d8c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f072c1aa920 con 0x55d6d2af41b0 2024-05-20T18:07:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f0739d8c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f072c1ad180 con 0x55d6d2acb2f0 2024-05-20T18:07:16.114 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f0739d8c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f072c1aaf70 con 0x55d6d2ad34d0 2024-05-20T18:07:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f072bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2af41b0 0x55d6d2e99600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f072bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2af41b0 0x55d6d2e99600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39146/0 (socket says 172.21.15.129:39146) 2024-05-20T18:07:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f072bfff640 1 -- 172.21.15.129:0/1230672280 learned_addr learned my addr 172.21.15.129:0/1230672280 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f072bfff640 1 --2- 172.21.15.129:0/1230672280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2acb2f0 0x55d6d2e95440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.115 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f0730ba9640 1 --2- 172.21.15.129:0/1230672280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2e990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f072bfff640 1 -- 172.21.15.129:0/1230672280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 msgr2=0x55d6d2e990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f072bfff640 1 --2- 172.21.15.129:0/1230672280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2e990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f072bfff640 1 -- 172.21.15.129:0/1230672280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2acb2f0 msgr2=0x55d6d2e95440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f072bfff640 1 --2- 172.21.15.129:0/1230672280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2acb2f0 0x55d6d2e95440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.114+0000 7f072bfff640 1 -- 172.21.15.129:0/1230672280 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6d2ea0c80 con 0x55d6d2af41b0 2024-05-20T18:07:16.116 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.115+0000 7f072bfff640 1 --2- 172.21.15.129:0/1230672280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2af41b0 0x55d6d2e99600 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0718005780 tx=0x7f0718005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e05f9941c8f31b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.115+0000 7f07167fc640 1 -- 172.21.15.129:0/1230672280 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0718027020 con 0x55d6d2af41b0 2024-05-20T18:07:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.115+0000 7f07167fc640 1 -- 172.21.15.129:0/1230672280 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0718005500 con 0x55d6d2af41b0 2024-05-20T18:07:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.115+0000 7f07167fc640 1 -- 172.21.15.129:0/1230672280 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071800f420 con 0x55d6d2af41b0 2024-05-20T18:07:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 -- 172.21.15.129:0/1230672280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2af41b0 msgr2=0x55d6d2e99600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 --2- 172.21.15.129:0/1230672280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2af41b0 0x55d6d2e99600 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0718005780 tx=0x7f0718005f90 comp rx=0 tx=0).stop 2024-05-20T18:07:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 -- 172.21.15.129:0/1230672280 shutdown_connections 2024-05-20T18:07:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 --2- 172.21.15.129:0/1230672280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2af41b0 0x55d6d2e99600 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.118 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 --2- 172.21.15.129:0/1230672280 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2e990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.118 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 --2- 172.21.15.129:0/1230672280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2acb2f0 0x55d6d2e95440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.118 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 -- 172.21.15.129:0/1230672280 >> 172.21.15.129:0/1230672280 conn(0x55d6d2ae8300 msgr2=0x55d6d2e94d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:16.118 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 -- 172.21.15.129:0/1230672280 shutdown_connections 2024-05-20T18:07:16.118 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 -- 172.21.15.129:0/1230672280 wait complete. 2024-05-20T18:07:16.118 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 -- start start 2024-05-20T18:07:16.119 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 Processor -- start 2024-05-20T18:07:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.116+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2eb4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2eb4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f0739d8c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2eb1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f0739d8c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f072c1ab640 con 0x55d6d2acb2f0 2024-05-20T18:07:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f0739d8c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f072c1acab0 con 0x55d6d2af41b0 2024-05-20T18:07:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f0739d8c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f072c1abd10 con 0x55d6d2ad34d0 2024-05-20T18:07:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f0739d8c740 10 client.0 _ll_register_callbacks cb 0x7f0718009970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f072bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2eb4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f072bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2eb4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39160/0 (socket says 172.21.15.129:39160) 2024-05-20T18:07:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f072bfff640 1 -- 172.21.15.129:0/2961552149 learned_addr learned my addr 172.21.15.129:0/2961552149 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f0730ba9640 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2eb4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f072bfff640 1 -- 172.21.15.129:0/2961552149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 msgr2=0x55d6d2eb4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f072bfff640 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2eb4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f072bfff640 1 -- 172.21.15.129:0/2961552149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 msgr2=0x55d6d2eb1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f072bfff640 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2eb1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f072bfff640 1 -- 172.21.15.129:0/2961552149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6d2eb2470 con 0x55d6d2acb2f0 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f0730ba9640 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2eb4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f072bfff640 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2eb4930 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0718001eb0 tx=0x7f0718021540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d517e8d8d58fe3 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f070bfff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f070bfff640 1 -- 172.21.15.129:0/2961552149 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0718027050 con 0x55d6d2acb2f0 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f070bfff640 1 -- 172.21.15.129:0/2961552149 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f0718011d40 con 0x55d6d2acb2f0 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f0739d8c740 1 -- 172.21.15.129:0/2961552149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0718005500 con 0x55d6d2acb2f0 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f070bfff640 1 -- 172.21.15.129:0/2961552149 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0718039050 con 0x55d6d2acb2f0 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f0739d8c740 10 client.16773 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.117+0000 7f0739d8c740 1 -- 172.21.15.129:0/2961552149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d6d2f19510 con 0x55d6d2acb2f0 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.118+0000 7f070bfff640 1 -- 172.21.15.129:0/2961552149 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0718005500 con 0x55d6d2acb2f0 2024-05-20T18:07:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.118+0000 7f070bfff640 1 -- 172.21.15.129:0/2961552149 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f071801ea30 con 0x55d6d2acb2f0 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.118+0000 7f0739d8c740 10 client.16773 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.118+0000 7f0739d8c740 10 client.16773 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.118+0000 7f0739d8c740 20 client.16773 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.118+0000 7f0739d8c740 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6d2a881c0 0x55d6d2f1bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.118+0000 7f0739d8c740 4 client.16773 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.118+0000 7f0739d8c740 1 -- 172.21.15.129:0/2961552149 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d6d2e8abe0 con 0x55d6d2a881c0 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.118+0000 7f0730ba9640 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6d2a881c0 0x55d6d2f1bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.119+0000 7f0730ba9640 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6d2a881c0 0x55d6d2f1bc00 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f7b10835307db85 server_cookie=1d8ecfeb9fae3e31 in_seq=0 out_seq=0 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.119+0000 7f070bfff640 10 client.16773 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f070bfff640 1 -- 172.21.15.129:0/2961552149 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d6d2e8abe0 con 0x55d6d2a881c0 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f070bfff640 10 client.16773 handle_command_reply: tid=0 2024-05-20T18:07:16.130 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 1 client.16773 shutdown 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 1 -- 172.21.15.129:0/2961552149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 msgr2=0x55d6d2eb4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2eb4930 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0718001eb0 tx=0x7f0718021540 comp rx=0 tx=0).stop 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 1 -- 172.21.15.129:0/2961552149 shutdown_connections 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6d2a881c0 0x55d6d2f1bc00 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6d2af41b0 0x55d6d2eb1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6d2ad34d0 0x55d6d2eb4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 1 --2- 172.21.15.129:0/2961552149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6d2acb2f0 0x55d6d2eb4930 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f070bfff640 0 client.16773 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 1 -- 172.21.15.129:0/2961552149 >> 172.21.15.129:0/2961552149 conn(0x55d6d2ae8300 msgr2=0x55d6d2ea0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 1 -- 172.21.15.129:0/2961552149 shutdown_connections 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 1 -- 172.21.15.129:0/2961552149 wait complete. 2024-05-20T18:07:16.131 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.129+0000 7f0739d8c740 20 client.16773 trim_cache size 0 max 16384 2024-05-20T18:07:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:07:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:07:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:07:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:07:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 -- 172.21.15.129:0/4042203093 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f071c0a2cf0 msgr2=0x7f071c0a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f071c0a2cf0 0x7f071c0a5200 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f07240ede90 tx=0x7f072408fcb0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 -- 172.21.15.129:0/4042203093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c107ed0 msgr2=0x7f072c19cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c107ed0 0x7f072c19cec0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0724031a20 tx=0x7f0724031a50 comp rx=0 tx=0).stop 2024-05-20T18:07:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 -- 172.21.15.129:0/4042203093 shutdown_connections 2024-05-20T18:07:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f071c0a2cf0 0x7f071c0a5200 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f072c107ed0 0x7f072c19cec0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f072c107a30 0x7f072c198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 --2- 172.21.15.129:0/4042203093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f072c102190 0x7f072c1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 -- 172.21.15.129:0/4042203093 >> 172.21.15.129:0/4042203093 conn(0x7f072c0fd210 msgr2=0x7f072c0ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 -- 172.21.15.129:0/4042203093 shutdown_connections 2024-05-20T18:07:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:16.130+0000 7f0732e0d640 1 -- 172.21.15.129:0/4042203093 wait complete. 2024-05-20T18:07:16.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:15 smithi154 ceph-mon[36482]: pgmap v268: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 30 MiB/s wr, 230 op/s 2024-05-20T18:07:16.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:15 smithi154 ceph-mon[36482]: osdmap e100: 12 total, 12 up, 12 in 2024-05-20T18:07:16.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:15 smithi129 ceph-mon[32029]: pgmap v268: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 30 MiB/s wr, 230 op/s 2024-05-20T18:07:16.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:15 smithi129 ceph-mon[32029]: osdmap e100: 12 total, 12 up, 12 in 2024-05-20T18:07:16.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:15 smithi204 ceph-mon[37076]: pgmap v268: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 30 MiB/s wr, 230 op/s 2024-05-20T18:07:16.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:15 smithi204 ceph-mon[37076]: osdmap e100: 12 total, 12 up, 12 in 2024-05-20T18:07:18.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:17 smithi154 ceph-mon[36482]: pgmap v270: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.4 MiB/s wr, 32 op/s 2024-05-20T18:07:18.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:17 smithi129 ceph-mon[32029]: pgmap v270: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.4 MiB/s wr, 32 op/s 2024-05-20T18:07:18.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:17 smithi204 ceph-mon[37076]: pgmap v270: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.4 MiB/s wr, 32 op/s 2024-05-20T18:07:20.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:19 smithi154 ceph-mon[36482]: pgmap v271: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.7 MiB/s wr, 41 op/s 2024-05-20T18:07:20.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:19 smithi129 ceph-mon[32029]: pgmap v271: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.7 MiB/s wr, 41 op/s 2024-05-20T18:07:20.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:19 smithi204 ceph-mon[37076]: pgmap v271: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.7 MiB/s wr, 41 op/s 2024-05-20T18:07:22.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:21 smithi154 ceph-mon[36482]: pgmap v272: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 32 MiB/s wr, 232 op/s 2024-05-20T18:07:22.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:21 smithi129 ceph-mon[32029]: pgmap v272: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 32 MiB/s wr, 232 op/s 2024-05-20T18:07:22.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:21 smithi204 ceph-mon[37076]: pgmap v272: 97 pgs: 97 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 32 MiB/s wr, 232 op/s 2024-05-20T18:07:24.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:23 smithi154 ceph-mon[36482]: pgmap v273: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 36 MiB/s wr, 261 op/s 2024-05-20T18:07:24.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:23 smithi129 ceph-mon[32029]: pgmap v273: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 36 MiB/s wr, 261 op/s 2024-05-20T18:07:24.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:23 smithi204 ceph-mon[37076]: pgmap v273: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 36 MiB/s wr, 261 op/s 2024-05-20T18:07:25.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:07:25.222 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:07:25.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.352+0000 7f2076467640 1 -- 172.21.15.129:0/4186327968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2070107a10 msgr2=0x7f20701015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.352+0000 7f2076467640 1 --2- 172.21.15.129:0/4186327968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2070107a10 0x7f20701015f0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f2060009900 tx=0x7f206002a410 comp rx=0 tx=0).stop 2024-05-20T18:07:25.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.352+0000 7f2076467640 1 -- 172.21.15.129:0/4186327968 shutdown_connections 2024-05-20T18:07:25.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.352+0000 7f2076467640 1 --2- 172.21.15.129:0/4186327968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f207010c370 0x7f207010e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.352+0000 7f2076467640 1 --2- 172.21.15.129:0/4186327968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2070101b30 0x7f2070101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.352+0000 7f2076467640 1 --2- 172.21.15.129:0/4186327968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2070107a10 0x7f20701015f0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.352+0000 7f2076467640 1 -- 172.21.15.129:0/4186327968 >> 172.21.15.129:0/4186327968 conn(0x7f2070077130 msgr2=0x7f2070077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.352+0000 7f2076467640 1 -- 172.21.15.129:0/4186327968 shutdown_connections 2024-05-20T18:07:25.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.352+0000 7f2076467640 1 -- 172.21.15.129:0/4186327968 wait complete. 2024-05-20T18:07:25.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f2076467640 1 Processor -- start 2024-05-20T18:07:25.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f2076467640 1 -- start start 2024-05-20T18:07:25.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f2076467640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2070101b30 0x7f20701a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f2076467640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2070107a10 0x7f20701a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f2076467640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f207010c370 0x7f207019b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f2076467640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2070111d20 con 0x7f2070101b30 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f2076467640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2070111ba0 con 0x7f2070107a10 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f2076467640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2070111ea0 con 0x7f207010c370 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f206ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2070101b30 0x7f20701a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f206ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2070101b30 0x7f20701a15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41136/0 (socket says 172.21.15.129:41136) 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f206ffff640 1 -- 172.21.15.129:0/3614836235 learned_addr learned my addr 172.21.15.129:0/3614836235 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f206f7fe640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2070107a10 0x7f20701a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f2074a04640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f207010c370 0x7f207019b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f206ffff640 1 -- 172.21.15.129:0/3614836235 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f207010c370 msgr2=0x7f207019b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f206ffff640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f207010c370 0x7f207019b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f206ffff640 1 -- 172.21.15.129:0/3614836235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2070107a10 msgr2=0x7f20701a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.353+0000 7f206ffff640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2070107a10 0x7f20701a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.354+0000 7f206ffff640 1 -- 172.21.15.129:0/3614836235 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207019bee0 con 0x7f2070101b30 2024-05-20T18:07:25.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.354+0000 7f2074a04640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f207010c370 0x7f207019b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:25.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.354+0000 7f206f7fe640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2070107a10 0x7f20701a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:25.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.354+0000 7f206ffff640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2070101b30 0x7f20701a15d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f2060005940 tx=0x7f2060004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.354+0000 7f206d7fa640 1 -- 172.21.15.129:0/3614836235 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206003f030 con 0x7f2070101b30 2024-05-20T18:07:25.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.354+0000 7f2076467640 1 -- 172.21.15.129:0/3614836235 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f207019c170 con 0x7f2070101b30 2024-05-20T18:07:25.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.354+0000 7f206d7fa640 1 -- 172.21.15.129:0/3614836235 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2060003ad0 con 0x7f2070101b30 2024-05-20T18:07:25.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.354+0000 7f206d7fa640 1 -- 172.21.15.129:0/3614836235 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060035070 con 0x7f2070101b30 2024-05-20T18:07:25.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.354+0000 7f2076467640 1 -- 172.21.15.129:0/3614836235 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20701a8420 con 0x7f2070101b30 2024-05-20T18:07:25.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.356+0000 7f206d7fa640 1 -- 172.21.15.129:0/3614836235 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f2060042030 con 0x7f2070101b30 2024-05-20T18:07:25.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f207d3e0740 1 Processor -- start 2024-05-20T18:07:25.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f207d3e0740 1 -- start start 2024-05-20T18:07:25.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f206d7fa640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20580a2d00 0x7f20580a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7a7ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76b74d0 0x5652a7a7f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76bf6b0 0x5652a7a7f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f207d3e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5652a79e4d00 con 0x5652a76e0390 2024-05-20T18:07:25.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f2074a04640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20580a2d00 0x7f20580a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f207d3e0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5652a798c4c0 con 0x5652a76bf6b0 2024-05-20T18:07:25.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f207d3e0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5652a79e0cc0 con 0x5652a76b74d0 2024-05-20T18:07:25.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f206d7fa640 1 -- 172.21.15.129:0/3614836235 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f20600efce0 con 0x7f2070101b30 2024-05-20T18:07:25.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f206ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7a7ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f206ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7a7ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41138/0 (socket says 172.21.15.129:41138) 2024-05-20T18:07:25.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f206ffff640 1 -- 172.21.15.129:0/625977119 learned_addr learned my addr 172.21.15.129:0/625977119 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:25.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.357+0000 7f206f7fe640 1 --2- 172.21.15.129:0/625977119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76b74d0 0x5652a7a7f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f2074a04640 1 --2- 172.21.15.129:0/625977119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76bf6b0 0x5652a7a7f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f206ffff640 1 -- 172.21.15.129:0/625977119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76b74d0 msgr2=0x5652a7a7f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f206ffff640 1 --2- 172.21.15.129:0/625977119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76b74d0 0x5652a7a7f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f206ffff640 1 -- 172.21.15.129:0/625977119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76bf6b0 msgr2=0x5652a7a7f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f206ffff640 1 --2- 172.21.15.129:0/625977119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76bf6b0 0x5652a7a7f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f206ffff640 1 -- 172.21.15.129:0/625977119 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652a77c27d0 con 0x5652a76e0390 2024-05-20T18:07:25.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f2074a04640 1 --2- 172.21.15.129:0/625977119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76bf6b0 0x5652a7a7f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f2074a04640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20580a2d00 0x7f20580a5210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2068005fd0 tx=0x7f206800f730 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f206ffff640 1 --2- 172.21.15.129:0/625977119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7a7ed30 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f206002a430 tx=0x7f2060003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a112e7403bcf4ae9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f20567fc640 1 -- 172.21.15.129:0/625977119 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060049040 con 0x5652a76e0390 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f20567fc640 1 -- 172.21.15.129:0/625977119 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20600bc1c0 con 0x5652a76e0390 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.358+0000 7f20567fc640 1 -- 172.21.15.129:0/625977119 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060031bd0 con 0x5652a76e0390 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.359+0000 7f207d3e0740 1 -- 172.21.15.129:0/625977119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 msgr2=0x5652a7a7ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.359+0000 7f207d3e0740 1 --2- 172.21.15.129:0/625977119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7a7ed30 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f206002a430 tx=0x7f2060003dc0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.359+0000 7f207d3e0740 1 -- 172.21.15.129:0/625977119 shutdown_connections 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.359+0000 7f207d3e0740 1 --2- 172.21.15.129:0/625977119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76bf6b0 0x5652a7a7f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.359+0000 7f207d3e0740 1 --2- 172.21.15.129:0/625977119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76b74d0 0x5652a7a7f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.359+0000 7f207d3e0740 1 --2- 172.21.15.129:0/625977119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7a7ed30 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.359+0000 7f207d3e0740 1 -- 172.21.15.129:0/625977119 >> 172.21.15.129:0/625977119 conn(0x5652a76d44e0 msgr2=0x5652a7a7b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.359+0000 7f207d3e0740 1 -- 172.21.15.129:0/625977119 shutdown_connections 2024-05-20T18:07:25.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.359+0000 7f207d3e0740 1 -- 172.21.15.129:0/625977119 wait complete. 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.359+0000 7f207d3e0740 1 -- start start 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f207d3e0740 1 Processor -- start 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76b74d0 0x5652a7a8af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76bf6b0 0x5652a7a8b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76e0390 0x5652a7a89330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f207d3e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5652a7a8bf00 con 0x5652a76b74d0 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f207d3e0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5652a799fb90 con 0x5652a76bf6b0 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f207d3e0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5652a7a8c080 con 0x5652a76e0390 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f207d3e0740 10 client.0 _ll_register_callbacks cb 0x5652a79dbda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f206ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76b74d0 0x5652a7a8af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f206ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76b74d0 0x5652a7a8af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41148/0 (socket says 172.21.15.129:41148) 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f206ffff640 1 -- 172.21.15.129:0/1804082693 learned_addr learned my addr 172.21.15.129:0/1804082693 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:25.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f206f7fe640 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76bf6b0 0x5652a7a8b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f2074a04640 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76e0390 0x5652a7a89330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.360+0000 7f206ffff640 1 -- 172.21.15.129:0/1804082693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76e0390 msgr2=0x5652a7a89330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f206ffff640 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76e0390 0x5652a7a89330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f206ffff640 1 -- 172.21.15.129:0/1804082693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76bf6b0 msgr2=0x5652a7a8b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f206ffff640 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76bf6b0 0x5652a7a8b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f206ffff640 1 -- 172.21.15.129:0/1804082693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652a76f83f0 con 0x5652a76b74d0 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f206ffff640 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76b74d0 0x5652a7a8af00 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2060046040 tx=0x7f20600bc030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200fc53e567e621b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f204f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f204f7fe640 1 -- 172.21.15.129:0/1804082693 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20600d4070 con 0x5652a76b74d0 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f204f7fe640 1 -- 172.21.15.129:0/1804082693 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f20600b88d0 con 0x5652a76b74d0 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f207d3e0740 1 -- 172.21.15.129:0/1804082693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20600bc1c0 con 0x5652a76b74d0 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f204f7fe640 1 -- 172.21.15.129:0/1804082693 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2060049040 con 0x5652a76b74d0 2024-05-20T18:07:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f207d3e0740 10 client.16785 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f207d3e0740 1 -- 172.21.15.129:0/1804082693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5652a76b80f0 con 0x5652a76b74d0 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f204f7fe640 1 -- 172.21.15.129:0/1804082693 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20600bc1c0 con 0x5652a76b74d0 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f204f7fe640 1 -- 172.21.15.129:0/1804082693 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f20600dc0f0 con 0x5652a76b74d0 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f207d3e0740 10 client.16785 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f207d3e0740 10 client.16785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f207d3e0740 20 client.16785 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f207d3e0740 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5652a769ef60 0x5652a7a9b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f207d3e0740 4 client.16785 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.361+0000 7f207d3e0740 1 -- 172.21.15.129:0/1804082693 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652a79f2430 con 0x5652a769ef60 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.362+0000 7f206f7fe640 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5652a769ef60 0x5652a7a9b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.362+0000 7f206f7fe640 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5652a769ef60 0x5652a7a9b830 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=701d6e8af814d06d server_cookie=2a262b80374bc188 in_seq=0 out_seq=0 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.362+0000 7f204f7fe640 10 client.16785 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:25.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f204f7fe640 1 -- 172.21.15.129:0/1804082693 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5652a79f2430 con 0x5652a769ef60 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f204f7fe640 10 client.16785 handle_command_reply: tid=0 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 1 client.16785 shutdown 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 1 -- 172.21.15.129:0/1804082693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76b74d0 msgr2=0x5652a7a8af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76b74d0 0x5652a7a8af00 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2060046040 tx=0x7f20600bc030 comp rx=0 tx=0).stop 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 1 -- 172.21.15.129:0/1804082693 shutdown_connections 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5652a769ef60 0x5652a7a9b830 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76bf6b0 0x5652a7a8b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f204f7fe640 0 client.16785 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76e0390 0x5652a7a89330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 1 --2- 172.21.15.129:0/1804082693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76b74d0 0x5652a7a8af00 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 1 -- 172.21.15.129:0/1804082693 >> 172.21.15.129:0/1804082693 conn(0x5652a76d44e0 msgr2=0x5652a7a80a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 1 -- 172.21.15.129:0/1804082693 shutdown_connections 2024-05-20T18:07:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 1 -- 172.21.15.129:0/1804082693 wait complete. 2024-05-20T18:07:25.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.363+0000 7f207d3e0740 20 client.16785 trim_cache size 0 max 16384 2024-05-20T18:07:25.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f207d3e0740 1 Processor -- start 2024-05-20T18:07:25.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f207d3e0740 1 -- start start 2024-05-20T18:07:25.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76b74d0 0x5652a7a88630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76bf6b0 0x5652a7a80ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7aa0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f207d3e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20701aec00 con 0x5652a76e0390 2024-05-20T18:07:25.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f207d3e0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20701b1460 con 0x5652a76b74d0 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f207d3e0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f20701af250 con 0x5652a76bf6b0 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f2074a04640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7aa0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f2074a04640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7aa0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41160/0 (socket says 172.21.15.129:41160) 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f2074a04640 1 -- 172.21.15.129:0/2098341245 learned_addr learned my addr 172.21.15.129:0/2098341245 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.381+0000 7f206ffff640 1 --2- 172.21.15.129:0/2098341245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76b74d0 0x5652a7a88630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.382+0000 7f206f7fe640 1 --2- 172.21.15.129:0/2098341245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76bf6b0 0x5652a7a80ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.382+0000 7f2074a04640 1 -- 172.21.15.129:0/2098341245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76bf6b0 msgr2=0x5652a7a80ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.382+0000 7f2074a04640 1 --2- 172.21.15.129:0/2098341245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76bf6b0 0x5652a7a80ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.382+0000 7f2074a04640 1 -- 172.21.15.129:0/2098341245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76b74d0 msgr2=0x5652a7a88630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.382+0000 7f2074a04640 1 --2- 172.21.15.129:0/2098341245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76b74d0 0x5652a7a88630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.382+0000 7f2074a04640 1 -- 172.21.15.129:0/2098341245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652a7a7eac0 con 0x5652a76e0390 2024-05-20T18:07:25.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.382+0000 7f2074a04640 1 --2- 172.21.15.129:0/2098341245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7aa0e00 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f2068011770 tx=0x7f2068014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98eaaae149493215 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.382+0000 7f2055ffb640 1 -- 172.21.15.129:0/2098341245 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2068023070 con 0x5652a76e0390 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.382+0000 7f2055ffb640 1 -- 172.21.15.129:0/2098341245 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2068004d20 con 0x5652a76e0390 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.382+0000 7f2055ffb640 1 -- 172.21.15.129:0/2098341245 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2068005560 con 0x5652a76e0390 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.383+0000 7f207d3e0740 1 -- 172.21.15.129:0/2098341245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 msgr2=0x5652a7aa0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.383+0000 7f207d3e0740 1 --2- 172.21.15.129:0/2098341245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7aa0e00 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f2068011770 tx=0x7f2068014000 comp rx=0 tx=0).stop 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.383+0000 7f207d3e0740 1 -- 172.21.15.129:0/2098341245 shutdown_connections 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.383+0000 7f207d3e0740 1 --2- 172.21.15.129:0/2098341245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76e0390 0x5652a7aa0e00 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.383+0000 7f207d3e0740 1 --2- 172.21.15.129:0/2098341245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76bf6b0 0x5652a7a80ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.383+0000 7f207d3e0740 1 --2- 172.21.15.129:0/2098341245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76b74d0 0x5652a7a88630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.383+0000 7f207d3e0740 1 -- 172.21.15.129:0/2098341245 >> 172.21.15.129:0/2098341245 conn(0x5652a76d44e0 msgr2=0x5652a7a807d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.383+0000 7f207d3e0740 1 -- 172.21.15.129:0/2098341245 shutdown_connections 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.383+0000 7f207d3e0740 1 -- 172.21.15.129:0/2098341245 wait complete. 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.383+0000 7f207d3e0740 1 -- start start 2024-05-20T18:07:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.383+0000 7f207d3e0740 1 Processor -- start 2024-05-20T18:07:25.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76b74d0 0x5652a7a8d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76bf6b0 0x5652a7a8dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f207d3e0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76e0390 0x5652a7b029b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f207d3e0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20701af920 con 0x5652a76bf6b0 2024-05-20T18:07:25.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f207d3e0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20701b0d90 con 0x5652a76e0390 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f207d3e0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f20701afff0 con 0x5652a76b74d0 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f207d3e0740 10 client.0 _ll_register_callbacks cb 0x7f20600c0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f206f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76bf6b0 0x5652a7a8dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f206f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76bf6b0 0x5652a7a8dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41172/0 (socket says 172.21.15.129:41172) 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f206f7fe640 1 -- 172.21.15.129:0/2607291899 learned_addr learned my addr 172.21.15.129:0/2607291899 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f2074a04640 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76e0390 0x5652a7b029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f206ffff640 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76b74d0 0x5652a7a8d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f206f7fe640 1 -- 172.21.15.129:0/2607291899 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76b74d0 msgr2=0x5652a7a8d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f206f7fe640 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76b74d0 0x5652a7a8d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f206f7fe640 1 -- 172.21.15.129:0/2607291899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76e0390 msgr2=0x5652a7b029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f206f7fe640 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76e0390 0x5652a7b029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f206f7fe640 1 -- 172.21.15.129:0/2607291899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652a7b02ef0 con 0x5652a76bf6b0 2024-05-20T18:07:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f206f7fe640 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76bf6b0 0x5652a7a8dab0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2064009c80 tx=0x7f2064012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc68d8d09d82be7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f204f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f204f7fe640 1 -- 172.21.15.129:0/2607291899 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206401e880 con 0x5652a76bf6b0 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f204f7fe640 1 -- 172.21.15.129:0/2607291899 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f206401ea20 con 0x5652a76bf6b0 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.384+0000 7f207d3e0740 1 -- 172.21.15.129:0/2607291899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2068004d20 con 0x5652a76bf6b0 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f204f7fe640 1 -- 172.21.15.129:0/2607291899 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f206402d050 con 0x5652a76bf6b0 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f207d3e0740 10 client.16791 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f207d3e0740 1 -- 172.21.15.129:0/2607291899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5652a7b04080 con 0x5652a76bf6b0 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f204f7fe640 1 -- 172.21.15.129:0/2607291899 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2068004d20 con 0x5652a76bf6b0 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f204f7fe640 1 -- 172.21.15.129:0/2607291899 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f206400f6c0 con 0x5652a76bf6b0 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f207d3e0740 10 client.16791 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f207d3e0740 10 client.16791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f207d3e0740 20 client.16791 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f207d3e0740 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5652a76743a0 0x5652a7b06840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f207d3e0740 4 client.16791 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f207d3e0740 1 -- 172.21.15.129:0/2607291899 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5652a79ec780 con 0x5652a76743a0 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f206ffff640 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5652a76743a0 0x5652a7b06840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f204f7fe640 10 client.16791 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.385+0000 7f206ffff640 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5652a76743a0 0x5652a7b06840 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43ebb2c83e9f5214 server_cookie=38a5a87525178fdf in_seq=0 out_seq=0 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f204f7fe640 1 -- 172.21.15.129:0/2607291899 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5652a79ec780 con 0x5652a76743a0 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f204f7fe640 10 client.16791 handle_command_reply: tid=0 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 1 client.16791 shutdown 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 1 -- 172.21.15.129:0/2607291899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76bf6b0 msgr2=0x5652a7a8dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76bf6b0 0x5652a7a8dab0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2064009c80 tx=0x7f2064012820 comp rx=0 tx=0).stop 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 1 -- 172.21.15.129:0/2607291899 shutdown_connections 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5652a76743a0 0x5652a7b06840 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5652a76e0390 0x5652a7b029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5652a76bf6b0 0x5652a7a8dab0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f204f7fe640 0 client.16791 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:25.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 1 --2- 172.21.15.129:0/2607291899 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5652a76b74d0 0x5652a7a8d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 1 -- 172.21.15.129:0/2607291899 >> 172.21.15.129:0/2607291899 conn(0x5652a76d44e0 msgr2=0x5652a7a88390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 1 -- 172.21.15.129:0/2607291899 shutdown_connections 2024-05-20T18:07:25.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 1 -- 172.21.15.129:0/2607291899 wait complete. 2024-05-20T18:07:25.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.386+0000 7f207d3e0740 20 client.16791 trim_cache size 0 max 16384 2024-05-20T18:07:25.391 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:07:25.391 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:07:25.391 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:07:25.392 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:07:25.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 -- 172.21.15.129:0/3614836235 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20580a2d00 msgr2=0x7f20580a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20580a2d00 0x7f20580a5210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2068005fd0 tx=0x7f206800f730 comp rx=0 tx=0).stop 2024-05-20T18:07:25.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 -- 172.21.15.129:0/3614836235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2070101b30 msgr2=0x7f20701a15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2070101b30 0x7f20701a15d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f2060005940 tx=0x7f2060004270 comp rx=0 tx=0).stop 2024-05-20T18:07:25.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 -- 172.21.15.129:0/3614836235 shutdown_connections 2024-05-20T18:07:25.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f20580a2d00 0x7f20580a5210 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2070107a10 0x7f20701a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f207010c370 0x7f207019b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 --2- 172.21.15.129:0/3614836235 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2070101b30 0x7f20701a15d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 -- 172.21.15.129:0/3614836235 >> 172.21.15.129:0/3614836235 conn(0x7f2070077130 msgr2=0x7f2070100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 -- 172.21.15.129:0/3614836235 shutdown_connections 2024-05-20T18:07:25.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.387+0000 7f2076467640 1 -- 172.21.15.129:0/3614836235 wait complete. 2024-05-20T18:07:25.396 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:07:25.396 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:07:25.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.509+0000 7f5ae45aa640 1 -- 172.21.15.129:0/1437056456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5adc101010 msgr2=0x7f5adc101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.509+0000 7f5ae45aa640 1 --2- 172.21.15.129:0/1437056456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5adc101010 0x7f5adc101420 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009900 tx=0x7f5acc02a410 comp rx=0 tx=0).stop 2024-05-20T18:07:25.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.509+0000 7f5ae45aa640 1 -- 172.21.15.129:0/1437056456 shutdown_connections 2024-05-20T18:07:25.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.509+0000 7f5ae45aa640 1 --2- 172.21.15.129:0/1437056456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc10d510 0x7f5adc10f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.509+0000 7f5ae45aa640 1 --2- 172.21.15.129:0/1437056456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5adc101010 0x7f5adc101420 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.509+0000 7f5ae45aa640 1 --2- 172.21.15.129:0/1437056456 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5adc108b60 0x7f5adc100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.509+0000 7f5ae45aa640 1 -- 172.21.15.129:0/1437056456 >> 172.21.15.129:0/1437056456 conn(0x7f5adc0fc800 msgr2=0x7f5adc0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.510+0000 7f5ae45aa640 1 -- 172.21.15.129:0/1437056456 shutdown_connections 2024-05-20T18:07:25.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.510+0000 7f5ae45aa640 1 -- 172.21.15.129:0/1437056456 wait complete. 2024-05-20T18:07:25.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.510+0000 7f5ae45aa640 1 Processor -- start 2024-05-20T18:07:25.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.510+0000 7f5ae45aa640 1 -- start start 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.510+0000 7f5ae45aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc101010 0x7f5adc1997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.510+0000 7f5ae45aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5adc108b60 0x7f5adc199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.510+0000 7f5ae45aa640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5adc10d510 0x7f5adc19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.510+0000 7f5ae45aa640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc112e20 con 0x7f5adc108b60 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.510+0000 7f5ae45aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc112ca0 con 0x7f5adc101010 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae45aa640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc112fa0 con 0x7f5adc10d510 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae1b45640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5adc108b60 0x7f5adc199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae1b45640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5adc108b60 0x7f5adc199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41184/0 (socket says 172.21.15.129:41184) 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae1b45640 1 -- 172.21.15.129:0/4063639003 learned_addr learned my addr 172.21.15.129:0/4063639003 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae2346640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc101010 0x7f5adc1997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae2b47640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5adc10d510 0x7f5adc19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae1b45640 1 -- 172.21.15.129:0/4063639003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5adc10d510 msgr2=0x7f5adc19dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae1b45640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5adc10d510 0x7f5adc19dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae1b45640 1 -- 172.21.15.129:0/4063639003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc101010 msgr2=0x7f5adc1997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae1b45640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc101010 0x7f5adc1997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae1b45640 1 -- 172.21.15.129:0/4063639003 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc19e600 con 0x7f5adc108b60 2024-05-20T18:07:25.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae2346640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc101010 0x7f5adc1997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:25.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae1b45640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5adc108b60 0x7f5adc199d00 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5acc005940 tx=0x7f5acc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ad37fe640 1 -- 172.21.15.129:0/4063639003 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc03f030 con 0x7f5adc108b60 2024-05-20T18:07:25.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ad37fe640 1 -- 172.21.15.129:0/4063639003 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5acc003ad0 con 0x7f5adc108b60 2024-05-20T18:07:25.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae45aa640 1 -- 172.21.15.129:0/4063639003 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5adc19e890 con 0x7f5adc108b60 2024-05-20T18:07:25.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ad37fe640 1 -- 172.21.15.129:0/4063639003 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc035070 con 0x7f5adc108b60 2024-05-20T18:07:25.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.511+0000 7f5ae45aa640 1 -- 172.21.15.129:0/4063639003 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5adc19edd0 con 0x7f5adc108b60 2024-05-20T18:07:25.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.512+0000 7f5ae45aa640 1 -- 172.21.15.129:0/4063639003 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5adc069140 con 0x7f5adc108b60 2024-05-20T18:07:25.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.514+0000 7f5ad37fe640 1 -- 172.21.15.129:0/4063639003 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5acc042030 con 0x7f5adc108b60 2024-05-20T18:07:25.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.514+0000 7f5ad37fe640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ac80a2d80 0x7f5ac80a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.515+0000 7f5ae2346640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ac80a2d80 0x7f5ac80a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.515+0000 7f5ad37fe640 1 -- 172.21.15.129:0/4063639003 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f5acc0efea0 con 0x7f5adc108b60 2024-05-20T18:07:25.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.515+0000 7f5ae2346640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ac80a2d80 0x7f5ac80a5290 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80058c0 tx=0x7f5ad8005850 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.518+0000 7f5ad37fe640 1 -- 172.21.15.129:0/4063639003 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5acc0bca80 con 0x7f5adc108b60 2024-05-20T18:07:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.732+0000 7f5ae45aa640 1 -- 172.21.15.129:0/4063639003 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5adc19aaf0 con 0x7f5adc108b60 2024-05-20T18:07:25.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.732+0000 7f5ad37fe640 1 -- 172.21.15.129:0/4063639003 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11829 (secure 0 0 0) 0x7f5acc0bc380 con 0x7f5adc108b60 2024-05-20T18:07:25.730 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:07:25.731 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":26255,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/2724397229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":2724397229},{"type":"v1","addr":"172.21.15.154:6837","nonce":2724397229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:06:10.247052+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:07:25.731 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-05-20T18:07:25.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.734+0000 7f5ae45aa640 1 -- 172.21.15.129:0/4063639003 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ac80a2d80 msgr2=0x7f5ac80a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.734+0000 7f5ae45aa640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ac80a2d80 0x7f5ac80a5290 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80058c0 tx=0x7f5ad8005850 comp rx=0 tx=0).stop 2024-05-20T18:07:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.734+0000 7f5ae45aa640 1 -- 172.21.15.129:0/4063639003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5adc108b60 msgr2=0x7f5adc199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.734+0000 7f5ae45aa640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5adc108b60 0x7f5adc199d00 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5acc005940 tx=0x7f5acc004270 comp rx=0 tx=0).stop 2024-05-20T18:07:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.735+0000 7f5ae45aa640 1 -- 172.21.15.129:0/4063639003 shutdown_connections 2024-05-20T18:07:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.735+0000 7f5ae45aa640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ac80a2d80 0x7f5ac80a5290 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.735+0000 7f5ae45aa640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5adc10d510 0x7f5adc19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.735+0000 7f5ae45aa640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5adc108b60 0x7f5adc199d00 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.735+0000 7f5ae45aa640 1 --2- 172.21.15.129:0/4063639003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc101010 0x7f5adc1997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.735+0000 7f5ae45aa640 1 -- 172.21.15.129:0/4063639003 >> 172.21.15.129:0/4063639003 conn(0x7f5adc0fc800 msgr2=0x7f5adc0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.735+0000 7f5ae45aa640 1 -- 172.21.15.129:0/4063639003 shutdown_connections 2024-05-20T18:07:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.735+0000 7f5ae45aa640 1 -- 172.21.15.129:0/4063639003 wait complete. 2024-05-20T18:07:25.745 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:07:25.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.862+0000 7f4561ee9640 1 -- 172.21.15.129:0/2501798840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f455c107a70 msgr2=0x7f455c101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.862+0000 7f4561ee9640 1 --2- 172.21.15.129:0/2501798840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f455c107a70 0x7f455c101c70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f454c009900 tx=0x7f454c02a410 comp rx=0 tx=0).stop 2024-05-20T18:07:25.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.862+0000 7f4561ee9640 1 -- 172.21.15.129:0/2501798840 shutdown_connections 2024-05-20T18:07:25.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.862+0000 7f4561ee9640 1 --2- 172.21.15.129:0/2501798840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f455c102820 0x7f455c10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.862+0000 7f4561ee9640 1 --2- 172.21.15.129:0/2501798840 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f455c107f10 0x7f455c1021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.862+0000 7f4561ee9640 1 --2- 172.21.15.129:0/2501798840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f455c107a70 0x7f455c101c70 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.862+0000 7f4561ee9640 1 -- 172.21.15.129:0/2501798840 >> 172.21.15.129:0/2501798840 conn(0x7f455c0768d0 msgr2=0x7f455c076d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.862+0000 7f4561ee9640 1 -- 172.21.15.129:0/2501798840 shutdown_connections 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.862+0000 7f4561ee9640 1 -- 172.21.15.129:0/2501798840 wait complete. 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.863+0000 7f4561ee9640 1 Processor -- start 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.863+0000 7f4561ee9640 1 -- start start 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.863+0000 7f4561ee9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f455c102820 0x7f455c19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.863+0000 7f4561ee9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f455c107a70 0x7f455c19fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.863+0000 7f4561ee9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f455c107f10 0x7f455c199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.863+0000 7f4561ee9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f455c112f00 con 0x7f455c102820 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.863+0000 7f4561ee9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f455c112d80 con 0x7f455c107f10 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.863+0000 7f4561ee9640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f455c113080 con 0x7f455c107a70 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f455c102820 0x7f455c19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f455c102820 0x7f455c19f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41196/0 (socket says 172.21.15.129:41196) 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455b7fe640 1 -- 172.21.15.129:0/754123539 learned_addr learned my addr 172.21.15.129:0/754123539 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:25.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455affd640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f455c107a70 0x7f455c19fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455bfff640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f455c107f10 0x7f455c199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455b7fe640 1 -- 172.21.15.129:0/754123539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f455c107a70 msgr2=0x7f455c19fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455b7fe640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f455c107a70 0x7f455c19fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455b7fe640 1 -- 172.21.15.129:0/754123539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f455c107f10 msgr2=0x7f455c199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455b7fe640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f455c107f10 0x7f455c199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455b7fe640 1 -- 172.21.15.129:0/754123539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455c199e50 con 0x7f455c102820 2024-05-20T18:07:25.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455bfff640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f455c107f10 0x7f455c199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:25.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f455b7fe640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f455c102820 0x7f455c19f5a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f454c02a930 tx=0x7f454c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f4558ff9640 1 -- 172.21.15.129:0/754123539 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c03f070 con 0x7f455c102820 2024-05-20T18:07:25.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f4561ee9640 1 -- 172.21.15.129:0/754123539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f455c19a080 con 0x7f455c102820 2024-05-20T18:07:25.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f4558ff9640 1 -- 172.21.15.129:0/754123539 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f454c031c90 con 0x7f455c102820 2024-05-20T18:07:25.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f4558ff9640 1 -- 172.21.15.129:0/754123539 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c035030 con 0x7f455c102820 2024-05-20T18:07:25.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.864+0000 7f4561ee9640 1 -- 172.21.15.129:0/754123539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f455c19a570 con 0x7f455c102820 2024-05-20T18:07:25.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.867+0000 7f4558ff9640 1 -- 172.21.15.129:0/754123539 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f454c042070 con 0x7f455c102820 2024-05-20T18:07:25.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.867+0000 7f4568d8d740 1 Processor -- start 2024-05-20T18:07:25.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.867+0000 7f4568d8d740 1 -- start start 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.867+0000 7f4558ff9640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f45440a2d00 0x7f45440a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.867+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a97b3390 0x55e5a9b51d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.867+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 0x55e5a9b52260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.867+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b527a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.867+0000 7f4568d8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e5a9ab7c40 con 0x55e5a97926b0 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.867+0000 7f4568d8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e5a9a5f580 con 0x55e5a97b3390 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.867+0000 7f4568d8d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e5a9ab3c00 con 0x55e5a978a4d0 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.867+0000 7f455bfff640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f45440a2d00 0x7f45440a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f4558ff9640 1 -- 172.21.15.129:0/754123539 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f454c0efd30 con 0x7f455c102820 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a97b3390 0x55e5a9b51d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455affd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 0x55e5a9b52260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455affd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 0x55e5a9b52260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:51122/0 (socket says 172.21.15.129:51122) 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455b7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a97b3390 0x55e5a9b51d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:40826/0 (socket says 172.21.15.129:40826) 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41208/0 (socket says 172.21.15.129:41208) 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455affd640 1 -- 172.21.15.129:0/3984341852 learned_addr learned my addr 172.21.15.129:0/3984341852 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455affd640 1 -- 172.21.15.129:0/3984341852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a97b3390 msgr2=0x55e5a9b51d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455affd640 1 --2- 172.21.15.129:0/3984341852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a97b3390 0x55e5a9b51d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455affd640 1 -- 172.21.15.129:0/3984341852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 msgr2=0x55e5a9b527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455affd640 1 --2- 172.21.15.129:0/3984341852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455affd640 1 -- 172.21.15.129:0/3984341852 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5a97cb3f0 con 0x55e5a978a4d0 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455bfff640 1 --2- 172.21.15.129:0/3984341852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455bfff640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f45440a2d00 0x7f45440a5210 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4554005fd0 tx=0x7f455400ac90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f455affd640 1 --2- 172.21.15.129:0/3984341852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 0x55e5a9b52260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f455000bb60 tx=0x7f455000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26723c8411fdb4e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f4541ffb640 1 -- 172.21.15.129:0/3984341852 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550015070 con 0x55e5a978a4d0 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f4541ffb640 1 -- 172.21.15.129:0/3984341852 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f455000be50 con 0x55e5a978a4d0 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.868+0000 7f4541ffb640 1 -- 172.21.15.129:0/3984341852 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550011440 con 0x55e5a978a4d0 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.869+0000 7f4568d8d740 1 -- 172.21.15.129:0/3984341852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 msgr2=0x55e5a9b52260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.869+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3984341852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 0x55e5a9b52260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f455000bb60 tx=0x7f455000f730 comp rx=0 tx=0).stop 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.869+0000 7f4568d8d740 1 -- 172.21.15.129:0/3984341852 shutdown_connections 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.869+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3984341852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.869+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3984341852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 0x55e5a9b52260 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.869+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3984341852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a97b3390 0x55e5a9b51d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.869+0000 7f4568d8d740 1 -- 172.21.15.129:0/3984341852 >> 172.21.15.129:0/3984341852 conn(0x55e5a97a74e0 msgr2=0x55e5a9b4e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.869+0000 7f4568d8d740 1 -- 172.21.15.129:0/3984341852 shutdown_connections 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.869+0000 7f4568d8d740 1 -- 172.21.15.129:0/3984341852 wait complete. 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f4568d8d740 1 -- start start 2024-05-20T18:07:25.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f4568d8d740 1 Processor -- start 2024-05-20T18:07:25.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a978a4d0 0x55e5a9b5de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a97926b0 0x55e5a9b5e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b5c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f4568d8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e5a9b5efb0 con 0x55e5a97b3390 2024-05-20T18:07:25.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f4568d8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e5a9a72c40 con 0x55e5a978a4d0 2024-05-20T18:07:25.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f4568d8d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e5a9b5f130 con 0x55e5a97926b0 2024-05-20T18:07:25.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f4568d8d740 10 client.0 _ll_register_callbacks cb 0x55e5a97cabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f455bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b5c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f455bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b5c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41212/0 (socket says 172.21.15.129:41212) 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f455affd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a97926b0 0x55e5a9b5e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f455bfff640 1 -- 172.21.15.129:0/3706406974 learned_addr learned my addr 172.21.15.129:0/3706406974 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.870+0000 7f455b7fe640 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a978a4d0 0x55e5a9b5de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f455affd640 1 -- 172.21.15.129:0/3706406974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a978a4d0 msgr2=0x55e5a9b5de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f455affd640 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a978a4d0 0x55e5a9b5de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f455affd640 1 -- 172.21.15.129:0/3706406974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 msgr2=0x55e5a9b5c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f455affd640 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b5c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f455affd640 1 -- 172.21.15.129:0/3706406974 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5a98957d0 con 0x55e5a97926b0 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f455bfff640 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b5c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f455affd640 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a97926b0 0x55e5a9b5e350 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x55e5a9b532e0 tx=0x7f4550003de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=948b44fb3491071d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f453affd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:07:25.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f453affd640 1 -- 172.21.15.129:0/3706406974 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550015030 con 0x55e5a97926b0 2024-05-20T18:07:25.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f4568d8d740 1 -- 172.21.15.129:0/3706406974 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f455000be50 con 0x55e5a97926b0 2024-05-20T18:07:25.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f453affd640 1 -- 172.21.15.129:0/3706406974 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f455000bc50 con 0x55e5a97926b0 2024-05-20T18:07:25.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f453affd640 1 -- 172.21.15.129:0/3706406974 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f45500035f0 con 0x55e5a97926b0 2024-05-20T18:07:25.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f4568d8d740 10 client.26515 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:25.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f4568d8d740 1 -- 172.21.15.129:0/3706406974 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e5a9b6b830 con 0x55e5a97926b0 2024-05-20T18:07:25.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.871+0000 7f453affd640 1 -- 172.21.15.129:0/3706406974 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f455000be50 con 0x55e5a97926b0 2024-05-20T18:07:25.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.872+0000 7f453affd640 1 -- 172.21.15.129:0/3706406974 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f455002a030 con 0x55e5a97926b0 2024-05-20T18:07:25.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.872+0000 7f4568d8d740 10 client.26515 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.872+0000 7f4568d8d740 10 client.26515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.872+0000 7f4568d8d740 20 client.26515 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.872+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e5a9771f60 0x55e5a9b6eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.872+0000 7f4568d8d740 4 client.26515 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.872+0000 7f4568d8d740 1 -- 172.21.15.129:0/3706406974 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5a9ac5370 con 0x55e5a9771f60 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.872+0000 7f455b7fe640 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e5a9771f60 0x55e5a9b6eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.872+0000 7f455b7fe640 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e5a9771f60 0x55e5a9b6eb90 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ac7383ccf74fa9 server_cookie=991e721c5733fa58 in_seq=0 out_seq=0 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.872+0000 7f453affd640 10 client.26515 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f453affd640 1 -- 172.21.15.129:0/3706406974 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e5a9ac5370 con 0x55e5a9771f60 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f453affd640 10 client.26515 handle_command_reply: tid=0 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f4568d8d740 1 client.26515 shutdown 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f4568d8d740 1 -- 172.21.15.129:0/3706406974 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a97926b0 msgr2=0x55e5a9b5e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a97926b0 0x55e5a9b5e350 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x55e5a9b532e0 tx=0x7f4550003de0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f4568d8d740 1 -- 172.21.15.129:0/3706406974 shutdown_connections 2024-05-20T18:07:25.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e5a9771f60 0x55e5a9b6eb90 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b5c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a97926b0 0x55e5a9b5e350 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3706406974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a978a4d0 0x55e5a9b5de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f453affd640 0 client.26515 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:25.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f4568d8d740 1 -- 172.21.15.129:0/3706406974 >> 172.21.15.129:0/3706406974 conn(0x55e5a97a74e0 msgr2=0x55e5a9b53a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.873+0000 7f4568d8d740 1 -- 172.21.15.129:0/3706406974 shutdown_connections 2024-05-20T18:07:25.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.874+0000 7f4568d8d740 1 -- 172.21.15.129:0/3706406974 wait complete. 2024-05-20T18:07:25.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.874+0000 7f4568d8d740 20 client.26515 trim_cache size 0 max 16384 2024-05-20T18:07:25.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.887+0000 7f4568d8d740 1 Processor -- start 2024-05-20T18:07:25.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.887+0000 7f4568d8d740 1 -- start start 2024-05-20T18:07:25.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.887+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a978a4d0 0x55e5a9b521c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.887+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a97926b0 0x55e5a9b52700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.887+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b74220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.887+0000 7f4568d8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f455c1acb60 con 0x55e5a97b3390 2024-05-20T18:07:25.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.887+0000 7f4568d8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f455c1af3c0 con 0x55e5a978a4d0 2024-05-20T18:07:25.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.887+0000 7f4568d8d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f455c1ad1b0 con 0x55e5a97926b0 2024-05-20T18:07:25.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.887+0000 7f455bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b74220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.887+0000 7f455bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b74220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41218/0 (socket says 172.21.15.129:41218) 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.887+0000 7f455bfff640 1 -- 172.21.15.129:0/3012357145 learned_addr learned my addr 172.21.15.129:0/3012357145 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.888+0000 7f455b7fe640 1 --2- 172.21.15.129:0/3012357145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a978a4d0 0x55e5a9b521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.888+0000 7f455affd640 1 --2- 172.21.15.129:0/3012357145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a97926b0 0x55e5a9b52700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.888+0000 7f455bfff640 1 -- 172.21.15.129:0/3012357145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a97926b0 msgr2=0x55e5a9b52700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.888+0000 7f455bfff640 1 --2- 172.21.15.129:0/3012357145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a97926b0 0x55e5a9b52700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.888+0000 7f455bfff640 1 -- 172.21.15.129:0/3012357145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a978a4d0 msgr2=0x55e5a9b521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.888+0000 7f455bfff640 1 --2- 172.21.15.129:0/3012357145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a978a4d0 0x55e5a9b521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.888+0000 7f455bfff640 1 -- 172.21.15.129:0/3012357145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5a9b5dc00 con 0x55e5a97b3390 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.888+0000 7f455bfff640 1 --2- 172.21.15.129:0/3012357145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b74220 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f455400fac0 tx=0x7f455401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2370e8f1069d49 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.888+0000 7f45417fa640 1 -- 172.21.15.129:0/3012357145 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4554011b00 con 0x55e5a97b3390 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.888+0000 7f45417fa640 1 -- 172.21.15.129:0/3012357145 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4554011ca0 con 0x55e5a97b3390 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.888+0000 7f45417fa640 1 -- 172.21.15.129:0/3012357145 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455401f650 con 0x55e5a97b3390 2024-05-20T18:07:25.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.889+0000 7f4568d8d740 1 -- 172.21.15.129:0/3012357145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 msgr2=0x55e5a9b74220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.889+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3012357145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b74220 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f455400fac0 tx=0x7f455401c000 comp rx=0 tx=0).stop 2024-05-20T18:07:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.889+0000 7f4568d8d740 1 -- 172.21.15.129:0/3012357145 shutdown_connections 2024-05-20T18:07:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.889+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3012357145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97b3390 0x55e5a9b74220 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.889+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3012357145 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a97926b0 0x55e5a9b52700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.889+0000 7f4568d8d740 1 --2- 172.21.15.129:0/3012357145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a978a4d0 0x55e5a9b521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.889+0000 7f4568d8d740 1 -- 172.21.15.129:0/3012357145 >> 172.21.15.129:0/3012357145 conn(0x55e5a97a74e0 msgr2=0x55e5a9b51ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.889+0000 7f4568d8d740 1 -- 172.21.15.129:0/3012357145 shutdown_connections 2024-05-20T18:07:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.889+0000 7f4568d8d740 1 -- 172.21.15.129:0/3012357145 wait complete. 2024-05-20T18:07:25.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.889+0000 7f4568d8d740 1 -- start start 2024-05-20T18:07:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.889+0000 7f4568d8d740 1 Processor -- start 2024-05-20T18:07:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 0x55e5a9b61720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b61c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f4568d8d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a97b3390 0x55e5a9bd4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f4568d8d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f455c1ad880 con 0x55e5a97926b0 2024-05-20T18:07:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f4568d8d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f455c1aecf0 con 0x55e5a97b3390 2024-05-20T18:07:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f4568d8d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f455c1adf50 con 0x55e5a978a4d0 2024-05-20T18:07:25.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f4568d8d740 10 client.0 _ll_register_callbacks cb 0x7f4554012570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f455affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b61c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f455affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b61c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41220/0 (socket says 172.21.15.129:41220) 2024-05-20T18:07:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f455affd640 1 -- 172.21.15.129:0/1678726491 learned_addr learned my addr 172.21.15.129:0/1678726491 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f455b7fe640 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 0x55e5a9b61720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f455bfff640 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a97b3390 0x55e5a9bd4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f455affd640 1 -- 172.21.15.129:0/1678726491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 msgr2=0x55e5a9b61720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f455affd640 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 0x55e5a9b61720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f455affd640 1 -- 172.21.15.129:0/1678726491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a97b3390 msgr2=0x55e5a9bd4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:25.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f455affd640 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a97b3390 0x55e5a9bd4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f455affd640 1 -- 172.21.15.129:0/1678726491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5a9bd5310 con 0x55e5a97926b0 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f455affd640 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b61c60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x55e5a9b48a00 tx=0x7f4550003a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8dcb19c3ee263b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.890+0000 7f453affd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f453affd640 1 -- 172.21.15.129:0/1678726491 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550015040 con 0x55e5a97926b0 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f453affd640 1 -- 172.21.15.129:0/1678726491 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f4550004370 con 0x55e5a97926b0 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f4568d8d740 1 -- 172.21.15.129:0/1678726491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4554011ca0 con 0x55e5a97926b0 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f453affd640 1 -- 172.21.15.129:0/1678726491 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f455003c050 con 0x55e5a97926b0 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f4568d8d740 10 client.16815 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f4568d8d740 1 -- 172.21.15.129:0/1678726491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e5a9bd6560 con 0x55e5a97926b0 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f453affd640 1 -- 172.21.15.129:0/1678726491 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4554011ca0 con 0x55e5a97926b0 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f453affd640 1 -- 172.21.15.129:0/1678726491 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f4550033070 con 0x55e5a97926b0 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f4568d8d740 10 client.16815 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:25.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f4568d8d740 10 client.16815 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:25.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f4568d8d740 20 client.16815 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:25.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f4568d8d740 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e5a97473a0 0x55e5a9bd8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:25.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f4568d8d740 4 client.16815 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:07:25.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f4568d8d740 1 -- 172.21.15.129:0/1678726491 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e5a9abf6c0 con 0x55e5a97473a0 2024-05-20T18:07:25.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.891+0000 7f455b7fe640 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e5a97473a0 0x55e5a9bd8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:25.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f453affd640 10 client.16815 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:25.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f455b7fe640 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e5a97473a0 0x55e5a9bd8d20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf6a5f0ec366f8df server_cookie=80ecb03e62a0513 in_seq=0 out_seq=0 2024-05-20T18:07:25.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f453affd640 1 -- 172.21.15.129:0/1678726491 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e5a9abf6c0 con 0x55e5a97473a0 2024-05-20T18:07:25.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f453affd640 10 client.16815 handle_command_reply: tid=0 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 1 client.16815 shutdown 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 1 -- 172.21.15.129:0/1678726491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 msgr2=0x55e5a9b61c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b61c60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x55e5a9b48a00 tx=0x7f4550003a40 comp rx=0 tx=0).stop 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 1 -- 172.21.15.129:0/1678726491 shutdown_connections 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e5a97473a0 0x55e5a9bd8d20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e5a97b3390 0x55e5a9bd4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e5a97926b0 0x55e5a9b61c60 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 1 --2- 172.21.15.129:0/1678726491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e5a978a4d0 0x55e5a9b61720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f453affd640 0 client.16815 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 1 -- 172.21.15.129:0/1678726491 >> 172.21.15.129:0/1678726491 conn(0x55e5a97a74e0 msgr2=0x55e5a9b524e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 1 -- 172.21.15.129:0/1678726491 shutdown_connections 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 1 -- 172.21.15.129:0/1678726491 wait complete. 2024-05-20T18:07:25.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.892+0000 7f4568d8d740 20 client.16815 trim_cache size 0 max 16384 2024-05-20T18:07:25.898 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:07:25.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 -- 172.21.15.129:0/754123539 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f45440a2d00 msgr2=0x7f45440a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f45440a2d00 0x7f45440a5210 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4554005fd0 tx=0x7f455400ac90 comp rx=0 tx=0).stop 2024-05-20T18:07:25.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 -- 172.21.15.129:0/754123539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f455c102820 msgr2=0x7f455c19f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:25.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f455c102820 0x7f455c19f5a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f454c02a930 tx=0x7f454c003b90 comp rx=0 tx=0).stop 2024-05-20T18:07:25.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 -- 172.21.15.129:0/754123539 shutdown_connections 2024-05-20T18:07:25.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f45440a2d00 0x7f45440a5210 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f455c107f10 0x7f455c199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f455c107a70 0x7f455c19fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 --2- 172.21.15.129:0/754123539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f455c102820 0x7f455c19f5a0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 -- 172.21.15.129:0/754123539 >> 172.21.15.129:0/754123539 conn(0x7f455c0768d0 msgr2=0x7f455c108470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 -- 172.21.15.129:0/754123539 shutdown_connections 2024-05-20T18:07:25.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:25.893+0000 7f4561ee9640 1 -- 172.21.15.129:0/754123539 wait complete. 2024-05-20T18:07:26.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:25 smithi154 ceph-mon[36482]: pgmap v274: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 36 MiB/s wr, 261 op/s 2024-05-20T18:07:26.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:25 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/4063639003' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:07:26.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:25 smithi129 ceph-mon[32029]: pgmap v274: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 36 MiB/s wr, 261 op/s 2024-05-20T18:07:26.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:25 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4063639003' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:07:26.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:25 smithi204 ceph-mon[37076]: pgmap v274: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 36 MiB/s wr, 261 op/s 2024-05-20T18:07:26.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:25 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/4063639003' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:07:26.903 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:07:26.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:07:26.903 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:07:27.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.034+0000 7f3cc3232640 1 -- 172.21.15.129:0/805166311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cbc1013f0 msgr2=0x7f3cbc101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.034+0000 7f3cc3232640 1 --2- 172.21.15.129:0/805166311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cbc1013f0 0x7f3cbc101800 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009900 tx=0x7f3cb802a410 comp rx=0 tx=0).stop 2024-05-20T18:07:27.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.034+0000 7f3cc3232640 1 -- 172.21.15.129:0/805166311 shutdown_connections 2024-05-20T18:07:27.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.034+0000 7f3cc3232640 1 --2- 172.21.15.129:0/805166311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cbc105930 0x7f3cbc105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.034+0000 7f3cc3232640 1 --2- 172.21.15.129:0/805166311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cbc1063f0 0x7f3cbc10db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.034+0000 7f3cc3232640 1 --2- 172.21.15.129:0/805166311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cbc1013f0 0x7f3cbc101800 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.034+0000 7f3cc3232640 1 -- 172.21.15.129:0/805166311 >> 172.21.15.129:0/805166311 conn(0x7f3cbc076830 msgr2=0x7f3cbc076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.035+0000 7f3cc3232640 1 -- 172.21.15.129:0/805166311 shutdown_connections 2024-05-20T18:07:27.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.035+0000 7f3cc3232640 1 -- 172.21.15.129:0/805166311 wait complete. 2024-05-20T18:07:27.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.035+0000 7f3cc3232640 1 Processor -- start 2024-05-20T18:07:27.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.035+0000 7f3cc3232640 1 -- start start 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.035+0000 7f3cc3232640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cbc1013f0 0x7f3cbc19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.035+0000 7f3cc3232640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cbc105930 0x7f3cbc19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.035+0000 7f3cc3232640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cbc1063f0 0x7f3cbc19b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc3232640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc1114a0 con 0x7f3cbc1063f0 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc3232640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc111320 con 0x7f3cbc1013f0 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc3232640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc111620 con 0x7f3cbc105930 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc17cf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cbc1063f0 0x7f3cbc19b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc17cf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cbc1063f0 0x7f3cbc19b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41238/0 (socket says 172.21.15.129:41238) 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc17cf640 1 -- 172.21.15.129:0/1238937789 learned_addr learned my addr 172.21.15.129:0/1238937789 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc0fce640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cbc1013f0 0x7f3cbc19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cbc105930 0x7f3cbc19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc17cf640 1 -- 172.21.15.129:0/1238937789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cbc105930 msgr2=0x7f3cbc19aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc17cf640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cbc105930 0x7f3cbc19aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc17cf640 1 -- 172.21.15.129:0/1238937789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cbc1013f0 msgr2=0x7f3cbc19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc17cf640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cbc1013f0 0x7f3cbc19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc17cf640 1 -- 172.21.15.129:0/1238937789 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cbc1a58e0 con 0x7f3cbc1063f0 2024-05-20T18:07:27.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc17cf640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cbc1063f0 0x7f3cbc19b0e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400cce0 tx=0x7f3cb400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cb1ffb640 1 -- 172.21.15.129:0/1238937789 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb400f450 con 0x7f3cbc1063f0 2024-05-20T18:07:27.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cb1ffb640 1 -- 172.21.15.129:0/1238937789 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cb400fad0 con 0x7f3cbc1063f0 2024-05-20T18:07:27.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cc3232640 1 -- 172.21.15.129:0/1238937789 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cbc1a5bd0 con 0x7f3cbc1063f0 2024-05-20T18:07:27.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.036+0000 7f3cb1ffb640 1 -- 172.21.15.129:0/1238937789 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4018c10 con 0x7f3cbc1063f0 2024-05-20T18:07:27.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.037+0000 7f3cc3232640 1 -- 172.21.15.129:0/1238937789 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cbc1a60d0 con 0x7f3cbc1063f0 2024-05-20T18:07:27.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.039+0000 7f3cb1ffb640 1 -- 172.21.15.129:0/1238937789 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f3cb400f5f0 con 0x7f3cbc1063f0 2024-05-20T18:07:27.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.039+0000 7f3cca1b9740 1 Processor -- start 2024-05-20T18:07:27.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.039+0000 7f3cca1b9740 1 -- start start 2024-05-20T18:07:27.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3268e410 0x55ca32a2d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a2db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a2e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ca32992990 con 0x55ca3266d730 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ca3293a5b0 con 0x55ca32665550 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ca3298e950 con 0x55ca3268e410 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a2e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb1ffb640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3ca40a2d00 0x7f3ca40a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a2e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41242/0 (socket says 172.21.15.129:41242) 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb3fff640 1 -- 172.21.15.129:0/4058470637 learned_addr learned my addr 172.21.15.129:0/4058470637 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/4058470637 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3268e410 0x55ca32a2d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cc0fce640 1 --2- 172.21.15.129:0/4058470637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a2db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cc17cf640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3ca40a2d00 0x7f3ca40a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb1ffb640 1 -- 172.21.15.129:0/1238937789 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f3cb40cfb20 con 0x7f3cbc1063f0 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb3fff640 1 -- 172.21.15.129:0/4058470637 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3268e410 msgr2=0x55ca32a2d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/4058470637 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3268e410 0x55ca32a2d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb3fff640 1 -- 172.21.15.129:0/4058470637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 msgr2=0x55ca32a2db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/4058470637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a2db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb3fff640 1 -- 172.21.15.129:0/4058470637 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca3276e7e0 con 0x55ca3266d730 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.040+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/4058470637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a2e0a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009560 tx=0x7f3cac0096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee69f3ff22a3709 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.041+0000 7f3cc17cf640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3ca40a2d00 0x7f3ca40a5210 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3cb409bf30 tx=0x7f3cb406f010 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.041+0000 7f3ca2ffd640 1 -- 172.21.15.129:0/4058470637 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac012ca0 con 0x55ca3266d730 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.041+0000 7f3ca2ffd640 1 -- 172.21.15.129:0/4058470637 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cac012e40 con 0x55ca3266d730 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.041+0000 7f3ca2ffd640 1 -- 172.21.15.129:0/4058470637 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac003830 con 0x55ca3266d730 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.041+0000 7f3cca1b9740 1 -- 172.21.15.129:0/4058470637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 msgr2=0x55ca32a2e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.041+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/4058470637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a2e0a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009560 tx=0x7f3cac0096e0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.042+0000 7f3cca1b9740 1 -- 172.21.15.129:0/4058470637 shutdown_connections 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.042+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/4058470637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a2db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.042+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/4058470637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a2e0a0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.042+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/4058470637 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3268e410 0x55ca32a2d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.042+0000 7f3cca1b9740 1 -- 172.21.15.129:0/4058470637 >> 172.21.15.129:0/4058470637 conn(0x55ca32682560 msgr2=0x55ca32a29dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.042+0000 7f3cca1b9740 1 -- 172.21.15.129:0/4058470637 shutdown_connections 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.042+0000 7f3cca1b9740 1 -- 172.21.15.129:0/4058470637 wait complete. 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.042+0000 7f3cca1b9740 1 -- start start 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.042+0000 7f3cca1b9740 1 Processor -- start 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.042+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a37ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a38240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3268e410 0x55ca32a3de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ca32a3a970 con 0x55ca3266d730 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ca3294db40 con 0x55ca32665550 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ca32a3aaf0 con 0x55ca3268e410 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cca1b9740 10 client.0 _ll_register_callbacks cb 0x55ca329420f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cc0fce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a38240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cc0fce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a38240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41258/0 (socket says 172.21.15.129:41258) 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cc0fce640 1 -- 172.21.15.129:0/2099827120 learned_addr learned my addr 172.21.15.129:0/2099827120 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a37ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3268e410 0x55ca32a3de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cc0fce640 1 -- 172.21.15.129:0/2099827120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3268e410 msgr2=0x55ca32a3de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cc0fce640 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3268e410 0x55ca32a3de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cc0fce640 1 -- 172.21.15.129:0/2099827120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 msgr2=0x55ca32a37ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cc0fce640 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a37ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cc0fce640 1 -- 172.21.15.129:0/2099827120 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca327cef70 con 0x55ca3266d730 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3268e410 0x55ca32a3de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cc0fce640 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a38240 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009900 tx=0x7f3cb80044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ee9a6d79ff67cc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3c9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3c9bfff640 1 -- 172.21.15.129:0/2099827120 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8035040 con 0x55ca3266d730 2024-05-20T18:07:27.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3c9bfff640 1 -- 172.21.15.129:0/2099827120 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f3cb8005510 con 0x55ca3266d730 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cca1b9740 1 -- 172.21.15.129:0/2099827120 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ca329a00c0 con 0x55ca3266d730 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3c9bfff640 1 -- 172.21.15.129:0/2099827120 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3cb80046c0 con 0x55ca3266d730 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cca1b9740 10 client.16827 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.043+0000 7f3cca1b9740 1 -- 172.21.15.129:0/2099827120 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ca32666170 con 0x55ca3266d730 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.044+0000 7f3c9bfff640 1 -- 172.21.15.129:0/2099827120 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ca329a00c0 con 0x55ca3266d730 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.044+0000 7f3c9bfff640 1 -- 172.21.15.129:0/2099827120 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f3cb8003bb0 con 0x55ca3266d730 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.044+0000 7f3cca1b9740 10 client.16827 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.044+0000 7f3cca1b9740 10 client.16827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.044+0000 7f3cca1b9740 20 client.16827 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.044+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca3264d1f0 0x55ca32a4a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.044+0000 7f3cca1b9740 4 client.16827 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.044+0000 7f3cca1b9740 1 -- 172.21.15.129:0/2099827120 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca3299a410 con 0x55ca3264d1f0 2024-05-20T18:07:27.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.044+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca3264d1f0 0x55ca32a4a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.045+0000 7f3c9bfff640 10 client.16827 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.045+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca3264d1f0 0x55ca32a4a1b0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82bdcfb4ec29b96 server_cookie=db3f4514c0a72874 in_seq=0 out_seq=0 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.045+0000 7f3c9bfff640 1 -- 172.21.15.129:0/2099827120 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ca3299a410 con 0x55ca3264d1f0 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.045+0000 7f3c9bfff640 10 client.16827 handle_command_reply: tid=0 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.045+0000 7f3cca1b9740 1 client.16827 shutdown 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3cca1b9740 1 -- 172.21.15.129:0/2099827120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 msgr2=0x55ca32a38240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a38240 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009900 tx=0x7f3cb80044f0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3cca1b9740 1 -- 172.21.15.129:0/2099827120 shutdown_connections 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca3264d1f0 0x55ca32a4a1b0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3268e410 0x55ca32a3de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a38240 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3c9bfff640 0 client.16827 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:27.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/2099827120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a37ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3cca1b9740 1 -- 172.21.15.129:0/2099827120 >> 172.21.15.129:0/2099827120 conn(0x55ca32682560 msgr2=0x55ca32a2f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3cca1b9740 1 -- 172.21.15.129:0/2099827120 shutdown_connections 2024-05-20T18:07:27.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3cca1b9740 1 -- 172.21.15.129:0/2099827120 wait complete. 2024-05-20T18:07:27.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.046+0000 7f3cca1b9740 20 client.16827 trim_cache size 0 max 16384 2024-05-20T18:07:27.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cca1b9740 1 Processor -- start 2024-05-20T18:07:27.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cca1b9740 1 -- start start 2024-05-20T18:07:27.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3264d1f0 0x55ca32a3dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a2f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a44290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc1ac9a0 con 0x55ca3266d730 2024-05-20T18:07:27.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc1af200 con 0x55ca32665550 2024-05-20T18:07:27.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc1acff0 con 0x55ca3264d1f0 2024-05-20T18:07:27.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cb3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a44290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cb3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a44290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41262/0 (socket says 172.21.15.129:41262) 2024-05-20T18:07:27.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cb3fff640 1 -- 172.21.15.129:0/11629664 learned_addr learned my addr 172.21.15.129:0/11629664 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/11629664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3264d1f0 0x55ca32a3dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cc0fce640 1 --2- 172.21.15.129:0/11629664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a2f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cb3fff640 1 -- 172.21.15.129:0/11629664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3264d1f0 msgr2=0x55ca32a3dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/11629664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3264d1f0 0x55ca32a3dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cb3fff640 1 -- 172.21.15.129:0/11629664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 msgr2=0x55ca32a2f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/11629664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a2f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cb3fff640 1 -- 172.21.15.129:0/11629664 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca32a2a6d0 con 0x55ca3266d730 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.064+0000 7f3cc0fce640 1 --2- 172.21.15.129:0/11629664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a2f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.065+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/11629664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a44290 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3cac015020 tx=0x7f3cac017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae21d7d3a73c5d71 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.065+0000 7f3ca27fc640 1 -- 172.21.15.129:0/11629664 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac02ce60 con 0x55ca3266d730 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.065+0000 7f3ca27fc640 1 -- 172.21.15.129:0/11629664 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cac02c3c0 con 0x55ca3266d730 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.065+0000 7f3ca27fc640 1 -- 172.21.15.129:0/11629664 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac003850 con 0x55ca3266d730 2024-05-20T18:07:27.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.065+0000 7f3cca1b9740 1 -- 172.21.15.129:0/11629664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 msgr2=0x55ca32a44290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.065+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/11629664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a44290 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3cac015020 tx=0x7f3cac017040 comp rx=0 tx=0).stop 2024-05-20T18:07:27.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 -- 172.21.15.129:0/11629664 shutdown_connections 2024-05-20T18:07:27.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/11629664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca32665550 0x55ca32a2f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/11629664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca3266d730 0x55ca32a44290 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/11629664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3264d1f0 0x55ca32a3dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 -- 172.21.15.129:0/11629664 >> 172.21.15.129:0/11629664 conn(0x55ca32682560 msgr2=0x55ca32a2f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 -- 172.21.15.129:0/11629664 shutdown_connections 2024-05-20T18:07:27.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 -- 172.21.15.129:0/11629664 wait complete. 2024-05-20T18:07:27.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 -- start start 2024-05-20T18:07:27.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 Processor -- start 2024-05-20T18:07:27.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3264d1f0 0x55ca32aaef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca32665550 0x55ca32aaf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca3266d730 0x55ca32a30e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc1ad6c0 con 0x55ca32665550 2024-05-20T18:07:27.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc1aeb30 con 0x55ca3266d730 2024-05-20T18:07:27.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.066+0000 7f3cca1b9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3cbc1add90 con 0x55ca3264d1f0 2024-05-20T18:07:27.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cca1b9740 10 client.0 _ll_register_callbacks cb 0x7f3cb8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:27.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cc0fce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca32665550 0x55ca32aaf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cc0fce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca32665550 0x55ca32aaf470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41274/0 (socket says 172.21.15.129:41274) 2024-05-20T18:07:27.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cc0fce640 1 -- 172.21.15.129:0/904002141 learned_addr learned my addr 172.21.15.129:0/904002141 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:27.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3264d1f0 0x55ca32aaef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cc0fce640 1 -- 172.21.15.129:0/904002141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3264d1f0 msgr2=0x55ca32aaef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cc0fce640 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3264d1f0 0x55ca32aaef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cc0fce640 1 -- 172.21.15.129:0/904002141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca3266d730 msgr2=0x55ca32a30e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cc0fce640 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca3266d730 0x55ca32a30e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cc0fce640 1 -- 172.21.15.129:0/904002141 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca32a31440 con 0x55ca32665550 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3264d1f0 0x55ca32aaef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cc0fce640 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca32665550 0x55ca32aaf470 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8006010 tx=0x7f3cb8004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4e370cdec8e6a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3c9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3c9bfff640 1 -- 172.21.15.129:0/904002141 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8035030 con 0x55ca32665550 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3c9bfff640 1 -- 172.21.15.129:0/904002141 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f3cb8005510 con 0x55ca32665550 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cca1b9740 1 -- 172.21.15.129:0/904002141 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cac02c3c0 con 0x55ca32665550 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3c9bfff640 1 -- 172.21.15.129:0/904002141 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3cb805d050 con 0x55ca32665550 2024-05-20T18:07:27.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cca1b9740 10 client.16833 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.067+0000 7f3cca1b9740 1 -- 172.21.15.129:0/904002141 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ca32ab65b0 con 0x55ca32665550 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.068+0000 7f3c9bfff640 1 -- 172.21.15.129:0/904002141 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cac02c3c0 con 0x55ca32665550 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.068+0000 7f3c9bfff640 1 -- 172.21.15.129:0/904002141 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f3cb80039d0 con 0x55ca32665550 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.068+0000 7f3cca1b9740 10 client.16833 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.068+0000 7f3cca1b9740 10 client.16833 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.068+0000 7f3cca1b9740 20 client.16833 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.068+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca3268e410 0x55ca32ab8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.068+0000 7f3cca1b9740 4 client.16833 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.068+0000 7f3cca1b9740 1 -- 172.21.15.129:0/904002141 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ca32a336d0 con 0x55ca3268e410 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.068+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca3268e410 0x55ca32ab8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.068+0000 7f3c9bfff640 10 client.16833 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.068+0000 7f3cb3fff640 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca3268e410 0x55ca32ab8df0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76548111f27b46d3 server_cookie=674f72730489a075 in_seq=0 out_seq=0 2024-05-20T18:07:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3c9bfff640 1 -- 172.21.15.129:0/904002141 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ca32a336d0 con 0x55ca3268e410 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3c9bfff640 10 client.16833 handle_command_reply: tid=0 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 1 client.16833 shutdown 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 1 -- 172.21.15.129:0/904002141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca32665550 msgr2=0x55ca32aaf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca32665550 0x55ca32aaf470 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8006010 tx=0x7f3cb8004230 comp rx=0 tx=0).stop 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 1 -- 172.21.15.129:0/904002141 shutdown_connections 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55ca3268e410 0x55ca32ab8df0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca3266d730 0x55ca32a30e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ca32665550 0x55ca32aaf470 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3c9bfff640 0 client.16833 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 1 --2- 172.21.15.129:0/904002141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ca3264d1f0 0x55ca32aaef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 1 -- 172.21.15.129:0/904002141 >> 172.21.15.129:0/904002141 conn(0x55ca32682560 msgr2=0x55ca32a2d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 1 -- 172.21.15.129:0/904002141 shutdown_connections 2024-05-20T18:07:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 1 -- 172.21.15.129:0/904002141 wait complete. 2024-05-20T18:07:27.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.070+0000 7f3cca1b9740 20 client.16833 trim_cache size 0 max 16384 2024-05-20T18:07:27.075 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:07:27.075 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:07:27.075 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "72ff0bed-5336-492d-a76e-c70c26c48c65", 2024-05-20T18:07:27.075 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:07:27.075 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:07:27.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 -- 172.21.15.129:0/1238937789 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3ca40a2d00 msgr2=0x7f3ca40a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3ca40a2d00 0x7f3ca40a5210 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3cb409bf30 tx=0x7f3cb406f010 comp rx=0 tx=0).stop 2024-05-20T18:07:27.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 -- 172.21.15.129:0/1238937789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cbc1063f0 msgr2=0x7f3cbc19b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cbc1063f0 0x7f3cbc19b0e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400cce0 tx=0x7f3cb400a620 comp rx=0 tx=0).stop 2024-05-20T18:07:27.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 -- 172.21.15.129:0/1238937789 shutdown_connections 2024-05-20T18:07:27.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f3ca40a2d00 0x7f3ca40a5210 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3cbc1063f0 0x7f3cbc19b0e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3cbc105930 0x7f3cbc19aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 --2- 172.21.15.129:0/1238937789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cbc1013f0 0x7f3cbc19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 -- 172.21.15.129:0/1238937789 >> 172.21.15.129:0/1238937789 conn(0x7f3cbc076830 msgr2=0x7f3cbc102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 -- 172.21.15.129:0/1238937789 shutdown_connections 2024-05-20T18:07:27.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.071+0000 7f3cc3232640 1 -- 172.21.15.129:0/1238937789 wait complete. 2024-05-20T18:07:27.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:07:27.081 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:07:27.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.196+0000 7fdc0947a640 1 -- 172.21.15.129:0/3297045930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc04103270 msgr2=0x7fdc0410f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.196+0000 7fdc0947a640 1 --2- 172.21.15.129:0/3297045930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc04103270 0x7fdc0410f5f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009ca0 tx=0x7fdbfc02a8c0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.196+0000 7fdc0947a640 1 -- 172.21.15.129:0/3297045930 shutdown_connections 2024-05-20T18:07:27.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.196+0000 7fdc0947a640 1 --2- 172.21.15.129:0/3297045930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc04103270 0x7fdc0410f5f0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.196+0000 7fdc0947a640 1 --2- 172.21.15.129:0/3297045930 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc04108f70 0x7fdc04102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.196+0000 7fdc0947a640 1 --2- 172.21.15.129:0/3297045930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc04108ad0 0x7fdc041026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.196+0000 7fdc0947a640 1 -- 172.21.15.129:0/3297045930 >> 172.21.15.129:0/3297045930 conn(0x7fdc040781c0 msgr2=0x7fdc04078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.196+0000 7fdc0947a640 1 -- 172.21.15.129:0/3297045930 shutdown_connections 2024-05-20T18:07:27.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.196+0000 7fdc0947a640 1 -- 172.21.15.129:0/3297045930 wait complete. 2024-05-20T18:07:27.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc0947a640 1 Processor -- start 2024-05-20T18:07:27.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc0947a640 1 -- start start 2024-05-20T18:07:27.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc0947a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc04103270 0x7fdc04199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc0947a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc04108ad0 0x7fdc04199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc0947a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc04108f70 0x7fdc0419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc0947a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdc04112e10 con 0x7fdc04108f70 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc0947a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc04112c90 con 0x7fdc04108ad0 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc0947a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc04112f90 con 0x7fdc04103270 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc037fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc04108f70 0x7fdc0419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc037fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc04108f70 0x7fdc0419df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41306/0 (socket says 172.21.15.129:41306) 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc037fe640 1 -- 172.21.15.129:0/3689940245 learned_addr learned my addr 172.21.15.129:0/3689940245 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.197+0000 7fdc027fc640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc04108ad0 0x7fdc04199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdc02ffd640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc04103270 0x7fdc04199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdc037fe640 1 -- 172.21.15.129:0/3689940245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc04103270 msgr2=0x7fdc04199770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdc037fe640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc04103270 0x7fdc04199770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdc037fe640 1 -- 172.21.15.129:0/3689940245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc04108ad0 msgr2=0x7fdc04199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdc037fe640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc04108ad0 0x7fdc04199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdc037fe640 1 -- 172.21.15.129:0/3689940245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0419e5d0 con 0x7fdc04108f70 2024-05-20T18:07:27.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdc037fe640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc04108f70 0x7fdc0419df90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc031a20 tx=0x7fdbfc031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdbfbfff640 1 -- 172.21.15.129:0/3689940245 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc003c30 con 0x7fdc04108f70 2024-05-20T18:07:27.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdbfbfff640 1 -- 172.21.15.129:0/3689940245 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbfc003dd0 con 0x7fdc04108f70 2024-05-20T18:07:27.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdc0947a640 1 -- 172.21.15.129:0/3689940245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc0419e860 con 0x7fdc04108f70 2024-05-20T18:07:27.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdbfbfff640 1 -- 172.21.15.129:0/3689940245 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc02fcf0 con 0x7fdc04108f70 2024-05-20T18:07:27.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.198+0000 7fdc0947a640 1 -- 172.21.15.129:0/3689940245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc0419edb0 con 0x7fdc04108f70 2024-05-20T18:07:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.200+0000 7fdbfbfff640 1 -- 172.21.15.129:0/3689940245 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fdbfc004050 con 0x7fdc04108f70 2024-05-20T18:07:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc10402740 1 Processor -- start 2024-05-20T18:07:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc10402740 1 -- start start 2024-05-20T18:07:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc10402740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf5f390 0x557c3d302140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc10402740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d302680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc10402740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 0x557c3d302bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc10402740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557c3d263d00 con 0x557c3cf5f390 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc10402740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557c3d20b4c0 con 0x557c3cf364d0 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc10402740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557c3d25fcc0 con 0x557c3cf3e6b0 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc027fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf5f390 0x557c3d302140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc027fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf5f390 0x557c3d302140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41314/0 (socket says 172.21.15.129:41314) 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc027fc640 1 -- 172.21.15.129:0/4292463507 learned_addr learned my addr 172.21.15.129:0/4292463507 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc02ffd640 1 --2- 172.21.15.129:0/4292463507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d302680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdbfbfff640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdbf00a2d00 0x7fdbf00a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc027fc640 1 --2- 172.21.15.129:0/4292463507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 0x557c3d302bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc037fe640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdbf00a2d00 0x7fdbf00a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc027fc640 1 -- 172.21.15.129:0/4292463507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 msgr2=0x557c3d302bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc027fc640 1 --2- 172.21.15.129:0/4292463507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 0x557c3d302bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.201+0000 7fdc027fc640 1 -- 172.21.15.129:0/4292463507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 msgr2=0x557c3d302680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.202+0000 7fdc027fc640 1 --2- 172.21.15.129:0/4292463507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d302680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.202+0000 7fdc027fc640 1 -- 172.21.15.129:0/4292463507 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c3d0417d0 con 0x557c3cf5f390 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.202+0000 7fdbfbfff640 1 -- 172.21.15.129:0/3689940245 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fdbfc0f0730 con 0x7fdc04108f70 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.202+0000 7fdc027fc640 1 --2- 172.21.15.129:0/4292463507 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf5f390 0x557c3d302140 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00afb0 tx=0x7fdbec00a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212c794758eb08d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.202+0000 7fdbf97fa640 1 -- 172.21.15.129:0/4292463507 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec019020 con 0x557c3cf5f390 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.202+0000 7fdc037fe640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdbf00a2d00 0x7fdbf00a5210 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0ee440 tx=0x7fdbfc02a8c0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.202+0000 7fdbf97fa640 1 -- 172.21.15.129:0/4292463507 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec004230 con 0x557c3cf5f390 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.202+0000 7fdbf97fa640 1 -- 172.21.15.129:0/4292463507 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec00f650 con 0x557c3cf5f390 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.203+0000 7fdc10402740 1 -- 172.21.15.129:0/4292463507 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf5f390 msgr2=0x557c3d302140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.203+0000 7fdc10402740 1 --2- 172.21.15.129:0/4292463507 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf5f390 0x557c3d302140 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00afb0 tx=0x7fdbec00a680 comp rx=0 tx=0).stop 2024-05-20T18:07:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.203+0000 7fdc10402740 1 -- 172.21.15.129:0/4292463507 shutdown_connections 2024-05-20T18:07:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.203+0000 7fdc10402740 1 --2- 172.21.15.129:0/4292463507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d302680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.203+0000 7fdc10402740 1 --2- 172.21.15.129:0/4292463507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 0x557c3d302bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.203+0000 7fdc10402740 1 --2- 172.21.15.129:0/4292463507 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf5f390 0x557c3d302140 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.203+0000 7fdc10402740 1 -- 172.21.15.129:0/4292463507 >> 172.21.15.129:0/4292463507 conn(0x557c3cf534e0 msgr2=0x557c3d2fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.203+0000 7fdc10402740 1 -- 172.21.15.129:0/4292463507 shutdown_connections 2024-05-20T18:07:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.203+0000 7fdc10402740 1 -- 172.21.15.129:0/4292463507 wait complete. 2024-05-20T18:07:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.203+0000 7fdc10402740 1 -- start start 2024-05-20T18:07:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.203+0000 7fdc10402740 1 Processor -- start 2024-05-20T18:07:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc10402740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d30c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc10402740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf3e6b0 0x557c3d30cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc10402740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf5f390 0x557c3d312980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc10402740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557c3d30f490 con 0x557c3cf3e6b0 2024-05-20T18:07:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc10402740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557c3d21eb90 con 0x557c3cf364d0 2024-05-20T18:07:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc10402740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557c3d30f610 con 0x557c3cf5f390 2024-05-20T18:07:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc10402740 10 client.0 _ll_register_callbacks cb 0x557c3d25ada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc02ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf3e6b0 0x557c3d30cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc02ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf3e6b0 0x557c3d30cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41320/0 (socket says 172.21.15.129:41320) 2024-05-20T18:07:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc02ffd640 1 -- 172.21.15.129:0/321893623 learned_addr learned my addr 172.21.15.129:0/321893623 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc027fc640 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d30c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc02ffd640 1 -- 172.21.15.129:0/321893623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf5f390 msgr2=0x557c3d312980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc02ffd640 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf5f390 0x557c3d312980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc02ffd640 1 -- 172.21.15.129:0/321893623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 msgr2=0x557c3d30c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc02ffd640 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d30c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc02ffd640 1 -- 172.21.15.129:0/321893623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c3cf773f0 con 0x557c3cf3e6b0 2024-05-20T18:07:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc02ffd640 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf3e6b0 0x557c3d30cd40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400be40 tx=0x7fdbf4012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab0844caabd267b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdbea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdbea7fc640 1 -- 172.21.15.129:0/321893623 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40035f0 con 0x557c3cf3e6b0 2024-05-20T18:07:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdbea7fc640 1 -- 172.21.15.129:0/321893623 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fdbf4003790 con 0x557c3cf3e6b0 2024-05-20T18:07:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.204+0000 7fdc10402740 1 -- 172.21.15.129:0/321893623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbec004230 con 0x557c3cf3e6b0 2024-05-20T18:07:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdbea7fc640 1 -- 172.21.15.129:0/321893623 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdbf4003990 con 0x557c3cf3e6b0 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdc10402740 10 client.16845 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdc10402740 1 -- 172.21.15.129:0/321893623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557c3cf370f0 con 0x557c3cf3e6b0 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdbea7fc640 1 -- 172.21.15.129:0/321893623 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec004230 con 0x557c3cf3e6b0 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdbea7fc640 1 -- 172.21.15.129:0/321893623 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fdbf4021030 con 0x557c3cf3e6b0 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdc10402740 10 client.16845 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdc10402740 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdc10402740 20 client.16845 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdc10402740 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557c3cf1df60 0x557c3d31f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdc10402740 4 client.16845 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdc10402740 1 -- 172.21.15.129:0/321893623 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c3d271430 con 0x557c3cf1df60 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.205+0000 7fdc027fc640 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557c3cf1df60 0x557c3d31f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdc027fc640 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557c3cf1df60 0x557c3d31f080 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bbc918f6300dcdc server_cookie=634d4c86c282a07b in_seq=0 out_seq=0 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdbea7fc640 10 client.16845 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdbea7fc640 1 -- 172.21.15.129:0/321893623 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557c3d271430 con 0x557c3cf1df60 2024-05-20T18:07:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdbea7fc640 10 client.16845 handle_command_reply: tid=0 2024-05-20T18:07:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdc10402740 1 client.16845 shutdown 2024-05-20T18:07:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdc10402740 1 -- 172.21.15.129:0/321893623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf3e6b0 msgr2=0x557c3d30cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdc10402740 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf3e6b0 0x557c3d30cd40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400be40 tx=0x7fdbf4012820 comp rx=0 tx=0).stop 2024-05-20T18:07:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdc10402740 1 -- 172.21.15.129:0/321893623 shutdown_connections 2024-05-20T18:07:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdc10402740 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557c3cf1df60 0x557c3d31f080 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdc10402740 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf5f390 0x557c3d312980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdbea7fc640 0 client.16845 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdc10402740 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf3e6b0 0x557c3d30cd40 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdc10402740 1 --2- 172.21.15.129:0/321893623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d30c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.206+0000 7fdc10402740 1 -- 172.21.15.129:0/321893623 >> 172.21.15.129:0/321893623 conn(0x557c3cf534e0 msgr2=0x557c3d2f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.207+0000 7fdc10402740 1 -- 172.21.15.129:0/321893623 shutdown_connections 2024-05-20T18:07:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.207+0000 7fdc10402740 1 -- 172.21.15.129:0/321893623 wait complete. 2024-05-20T18:07:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.207+0000 7fdc10402740 20 client.16845 trim_cache size 0 max 16384 2024-05-20T18:07:27.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.224+0000 7fdc10402740 1 Processor -- start 2024-05-20T18:07:27.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.224+0000 7fdc10402740 1 -- start start 2024-05-20T18:07:27.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.224+0000 7fdc10402740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf1df60 0x557c3d312880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.224+0000 7fdc10402740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d30aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.224+0000 7fdc10402740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 0x557c3d30b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.224+0000 7fdc10402740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdc041aa920 con 0x557c3cf1df60 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.224+0000 7fdc10402740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc041ad180 con 0x557c3cf364d0 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.224+0000 7fdc10402740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc041aaf70 con 0x557c3cf3e6b0 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdc027fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf1df60 0x557c3d312880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdc027fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf1df60 0x557c3d312880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41324/0 (socket says 172.21.15.129:41324) 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdc027fc640 1 -- 172.21.15.129:0/431149915 learned_addr learned my addr 172.21.15.129:0/431149915 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdc02ffd640 1 --2- 172.21.15.129:0/431149915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d30aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdc027fc640 1 --2- 172.21.15.129:0/431149915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 0x557c3d30b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdc027fc640 1 -- 172.21.15.129:0/431149915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 msgr2=0x557c3d30b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdc027fc640 1 --2- 172.21.15.129:0/431149915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 0x557c3d30b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdc027fc640 1 -- 172.21.15.129:0/431149915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 msgr2=0x557c3d30aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdc027fc640 1 --2- 172.21.15.129:0/431149915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d30aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdc027fc640 1 -- 172.21.15.129:0/431149915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c3d30d500 con 0x557c3cf1df60 2024-05-20T18:07:27.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdc027fc640 1 --2- 172.21.15.129:0/431149915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf1df60 0x557c3d312880 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00afb0 tx=0x7fdbec003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c672d6442358a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdbf8ff9640 1 -- 172.21.15.129:0/431149915 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec02c070 con 0x557c3cf1df60 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdbf8ff9640 1 -- 172.21.15.129:0/431149915 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec00a5b0 con 0x557c3cf1df60 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.225+0000 7fdbf8ff9640 1 -- 172.21.15.129:0/431149915 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec019040 con 0x557c3cf1df60 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.226+0000 7fdc10402740 1 -- 172.21.15.129:0/431149915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf1df60 msgr2=0x557c3d312880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.226+0000 7fdc10402740 1 --2- 172.21.15.129:0/431149915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf1df60 0x557c3d312880 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00afb0 tx=0x7fdbec003c10 comp rx=0 tx=0).stop 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.226+0000 7fdc10402740 1 -- 172.21.15.129:0/431149915 shutdown_connections 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.226+0000 7fdc10402740 1 --2- 172.21.15.129:0/431149915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf364d0 0x557c3d30aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.226+0000 7fdc10402740 1 --2- 172.21.15.129:0/431149915 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 0x557c3d30b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.226+0000 7fdc10402740 1 --2- 172.21.15.129:0/431149915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf1df60 0x557c3d312880 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.226+0000 7fdc10402740 1 -- 172.21.15.129:0/431149915 >> 172.21.15.129:0/431149915 conn(0x557c3cf534e0 msgr2=0x557c3d307760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.226+0000 7fdc10402740 1 -- 172.21.15.129:0/431149915 shutdown_connections 2024-05-20T18:07:27.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.226+0000 7fdc10402740 1 -- 172.21.15.129:0/431149915 wait complete. 2024-05-20T18:07:27.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.226+0000 7fdc10402740 1 -- start start 2024-05-20T18:07:27.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.226+0000 7fdc10402740 1 Processor -- start 2024-05-20T18:07:27.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc10402740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf1df60 0x557c3d314100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc10402740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf364d0 0x557c3d314640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc10402740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 0x557c3d3853e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc10402740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdc041ab640 con 0x557c3cf364d0 2024-05-20T18:07:27.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc10402740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc041acab0 con 0x557c3cf1df60 2024-05-20T18:07:27.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc10402740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc041abd10 con 0x557c3cf3e6b0 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc10402740 10 client.0 _ll_register_callbacks cb 0x7fdbf400b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc02ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf364d0 0x557c3d314640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc02ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf364d0 0x557c3d314640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41334/0 (socket says 172.21.15.129:41334) 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc02ffd640 1 -- 172.21.15.129:0/3859820983 learned_addr learned my addr 172.21.15.129:0/3859820983 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc027fc640 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf1df60 0x557c3d314100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc02ffd640 1 -- 172.21.15.129:0/3859820983 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 msgr2=0x557c3d3853e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc02ffd640 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 0x557c3d3853e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc02ffd640 1 -- 172.21.15.129:0/3859820983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf1df60 msgr2=0x557c3d314100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc02ffd640 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf1df60 0x557c3d314100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc02ffd640 1 -- 172.21.15.129:0/3859820983 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c3d385920 con 0x557c3cf364d0 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdc02ffd640 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf364d0 0x557c3d314640 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4001eb0 tx=0x7fdbf40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970de9b748a0dc08 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdbea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.227+0000 7fdbea7fc640 1 -- 172.21.15.129:0/3859820983 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4005040 con 0x557c3cf364d0 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdbea7fc640 1 -- 172.21.15.129:0/3859820983 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fdbf40051e0 con 0x557c3cf364d0 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdc10402740 1 -- 172.21.15.129:0/3859820983 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557c3d26b780 con 0x557c3cf364d0 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdbea7fc640 1 -- 172.21.15.129:0/3859820983 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdbf4039050 con 0x557c3cf364d0 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdc10402740 10 client.16851 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdc10402740 1 -- 172.21.15.129:0/3859820983 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557c3d386b20 con 0x557c3cf364d0 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdbea7fc640 1 -- 172.21.15.129:0/3859820983 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557c3d26b780 con 0x557c3cf364d0 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdbea7fc640 1 -- 172.21.15.129:0/3859820983 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fdbf4019070 con 0x557c3cf364d0 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdc10402740 10 client.16851 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdc10402740 10 client.16851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdc10402740 20 client.16851 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdc10402740 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557c3cf5f390 0x557c3d389300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdc10402740 4 client.16851 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:07:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdc10402740 1 -- 172.21.15.129:0/3859820983 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557c3d1e86f0 con 0x557c3cf5f390 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.228+0000 7fdc027fc640 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557c3cf5f390 0x557c3d389300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc027fc640 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557c3cf5f390 0x557c3d389300 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a4eb14f5f7c52b6 server_cookie=7ce9642089b55b44 in_seq=0 out_seq=0 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdbea7fc640 10 client.16851 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdbea7fc640 1 -- 172.21.15.129:0/3859820983 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557c3d1e86f0 con 0x557c3cf5f390 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdbea7fc640 10 client.16851 handle_command_reply: tid=0 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 1 client.16851 shutdown 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 1 -- 172.21.15.129:0/3859820983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf364d0 msgr2=0x557c3d314640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf364d0 0x557c3d314640 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4001eb0 tx=0x7fdbf40038a0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 1 -- 172.21.15.129:0/3859820983 shutdown_connections 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557c3cf5f390 0x557c3d389300 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c3cf3e6b0 0x557c3d3853e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557c3cf364d0 0x557c3d314640 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdbea7fc640 0 client.16851 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:27.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 1 --2- 172.21.15.129:0/3859820983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557c3cf1df60 0x557c3d314100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 1 -- 172.21.15.129:0/3859820983 >> 172.21.15.129:0/3859820983 conn(0x557c3cf534e0 msgr2=0x557c3d307760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 1 -- 172.21.15.129:0/3859820983 shutdown_connections 2024-05-20T18:07:27.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 1 -- 172.21.15.129:0/3859820983 wait complete. 2024-05-20T18:07:27.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.229+0000 7fdc10402740 20 client.16851 trim_cache size 0 max 16384 2024-05-20T18:07:27.234 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:07:27.234 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:07:27.234 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:07:27.235 INFO:teuthology.orchestra.run.smithi129.stdout: "72ff0bed-5336-492d-a76e-c70c26c48c65": { 2024-05-20T18:07:27.235 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:07:27.235 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "72ff0bed-5336-492d-a76e-c70c26c48c65", 2024-05-20T18:07:27.235 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:07:27.235 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:07:27.235 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:07:27.235 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:07:27.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 -- 172.21.15.129:0/3689940245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdbf00a2d00 msgr2=0x7fdbf00a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdbf00a2d00 0x7fdbf00a5210 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0ee440 tx=0x7fdbfc02a8c0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 -- 172.21.15.129:0/3689940245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc04108f70 msgr2=0x7fdc0419df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:27.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc04108f70 0x7fdc0419df90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc031a20 tx=0x7fdbfc031a50 comp rx=0 tx=0).stop 2024-05-20T18:07:27.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 -- 172.21.15.129:0/3689940245 shutdown_connections 2024-05-20T18:07:27.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdbf00a2d00 0x7fdbf00a5210 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc04108ad0 0x7fdc04199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc04108f70 0x7fdc0419df90 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 --2- 172.21.15.129:0/3689940245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc04103270 0x7fdc04199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:27.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 -- 172.21.15.129:0/3689940245 >> 172.21.15.129:0/3689940245 conn(0x7fdc040781c0 msgr2=0x7fdc040ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:27.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 -- 172.21.15.129:0/3689940245 shutdown_connections 2024-05-20T18:07:27.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:27.230+0000 7fdc0947a640 1 -- 172.21.15.129:0/3689940245 wait complete. 2024-05-20T18:07:27.239 INFO:tasks.cephfs.filesystem:scrub status for tag:72ff0bed-5336-492d-a76e-c70c26c48c65 - {'path': '/', 'tag': '72ff0bed-5336-492d-a76e-c70c26c48c65', 'options': 'recursive,force'} 2024-05-20T18:07:28.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:27 smithi154 ceph-mon[36482]: pgmap v275: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 33 MiB/s wr, 241 op/s 2024-05-20T18:07:28.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:27 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:07:28.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:27 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:07:28.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:27 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:07:28.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:27 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:07:28.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:27 smithi129 ceph-mon[32029]: pgmap v275: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 33 MiB/s wr, 241 op/s 2024-05-20T18:07:28.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:27 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:07:28.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:27 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:07:28.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:27 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:07:28.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:27 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:07:28.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:27 smithi204 ceph-mon[37076]: pgmap v275: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 33 MiB/s wr, 241 op/s 2024-05-20T18:07:28.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:27 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:07:28.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:27 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:07:28.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:27 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:07:28.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:27 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:07:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.240+0000 7fed9238f640 1 -- 172.21.15.129:0/1947664328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c107a30 msgr2=0x7fed8c1015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.240+0000 7fed9238f640 1 --2- 172.21.15.129:0/1947664328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c107a30 0x7fed8c1015e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fed7c009900 tx=0x7fed7c02a410 comp rx=0 tx=0).stop 2024-05-20T18:07:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.240+0000 7fed9238f640 1 -- 172.21.15.129:0/1947664328 shutdown_connections 2024-05-20T18:07:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.240+0000 7fed9238f640 1 --2- 172.21.15.129:0/1947664328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed8c107ed0 0x7fed8c101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.240+0000 7fed9238f640 1 --2- 172.21.15.129:0/1947664328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed8c102190 0x7fed8c10e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.240+0000 7fed9238f640 1 --2- 172.21.15.129:0/1947664328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c107a30 0x7fed8c1015e0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.240+0000 7fed9238f640 1 -- 172.21.15.129:0/1947664328 >> 172.21.15.129:0/1947664328 conn(0x7fed8c0fd210 msgr2=0x7fed8c0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.240+0000 7fed9238f640 1 -- 172.21.15.129:0/1947664328 shutdown_connections 2024-05-20T18:07:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.240+0000 7fed9238f640 1 -- 172.21.15.129:0/1947664328 wait complete. 2024-05-20T18:07:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.240+0000 7fed9238f640 1 Processor -- start 2024-05-20T18:07:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.240+0000 7fed9238f640 1 -- start start 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed9238f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed8c102190 0x7fed8c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed9238f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c107a30 0x7fed8c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed9238f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed8c107ed0 0x7fed8c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed9238f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c111dd0 con 0x7fed8c107a30 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed9238f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c111c50 con 0x7fed8c102190 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed9238f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c111f50 con 0x7fed8c107ed0 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed8b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c107a30 0x7fed8c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed8b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c107a30 0x7fed8c198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41354/0 (socket says 172.21.15.129:41354) 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed8b7fe640 1 -- 172.21.15.129:0/3810826413 learned_addr learned my addr 172.21.15.129:0/3810826413 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed8bfff640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed8c102190 0x7fed8c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed9092c640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed8c107ed0 0x7fed8c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed8b7fe640 1 -- 172.21.15.129:0/3810826413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed8c107ed0 msgr2=0x7fed8c19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed8c107ed0 0x7fed8c19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed8b7fe640 1 -- 172.21.15.129:0/3810826413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed8c102190 msgr2=0x7fed8c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed8c102190 0x7fed8c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.241+0000 7fed8b7fe640 1 -- 172.21.15.129:0/3810826413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8c19d530 con 0x7fed8c107a30 2024-05-20T18:07:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.242+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c107a30 0x7fed8c198c30 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fed80005fd0 tx=0x7fed8000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.242+0000 7fed897fa640 1 -- 172.21.15.129:0/3810826413 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80017070 con 0x7fed8c107a30 2024-05-20T18:07:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.242+0000 7fed9238f640 1 -- 172.21.15.129:0/3810826413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed8c19d820 con 0x7fed8c107a30 2024-05-20T18:07:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.242+0000 7fed897fa640 1 -- 172.21.15.129:0/3810826413 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed80009c30 con 0x7fed8c107a30 2024-05-20T18:07:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.242+0000 7fed897fa640 1 -- 172.21.15.129:0/3810826413 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80003e40 con 0x7fed8c107a30 2024-05-20T18:07:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.242+0000 7fed9238f640 1 -- 172.21.15.129:0/3810826413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed8c1a51a0 con 0x7fed8c107a30 2024-05-20T18:07:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.244+0000 7fed897fa640 1 -- 172.21.15.129:0/3810826413 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fed80010460 con 0x7fed8c107a30 2024-05-20T18:07:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.244+0000 7fed99236740 1 Processor -- start 2024-05-20T18:07:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed99236740 1 -- start start 2024-05-20T18:07:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed99236740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156fd1b0 0x562c15a9c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed99236740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156d42f0 0x562c15a9c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed99236740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156dc4d0 0x562c15a9cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed99236740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562c15a01340 con 0x562c156fd1b0 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed99236740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562c159a9290 con 0x562c156d42f0 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed99236740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x562c159fd300 con 0x562c156dc4d0 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed9092c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156fd1b0 0x562c15a9c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed9092c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156fd1b0 0x562c15a9c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41358/0 (socket says 172.21.15.129:41358) 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed9092c640 1 -- 172.21.15.129:0/2554887246 learned_addr learned my addr 172.21.15.129:0/2554887246 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/2554887246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156dc4d0 0x562c15a9cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed8bfff640 1 --2- 172.21.15.129:0/2554887246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156d42f0 0x562c15a9c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed9092c640 1 -- 172.21.15.129:0/2554887246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156dc4d0 msgr2=0x562c15a9cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed897fa640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed740a2cf0 0x7fed740a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed9092c640 1 --2- 172.21.15.129:0/2554887246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156dc4d0 0x562c15a9cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed9092c640 1 -- 172.21.15.129:0/2554887246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156d42f0 msgr2=0x562c15a9c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed9092c640 1 --2- 172.21.15.129:0/2554887246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156d42f0 0x562c15a9c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed9092c640 1 -- 172.21.15.129:0/2554887246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c159cd8e0 con 0x562c156fd1b0 2024-05-20T18:07:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.245+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/2554887246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156dc4d0 0x562c15a9cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.246+0000 7fed897fa640 1 -- 172.21.15.129:0/3810826413 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fed800cf140 con 0x7fed8c107a30 2024-05-20T18:07:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.246+0000 7fed9092c640 1 --2- 172.21.15.129:0/2554887246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156fd1b0 0x562c15a9c080 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fed84009fd0 tx=0x7fed840118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6e4ddc35bcd19a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.246+0000 7fed9092c640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed740a2cf0 0x7fed740a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.246+0000 7fed727fc640 1 -- 172.21.15.129:0/2554887246 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8401e070 con 0x562c156fd1b0 2024-05-20T18:07:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.246+0000 7fed727fc640 1 -- 172.21.15.129:0/2554887246 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed8400fcc0 con 0x562c156fd1b0 2024-05-20T18:07:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.246+0000 7fed727fc640 1 -- 172.21.15.129:0/2554887246 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8400e5d0 con 0x562c156fd1b0 2024-05-20T18:07:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.247+0000 7fed99236740 1 -- 172.21.15.129:0/2554887246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156fd1b0 msgr2=0x562c15a9c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.247+0000 7fed99236740 1 --2- 172.21.15.129:0/2554887246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156fd1b0 0x562c15a9c080 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fed84009fd0 tx=0x7fed840118d0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.247+0000 7fed9092c640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed740a2cf0 0x7fed740a5200 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed8401b000 tx=0x7fed8400ecf0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.247+0000 7fed99236740 1 -- 172.21.15.129:0/2554887246 shutdown_connections 2024-05-20T18:07:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.247+0000 7fed99236740 1 --2- 172.21.15.129:0/2554887246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156d42f0 0x562c15a9c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.247+0000 7fed99236740 1 --2- 172.21.15.129:0/2554887246 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156dc4d0 0x562c15a9cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.247+0000 7fed99236740 1 --2- 172.21.15.129:0/2554887246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156fd1b0 0x562c15a9c080 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.247+0000 7fed99236740 1 -- 172.21.15.129:0/2554887246 >> 172.21.15.129:0/2554887246 conn(0x562c156f1300 msgr2=0x562c15a98830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.247+0000 7fed99236740 1 -- 172.21.15.129:0/2554887246 shutdown_connections 2024-05-20T18:07:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.247+0000 7fed99236740 1 -- 172.21.15.129:0/2554887246 wait complete. 2024-05-20T18:07:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.247+0000 7fed99236740 1 -- start start 2024-05-20T18:07:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed99236740 1 Processor -- start 2024-05-20T18:07:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed99236740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156d42f0 0x562c15aada90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed99236740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156dc4d0 0x562c15aadfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed99236740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156fd1b0 0x562c15aa9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed99236740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562c15aa91b0 con 0x562c156d42f0 2024-05-20T18:07:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed99236740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562c159bc790 con 0x562c156dc4d0 2024-05-20T18:07:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed99236740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x562c15aa9330 con 0x562c156fd1b0 2024-05-20T18:07:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed99236740 10 client.0 _ll_register_callbacks cb 0x562c159f7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed8bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156d42f0 0x562c15aada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed8bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156d42f0 0x562c15aada90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41374/0 (socket says 172.21.15.129:41374) 2024-05-20T18:07:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed8bfff640 1 -- 172.21.15.129:0/2902820835 learned_addr learned my addr 172.21.15.129:0/2902820835 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156dc4d0 0x562c15aadfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed9092c640 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156fd1b0 0x562c15aa9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed8bfff640 1 -- 172.21.15.129:0/2902820835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156fd1b0 msgr2=0x562c15aa9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed8bfff640 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156fd1b0 0x562c15aa9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed8bfff640 1 -- 172.21.15.129:0/2902820835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156dc4d0 msgr2=0x562c15aadfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed8bfff640 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156dc4d0 0x562c15aadfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed8bfff640 1 -- 172.21.15.129:0/2902820835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c15715210 con 0x562c156d42f0 2024-05-20T18:07:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.248+0000 7fed9092c640 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156fd1b0 0x562c15aa9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed8bfff640 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156d42f0 0x562c15aada90 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fed7c031550 tx=0x7fed7c0042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3cce9d134e54537 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2902820835 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c035040 con 0x562c156d42f0 2024-05-20T18:07:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2902820835 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fed7c005230 con 0x562c156d42f0 2024-05-20T18:07:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed99236740 1 -- 172.21.15.129:0/2902820835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed8400fcc0 con 0x562c156d42f0 2024-05-20T18:07:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2902820835 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fed7c005430 con 0x562c156d42f0 2024-05-20T18:07:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed99236740 10 client.16863 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed99236740 1 -- 172.21.15.129:0/2902820835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562c157e1620 con 0x562c156d42f0 2024-05-20T18:07:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2902820835 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed8400fcc0 con 0x562c156d42f0 2024-05-20T18:07:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2902820835 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fed7c0039c0 con 0x562c156d42f0 2024-05-20T18:07:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed99236740 10 client.16863 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed99236740 10 client.16863 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed99236740 20 client.16863 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed99236740 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562c156bbd80 0x562c15abac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed99236740 4 client.16863 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.249+0000 7fed99236740 1 -- 172.21.15.129:0/2902820835 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c15a0ea70 con 0x562c156bbd80 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.250+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562c156bbd80 0x562c15abac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.250+0000 7fed6b7fe640 10 client.16863 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.250+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562c156bbd80 0x562c15abac90 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdee0d84d7be5b2b server_cookie=c484af1a33f84595 in_seq=0 out_seq=0 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2902820835 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562c15a0ea70 con 0x562c156bbd80 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed6b7fe640 10 client.16863 handle_command_reply: tid=0 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 1 client.16863 shutdown 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 1 -- 172.21.15.129:0/2902820835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156d42f0 msgr2=0x562c15aada90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156d42f0 0x562c15aada90 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fed7c031550 tx=0x7fed7c0042e0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 1 -- 172.21.15.129:0/2902820835 shutdown_connections 2024-05-20T18:07:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562c156bbd80 0x562c15abac90 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156dc4d0 0x562c15aadfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156fd1b0 0x562c15aa9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed6b7fe640 0 client.16863 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 1 --2- 172.21.15.129:0/2902820835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156d42f0 0x562c15aada90 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 1 -- 172.21.15.129:0/2902820835 >> 172.21.15.129:0/2902820835 conn(0x562c156f1300 msgr2=0x562c15a9be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 1 -- 172.21.15.129:0/2902820835 shutdown_connections 2024-05-20T18:07:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 1 -- 172.21.15.129:0/2902820835 wait complete. 2024-05-20T18:07:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.251+0000 7fed99236740 20 client.16863 trim_cache size 0 max 16384 2024-05-20T18:07:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.266+0000 7fed99236740 1 Processor -- start 2024-05-20T18:07:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.266+0000 7fed99236740 1 -- start start 2024-05-20T18:07:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed99236740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156f1300 0x562c15aa0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed99236740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156d42f0 0x562c15ac0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed99236740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156dc4d0 0x562c15ac09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed99236740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1aaa40 con 0x562c156f1300 2024-05-20T18:07:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed99236740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1ad2a0 con 0x562c156d42f0 2024-05-20T18:07:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed99236740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1ab090 con 0x562c156dc4d0 2024-05-20T18:07:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed8bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156f1300 0x562c15aa0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed8bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156f1300 0x562c15aa0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41376/0 (socket says 172.21.15.129:41376) 2024-05-20T18:07:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed8bfff640 1 -- 172.21.15.129:0/2705373718 learned_addr learned my addr 172.21.15.129:0/2705373718 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/2705373718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156d42f0 0x562c15ac0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed8bfff640 1 -- 172.21.15.129:0/2705373718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156dc4d0 msgr2=0x562c15ac09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed8bfff640 1 --2- 172.21.15.129:0/2705373718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156dc4d0 0x562c15ac09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed8bfff640 1 -- 172.21.15.129:0/2705373718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156d42f0 msgr2=0x562c15ac0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed8bfff640 1 --2- 172.21.15.129:0/2705373718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156d42f0 0x562c15ac0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed8bfff640 1 -- 172.21.15.129:0/2705373718 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c15aa86e0 con 0x562c156f1300 2024-05-20T18:07:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed8bfff640 1 --2- 172.21.15.129:0/2705373718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156f1300 0x562c15aa0450 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fed7c000c00 tx=0x7fed7c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1f5eae1a871394 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed71ffb640 1 -- 172.21.15.129:0/2705373718 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c035030 con 0x562c156f1300 2024-05-20T18:07:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed71ffb640 1 -- 172.21.15.129:0/2705373718 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed7c040070 con 0x562c156f1300 2024-05-20T18:07:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.267+0000 7fed71ffb640 1 -- 172.21.15.129:0/2705373718 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c050070 con 0x562c156f1300 2024-05-20T18:07:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.268+0000 7fed99236740 1 -- 172.21.15.129:0/2705373718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156f1300 msgr2=0x562c15aa0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.268+0000 7fed99236740 1 --2- 172.21.15.129:0/2705373718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156f1300 0x562c15aa0450 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fed7c000c00 tx=0x7fed7c001e50 comp rx=0 tx=0).stop 2024-05-20T18:07:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.268+0000 7fed99236740 1 -- 172.21.15.129:0/2705373718 shutdown_connections 2024-05-20T18:07:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.268+0000 7fed99236740 1 --2- 172.21.15.129:0/2705373718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156d42f0 0x562c15ac0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.268+0000 7fed99236740 1 --2- 172.21.15.129:0/2705373718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156dc4d0 0x562c15ac09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.268+0000 7fed99236740 1 --2- 172.21.15.129:0/2705373718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156f1300 0x562c15aa0450 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.268+0000 7fed99236740 1 -- 172.21.15.129:0/2705373718 >> 172.21.15.129:0/2705373718 conn(0x562c15a93870 msgr2=0x562c15a9fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.268+0000 7fed99236740 1 -- 172.21.15.129:0/2705373718 shutdown_connections 2024-05-20T18:07:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.268+0000 7fed99236740 1 -- 172.21.15.129:0/2705373718 wait complete. 2024-05-20T18:07:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.268+0000 7fed99236740 1 -- start start 2024-05-20T18:07:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed99236740 1 Processor -- start 2024-05-20T18:07:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed99236740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156d42f0 0x562c15ab5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed99236740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156dc4d0 0x562c15ab5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed99236740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156f1300 0x562c15ab0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed99236740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1ab760 con 0x562c156dc4d0 2024-05-20T18:07:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed99236740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1acbd0 con 0x562c156f1300 2024-05-20T18:07:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed99236740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1abe30 con 0x562c156d42f0 2024-05-20T18:07:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed99236740 10 client.0 _ll_register_callbacks cb 0x7fed7c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed8b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156dc4d0 0x562c15ab5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed8b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156dc4d0 0x562c15ab5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41384/0 (socket says 172.21.15.129:41384) 2024-05-20T18:07:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed8b7fe640 1 -- 172.21.15.129:0/2869369563 learned_addr learned my addr 172.21.15.129:0/2869369563 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed9092c640 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156f1300 0x562c15ab0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed8bfff640 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156d42f0 0x562c15ab5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed8b7fe640 1 -- 172.21.15.129:0/2869369563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156d42f0 msgr2=0x562c15ab5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156d42f0 0x562c15ab5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed8b7fe640 1 -- 172.21.15.129:0/2869369563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156f1300 msgr2=0x562c15ab0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156f1300 0x562c15ab0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.269+0000 7fed8b7fe640 1 -- 172.21.15.129:0/2869369563 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c15ab0d20 con 0x562c156dc4d0 2024-05-20T18:07:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed8bfff640 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156d42f0 0x562c15ab5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed8b7fe640 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156dc4d0 0x562c15ab5670 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fed800cc850 tx=0x7fed8009af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd531e2ec2b5cc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2869369563 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8009bcc0 con 0x562c156dc4d0 2024-05-20T18:07:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2869369563 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fed800cca70 con 0x562c156dc4d0 2024-05-20T18:07:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed99236740 1 -- 172.21.15.129:0/2869369563 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed7c040070 con 0x562c156dc4d0 2024-05-20T18:07:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2869369563 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fed800ccc70 con 0x562c156dc4d0 2024-05-20T18:07:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed99236740 10 client.16869 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed99236740 1 -- 172.21.15.129:0/2869369563 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562c15b22b70 con 0x562c156dc4d0 2024-05-20T18:07:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2869369563 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed7c040070 con 0x562c156dc4d0 2024-05-20T18:07:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2869369563 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fed800d11d0 con 0x562c156dc4d0 2024-05-20T18:07:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed99236740 10 client.16869 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed99236740 10 client.16869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed99236740 20 client.16869 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed99236740 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562c156fd1b0 0x562c15b252e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed99236740 4 client.16869 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:07:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.270+0000 7fed99236740 1 -- 172.21.15.129:0/2869369563 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c15a08dc0 con 0x562c156fd1b0 2024-05-20T18:07:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.271+0000 7fed8bfff640 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562c156fd1b0 0x562c15b252e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.271+0000 7fed6b7fe640 10 client.16869 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.271+0000 7fed8bfff640 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562c156fd1b0 0x562c15b252e0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5de98cde2899fabf server_cookie=ce31e849244de418 in_seq=0 out_seq=0 2024-05-20T18:07:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed6b7fe640 1 -- 172.21.15.129:0/2869369563 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c15a08dc0 con 0x562c156fd1b0 2024-05-20T18:07:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed6b7fe640 10 client.16869 handle_command_reply: tid=0 2024-05-20T18:07:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 1 client.16869 shutdown 2024-05-20T18:07:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 1 -- 172.21.15.129:0/2869369563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156dc4d0 msgr2=0x562c15ab5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156dc4d0 0x562c15ab5670 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fed800cc850 tx=0x7fed8009af20 comp rx=0 tx=0).stop 2024-05-20T18:07:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 1 -- 172.21.15.129:0/2869369563 shutdown_connections 2024-05-20T18:07:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562c156fd1b0 0x562c15b252e0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c156f1300 0x562c15ab0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed6b7fe640 0 client.16869 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562c156dc4d0 0x562c15ab5670 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 1 --2- 172.21.15.129:0/2869369563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562c156d42f0 0x562c15ab5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 1 -- 172.21.15.129:0/2869369563 >> 172.21.15.129:0/2869369563 conn(0x562c15a93870 msgr2=0x562c15a9be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 1 -- 172.21.15.129:0/2869369563 shutdown_connections 2024-05-20T18:07:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 1 -- 172.21.15.129:0/2869369563 wait complete. 2024-05-20T18:07:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.276+0000 7fed99236740 20 client.16869 trim_cache size 0 max 16384 2024-05-20T18:07:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:07:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:07:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:07:29.280 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:07:29.280 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 -- 172.21.15.129:0/3810826413 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed740a2cf0 msgr2=0x7fed740a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:29.280 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed740a2cf0 0x7fed740a5200 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed8401b000 tx=0x7fed8400ecf0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 -- 172.21.15.129:0/3810826413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c107a30 msgr2=0x7fed8c198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c107a30 0x7fed8c198c30 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fed80005fd0 tx=0x7fed8000dde0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 -- 172.21.15.129:0/3810826413 shutdown_connections 2024-05-20T18:07:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fed740a2cf0 0x7fed740a5200 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed8c107ed0 0x7fed8c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed8c107a30 0x7fed8c198c30 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 --2- 172.21.15.129:0/3810826413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed8c102190 0x7fed8c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 -- 172.21.15.129:0/3810826413 >> 172.21.15.129:0/3810826413 conn(0x7fed8c0fd210 msgr2=0x7fed8c0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 -- 172.21.15.129:0/3810826413 shutdown_connections 2024-05-20T18:07:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:29.277+0000 7fed9238f640 1 -- 172.21.15.129:0/3810826413 wait complete. 2024-05-20T18:07:30.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:29 smithi154 ceph-mon[36482]: pgmap v276: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 30 MiB/s wr, 217 op/s 2024-05-20T18:07:30.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:29 smithi129 ceph-mon[32029]: pgmap v276: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 30 MiB/s wr, 217 op/s 2024-05-20T18:07:30.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:29 smithi204 ceph-mon[37076]: pgmap v276: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 30 MiB/s wr, 217 op/s 2024-05-20T18:07:31.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:30 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:07:31.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:30 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:07:31.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:30 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:07:32.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:31 smithi154 ceph-mon[36482]: pgmap v277: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 26 MiB/s wr, 194 op/s 2024-05-20T18:07:32.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:31 smithi129 ceph-mon[32029]: pgmap v277: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 26 MiB/s wr, 194 op/s 2024-05-20T18:07:32.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:31 smithi204 ceph-mon[37076]: pgmap v277: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 26 MiB/s wr, 194 op/s 2024-05-20T18:07:34.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:33 smithi154 ceph-mon[36482]: pgmap v278: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3.3 MiB/s wr, 28 op/s 2024-05-20T18:07:34.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:33 smithi129 ceph-mon[32029]: pgmap v278: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3.3 MiB/s wr, 28 op/s 2024-05-20T18:07:34.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:33 smithi204 ceph-mon[37076]: pgmap v278: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3.3 MiB/s wr, 28 op/s 2024-05-20T18:07:36.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:35 smithi154 ceph-mon[36482]: pgmap v279: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 198 KiB/s wr, 6 op/s 2024-05-20T18:07:36.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:35 smithi129 ceph-mon[32029]: pgmap v279: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 198 KiB/s wr, 6 op/s 2024-05-20T18:07:36.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:35 smithi204 ceph-mon[37076]: pgmap v279: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 198 KiB/s wr, 6 op/s 2024-05-20T18:07:38.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:37 smithi154 ceph-mon[36482]: pgmap v280: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.9 MiB/s wr, 19 op/s 2024-05-20T18:07:38.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:37 smithi129 ceph-mon[32029]: pgmap v280: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.9 MiB/s wr, 19 op/s 2024-05-20T18:07:38.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:37 smithi204 ceph-mon[37076]: pgmap v280: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.9 MiB/s wr, 19 op/s 2024-05-20T18:07:40.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:39 smithi154 ceph-mon[36482]: pgmap v281: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.1 MiB/s wr, 19 op/s 2024-05-20T18:07:40.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:39 smithi129 ceph-mon[32029]: pgmap v281: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.1 MiB/s wr, 19 op/s 2024-05-20T18:07:40.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:39 smithi204 ceph-mon[37076]: pgmap v281: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.1 MiB/s wr, 19 op/s 2024-05-20T18:07:42.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:41 smithi154 ceph-mon[36482]: pgmap v282: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-05-20T18:07:42.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:41 smithi129 ceph-mon[32029]: pgmap v282: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-05-20T18:07:42.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:41 smithi204 ceph-mon[37076]: pgmap v282: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.1 MiB/s wr, 20 op/s 2024-05-20T18:07:42.387 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.388+0000 7fb1125aa640 1 -- 172.21.15.129:0/2315868755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c10d070 msgr2=0x7fb10c10f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:42.430 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.388+0000 7fb1125aa640 1 --2- 172.21.15.129:0/2315868755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c10d070 0x7fb10c10f560 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb104009ca0 tx=0x7fb10402a8c0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.388+0000 7fb1125aa640 1 -- 172.21.15.129:0/2315868755 shutdown_connections 2024-05-20T18:07:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.388+0000 7fb1125aa640 1 --2- 172.21.15.129:0/2315868755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c10d070 0x7fb10c10f560 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.388+0000 7fb1125aa640 1 --2- 172.21.15.129:0/2315868755 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb10c10a680 0x7fb10c10caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.388+0000 7fb1125aa640 1 --2- 172.21.15.129:0/2315868755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c103c20 0x7fb10c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.431 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.388+0000 7fb1125aa640 1 -- 172.21.15.129:0/2315868755 >> 172.21.15.129:0/2315868755 conn(0x7fb10c077160 msgr2=0x7fb10c0775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.388+0000 7fb1125aa640 1 -- 172.21.15.129:0/2315868755 shutdown_connections 2024-05-20T18:07:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.388+0000 7fb1125aa640 1 -- 172.21.15.129:0/2315868755 wait complete. 2024-05-20T18:07:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.389+0000 7fb1125aa640 1 Processor -- start 2024-05-20T18:07:42.432 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.389+0000 7fb1125aa640 1 -- start start 2024-05-20T18:07:42.458 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.389+0000 7fb1125aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c103c20 0x7fb10c199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.459 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.389+0000 7fb1125aa640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb10c10a680 0x7fb10c199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.459 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.389+0000 7fb1125aa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c10d070 0x7fb10c19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.459 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.389+0000 7fb1125aa640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c112e20 con 0x7fb10c10d070 2024-05-20T18:07:42.459 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.389+0000 7fb1125aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c112ca0 con 0x7fb10c103c20 2024-05-20T18:07:42.459 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.389+0000 7fb1125aa640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c112fa0 con 0x7fb10c10a680 2024-05-20T18:07:42.459 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.389+0000 7fb110b47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c10d070 0x7fb10c19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb110b47640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c10d070 0x7fb10c19df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47962/0 (socket says 172.21.15.129:47962) 2024-05-20T18:07:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb110b47640 1 -- 172.21.15.129:0/2731545146 learned_addr learned my addr 172.21.15.129:0/2731545146 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb103fff640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb10c10a680 0x7fb10c199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb10bfff640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c103c20 0x7fb10c199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb110b47640 1 -- 172.21.15.129:0/2731545146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb10c10a680 msgr2=0x7fb10c199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb110b47640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb10c10a680 0x7fb10c199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb110b47640 1 -- 172.21.15.129:0/2731545146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c103c20 msgr2=0x7fb10c199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb110b47640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c103c20 0x7fb10c199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb110b47640 1 -- 172.21.15.129:0/2731545146 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c19e5a0 con 0x7fb10c10d070 2024-05-20T18:07:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb10bfff640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c103c20 0x7fb10c199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb110b47640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c10d070 0x7fb10c19df60 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb104031a60 tx=0x7fb104031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb109ffb640 1 -- 172.21.15.129:0/2731545146 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb104003c30 con 0x7fb10c10d070 2024-05-20T18:07:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb109ffb640 1 -- 172.21.15.129:0/2731545146 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb104003dd0 con 0x7fb10c10d070 2024-05-20T18:07:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb1125aa640 1 -- 172.21.15.129:0/2731545146 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10c19e830 con 0x7fb10c10d070 2024-05-20T18:07:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb109ffb640 1 -- 172.21.15.129:0/2731545146 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1040047e0 con 0x7fb10c10d070 2024-05-20T18:07:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.390+0000 7fb1125aa640 1 -- 172.21.15.129:0/2731545146 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10c19ed80 con 0x7fb10c10d070 2024-05-20T18:07:42.462 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.393+0000 7fb109ffb640 1 -- 172.21.15.129:0/2731545146 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fb10402f790 con 0x7fb10c10d070 2024-05-20T18:07:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.393+0000 7fb11944c740 1 Processor -- start 2024-05-20T18:07:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.393+0000 7fb11944c740 1 -- start start 2024-05-20T18:07:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.393+0000 7fb11944c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b691b0 0x558593f08080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.393+0000 7fb11944c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b402f0 0x558593f085c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.393+0000 7fb11944c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f08b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.393+0000 7fb11944c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558593e6d340 con 0x558593b484d0 2024-05-20T18:07:42.463 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.393+0000 7fb11944c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558593e15290 con 0x558593b691b0 2024-05-20T18:07:42.464 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.393+0000 7fb11944c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558593e69300 con 0x558593b402f0 2024-05-20T18:07:42.464 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb103fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f08b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.464 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b402f0 0x558593f085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.464 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb103fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f08b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47964/0 (socket says 172.21.15.129:47964) 2024-05-20T18:07:42.464 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb103fff640 1 -- 172.21.15.129:0/1008772113 learned_addr learned my addr 172.21.15.129:0/1008772113 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:42.464 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b402f0 0x558593f085c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:53338/0 (socket says 172.21.15.129:53338) 2024-05-20T18:07:42.465 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb103fff640 1 --2- 172.21.15.129:0/1008772113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b691b0 0x558593f08080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.465 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb109ffb640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb0f80a2cf0 0x7fb0f80a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.465 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb110b47640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb0f80a2cf0 0x7fb0f80a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.465 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb103fff640 1 -- 172.21.15.129:0/1008772113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b402f0 msgr2=0x558593f085c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:42.465 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb103fff640 1 --2- 172.21.15.129:0/1008772113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b402f0 0x558593f085c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.465 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb103fff640 1 -- 172.21.15.129:0/1008772113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b691b0 msgr2=0x558593f08080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:42.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb103fff640 1 --2- 172.21.15.129:0/1008772113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b691b0 0x558593f08080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb103fff640 1 -- 172.21.15.129:0/1008772113 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558593e398e0 con 0x558593b484d0 2024-05-20T18:07:42.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb109ffb640 1 -- 172.21.15.129:0/2731545146 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fb1040f0860 con 0x7fb10c10d070 2024-05-20T18:07:42.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb103fff640 1 --2- 172.21.15.129:0/1008772113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f08b00 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4005fd0 tx=0x7fb0f400e3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b791ccab2be3b590 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:42.466 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb1027fc640 1 -- 172.21.15.129:0/1008772113 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4004020 con 0x558593b484d0 2024-05-20T18:07:42.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb110b47640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb0f80a2cf0 0x7fb0f80a5200 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb1040b92b0 tx=0x7fb10408fec0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:42.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb1027fc640 1 -- 172.21.15.129:0/1008772113 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0f400bdb0 con 0x558593b484d0 2024-05-20T18:07:42.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.394+0000 7fb1027fc640 1 -- 172.21.15.129:0/1008772113 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f40038e0 con 0x558593b484d0 2024-05-20T18:07:42.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.395+0000 7fb11944c740 1 -- 172.21.15.129:0/1008772113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 msgr2=0x558593f08b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:42.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.395+0000 7fb11944c740 1 --2- 172.21.15.129:0/1008772113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f08b00 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4005fd0 tx=0x7fb0f400e3f0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.467 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.395+0000 7fb11944c740 1 -- 172.21.15.129:0/1008772113 shutdown_connections 2024-05-20T18:07:42.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.395+0000 7fb11944c740 1 --2- 172.21.15.129:0/1008772113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f08b00 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.395+0000 7fb11944c740 1 --2- 172.21.15.129:0/1008772113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b402f0 0x558593f085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.395+0000 7fb11944c740 1 --2- 172.21.15.129:0/1008772113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b691b0 0x558593f08080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.395+0000 7fb11944c740 1 -- 172.21.15.129:0/1008772113 >> 172.21.15.129:0/1008772113 conn(0x558593b5d300 msgr2=0x558593f04830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:42.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.395+0000 7fb11944c740 1 -- 172.21.15.129:0/1008772113 shutdown_connections 2024-05-20T18:07:42.468 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.395+0000 7fb11944c740 1 -- 172.21.15.129:0/1008772113 wait complete. 2024-05-20T18:07:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb11944c740 1 -- start start 2024-05-20T18:07:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb11944c740 1 Processor -- start 2024-05-20T18:07:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb11944c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b402f0 0x558593f14130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb11944c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b484d0 0x558593f14670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb11944c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b691b0 0x558593f12640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.469 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb11944c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558593f151b0 con 0x558593b691b0 2024-05-20T18:07:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb11944c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558593e28790 con 0x558593b484d0 2024-05-20T18:07:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb11944c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558593f15330 con 0x558593b402f0 2024-05-20T18:07:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb11944c740 10 client.0 _ll_register_callbacks cb 0x558593e63a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb103fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b691b0 0x558593f12640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.470 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb103fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b691b0 0x558593f12640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47966/0 (socket says 172.21.15.129:47966) 2024-05-20T18:07:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.396+0000 7fb103fff640 1 -- 172.21.15.129:0/1903218028 learned_addr learned my addr 172.21.15.129:0/1903218028 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb103fff640 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b402f0 0x558593f14130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb10bfff640 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b484d0 0x558593f14670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb103fff640 1 -- 172.21.15.129:0/1903218028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b402f0 msgr2=0x558593f14130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:42.471 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb103fff640 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b402f0 0x558593f14130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb103fff640 1 -- 172.21.15.129:0/1903218028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b484d0 msgr2=0x558593f14670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb103fff640 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b484d0 0x558593f14670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb103fff640 1 -- 172.21.15.129:0/1903218028 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558593b81210 con 0x558593b691b0 2024-05-20T18:07:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb103fff640 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b691b0 0x558593f12640 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4004610 tx=0x7fb0f4004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0602850d081e06 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb0eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:42.472 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/1903218028 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f400bad0 con 0x558593b691b0 2024-05-20T18:07:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb11944c740 1 -- 172.21.15.129:0/1903218028 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0f400bdb0 con 0x558593b691b0 2024-05-20T18:07:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/1903218028 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fb0f4003660 con 0x558593b691b0 2024-05-20T18:07:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/1903218028 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb0f4013550 con 0x558593b691b0 2024-05-20T18:07:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb11944c740 10 client.16881 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.397+0000 7fb11944c740 1 -- 172.21.15.129:0/1903218028 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558593c4d620 con 0x558593b691b0 2024-05-20T18:07:42.473 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.398+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/1903218028 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0f400bdb0 con 0x558593b691b0 2024-05-20T18:07:42.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.398+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/1903218028 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fb0f4020780 con 0x558593b691b0 2024-05-20T18:07:42.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.398+0000 7fb11944c740 10 client.16881 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:42.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.398+0000 7fb11944c740 10 client.16881 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:42.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.398+0000 7fb11944c740 20 client.16881 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:42.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.398+0000 7fb11944c740 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558593b27d80 0x558593f24910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.474 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.398+0000 7fb11944c740 4 client.16881 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.398+0000 7fb11944c740 1 -- 172.21.15.129:0/1903218028 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558593e7aa70 con 0x558593b27d80 2024-05-20T18:07:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.398+0000 7fb10bfff640 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558593b27d80 0x558593f24910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.399+0000 7fb10bfff640 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558593b27d80 0x558593f24910 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b0030c2870355b9 server_cookie=7bf1a220f9d187eb in_seq=0 out_seq=0 2024-05-20T18:07:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.399+0000 7fb0eb7fe640 10 client.16881 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:42.475 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.399+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/1903218028 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558593e7aa70 con 0x558593b27d80 2024-05-20T18:07:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.399+0000 7fb0eb7fe640 10 client.16881 handle_command_reply: tid=0 2024-05-20T18:07:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.399+0000 7fb11944c740 1 client.16881 shutdown 2024-05-20T18:07:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb11944c740 1 -- 172.21.15.129:0/1903218028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b691b0 msgr2=0x558593f12640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb11944c740 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b691b0 0x558593f12640 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4004610 tx=0x7fb0f4004510 comp rx=0 tx=0).stop 2024-05-20T18:07:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb11944c740 1 -- 172.21.15.129:0/1903218028 shutdown_connections 2024-05-20T18:07:42.476 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb11944c740 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558593b27d80 0x558593f24910 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb11944c740 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b484d0 0x558593f14670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb11944c740 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b691b0 0x558593f12640 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb11944c740 1 --2- 172.21.15.129:0/1903218028 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b402f0 0x558593f14130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb0eb7fe640 0 client.16881 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb11944c740 1 -- 172.21.15.129:0/1903218028 >> 172.21.15.129:0/1903218028 conn(0x558593b5d300 msgr2=0x558593f09da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:42.477 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb11944c740 1 -- 172.21.15.129:0/1903218028 shutdown_connections 2024-05-20T18:07:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb11944c740 1 -- 172.21.15.129:0/1903218028 wait complete. 2024-05-20T18:07:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.400+0000 7fb11944c740 20 client.16881 trim_cache size 0 max 16384 2024-05-20T18:07:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.415+0000 7fb11944c740 1 Processor -- start 2024-05-20T18:07:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.415+0000 7fb11944c740 1 -- start start 2024-05-20T18:07:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.415+0000 7fb11944c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b402f0 0x558593f0bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.478 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.415+0000 7fb11944c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f0c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.415+0000 7fb11944c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b691b0 0x558593f09cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.415+0000 7fb11944c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c1acb00 con 0x558593b484d0 2024-05-20T18:07:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.415+0000 7fb11944c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c1af360 con 0x558593b402f0 2024-05-20T18:07:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.415+0000 7fb11944c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c1ad150 con 0x558593b691b0 2024-05-20T18:07:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.415+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f0c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.479 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.415+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f0c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47974/0 (socket says 172.21.15.129:47974) 2024-05-20T18:07:42.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.415+0000 7fb10bfff640 1 -- 172.21.15.129:0/2120682644 learned_addr learned my addr 172.21.15.129:0/2120682644 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:42.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb103fff640 1 --2- 172.21.15.129:0/2120682644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b691b0 0x558593f09cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb103fff640 1 --2- 172.21.15.129:0/2120682644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b402f0 0x558593f0bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb10bfff640 1 -- 172.21.15.129:0/2120682644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b691b0 msgr2=0x558593f09cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:42.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb10bfff640 1 --2- 172.21.15.129:0/2120682644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b691b0 0x558593f09cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.480 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb10bfff640 1 -- 172.21.15.129:0/2120682644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b402f0 msgr2=0x558593f0bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:42.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb10bfff640 1 --2- 172.21.15.129:0/2120682644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b402f0 0x558593f0bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb10bfff640 1 -- 172.21.15.129:0/2120682644 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558593f07e10 con 0x558593b484d0 2024-05-20T18:07:42.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb103fff640 1 --2- 172.21.15.129:0/2120682644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b402f0 0x558593f0bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:42.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb10bfff640 1 --2- 172.21.15.129:0/2120682644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f0c270 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00cd70 tx=0x7fb0fc00c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b9a174cd49a1fd server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:42.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb101ffb640 1 -- 172.21.15.129:0/2120682644 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc004220 con 0x558593b484d0 2024-05-20T18:07:42.481 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb101ffb640 1 -- 172.21.15.129:0/2120682644 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0fc0043c0 con 0x558593b484d0 2024-05-20T18:07:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.416+0000 7fb101ffb640 1 -- 172.21.15.129:0/2120682644 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc003cd0 con 0x558593b484d0 2024-05-20T18:07:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.417+0000 7fb11944c740 1 -- 172.21.15.129:0/2120682644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 msgr2=0x558593f0c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.417+0000 7fb11944c740 1 --2- 172.21.15.129:0/2120682644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f0c270 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00cd70 tx=0x7fb0fc00c4f0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.417+0000 7fb11944c740 1 -- 172.21.15.129:0/2120682644 shutdown_connections 2024-05-20T18:07:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.417+0000 7fb11944c740 1 --2- 172.21.15.129:0/2120682644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b691b0 0x558593f09cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.417+0000 7fb11944c740 1 --2- 172.21.15.129:0/2120682644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b484d0 0x558593f0c270 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.417+0000 7fb11944c740 1 --2- 172.21.15.129:0/2120682644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b402f0 0x558593f0bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.417+0000 7fb11944c740 1 -- 172.21.15.129:0/2120682644 >> 172.21.15.129:0/2120682644 conn(0x558593b5d300 msgr2=0x558593f10f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.417+0000 7fb11944c740 1 -- 172.21.15.129:0/2120682644 shutdown_connections 2024-05-20T18:07:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.417+0000 7fb11944c740 1 -- 172.21.15.129:0/2120682644 wait complete. 2024-05-20T18:07:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.417+0000 7fb11944c740 1 -- start start 2024-05-20T18:07:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb11944c740 1 Processor -- start 2024-05-20T18:07:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb11944c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b402f0 0x558593f1a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb11944c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b484d0 0x558593f17ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb11944c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b691b0 0x558593f18220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb11944c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c1ad820 con 0x558593b402f0 2024-05-20T18:07:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb11944c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c1aec90 con 0x558593b484d0 2024-05-20T18:07:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb11944c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c1adef0 con 0x558593b691b0 2024-05-20T18:07:42.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb11944c740 10 client.0 _ll_register_callbacks cb 0x7fb0f400ee50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:42.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb103fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b402f0 0x558593f1a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb103fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b402f0 0x558593f1a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47988/0 (socket says 172.21.15.129:47988) 2024-05-20T18:07:42.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb103fff640 1 -- 172.21.15.129:0/341353517 learned_addr learned my addr 172.21.15.129:0/341353517 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:42.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb103fff640 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b691b0 0x558593f18220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.485 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb10bfff640 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b484d0 0x558593f17ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb103fff640 1 -- 172.21.15.129:0/341353517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b691b0 msgr2=0x558593f18220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:42.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb103fff640 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b691b0 0x558593f18220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb103fff640 1 -- 172.21.15.129:0/341353517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b484d0 msgr2=0x558593f17ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:42.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb103fff640 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b484d0 0x558593f17ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb103fff640 1 -- 172.21.15.129:0/341353517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558593f18760 con 0x558593b402f0 2024-05-20T18:07:42.486 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb10bfff640 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b484d0 0x558593f17ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:42.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb103fff640 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b402f0 0x558593f1a890 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4020c00 tx=0x7fb0f4004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d360c4a7a13875 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:42.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb0eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:42.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.418+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/341353517 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f403e020 con 0x558593b402f0 2024-05-20T18:07:42.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/341353517 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fb0f4020450 con 0x558593b402f0 2024-05-20T18:07:42.487 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb11944c740 1 -- 172.21.15.129:0/341353517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0fc0043c0 con 0x558593b402f0 2024-05-20T18:07:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/341353517 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb0f4046050 con 0x558593b402f0 2024-05-20T18:07:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb11944c740 10 client.16887 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb11944c740 1 -- 172.21.15.129:0/341353517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558593f8c750 con 0x558593b402f0 2024-05-20T18:07:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/341353517 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0fc0043c0 con 0x558593b402f0 2024-05-20T18:07:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/341353517 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fb0f4004660 con 0x558593b402f0 2024-05-20T18:07:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb11944c740 10 client.16887 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb11944c740 10 client.16887 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb11944c740 20 client.16887 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb11944c740 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558593afd1c0 0x558593f8ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb11944c740 4 client.16887 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:07:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb11944c740 1 -- 172.21.15.129:0/341353517 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558593e74dc0 con 0x558593afd1c0 2024-05-20T18:07:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.419+0000 7fb10bfff640 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558593afd1c0 0x558593f8ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.420+0000 7fb0eb7fe640 10 client.16887 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:42.490 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.420+0000 7fb10bfff640 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558593afd1c0 0x558593f8ee80 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baf59321b54ac17 server_cookie=646fb8d8b3e3e759 in_seq=0 out_seq=0 2024-05-20T18:07:42.490 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb0eb7fe640 1 -- 172.21.15.129:0/341353517 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558593e74dc0 con 0x558593afd1c0 2024-05-20T18:07:42.490 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb0eb7fe640 10 client.16887 handle_command_reply: tid=0 2024-05-20T18:07:42.490 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb11944c740 1 client.16887 shutdown 2024-05-20T18:07:42.490 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb11944c740 1 -- 172.21.15.129:0/341353517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b402f0 msgr2=0x558593f1a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb11944c740 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b402f0 0x558593f1a890 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4020c00 tx=0x7fb0f4004510 comp rx=0 tx=0).stop 2024-05-20T18:07:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb11944c740 1 -- 172.21.15.129:0/341353517 shutdown_connections 2024-05-20T18:07:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb11944c740 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558593afd1c0 0x558593f8ee80 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb11944c740 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558593b484d0 0x558593f17ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb11944c740 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558593b691b0 0x558593f18220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.491 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb11944c740 1 --2- 172.21.15.129:0/341353517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558593b402f0 0x558593f1a890 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.492 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb0eb7fe640 0 client.16887 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:42.492 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb11944c740 1 -- 172.21.15.129:0/341353517 >> 172.21.15.129:0/341353517 conn(0x558593b5d300 msgr2=0x558593f09a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:42.492 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb11944c740 1 -- 172.21.15.129:0/341353517 shutdown_connections 2024-05-20T18:07:42.492 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.429+0000 7fb11944c740 1 -- 172.21.15.129:0/341353517 wait complete. 2024-05-20T18:07:42.492 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb11944c740 20 client.16887 trim_cache size 0 max 16384 2024-05-20T18:07:42.492 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:07:42.492 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:07:42.492 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:07:42.493 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:07:42.493 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 -- 172.21.15.129:0/2731545146 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb0f80a2cf0 msgr2=0x7fb0f80a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:42.493 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb0f80a2cf0 0x7fb0f80a5200 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb1040b92b0 tx=0x7fb10408fec0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 -- 172.21.15.129:0/2731545146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c10d070 msgr2=0x7fb10c19df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c10d070 0x7fb10c19df60 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb104031a60 tx=0x7fb104031a90 comp rx=0 tx=0).stop 2024-05-20T18:07:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 -- 172.21.15.129:0/2731545146 shutdown_connections 2024-05-20T18:07:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb0f80a2cf0 0x7fb0f80a5200 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c10d070 0x7fb10c19df60 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb10c10a680 0x7fb10c199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 --2- 172.21.15.129:0/2731545146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb10c103c20 0x7fb10c199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 -- 172.21.15.129:0/2731545146 >> 172.21.15.129:0/2731545146 conn(0x7fb10c077160 msgr2=0x7fb10c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 -- 172.21.15.129:0/2731545146 shutdown_connections 2024-05-20T18:07:42.494 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:42.430+0000 7fb1125aa640 1 -- 172.21.15.129:0/2731545146 wait complete. 2024-05-20T18:07:44.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:43 smithi154 ceph-mon[36482]: pgmap v283: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 MiB/s wr, 17 op/s 2024-05-20T18:07:44.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:43 smithi129 ceph-mon[32029]: pgmap v283: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 MiB/s wr, 17 op/s 2024-05-20T18:07:44.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:43 smithi204 ceph-mon[37076]: pgmap v283: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 MiB/s wr, 17 op/s 2024-05-20T18:07:46.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:45 smithi154 ceph-mon[36482]: pgmap v284: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.1 MiB/s wr, 17 op/s 2024-05-20T18:07:46.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:45 smithi129 ceph-mon[32029]: pgmap v284: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.1 MiB/s wr, 17 op/s 2024-05-20T18:07:46.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:45 smithi204 ceph-mon[37076]: pgmap v284: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.1 MiB/s wr, 17 op/s 2024-05-20T18:07:48.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:47 smithi129 ceph-mon[32029]: pgmap v285: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-05-20T18:07:48.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:47 smithi204 ceph-mon[37076]: pgmap v285: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-05-20T18:07:48.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:47 smithi154 ceph-mon[36482]: pgmap v285: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.9 MiB/s wr, 18 op/s 2024-05-20T18:07:50.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:49 smithi129 ceph-mon[32029]: pgmap v286: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 187 KiB/s wr, 5 op/s 2024-05-20T18:07:50.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:49 smithi204 ceph-mon[37076]: pgmap v286: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 187 KiB/s wr, 5 op/s 2024-05-20T18:07:50.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:49 smithi154 ceph-mon[36482]: pgmap v286: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 187 KiB/s wr, 5 op/s 2024-05-20T18:07:52.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:51 smithi129 ceph-mon[32029]: pgmap v287: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 6.1 KiB/s wr, 5 op/s 2024-05-20T18:07:52.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:51 smithi204 ceph-mon[37076]: pgmap v287: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 6.1 KiB/s wr, 5 op/s 2024-05-20T18:07:52.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:51 smithi154 ceph-mon[36482]: pgmap v287: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 6.1 KiB/s wr, 5 op/s 2024-05-20T18:07:54.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:53 smithi129 ceph-mon[32029]: pgmap v288: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 3.2 KiB/s wr, 4 op/s 2024-05-20T18:07:54.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:53 smithi204 ceph-mon[37076]: pgmap v288: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 3.2 KiB/s wr, 4 op/s 2024-05-20T18:07:54.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:53 smithi154 ceph-mon[36482]: pgmap v288: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 3.2 KiB/s wr, 4 op/s 2024-05-20T18:07:55.538 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.540+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1426384366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc38107a30 msgr2=0x7fdc381015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.540+0000 7fdc3fce2640 1 --2- 172.21.15.129:0/1426384366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc38107a30 0x7fdc381015e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009900 tx=0x7fdc2802a410 comp rx=0 tx=0).stop 2024-05-20T18:07:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.540+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1426384366 shutdown_connections 2024-05-20T18:07:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.540+0000 7fdc3fce2640 1 --2- 172.21.15.129:0/1426384366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc38102190 0x7fdc3810e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.540+0000 7fdc3fce2640 1 --2- 172.21.15.129:0/1426384366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc38107ed0 0x7fdc38101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.540+0000 7fdc3fce2640 1 --2- 172.21.15.129:0/1426384366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc38107a30 0x7fdc381015e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.540+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1426384366 >> 172.21.15.129:0/1426384366 conn(0x7fdc380fd210 msgr2=0x7fdc380ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.540+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1426384366 shutdown_connections 2024-05-20T18:07:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.540+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1426384366 wait complete. 2024-05-20T18:07:55.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.541+0000 7fdc3fce2640 1 Processor -- start 2024-05-20T18:07:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.541+0000 7fdc3fce2640 1 -- start start 2024-05-20T18:07:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.541+0000 7fdc3fce2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc38102190 0x7fdc381986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.541+0000 7fdc3fce2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc38107a30 0x7fdc38198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.541+0000 7fdc3fce2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc38107ed0 0x7fdc3819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.541+0000 7fdc3fce2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdc38111d80 con 0x7fdc38102190 2024-05-20T18:07:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.541+0000 7fdc3fce2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc38111c00 con 0x7fdc38107ed0 2024-05-20T18:07:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.541+0000 7fdc3fce2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc38111f00 con 0x7fdc38107a30 2024-05-20T18:07:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.541+0000 7fdc3da7e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc38102190 0x7fdc381986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.541+0000 7fdc3da7e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc38102190 0x7fdc381986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36754/0 (socket says 172.21.15.129:36754) 2024-05-20T18:07:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.541+0000 7fdc3da7e640 1 -- 172.21.15.129:0/1909832328 learned_addr learned my addr 172.21.15.129:0/1909832328 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:55.542 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc3d27d640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc38107a30 0x7fdc38198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc38107ed0 0x7fdc3819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc3da7e640 1 -- 172.21.15.129:0/1909832328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc38107a30 msgr2=0x7fdc38198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc38107a30 0x7fdc38198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc3da7e640 1 -- 172.21.15.129:0/1909832328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc38107ed0 msgr2=0x7fdc3819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc38107ed0 0x7fdc3819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc3da7e640 1 -- 172.21.15.129:0/1909832328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3819d500 con 0x7fdc38102190 2024-05-20T18:07:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc38102190 0x7fdc381986a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdc28005940 tx=0x7fdc28004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc36ffd640 1 -- 172.21.15.129:0/1909832328 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc28035070 con 0x7fdc38102190 2024-05-20T18:07:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1909832328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3819d790 con 0x7fdc38102190 2024-05-20T18:07:55.543 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc36ffd640 1 -- 172.21.15.129:0/1909832328 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc28003ad0 con 0x7fdc38102190 2024-05-20T18:07:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc36ffd640 1 -- 172.21.15.129:0/1909832328 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2803bbf0 con 0x7fdc38102190 2024-05-20T18:07:55.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.542+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1909832328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3819dcd0 con 0x7fdc38102190 2024-05-20T18:07:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.545+0000 7fdc46b86740 1 Processor -- start 2024-05-20T18:07:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.545+0000 7fdc36ffd640 1 -- 172.21.15.129:0/1909832328 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fdc2804d440 con 0x7fdc38102190 2024-05-20T18:07:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc46b86740 1 -- start start 2024-05-20T18:07:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7852080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a748a2f0 0x5636a78525c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a74924d0 0x5636a7852b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc46b86740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5636a77b7340 con 0x5636a74b31b0 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc46b86740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5636a775f290 con 0x5636a74924d0 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc46b86740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5636a77b3300 con 0x5636a748a2f0 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc3e27f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7852080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc3e27f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7852080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36764/0 (socket says 172.21.15.129:36764) 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc3e27f640 1 -- 172.21.15.129:0/513993841 learned_addr learned my addr 172.21.15.129:0/513993841 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc3d27d640 1 --2- 172.21.15.129:0/513993841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a74924d0 0x5636a7852b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc36ffd640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdc200a2cf0 0x7fdc200a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.546+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/513993841 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a748a2f0 0x5636a78525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc3e27f640 1 -- 172.21.15.129:0/513993841 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a748a2f0 msgr2=0x5636a78525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/513993841 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a748a2f0 0x5636a78525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc3e27f640 1 -- 172.21.15.129:0/513993841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a74924d0 msgr2=0x5636a7852b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:55.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/513993841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a74924d0 0x5636a7852b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.553 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc3e27f640 1 -- 172.21.15.129:0/513993841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636a77838e0 con 0x5636a74b31b0 2024-05-20T18:07:55.553 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc36ffd640 1 -- 172.21.15.129:0/1909832328 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fdc280f1cc0 con 0x7fdc38102190 2024-05-20T18:07:55.553 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc3d27d640 1 --2- 172.21.15.129:0/513993841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a74924d0 0x5636a7852b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:07:55.553 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdc200a2cf0 0x7fdc200a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/513993841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7852080 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fdc300104f0 tx=0x7fdc300109b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4810bd3365e7293 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc1ffff640 1 -- 172.21.15.129:0/513993841 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc30019070 con 0x5636a74b31b0 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc1ffff640 1 -- 172.21.15.129:0/513993841 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc3000ede0 con 0x5636a74b31b0 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.547+0000 7fdc1ffff640 1 -- 172.21.15.129:0/513993841 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc30011ae0 con 0x5636a74b31b0 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.548+0000 7fdc46b86740 1 -- 172.21.15.129:0/513993841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 msgr2=0x5636a7852080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.548+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdc200a2cf0 0x7fdc200a5200 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdc30021000 tx=0x7fdc3001e000 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.548+0000 7fdc46b86740 1 --2- 172.21.15.129:0/513993841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7852080 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fdc300104f0 tx=0x7fdc300109b0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.548+0000 7fdc46b86740 1 -- 172.21.15.129:0/513993841 shutdown_connections 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.548+0000 7fdc46b86740 1 --2- 172.21.15.129:0/513993841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a74924d0 0x5636a7852b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.548+0000 7fdc46b86740 1 --2- 172.21.15.129:0/513993841 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a748a2f0 0x5636a78525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.548+0000 7fdc46b86740 1 --2- 172.21.15.129:0/513993841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7852080 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.548+0000 7fdc46b86740 1 -- 172.21.15.129:0/513993841 >> 172.21.15.129:0/513993841 conn(0x5636a74a7300 msgr2=0x5636a784e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.548+0000 7fdc46b86740 1 -- 172.21.15.129:0/513993841 shutdown_connections 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.548+0000 7fdc46b86740 1 -- 172.21.15.129:0/513993841 wait complete. 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc46b86740 1 -- start start 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc46b86740 1 Processor -- start 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a748a2f0 0x5636a785e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a785e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a74b31b0 0x5636a785c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc46b86740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5636a785f1b0 con 0x5636a74924d0 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc46b86740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5636a7772790 con 0x5636a74b31b0 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc46b86740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5636a785f330 con 0x5636a748a2f0 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc46b86740 10 client.0 _ll_register_callbacks cb 0x5636a77ada20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc3d27d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a785e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc3d27d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a785e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36772/0 (socket says 172.21.15.129:36772) 2024-05-20T18:07:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc3d27d640 1 -- 172.21.15.129:0/2698513293 learned_addr learned my addr 172.21.15.129:0/2698513293 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.549+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a748a2f0 0x5636a785e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a74b31b0 0x5636a785c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc3d27d640 1 -- 172.21.15.129:0/2698513293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a748a2f0 msgr2=0x5636a785e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc3d27d640 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a748a2f0 0x5636a785e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc3d27d640 1 -- 172.21.15.129:0/2698513293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a74b31b0 msgr2=0x5636a785c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc3d27d640 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a74b31b0 0x5636a785c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc3d27d640 1 -- 172.21.15.129:0/2698513293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636a74cb210 con 0x5636a74924d0 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc3d27d640 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a785e680 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00bf50 tx=0x7fdc2c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b967f963cccac05 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/2698513293 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c004030 con 0x5636a74924d0 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/2698513293 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fdc2c0041d0 con 0x5636a74924d0 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc46b86740 1 -- 172.21.15.129:0/2698513293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc3000ede0 con 0x5636a74924d0 2024-05-20T18:07:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/2698513293 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdc2c0043d0 con 0x5636a74924d0 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc46b86740 10 client.16899 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc46b86740 1 -- 172.21.15.129:0/2698513293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5636a7597620 con 0x5636a74924d0 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.550+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/2698513293 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc3000ede0 con 0x5636a74924d0 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.551+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/2698513293 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fdc2c020030 con 0x5636a74924d0 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.551+0000 7fdc46b86740 10 client.16899 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.551+0000 7fdc46b86740 10 client.16899 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.551+0000 7fdc46b86740 20 client.16899 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.551+0000 7fdc46b86740 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5636a7471d80 0x5636a786ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.551+0000 7fdc46b86740 4 client.16899 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.551+0000 7fdc46b86740 1 -- 172.21.15.129:0/2698513293 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636a77c4a70 con 0x5636a7471d80 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.551+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5636a7471d80 0x5636a786ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.551+0000 7fdc1d7fa640 10 client.16899 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.551+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5636a7471d80 0x5636a786ea70 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1c3cafdf37c1f7d server_cookie=8357bf9e5e6d8ff0 in_seq=0 out_seq=0 2024-05-20T18:07:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/2698513293 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5636a77c4a70 con 0x5636a7471d80 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc1d7fa640 10 client.16899 handle_command_reply: tid=0 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc46b86740 1 client.16899 shutdown 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc46b86740 1 -- 172.21.15.129:0/2698513293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 msgr2=0x5636a785e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc46b86740 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a785e680 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00bf50 tx=0x7fdc2c0118d0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc46b86740 1 -- 172.21.15.129:0/2698513293 shutdown_connections 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc46b86740 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5636a7471d80 0x5636a786ea70 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc46b86740 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a74b31b0 0x5636a785c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc46b86740 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a785e680 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc1d7fa640 0 client.16899 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc46b86740 1 --2- 172.21.15.129:0/2698513293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a748a2f0 0x5636a785e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc46b86740 1 -- 172.21.15.129:0/2698513293 >> 172.21.15.129:0/2698513293 conn(0x5636a74a7300 msgr2=0x5636a7851e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.552+0000 7fdc46b86740 1 -- 172.21.15.129:0/2698513293 shutdown_connections 2024-05-20T18:07:55.558 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.553+0000 7fdc46b86740 1 -- 172.21.15.129:0/2698513293 wait complete. 2024-05-20T18:07:55.559 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.553+0000 7fdc46b86740 20 client.16899 trim_cache size 0 max 16384 2024-05-20T18:07:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc46b86740 1 Processor -- start 2024-05-20T18:07:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc46b86740 1 -- start start 2024-05-20T18:07:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a748a2f0 0x5636a784e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a74924d0 0x5636a7874250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7874790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc46b86740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381aa9c0 con 0x5636a74b31b0 2024-05-20T18:07:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc46b86740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381ad220 con 0x5636a748a2f0 2024-05-20T18:07:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc46b86740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381ab010 con 0x5636a74924d0 2024-05-20T18:07:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc3e27f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7874790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.571 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc3e27f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7874790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36780/0 (socket says 172.21.15.129:36780) 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc3e27f640 1 -- 172.21.15.129:0/3947974126 learned_addr learned my addr 172.21.15.129:0/3947974126 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/3947974126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a748a2f0 0x5636a784e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc3d27d640 1 --2- 172.21.15.129:0/3947974126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a74924d0 0x5636a7874250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc3e27f640 1 -- 172.21.15.129:0/3947974126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a74924d0 msgr2=0x5636a7874250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/3947974126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a74924d0 0x5636a7874250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc3e27f640 1 -- 172.21.15.129:0/3947974126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a748a2f0 msgr2=0x5636a784e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/3947974126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a748a2f0 0x5636a784e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc3e27f640 1 -- 172.21.15.129:0/3947974126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636a785b950 con 0x5636a74b31b0 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.568+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/3947974126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a748a2f0 0x5636a784e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/3947974126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7874790 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdc30009590 tx=0x7fdc30010480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61875e13764696c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc1f7fe640 1 -- 172.21.15.129:0/3947974126 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc30019070 con 0x5636a74b31b0 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc1f7fe640 1 -- 172.21.15.129:0/3947974126 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc30013580 con 0x5636a74b31b0 2024-05-20T18:07:55.572 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc1f7fe640 1 -- 172.21.15.129:0/3947974126 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3000ec00 con 0x5636a74b31b0 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc46b86740 1 -- 172.21.15.129:0/3947974126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 msgr2=0x5636a7874790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc46b86740 1 --2- 172.21.15.129:0/3947974126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7874790 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdc30009590 tx=0x7fdc30010480 comp rx=0 tx=0).stop 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc46b86740 1 -- 172.21.15.129:0/3947974126 shutdown_connections 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc46b86740 1 --2- 172.21.15.129:0/3947974126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74b31b0 0x5636a7874790 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc46b86740 1 --2- 172.21.15.129:0/3947974126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a74924d0 0x5636a7874250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc46b86740 1 --2- 172.21.15.129:0/3947974126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a748a2f0 0x5636a784e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc46b86740 1 -- 172.21.15.129:0/3947974126 >> 172.21.15.129:0/3947974126 conn(0x5636a74a7300 msgr2=0x5636a784e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc46b86740 1 -- 172.21.15.129:0/3947974126 shutdown_connections 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.569+0000 7fdc46b86740 1 -- 172.21.15.129:0/3947974126 wait complete. 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc46b86740 1 -- start start 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc46b86740 1 Processor -- start 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a748a2f0 0x5636a78648c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a7861c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc46b86740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a74b31b0 0x5636a7862170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc46b86740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381ab6e0 con 0x5636a74924d0 2024-05-20T18:07:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc46b86740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381acb50 con 0x5636a748a2f0 2024-05-20T18:07:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc46b86740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381abdb0 con 0x5636a74b31b0 2024-05-20T18:07:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc46b86740 10 client.0 _ll_register_callbacks cb 0x7fdc2c00b820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc3d27d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a7861c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc3d27d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a7861c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36788/0 (socket says 172.21.15.129:36788) 2024-05-20T18:07:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.570+0000 7fdc3d27d640 1 -- 172.21.15.129:0/1012459993 learned_addr learned my addr 172.21.15.129:0/1012459993 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a748a2f0 0x5636a78648c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a74b31b0 0x5636a7862170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.574 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc3d27d640 1 -- 172.21.15.129:0/1012459993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a74b31b0 msgr2=0x5636a7862170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:55.575 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc3d27d640 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a74b31b0 0x5636a7862170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.575 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc3d27d640 1 -- 172.21.15.129:0/1012459993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a748a2f0 msgr2=0x5636a78648c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc3d27d640 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a748a2f0 0x5636a78648c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc3d27d640 1 -- 172.21.15.129:0/1012459993 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636a78626b0 con 0x5636a74924d0 2024-05-20T18:07:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc3e27f640 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a74b31b0 0x5636a7862170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc3d27d640 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a7861c30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c000c00 tx=0x7fdc2c010e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87923d6b05b5d889 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:55.576 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/1012459993 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c02f060 con 0x5636a74924d0 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/1012459993 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fdc2c0051d0 con 0x5636a74924d0 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc46b86740 1 -- 172.21.15.129:0/1012459993 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc30013580 con 0x5636a74924d0 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/1012459993 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdc2c0053f0 con 0x5636a74924d0 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc46b86740 10 client.16905 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc46b86740 1 -- 172.21.15.129:0/1012459993 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5636a78d69f0 con 0x5636a74924d0 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.571+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/1012459993 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc30013580 con 0x5636a74924d0 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.572+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/1012459993 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fdc2c02a570 con 0x5636a74924d0 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.572+0000 7fdc46b86740 10 client.16905 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.572+0000 7fdc46b86740 10 client.16905 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.572+0000 7fdc46b86740 20 client.16905 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:55.577 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.572+0000 7fdc46b86740 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5636a74471c0 0x5636a78d9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.572+0000 7fdc46b86740 4 client.16905 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:07:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.572+0000 7fdc46b86740 1 -- 172.21.15.129:0/1012459993 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5636a77bedc0 con 0x5636a74471c0 2024-05-20T18:07:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.572+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5636a74471c0 0x5636a78d9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.572+0000 7fdc3da7e640 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5636a74471c0 0x5636a78d9130 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7590e43520e1765 server_cookie=b9d2ba6a8c512e15 in_seq=0 out_seq=0 2024-05-20T18:07:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.572+0000 7fdc1d7fa640 10 client.16905 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc1d7fa640 1 -- 172.21.15.129:0/1012459993 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5636a77bedc0 con 0x5636a74471c0 2024-05-20T18:07:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc1d7fa640 10 client.16905 handle_command_reply: tid=0 2024-05-20T18:07:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 1 client.16905 shutdown 2024-05-20T18:07:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 1 -- 172.21.15.129:0/1012459993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 msgr2=0x5636a7861c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a7861c30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c000c00 tx=0x7fdc2c010e30 comp rx=0 tx=0).stop 2024-05-20T18:07:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 1 -- 172.21.15.129:0/1012459993 shutdown_connections 2024-05-20T18:07:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5636a74471c0 0x5636a78d9130 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5636a74b31b0 0x5636a7862170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5636a74924d0 0x5636a7861c30 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 1 --2- 172.21.15.129:0/1012459993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5636a748a2f0 0x5636a78648c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc1d7fa640 0 client.16905 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 1 -- 172.21.15.129:0/1012459993 >> 172.21.15.129:0/1012459993 conn(0x5636a74a7300 msgr2=0x5636a7855cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 1 -- 172.21.15.129:0/1012459993 shutdown_connections 2024-05-20T18:07:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 1 -- 172.21.15.129:0/1012459993 wait complete. 2024-05-20T18:07:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.577+0000 7fdc46b86740 20 client.16905 trim_cache size 0 max 16384 2024-05-20T18:07:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:07:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:07:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:07:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:07:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1909832328 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdc200a2cf0 msgr2=0x7fdc200a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdc200a2cf0 0x7fdc200a5200 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdc30021000 tx=0x7fdc3001e000 comp rx=0 tx=0).stop 2024-05-20T18:07:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1909832328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc38102190 msgr2=0x7fdc381986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc38102190 0x7fdc381986a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdc28005940 tx=0x7fdc28004270 comp rx=0 tx=0).stop 2024-05-20T18:07:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1909832328 shutdown_connections 2024-05-20T18:07:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fdc200a2cf0 0x7fdc200a5200 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc38107ed0 0x7fdc3819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdc38107a30 0x7fdc38198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 --2- 172.21.15.129:0/1909832328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc38102190 0x7fdc381986a0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1909832328 >> 172.21.15.129:0/1909832328 conn(0x7fdc380fd210 msgr2=0x7fdc380fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1909832328 shutdown_connections 2024-05-20T18:07:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:07:55.578+0000 7fdc3fce2640 1 -- 172.21.15.129:0/1909832328 wait complete. 2024-05-20T18:07:56.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:55 smithi129 ceph-mon[32029]: pgmap v289: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 3.2 KiB/s wr, 4 op/s 2024-05-20T18:07:56.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:55 smithi204 ceph-mon[37076]: pgmap v289: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 3.2 KiB/s wr, 4 op/s 2024-05-20T18:07:56.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:55 smithi154 ceph-mon[36482]: pgmap v289: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 3.2 KiB/s wr, 4 op/s 2024-05-20T18:07:57.241 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:07:57.241 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:07:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.371+0000 7f18d0707640 1 -- 172.21.15.129:0/2575821091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c8101000 msgr2=0x7f18c8101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.371+0000 7f18d0707640 1 --2- 172.21.15.129:0/2575821091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c8101000 0x7f18c8101410 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009900 tx=0x7f18bc02a410 comp rx=0 tx=0).stop 2024-05-20T18:07:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.371+0000 7f18d0707640 1 -- 172.21.15.129:0/2575821091 shutdown_connections 2024-05-20T18:07:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.371+0000 7f18d0707640 1 --2- 172.21.15.129:0/2575821091 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c810d500 0x7f18c810f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.371+0000 7f18d0707640 1 --2- 172.21.15.129:0/2575821091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c8101000 0x7f18c8101410 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.371+0000 7f18d0707640 1 --2- 172.21.15.129:0/2575821091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c8108b50 0x7f18c8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.371+0000 7f18d0707640 1 -- 172.21.15.129:0/2575821091 >> 172.21.15.129:0/2575821091 conn(0x7f18c8076830 msgr2=0x7f18c8076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.371+0000 7f18d0707640 1 -- 172.21.15.129:0/2575821091 shutdown_connections 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.371+0000 7f18d0707640 1 -- 172.21.15.129:0/2575821091 wait complete. 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18d0707640 1 Processor -- start 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18d0707640 1 -- start start 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18d0707640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c8101000 0x7f18c81997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18d0707640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c8108b50 0x7f18c8199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18d0707640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c810d500 0x7f18c819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18d0707640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f18c8112e10 con 0x7f18c810d500 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18d0707640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18c8112c90 con 0x7f18c8101000 2024-05-20T18:07:57.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18d0707640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f18c8112f90 con 0x7f18c8108b50 2024-05-20T18:07:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18ceca4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c810d500 0x7f18c819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18ceca4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c810d500 0x7f18c819e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36796/0 (socket says 172.21.15.129:36796) 2024-05-20T18:07:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18ceca4640 1 -- 172.21.15.129:0/723647077 learned_addr learned my addr 172.21.15.129:0/723647077 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18cdca2640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c8108b50 0x7f18c8199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18ce4a3640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c8101000 0x7f18c81997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18ceca4640 1 -- 172.21.15.129:0/723647077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c8108b50 msgr2=0x7f18c8199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18ceca4640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c8108b50 0x7f18c8199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18ceca4640 1 -- 172.21.15.129:0/723647077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c8101000 msgr2=0x7f18c81997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18ceca4640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c8101000 0x7f18c81997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18ceca4640 1 -- 172.21.15.129:0/723647077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c819e650 con 0x7f18c810d500 2024-05-20T18:07:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18cdca2640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c8108b50 0x7f18c8199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:57.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.372+0000 7f18ce4a3640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c8101000 0x7f18c81997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:57.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.373+0000 7f18ceca4640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c810d500 0x7f18c819e010 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f18c000d720 tx=0x7f18c000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.373+0000 7f18c77fe640 1 -- 172.21.15.129:0/723647077 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c0017070 con 0x7f18c810d500 2024-05-20T18:07:57.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.373+0000 7f18c77fe640 1 -- 172.21.15.129:0/723647077 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18c000de30 con 0x7f18c810d500 2024-05-20T18:07:57.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.373+0000 7f18d0707640 1 -- 172.21.15.129:0/723647077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c819e940 con 0x7f18c810d500 2024-05-20T18:07:57.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.373+0000 7f18c77fe640 1 -- 172.21.15.129:0/723647077 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c0004d90 con 0x7f18c810d500 2024-05-20T18:07:57.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.373+0000 7f18d0707640 1 -- 172.21.15.129:0/723647077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c81a62c0 con 0x7f18c810d500 2024-05-20T18:07:57.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.375+0000 7f18d75a8740 1 Processor -- start 2024-05-20T18:07:57.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.375+0000 7f18c77fe640 1 -- 172.21.15.129:0/723647077 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f18c0010460 con 0x7f18c810d500 2024-05-20T18:07:57.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18d75a8740 1 -- start start 2024-05-20T18:07:57.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 0x55dc32587d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc32588270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c86b0 0x55dc325887b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18d75a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dc324edd00 con 0x55dc321c04d0 2024-05-20T18:07:57.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18d75a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dc324954c0 con 0x55dc321c86b0 2024-05-20T18:07:57.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18d75a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55dc324e9cc0 con 0x55dc321e9390 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18ce4a3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc32588270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18ce4a3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc32588270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36804/0 (socket says 172.21.15.129:36804) 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18ce4a3640 1 -- 172.21.15.129:0/3169426436 learned_addr learned my addr 172.21.15.129:0/3169426436 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18cdca2640 1 --2- 172.21.15.129:0/3169426436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 0x55dc32587d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18cdca2640 1 --2- 172.21.15.129:0/3169426436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c86b0 0x55dc325887b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.376+0000 7f18c77fe640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f18b00a2d00 0x7f18b00a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.377+0000 7f18ce4a3640 1 -- 172.21.15.129:0/3169426436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 msgr2=0x55dc32587d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.377+0000 7f18ce4a3640 1 --2- 172.21.15.129:0/3169426436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 0x55dc32587d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.377+0000 7f18ceca4640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f18b00a2d00 0x7f18b00a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.377+0000 7f18ce4a3640 1 -- 172.21.15.129:0/3169426436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c86b0 msgr2=0x55dc325887b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.377+0000 7f18ce4a3640 1 --2- 172.21.15.129:0/3169426436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c86b0 0x55dc325887b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.377+0000 7f18ce4a3640 1 -- 172.21.15.129:0/3169426436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc322cb7d0 con 0x55dc321c04d0 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.377+0000 7f18c77fe640 1 -- 172.21.15.129:0/723647077 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f18c00cee50 con 0x7f18c810d500 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.377+0000 7f18ce4a3640 1 --2- 172.21.15.129:0/3169426436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc32588270 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f18b8008e40 tx=0x7f18b800d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c0ab9ab9b5cc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.377+0000 7f18c4ff9640 1 -- 172.21.15.129:0/3169426436 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b800de60 con 0x55dc321c04d0 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.377+0000 7f18c4ff9640 1 -- 172.21.15.129:0/3169426436 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b8004d20 con 0x55dc321c04d0 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.377+0000 7f18c4ff9640 1 -- 172.21.15.129:0/3169426436 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b80055c0 con 0x55dc321c04d0 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.378+0000 7f18d75a8740 1 -- 172.21.15.129:0/3169426436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 msgr2=0x55dc32588270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.378+0000 7f18d75a8740 1 --2- 172.21.15.129:0/3169426436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc32588270 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f18b8008e40 tx=0x7f18b800d710 comp rx=0 tx=0).stop 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.378+0000 7f18ceca4640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f18b00a2d00 0x7f18b00a5210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f18c009b030 tx=0x7f18c006a260 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.378+0000 7f18d75a8740 1 -- 172.21.15.129:0/3169426436 shutdown_connections 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.378+0000 7f18d75a8740 1 --2- 172.21.15.129:0/3169426436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c86b0 0x55dc325887b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.378+0000 7f18d75a8740 1 --2- 172.21.15.129:0/3169426436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc32588270 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.378+0000 7f18d75a8740 1 --2- 172.21.15.129:0/3169426436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 0x55dc32587d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.378+0000 7f18d75a8740 1 -- 172.21.15.129:0/3169426436 >> 172.21.15.129:0/3169426436 conn(0x55dc321dd4e0 msgr2=0x55dc325844e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.378+0000 7f18d75a8740 1 -- 172.21.15.129:0/3169426436 shutdown_connections 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.378+0000 7f18d75a8740 1 -- 172.21.15.129:0/3169426436 wait complete. 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.378+0000 7f18d75a8740 1 -- start start 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18d75a8740 1 Processor -- start 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c04d0 0x55dc32593e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc325943a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 0x55dc32592370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18d75a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dc32594fc0 con 0x55dc321c86b0 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18d75a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dc324a8b90 con 0x55dc321c04d0 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18d75a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55dc32595140 con 0x55dc321e9390 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18d75a8740 10 client.0 _ll_register_callbacks cb 0x55dc324e4da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:57.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18ce4a3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc325943a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18ce4a3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc325943a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36820/0 (socket says 172.21.15.129:36820) 2024-05-20T18:07:57.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18ce4a3640 1 -- 172.21.15.129:0/3894684782 learned_addr learned my addr 172.21.15.129:0/3894684782 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:57.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18cdca2640 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c04d0 0x55dc32593e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18ce4a3640 1 -- 172.21.15.129:0/3894684782 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 msgr2=0x55dc32592370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18ce4a3640 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 0x55dc32592370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18ce4a3640 1 -- 172.21.15.129:0/3894684782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c04d0 msgr2=0x55dc32593e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18ce4a3640 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c04d0 0x55dc32593e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18ce4a3640 1 -- 172.21.15.129:0/3894684782 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc322013f0 con 0x55dc321c86b0 2024-05-20T18:07:57.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18cdca2640 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c04d0 0x55dc32593e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:57.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18ce4a3640 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc325943a0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f18b800dc10 tx=0x7f18b800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ffce177667a17fa server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.379+0000 7f18adffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18adffb640 1 -- 172.21.15.129:0/3894684782 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b8027060 con 0x55dc321c86b0 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18adffb640 1 -- 172.21.15.129:0/3894684782 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f18b8004a40 con 0x55dc321c86b0 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18d75a8740 1 -- 172.21.15.129:0/3894684782 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18b8004d20 con 0x55dc321c86b0 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18adffb640 1 -- 172.21.15.129:0/3894684782 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f18b800de60 con 0x55dc321c86b0 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18d75a8740 10 client.16917 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18d75a8740 1 -- 172.21.15.129:0/3894684782 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dc321c10f0 con 0x55dc321c86b0 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18adffb640 1 -- 172.21.15.129:0/3894684782 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b8004d20 con 0x55dc321c86b0 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18adffb640 1 -- 172.21.15.129:0/3894684782 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f18b8017550 con 0x55dc321c86b0 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18d75a8740 10 client.16917 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18d75a8740 10 client.16917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18d75a8740 20 client.16917 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:57.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18d75a8740 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dc321a7f60 0x55dc325a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18d75a8740 4 client.16917 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.380+0000 7f18d75a8740 1 -- 172.21.15.129:0/3894684782 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc324fb430 con 0x55dc321a7f60 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.381+0000 7f18cdca2640 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dc321a7f60 0x55dc325a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.381+0000 7f18adffb640 10 client.16917 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.381+0000 7f18cdca2640 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dc321a7f60 0x55dc325a4b10 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c2cb730c6abc830 server_cookie=1a4877b22e334db6 in_seq=0 out_seq=0 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18adffb640 1 -- 172.21.15.129:0/3894684782 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55dc324fb430 con 0x55dc321a7f60 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18adffb640 10 client.16917 handle_command_reply: tid=0 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 1 client.16917 shutdown 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 1 -- 172.21.15.129:0/3894684782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 msgr2=0x55dc325943a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc325943a0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f18b800dc10 tx=0x7f18b800fcb0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 1 -- 172.21.15.129:0/3894684782 shutdown_connections 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dc321a7f60 0x55dc325a4b10 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 0x55dc32592370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc325943a0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18adffb640 0 client.16917 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:57.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 1 --2- 172.21.15.129:0/3894684782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c04d0 0x55dc32593e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 1 -- 172.21.15.129:0/3894684782 >> 172.21.15.129:0/3894684782 conn(0x55dc321dd4e0 msgr2=0x55dc32589a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 1 -- 172.21.15.129:0/3894684782 shutdown_connections 2024-05-20T18:07:57.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 1 -- 172.21.15.129:0/3894684782 wait complete. 2024-05-20T18:07:57.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.382+0000 7f18d75a8740 20 client.16917 trim_cache size 0 max 16384 2024-05-20T18:07:57.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18d75a8740 1 Processor -- start 2024-05-20T18:07:57.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18d75a8740 1 -- start start 2024-05-20T18:07:57.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc325912b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c86b0 0x55dc32591fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 0x55dc325a7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18d75a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81acc50 con 0x55dc321c04d0 2024-05-20T18:07:57.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18d75a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81af4b0 con 0x55dc321c86b0 2024-05-20T18:07:57.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18d75a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81ad2a0 con 0x55dc321e9390 2024-05-20T18:07:57.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18cdca2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc325912b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18cdca2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc325912b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36832/0 (socket says 172.21.15.129:36832) 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18cdca2640 1 -- 172.21.15.129:0/2226762121 learned_addr learned my addr 172.21.15.129:0/2226762121 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18cdca2640 1 -- 172.21.15.129:0/2226762121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 msgr2=0x55dc325a7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18cdca2640 1 --2- 172.21.15.129:0/2226762121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 0x55dc325a7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18cdca2640 1 -- 172.21.15.129:0/2226762121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c86b0 msgr2=0x55dc32591fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18cdca2640 1 --2- 172.21.15.129:0/2226762121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c86b0 0x55dc32591fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.400+0000 7f18cdca2640 1 -- 172.21.15.129:0/2226762121 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc32587ac0 con 0x55dc321c04d0 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.401+0000 7f18cdca2640 1 --2- 172.21.15.129:0/2226762121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc325912b0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f18bc032fd0 tx=0x7f18bc0095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b623d612fd00705 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.401+0000 7f18affff640 1 -- 172.21.15.129:0/2226762121 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc04d070 con 0x55dc321c04d0 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.401+0000 7f18affff640 1 -- 172.21.15.129:0/2226762121 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18bc005520 con 0x55dc321c04d0 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.401+0000 7f18affff640 1 -- 172.21.15.129:0/2226762121 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc035040 con 0x55dc321c04d0 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.401+0000 7f18d75a8740 1 -- 172.21.15.129:0/2226762121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 msgr2=0x55dc325912b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.401+0000 7f18d75a8740 1 --2- 172.21.15.129:0/2226762121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc325912b0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f18bc032fd0 tx=0x7f18bc0095d0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 -- 172.21.15.129:0/2226762121 shutdown_connections 2024-05-20T18:07:57.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 --2- 172.21.15.129:0/2226762121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321c86b0 0x55dc32591fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 --2- 172.21.15.129:0/2226762121 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321e9390 0x55dc325a7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 --2- 172.21.15.129:0/2226762121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c04d0 0x55dc325912b0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 -- 172.21.15.129:0/2226762121 >> 172.21.15.129:0/2226762121 conn(0x55dc321dd4e0 msgr2=0x55dc325918b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 -- 172.21.15.129:0/2226762121 shutdown_connections 2024-05-20T18:07:57.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 -- 172.21.15.129:0/2226762121 wait complete. 2024-05-20T18:07:57.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 -- start start 2024-05-20T18:07:57.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 Processor -- start 2024-05-20T18:07:57.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321c04d0 0x55dc32600bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc32601110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321e9390 0x55dc32601650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81ad970 con 0x55dc321c86b0 2024-05-20T18:07:57.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81aede0 con 0x55dc321e9390 2024-05-20T18:07:57.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81ae040 con 0x55dc321c04d0 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18d75a8740 10 client.0 _ll_register_callbacks cb 0x7f18bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18ce4a3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc32601110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18ce4a3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc32601110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36848/0 (socket says 172.21.15.129:36848) 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18ce4a3640 1 -- 172.21.15.129:0/2912564715 learned_addr learned my addr 172.21.15.129:0/2912564715 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.402+0000 7f18cdca2640 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321e9390 0x55dc32601650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18cdca2640 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321c04d0 0x55dc32600bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18ce4a3640 1 -- 172.21.15.129:0/2912564715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321c04d0 msgr2=0x55dc32600bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18ce4a3640 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321c04d0 0x55dc32600bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18ce4a3640 1 -- 172.21.15.129:0/2912564715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321e9390 msgr2=0x55dc32601650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18ce4a3640 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321e9390 0x55dc32601650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18ce4a3640 1 -- 172.21.15.129:0/2912564715 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc32588cc0 con 0x55dc321c86b0 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18cdca2640 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321c04d0 0x55dc32600bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:57.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18ce4a3640 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc32601110 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f18b8008810 tx=0x7f18b8008270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6681ad15fe6655 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18adffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:57.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18adffb640 1 -- 172.21.15.129:0/2912564715 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b801b070 con 0x55dc321c86b0 2024-05-20T18:07:57.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18adffb640 1 -- 172.21.15.129:0/2912564715 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f18b8004c10 con 0x55dc321c86b0 2024-05-20T18:07:57.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.403+0000 7f18d75a8740 1 -- 172.21.15.129:0/2912564715 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18bc005520 con 0x55dc321c86b0 2024-05-20T18:07:57.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18adffb640 1 -- 172.21.15.129:0/2912564715 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f18b803a050 con 0x55dc321c86b0 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18d75a8740 10 client.16923 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18d75a8740 1 -- 172.21.15.129:0/2912564715 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dc3260cf10 con 0x55dc321c86b0 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18adffb640 1 -- 172.21.15.129:0/2912564715 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18bc005520 con 0x55dc321c86b0 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18adffb640 1 -- 172.21.15.129:0/2912564715 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f18b8004140 con 0x55dc321c86b0 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18d75a8740 10 client.16923 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18d75a8740 10 client.16923 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18d75a8740 20 client.16923 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18d75a8740 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dc3217d3a0 0x55dc3260f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18d75a8740 4 client.16923 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18d75a8740 1 -- 172.21.15.129:0/2912564715 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dc324f5780 con 0x55dc3217d3a0 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.404+0000 7f18cdca2640 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dc3217d3a0 0x55dc3260f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18adffb640 10 client.16923 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18cdca2640 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dc3217d3a0 0x55dc3260f670 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=335e12f57ce5d374 server_cookie=fcaacb92ef8ee6cd in_seq=0 out_seq=0 2024-05-20T18:07:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18adffb640 1 -- 172.21.15.129:0/2912564715 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55dc324f5780 con 0x55dc3217d3a0 2024-05-20T18:07:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18adffb640 10 client.16923 handle_command_reply: tid=0 2024-05-20T18:07:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 1 client.16923 shutdown 2024-05-20T18:07:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 1 -- 172.21.15.129:0/2912564715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 msgr2=0x55dc32601110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc32601110 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f18b8008810 tx=0x7f18b8008270 comp rx=0 tx=0).stop 2024-05-20T18:07:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 1 -- 172.21.15.129:0/2912564715 shutdown_connections 2024-05-20T18:07:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dc3217d3a0 0x55dc3260f670 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dc321e9390 0x55dc32601650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dc321c86b0 0x55dc32601110 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18adffb640 0 client.16923 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 1 --2- 172.21.15.129:0/2912564715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dc321c04d0 0x55dc32600bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 1 -- 172.21.15.129:0/2912564715 >> 172.21.15.129:0/2912564715 conn(0x55dc321dd4e0 msgr2=0x55dc3258a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 1 -- 172.21.15.129:0/2912564715 shutdown_connections 2024-05-20T18:07:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 1 -- 172.21.15.129:0/2912564715 wait complete. 2024-05-20T18:07:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.405+0000 7f18d75a8740 20 client.16923 trim_cache size 0 max 16384 2024-05-20T18:07:57.412 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:07:57.413 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:07:57.413 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:07:57.413 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:07:57.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 -- 172.21.15.129:0/723647077 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f18b00a2d00 msgr2=0x7f18b00a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f18b00a2d00 0x7f18b00a5210 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f18c009b030 tx=0x7f18c006a260 comp rx=0 tx=0).stop 2024-05-20T18:07:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 -- 172.21.15.129:0/723647077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c810d500 msgr2=0x7f18c819e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c810d500 0x7f18c819e010 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f18c000d720 tx=0x7f18c000dbe0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 -- 172.21.15.129:0/723647077 shutdown_connections 2024-05-20T18:07:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f18b00a2d00 0x7f18b00a5210 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18c810d500 0x7f18c819e010 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c8108b50 0x7f18c8199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 --2- 172.21.15.129:0/723647077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c8101000 0x7f18c81997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 -- 172.21.15.129:0/723647077 >> 172.21.15.129:0/723647077 conn(0x7f18c8076830 msgr2=0x7f18c81094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 -- 172.21.15.129:0/723647077 shutdown_connections 2024-05-20T18:07:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.406+0000 7f18d0707640 1 -- 172.21.15.129:0/723647077 wait complete. 2024-05-20T18:07:57.415 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:07:57.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:07:57.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.528+0000 7fe511251640 1 -- 172.21.15.129:0/149831698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe50c101b30 msgr2=0x7fe50c101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.528+0000 7fe511251640 1 --2- 172.21.15.129:0/149831698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe50c101b30 0x7fe50c101f40 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4009900 tx=0x7fe4f402a410 comp rx=0 tx=0).stop 2024-05-20T18:07:57.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.528+0000 7fe511251640 1 -- 172.21.15.129:0/149831698 shutdown_connections 2024-05-20T18:07:57.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.528+0000 7fe511251640 1 --2- 172.21.15.129:0/149831698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe50c10c370 0x7fe50c10e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.528+0000 7fe511251640 1 --2- 172.21.15.129:0/149831698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe50c101b30 0x7fe50c101f40 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.528+0000 7fe511251640 1 --2- 172.21.15.129:0/149831698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50c107a10 0x7fe50c1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.528+0000 7fe511251640 1 -- 172.21.15.129:0/149831698 >> 172.21.15.129:0/149831698 conn(0x7fe50c077130 msgr2=0x7fe50c077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.529+0000 7fe511251640 1 -- 172.21.15.129:0/149831698 shutdown_connections 2024-05-20T18:07:57.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.529+0000 7fe511251640 1 -- 172.21.15.129:0/149831698 wait complete. 2024-05-20T18:07:57.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.529+0000 7fe511251640 1 Processor -- start 2024-05-20T18:07:57.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe511251640 1 -- start start 2024-05-20T18:07:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe511251640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe50c101b30 0x7fe50c19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe511251640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe50c107a10 0x7fe50c19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe511251640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50c10c370 0x7fe50c19b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe511251640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c111d20 con 0x7fe50c101b30 2024-05-20T18:07:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe511251640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c111ba0 con 0x7fe50c10c370 2024-05-20T18:07:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe511251640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c111ea0 con 0x7fe50c107a10 2024-05-20T18:07:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe50c101b30 0x7fe50c19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe50c101b30 0x7fe50c19a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36864/0 (socket says 172.21.15.129:36864) 2024-05-20T18:07:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50ad9d640 1 -- 172.21.15.129:0/4050724734 learned_addr learned my addr 172.21.15.129:0/4050724734 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50a59c640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe50c107a10 0x7fe50c19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50b59e640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50c10c370 0x7fe50c19b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50ad9d640 1 -- 172.21.15.129:0/4050724734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe50c107a10 msgr2=0x7fe50c19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50ad9d640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe50c107a10 0x7fe50c19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50ad9d640 1 -- 172.21.15.129:0/4050724734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50c10c370 msgr2=0x7fe50c19b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50ad9d640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50c10c370 0x7fe50c19b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50ad9d640 1 -- 172.21.15.129:0/4050724734 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50c1a5900 con 0x7fe50c101b30 2024-05-20T18:07:57.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50b59e640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50c10c370 0x7fe50c19b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:57.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.530+0000 7fe50ad9d640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe50c101b30 0x7fe50c19a5c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c970 tx=0x7fe4fc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.531+0000 7fe503fff640 1 -- 172.21.15.129:0/4050724734 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc007bb0 con 0x7fe50c101b30 2024-05-20T18:07:57.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.531+0000 7fe503fff640 1 -- 172.21.15.129:0/4050724734 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4fc007d50 con 0x7fe50c101b30 2024-05-20T18:07:57.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.531+0000 7fe511251640 1 -- 172.21.15.129:0/4050724734 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe50c1a5bf0 con 0x7fe50c101b30 2024-05-20T18:07:57.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.531+0000 7fe503fff640 1 -- 172.21.15.129:0/4050724734 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc00f450 con 0x7fe50c101b30 2024-05-20T18:07:57.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.531+0000 7fe511251640 1 -- 172.21.15.129:0/4050724734 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe50c1a6110 con 0x7fe50c101b30 2024-05-20T18:07:57.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.532+0000 7fe511251640 1 -- 172.21.15.129:0/4050724734 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe50c069140 con 0x7fe50c101b30 2024-05-20T18:07:57.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.533+0000 7fe503fff640 1 -- 172.21.15.129:0/4050724734 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fe4fc0046c0 con 0x7fe50c101b30 2024-05-20T18:07:57.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.534+0000 7fe503fff640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4f80a2d00 0x7fe4f80a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.534+0000 7fe50a59c640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4f80a2d00 0x7fe4f80a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.534+0000 7fe503fff640 1 -- 172.21.15.129:0/4050724734 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fe4fc0cf4c0 con 0x7fe50c101b30 2024-05-20T18:07:57.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.534+0000 7fe50a59c640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4f80a2d00 0x7fe4f80a5210 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4005940 tx=0x7fe4f40058d0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.539+0000 7fe503fff640 1 -- 172.21.15.129:0/4050724734 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe4fc014030 con 0x7fe50c101b30 2024-05-20T18:07:57.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.753+0000 7fe511251640 1 -- 172.21.15.129:0/4050724734 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe50c074390 con 0x7fe50c101b30 2024-05-20T18:07:57.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.753+0000 7fe503fff640 1 -- 172.21.15.129:0/4050724734 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11829 (secure 0 0 0) 0x7fe4fc09c0a0 con 0x7fe50c101b30 2024-05-20T18:07:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:07:57.751 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":26255,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/2724397229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":2724397229},{"type":"v1","addr":"172.21.15.154:6837","nonce":2724397229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:06:10.247052+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:07:57.752 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-05-20T18:07:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.755+0000 7fe511251640 1 -- 172.21.15.129:0/4050724734 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4f80a2d00 msgr2=0x7fe4f80a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.755+0000 7fe511251640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4f80a2d00 0x7fe4f80a5210 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4005940 tx=0x7fe4f40058d0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.755+0000 7fe511251640 1 -- 172.21.15.129:0/4050724734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe50c101b30 msgr2=0x7fe50c19a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.755+0000 7fe511251640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe50c101b30 0x7fe50c19a5c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c970 tx=0x7fe4fc00ce30 comp rx=0 tx=0).stop 2024-05-20T18:07:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.755+0000 7fe511251640 1 -- 172.21.15.129:0/4050724734 shutdown_connections 2024-05-20T18:07:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.755+0000 7fe511251640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe4f80a2d00 0x7fe4f80a5210 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.755+0000 7fe511251640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50c10c370 0x7fe50c19b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.755+0000 7fe511251640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe50c107a10 0x7fe50c19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.755+0000 7fe511251640 1 --2- 172.21.15.129:0/4050724734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe50c101b30 0x7fe50c19a5c0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.755+0000 7fe511251640 1 -- 172.21.15.129:0/4050724734 >> 172.21.15.129:0/4050724734 conn(0x7fe50c077130 msgr2=0x7fe50c1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.756+0000 7fe511251640 1 -- 172.21.15.129:0/4050724734 shutdown_connections 2024-05-20T18:07:57.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.756+0000 7fe511251640 1 -- 172.21.15.129:0/4050724734 wait complete. 2024-05-20T18:07:57.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:07:57.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.882+0000 7f99ce85a640 1 -- 172.21.15.129:0/2640018519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c8108f60 msgr2=0x7f99c8102ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.882+0000 7f99ce85a640 1 --2- 172.21.15.129:0/2640018519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c8108f60 0x7f99c8102ba0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009900 tx=0x7f99bc02a410 comp rx=0 tx=0).stop 2024-05-20T18:07:57.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.882+0000 7f99ce85a640 1 -- 172.21.15.129:0/2640018519 shutdown_connections 2024-05-20T18:07:57.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.882+0000 7f99ce85a640 1 --2- 172.21.15.129:0/2640018519 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99c8103210 0x7f99c810f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.882+0000 7f99ce85a640 1 --2- 172.21.15.129:0/2640018519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c8108f60 0x7f99c8102ba0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.882+0000 7f99ce85a640 1 --2- 172.21.15.129:0/2640018519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99c8108ac0 0x7f99c8102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.882+0000 7f99ce85a640 1 -- 172.21.15.129:0/2640018519 >> 172.21.15.129:0/2640018519 conn(0x7f99c80fe1c0 msgr2=0x7f99c8100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.882+0000 7f99ce85a640 1 -- 172.21.15.129:0/2640018519 shutdown_connections 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.882+0000 7f99ce85a640 1 -- 172.21.15.129:0/2640018519 wait complete. 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.882+0000 7f99ce85a640 1 Processor -- start 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.882+0000 7f99ce85a640 1 -- start start 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99ce85a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c8103210 0x7f99c819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99ce85a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99c8108ac0 0x7f99c819fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99ce85a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99c8108f60 0x7f99c8199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99ce85a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99c8112e00 con 0x7f99c8103210 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99ce85a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99c8112c80 con 0x7f99c8108ac0 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99ce85a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f99c8112f80 con 0x7f99c8108f60 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99c7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c8103210 0x7f99c819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99c7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c8103210 0x7f99c819f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36880/0 (socket says 172.21.15.129:36880) 2024-05-20T18:07:57.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99c7fff640 1 -- 172.21.15.129:0/2910325075 learned_addr learned my addr 172.21.15.129:0/2910325075 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:57.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99c77fe640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99c8108ac0 0x7f99c819fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99c8108f60 0x7f99c8199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99c7fff640 1 -- 172.21.15.129:0/2910325075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99c8108f60 msgr2=0x7f99c8199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99c7fff640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99c8108f60 0x7f99c8199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99c7fff640 1 -- 172.21.15.129:0/2910325075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99c8108ac0 msgr2=0x7f99c819fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99c7fff640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99c8108ac0 0x7f99c819fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99c7fff640 1 -- 172.21.15.129:0/2910325075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c8199e30 con 0x7f99c8103210 2024-05-20T18:07:57.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99c8108f60 0x7f99c8199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:57.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99c77fe640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99c8108ac0 0x7f99c819fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:57.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.883+0000 7f99c7fff640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c8103210 0x7f99c819f580 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f99b8005fd0 tx=0x7f99b800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.884+0000 7f99c57fa640 1 -- 172.21.15.129:0/2910325075 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8017070 con 0x7f99c8103210 2024-05-20T18:07:57.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.884+0000 7f99c57fa640 1 -- 172.21.15.129:0/2910325075 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b8009c30 con 0x7f99c8103210 2024-05-20T18:07:57.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.884+0000 7f99ce85a640 1 -- 172.21.15.129:0/2910325075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c819a0c0 con 0x7f99c8103210 2024-05-20T18:07:57.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.884+0000 7f99c57fa640 1 -- 172.21.15.129:0/2910325075 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8010460 con 0x7f99c8103210 2024-05-20T18:07:57.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.884+0000 7f99ce85a640 1 -- 172.21.15.129:0/2910325075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c81a6320 con 0x7f99c8103210 2024-05-20T18:07:57.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.887+0000 7f99c57fa640 1 -- 172.21.15.129:0/2910325075 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f99b8003e70 con 0x7f99c8103210 2024-05-20T18:07:57.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.887+0000 7f99d57dd740 1 Processor -- start 2024-05-20T18:07:57.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.887+0000 7f99d57dd740 1 -- start start 2024-05-20T18:07:57.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f926fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f9270110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9270650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99d57dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5591f91d5c40 con 0x5591f8eb06b0 2024-05-20T18:07:57.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99d57dd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5591f917d580 con 0x5591f8ed1390 2024-05-20T18:07:57.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99d57dd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5591f91d1c00 con 0x5591f8ea84d0 2024-05-20T18:07:57.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c57fa640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f99b00a2d80 0x7f99b00a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9270650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9270650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36892/0 (socket says 172.21.15.129:36892) 2024-05-20T18:07:57.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c77fe640 1 -- 172.21.15.129:0/4018129242 learned_addr learned my addr 172.21.15.129:0/4018129242 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:57.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f99b00a2d80 0x7f99b00a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/4018129242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f926fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c7fff640 1 --2- 172.21.15.129:0/4018129242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f9270110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c57fa640 1 -- 172.21.15.129:0/2910325075 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f99b80cf0e0 con 0x7f99c8103210 2024-05-20T18:07:57.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c77fe640 1 -- 172.21.15.129:0/4018129242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 msgr2=0x5591f9270110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c77fe640 1 --2- 172.21.15.129:0/4018129242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f9270110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c77fe640 1 -- 172.21.15.129:0/4018129242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 msgr2=0x5591f926fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c77fe640 1 --2- 172.21.15.129:0/4018129242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f926fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c77fe640 1 -- 172.21.15.129:0/4018129242 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591f8ee93f0 con 0x5591f8eb06b0 2024-05-20T18:07:57.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99c77fe640 1 --2- 172.21.15.129:0/4018129242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9270650 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009900 tx=0x7f99bc032cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa4a01ef1889021 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f99b00a2d80 0x7f99b00a5290 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009fd0 tx=0x7f99c0009440 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99ae7fc640 1 -- 172.21.15.129:0/4018129242 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc03f070 con 0x5591f8eb06b0 2024-05-20T18:07:57.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.888+0000 7f99ae7fc640 1 -- 172.21.15.129:0/4018129242 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99bc0317b0 con 0x5591f8eb06b0 2024-05-20T18:07:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.889+0000 7f99ae7fc640 1 -- 172.21.15.129:0/4018129242 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc035070 con 0x5591f8eb06b0 2024-05-20T18:07:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.889+0000 7f99d57dd740 1 -- 172.21.15.129:0/4018129242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 msgr2=0x5591f9270650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.889+0000 7f99d57dd740 1 --2- 172.21.15.129:0/4018129242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9270650 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009900 tx=0x7f99bc032cb0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.889+0000 7f99d57dd740 1 -- 172.21.15.129:0/4018129242 shutdown_connections 2024-05-20T18:07:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.889+0000 7f99d57dd740 1 --2- 172.21.15.129:0/4018129242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9270650 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.889+0000 7f99d57dd740 1 --2- 172.21.15.129:0/4018129242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f9270110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.889+0000 7f99d57dd740 1 --2- 172.21.15.129:0/4018129242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f926fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.889+0000 7f99d57dd740 1 -- 172.21.15.129:0/4018129242 >> 172.21.15.129:0/4018129242 conn(0x5591f8ec54e0 msgr2=0x5591f926c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.889+0000 7f99d57dd740 1 -- 172.21.15.129:0/4018129242 shutdown_connections 2024-05-20T18:07:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.889+0000 7f99d57dd740 1 -- 172.21.15.129:0/4018129242 wait complete. 2024-05-20T18:07:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99d57dd740 1 -- start start 2024-05-20T18:07:57.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99d57dd740 1 Processor -- start 2024-05-20T18:07:57.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f927e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8eb06b0 0x5591f927eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ed1390 0x5591f92848a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99d57dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5591f9281420 con 0x5591f8ed1390 2024-05-20T18:07:57.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99d57dd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5591f9190c40 con 0x5591f8eb06b0 2024-05-20T18:07:57.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99d57dd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5591f92815a0 con 0x5591f8ea84d0 2024-05-20T18:07:57.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99d57dd740 10 client.0 _ll_register_callbacks cb 0x5591f8ee8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:57.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99ccdf7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ed1390 0x5591f92848a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99ccdf7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ed1390 0x5591f92848a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36900/0 (socket says 172.21.15.129:36900) 2024-05-20T18:07:57.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99ccdf7640 1 -- 172.21.15.129:0/978030143 learned_addr learned my addr 172.21.15.129:0/978030143 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:57.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.890+0000 7f99c77fe640 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8eb06b0 0x5591f927eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99c7fff640 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f927e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99ccdf7640 1 -- 172.21.15.129:0/978030143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 msgr2=0x5591f927e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f927e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99ccdf7640 1 -- 172.21.15.129:0/978030143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8eb06b0 msgr2=0x5591f927eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8eb06b0 0x5591f927eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99ccdf7640 1 -- 172.21.15.129:0/978030143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591f8fb37d0 con 0x5591f8ed1390 2024-05-20T18:07:57.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99c77fe640 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8eb06b0 0x5591f927eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:57.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ed1390 0x5591f92848a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f99c0010a90 tx=0x7f99c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5428698b045991 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:57.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99a77fe640 1 -- 172.21.15.129:0/978030143 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0024030 con 0x5591f8ed1390 2024-05-20T18:07:57.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99a77fe640 1 -- 172.21.15.129:0/978030143 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f99c0011960 con 0x5591f8ed1390 2024-05-20T18:07:57.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99d57dd740 1 -- 172.21.15.129:0/978030143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99bc0317b0 con 0x5591f8ed1390 2024-05-20T18:07:57.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99a77fe640 1 -- 172.21.15.129:0/978030143 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f99c001a040 con 0x5591f8ed1390 2024-05-20T18:07:57.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99d57dd740 10 client.16941 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:57.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99d57dd740 1 -- 172.21.15.129:0/978030143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5591f928db90 con 0x5591f8ed1390 2024-05-20T18:07:57.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.891+0000 7f99a77fe640 1 -- 172.21.15.129:0/978030143 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99bc0317b0 con 0x5591f8ed1390 2024-05-20T18:07:57.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.892+0000 7f99a77fe640 1 -- 172.21.15.129:0/978030143 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f99c0029c90 con 0x5591f8ed1390 2024-05-20T18:07:57.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.892+0000 7f99d57dd740 10 client.16941 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:57.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.892+0000 7f99d57dd740 10 client.16941 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:57.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.892+0000 7f99d57dd740 20 client.16941 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:57.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.892+0000 7f99d57dd740 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591f8e8ff60 0x5591f9291160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.892+0000 7f99d57dd740 4 client.16941 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:57.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.892+0000 7f99d57dd740 1 -- 172.21.15.129:0/978030143 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591f91e3370 con 0x5591f8e8ff60 2024-05-20T18:07:57.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.892+0000 7f99c7fff640 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591f8e8ff60 0x5591f9291160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99a77fe640 10 client.16941 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:57.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99c7fff640 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591f8e8ff60 0x5591f9291160 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f31f632f56d7c5c0 server_cookie=e67f3452388ed445 in_seq=0 out_seq=0 2024-05-20T18:07:57.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99a77fe640 1 -- 172.21.15.129:0/978030143 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5591f91e3370 con 0x5591f8e8ff60 2024-05-20T18:07:57.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99a77fe640 10 client.16941 handle_command_reply: tid=0 2024-05-20T18:07:57.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99d57dd740 1 client.16941 shutdown 2024-05-20T18:07:57.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99d57dd740 1 -- 172.21.15.129:0/978030143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ed1390 msgr2=0x5591f92848a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99d57dd740 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ed1390 0x5591f92848a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f99c0010a90 tx=0x7f99c0004a40 comp rx=0 tx=0).stop 2024-05-20T18:07:57.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99d57dd740 1 -- 172.21.15.129:0/978030143 shutdown_connections 2024-05-20T18:07:57.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99d57dd740 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591f8e8ff60 0x5591f9291160 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99d57dd740 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8eb06b0 0x5591f927eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99d57dd740 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ed1390 0x5591f92848a0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99a77fe640 0 client.16941 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:57.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.893+0000 7f99d57dd740 1 --2- 172.21.15.129:0/978030143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f927e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.894+0000 7f99d57dd740 1 -- 172.21.15.129:0/978030143 >> 172.21.15.129:0/978030143 conn(0x5591f8ec54e0 msgr2=0x5591f92719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.894+0000 7f99d57dd740 1 -- 172.21.15.129:0/978030143 shutdown_connections 2024-05-20T18:07:57.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.894+0000 7f99d57dd740 1 -- 172.21.15.129:0/978030143 wait complete. 2024-05-20T18:07:57.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.894+0000 7f99d57dd740 20 client.16941 trim_cache size 0 max 16384 2024-05-20T18:07:57.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.907+0000 7f99d57dd740 1 Processor -- start 2024-05-20T18:07:57.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.907+0000 7f99d57dd740 1 -- start start 2024-05-20T18:07:57.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.907+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ea84d0 0x5591f927d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.907+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8eb06b0 0x5591f9284810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.907+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f92701c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.907+0000 7f99d57dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99c81aa900 con 0x5591f8ea84d0 2024-05-20T18:07:57.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.907+0000 7f99d57dd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99c81ad160 con 0x5591f8ed1390 2024-05-20T18:07:57.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.907+0000 7f99d57dd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f99c81aaf50 con 0x5591f8eb06b0 2024-05-20T18:07:57.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.907+0000 7f99c7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ea84d0 0x5591f927d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.907+0000 7f99c7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ea84d0 0x5591f927d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36904/0 (socket says 172.21.15.129:36904) 2024-05-20T18:07:57.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.907+0000 7f99c7fff640 1 -- 172.21.15.129:0/1636202591 learned_addr learned my addr 172.21.15.129:0/1636202591 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:57.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99c77fe640 1 --2- 172.21.15.129:0/1636202591 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8eb06b0 0x5591f9284810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/1636202591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f92701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99c7fff640 1 -- 172.21.15.129:0/1636202591 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8eb06b0 msgr2=0x5591f9284810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99c7fff640 1 --2- 172.21.15.129:0/1636202591 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8eb06b0 0x5591f9284810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99c7fff640 1 -- 172.21.15.129:0/1636202591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 msgr2=0x5591f92701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99c7fff640 1 --2- 172.21.15.129:0/1636202591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f92701c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99c7fff640 1 -- 172.21.15.129:0/1636202591 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591f9272960 con 0x5591f8ea84d0 2024-05-20T18:07:57.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/1636202591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f92701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:57.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99c77fe640 1 --2- 172.21.15.129:0/1636202591 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8eb06b0 0x5591f9284810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:57.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99c7fff640 1 --2- 172.21.15.129:0/1636202591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ea84d0 0x5591f927d850 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f99b80cc8a0 tx=0x7f99b8016b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5166b60716db244 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99adffb640 1 -- 172.21.15.129:0/1636202591 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b80b3070 con 0x5591f8ea84d0 2024-05-20T18:07:57.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99adffb640 1 -- 172.21.15.129:0/1636202591 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b8009de0 con 0x5591f8ea84d0 2024-05-20T18:07:57.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.908+0000 7f99adffb640 1 -- 172.21.15.129:0/1636202591 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b80cca70 con 0x5591f8ea84d0 2024-05-20T18:07:57.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.909+0000 7f99d57dd740 1 -- 172.21.15.129:0/1636202591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ea84d0 msgr2=0x5591f927d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.909+0000 7f99d57dd740 1 --2- 172.21.15.129:0/1636202591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ea84d0 0x5591f927d850 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f99b80cc8a0 tx=0x7f99b8016b40 comp rx=0 tx=0).stop 2024-05-20T18:07:57.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.909+0000 7f99d57dd740 1 -- 172.21.15.129:0/1636202591 shutdown_connections 2024-05-20T18:07:57.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.909+0000 7f99d57dd740 1 --2- 172.21.15.129:0/1636202591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f92701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.909+0000 7f99d57dd740 1 --2- 172.21.15.129:0/1636202591 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8eb06b0 0x5591f9284810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.909+0000 7f99d57dd740 1 --2- 172.21.15.129:0/1636202591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8ea84d0 0x5591f927d850 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.909+0000 7f99d57dd740 1 -- 172.21.15.129:0/1636202591 >> 172.21.15.129:0/1636202591 conn(0x5591f8ec54e0 msgr2=0x5591f926fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.909+0000 7f99d57dd740 1 -- 172.21.15.129:0/1636202591 shutdown_connections 2024-05-20T18:07:57.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.909+0000 7f99d57dd740 1 -- 172.21.15.129:0/1636202591 wait complete. 2024-05-20T18:07:57.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.909+0000 7f99d57dd740 1 -- start start 2024-05-20T18:07:57.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.909+0000 7f99d57dd740 1 Processor -- start 2024-05-20T18:07:57.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f9285e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9286370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99d57dd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f92868b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99d57dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99c81ab620 con 0x5591f8eb06b0 2024-05-20T18:07:57.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99d57dd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99c81aca90 con 0x5591f8ed1390 2024-05-20T18:07:57.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99d57dd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f99c81abcf0 con 0x5591f8ea84d0 2024-05-20T18:07:57.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99d57dd740 10 client.0 _ll_register_callbacks cb 0x7f99c00134e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:57.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99c77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9286370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99c77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9286370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36906/0 (socket says 172.21.15.129:36906) 2024-05-20T18:07:57.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99c77fe640 1 -- 172.21.15.129:0/2418139571 learned_addr learned my addr 172.21.15.129:0/2418139571 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:57.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f92868b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99c7fff640 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f9285e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99c77fe640 1 -- 172.21.15.129:0/2418139571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 msgr2=0x5591f9285e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99c77fe640 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f9285e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99c77fe640 1 -- 172.21.15.129:0/2418139571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 msgr2=0x5591f92868b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:57.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99c77fe640 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f92868b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99c77fe640 1 -- 172.21.15.129:0/2418139571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591f92f7570 con 0x5591f8eb06b0 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99ccdf7640 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f92868b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99c77fe640 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9286370 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009900 tx=0x7f99bc004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cfeaea8750a3d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99a77fe640 1 -- 172.21.15.129:0/2418139571 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc035030 con 0x5591f8eb06b0 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99a77fe640 1 -- 172.21.15.129:0/2418139571 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f99bc005240 con 0x5591f8eb06b0 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.910+0000 7f99d57dd740 1 -- 172.21.15.129:0/2418139571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99b8009de0 con 0x5591f8eb06b0 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.911+0000 7f99a77fe640 1 -- 172.21.15.129:0/2418139571 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f99bc055050 con 0x5591f8eb06b0 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.911+0000 7f99d57dd740 10 client.16947 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.911+0000 7f99d57dd740 1 -- 172.21.15.129:0/2418139571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5591f92f8760 con 0x5591f8eb06b0 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.911+0000 7f99a77fe640 1 -- 172.21.15.129:0/2418139571 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b8009de0 con 0x5591f8eb06b0 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.911+0000 7f99a77fe640 1 -- 172.21.15.129:0/2418139571 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f99bc0035f0 con 0x5591f8eb06b0 2024-05-20T18:07:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.911+0000 7f99d57dd740 10 client.16947 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:57.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.911+0000 7f99d57dd740 10 client.16947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:57.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.911+0000 7f99d57dd740 20 client.16947 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:57.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.911+0000 7f99d57dd740 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591f8e653a0 0x5591f92faa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:57.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.911+0000 7f99d57dd740 4 client.16947 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:07:57.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.911+0000 7f99d57dd740 1 -- 172.21.15.129:0/2418139571 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5591f91dd6c0 con 0x5591f8e653a0 2024-05-20T18:07:57.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.912+0000 7f99c7fff640 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591f8e653a0 0x5591f92faa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:57.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.912+0000 7f99a77fe640 10 client.16947 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:57.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.912+0000 7f99c7fff640 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591f8e653a0 0x5591f92faa40 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6554ea77bebced9 server_cookie=c1b4d08684fc7190 in_seq=0 out_seq=0 2024-05-20T18:07:57.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.912+0000 7f99a77fe640 1 -- 172.21.15.129:0/2418139571 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5591f91dd6c0 con 0x5591f8e653a0 2024-05-20T18:07:57.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.912+0000 7f99a77fe640 10 client.16947 handle_command_reply: tid=0 2024-05-20T18:07:57.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.912+0000 7f99d57dd740 1 client.16947 shutdown 2024-05-20T18:07:57.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.912+0000 7f99d57dd740 1 -- 172.21.15.129:0/2418139571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 msgr2=0x5591f9286370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.912+0000 7f99d57dd740 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9286370 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009900 tx=0x7f99bc004250 comp rx=0 tx=0).stop 2024-05-20T18:07:57.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99d57dd740 1 -- 172.21.15.129:0/2418139571 shutdown_connections 2024-05-20T18:07:57.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99d57dd740 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5591f8e653a0 0x5591f92faa40 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99d57dd740 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5591f8ed1390 0x5591f92868b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99d57dd740 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591f8eb06b0 0x5591f9286370 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99a77fe640 0 client.16947 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:57.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99d57dd740 1 --2- 172.21.15.129:0/2418139571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5591f8ea84d0 0x5591f9285e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99d57dd740 1 -- 172.21.15.129:0/2418139571 >> 172.21.15.129:0/2418139571 conn(0x5591f8ec54e0 msgr2=0x5591f927d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99d57dd740 1 -- 172.21.15.129:0/2418139571 shutdown_connections 2024-05-20T18:07:57.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99d57dd740 1 -- 172.21.15.129:0/2418139571 wait complete. 2024-05-20T18:07:57.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99d57dd740 20 client.16947 trim_cache size 0 max 16384 2024-05-20T18:07:57.920 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:07:57.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 -- 172.21.15.129:0/2910325075 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f99b00a2d80 msgr2=0x7f99b00a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f99b00a2d80 0x7f99b00a5290 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009fd0 tx=0x7f99c0009440 comp rx=0 tx=0).stop 2024-05-20T18:07:57.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 -- 172.21.15.129:0/2910325075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c8103210 msgr2=0x7f99c819f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:57.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c8103210 0x7f99c819f580 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f99b8005fd0 tx=0x7f99b800dde0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 -- 172.21.15.129:0/2910325075 shutdown_connections 2024-05-20T18:07:57.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f99b00a2d80 0x7f99b00a5290 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99c8108ac0 0x7f99c819fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f99c8108f60 0x7f99c8199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 --2- 172.21.15.129:0/2910325075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c8103210 0x7f99c819f580 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:57.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 -- 172.21.15.129:0/2910325075 >> 172.21.15.129:0/2910325075 conn(0x7f99c80fe1c0 msgr2=0x7f99c810e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:57.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 -- 172.21.15.129:0/2910325075 shutdown_connections 2024-05-20T18:07:57.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:57.913+0000 7f99ce85a640 1 -- 172.21.15.129:0/2910325075 wait complete. 2024-05-20T18:07:58.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:57 smithi129 ceph-mon[32029]: pgmap v290: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-05-20T18:07:58.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:57 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4050724734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:07:58.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:57 smithi204 ceph-mon[37076]: pgmap v290: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-05-20T18:07:58.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:57 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/4050724734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:07:58.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:57 smithi154 ceph-mon[36482]: pgmap v290: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-05-20T18:07:58.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:57 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/4050724734' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:07:58.923 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:07:58.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:07:58.924 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:07:59.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.048+0000 7f950f473640 1 -- 172.21.15.129:0/3221515123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f950810a690 msgr2=0x7f950810cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.048+0000 7f950f473640 1 --2- 172.21.15.129:0/3221515123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f950810a690 0x7f950810cab0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009900 tx=0x7f94f802a480 comp rx=0 tx=0).stop 2024-05-20T18:07:59.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.048+0000 7f950f473640 1 -- 172.21.15.129:0/3221515123 shutdown_connections 2024-05-20T18:07:59.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.048+0000 7f950f473640 1 --2- 172.21.15.129:0/3221515123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f950810d040 0x7f950810f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.048+0000 7f950f473640 1 --2- 172.21.15.129:0/3221515123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f950810a690 0x7f950810cab0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.048+0000 7f950f473640 1 --2- 172.21.15.129:0/3221515123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9508103c30 0x7f9508104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.048+0000 7f950f473640 1 -- 172.21.15.129:0/3221515123 >> 172.21.15.129:0/3221515123 conn(0x7f9508077130 msgr2=0x7f9508077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.049+0000 7f950f473640 1 -- 172.21.15.129:0/3221515123 shutdown_connections 2024-05-20T18:07:59.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.049+0000 7f950f473640 1 -- 172.21.15.129:0/3221515123 wait complete. 2024-05-20T18:07:59.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.049+0000 7f950f473640 1 Processor -- start 2024-05-20T18:07:59.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.049+0000 7f950f473640 1 -- start start 2024-05-20T18:07:59.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.049+0000 7f950f473640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9508103c30 0x7f950819f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.049+0000 7f950f473640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f950810a690 0x7f950819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.049+0000 7f950f473640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f950810d040 0x7f95081996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.049+0000 7f950f473640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112e30 con 0x7f9508103c30 2024-05-20T18:07:59.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.049+0000 7f950f473640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112cb0 con 0x7f950810d040 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.049+0000 7f950f473640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112fb0 con 0x7f950810a690 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950d20f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9508103c30 0x7f950819f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950d20f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9508103c30 0x7f950819f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36920/0 (socket says 172.21.15.129:36920) 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950d20f640 1 -- 172.21.15.129:0/2221705311 learned_addr learned my addr 172.21.15.129:0/2221705311 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950ca0e640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f950810a690 0x7f950819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950da10640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f950810d040 0x7f95081996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950d20f640 1 -- 172.21.15.129:0/2221705311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f950810a690 msgr2=0x7f950819fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950d20f640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f950810a690 0x7f950819fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950d20f640 1 -- 172.21.15.129:0/2221705311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f950810d040 msgr2=0x7f95081996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950d20f640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f950810d040 0x7f95081996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950d20f640 1 -- 172.21.15.129:0/2221705311 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9508199e20 con 0x7f9508103c30 2024-05-20T18:07:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950da10640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f950810d040 0x7f95081996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:59.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950d20f640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9508103c30 0x7f950819f570 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f950400d950 tx=0x7f950400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f94fe7fc640 1 -- 172.21.15.129:0/2221705311 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504017070 con 0x7f9508103c30 2024-05-20T18:07:59.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f94fe7fc640 1 -- 172.21.15.129:0/2221705311 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9504009c30 con 0x7f9508103c30 2024-05-20T18:07:59.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950f473640 1 -- 172.21.15.129:0/2221705311 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950819a0b0 con 0x7f9508103c30 2024-05-20T18:07:59.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f94fe7fc640 1 -- 172.21.15.129:0/2221705311 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950400f6b0 con 0x7f9508103c30 2024-05-20T18:07:59.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.050+0000 7f950f473640 1 -- 172.21.15.129:0/2221705311 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95081a6310 con 0x7f9508103c30 2024-05-20T18:07:59.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.053+0000 7f94fe7fc640 1 -- 172.21.15.129:0/2221705311 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f95040109d0 con 0x7f9508103c30 2024-05-20T18:07:59.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.053+0000 7f95163fb740 1 Processor -- start 2024-05-20T18:07:59.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.053+0000 7f95163fb740 1 -- start start 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f95163fb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74d1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f95163fb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7109550 0x5613a74d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f95163fb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7111730 0x5613a74d20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f95163fb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5613a7436990 con 0x5613a7109550 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f95163fb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5613a73de5b0 con 0x5613a7132410 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f95163fb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5613a7432950 con 0x5613a7111730 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f94fe7fc640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f94f40a2d80 0x7f94f40a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950d20f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7109550 0x5613a74d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950da10640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f94f40a2d80 0x7f94f40a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950d20f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7109550 0x5613a74d1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36926/0 (socket says 172.21.15.129:36926) 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950d20f640 1 -- 172.21.15.129:0/1129985523 learned_addr learned my addr 172.21.15.129:0/1129985523 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950ca0e640 1 --2- 172.21.15.129:0/1129985523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7111730 0x5613a74d20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f94fe7fc640 1 -- 172.21.15.129:0/2221705311 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f95040cf0e0 con 0x7f9508103c30 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950da10640 1 --2- 172.21.15.129:0/1129985523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74d1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950ca0e640 1 -- 172.21.15.129:0/1129985523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 msgr2=0x5613a74d1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950ca0e640 1 --2- 172.21.15.129:0/1129985523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74d1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950ca0e640 1 -- 172.21.15.129:0/1129985523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7109550 msgr2=0x5613a74d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950ca0e640 1 --2- 172.21.15.129:0/1129985523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7109550 0x5613a74d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950ca0e640 1 -- 172.21.15.129:0/1129985523 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613a72127e0 con 0x5613a7111730 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950d20f640 1 --2- 172.21.15.129:0/1129985523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7109550 0x5613a74d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950da10640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f94f40a2d80 0x7f94f40a5290 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9500009440 tx=0x7f950000ab20 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.054+0000 7f950ca0e640 1 --2- 172.21.15.129:0/1129985523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7111730 0x5613a74d20a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f94f8005940 tx=0x7f94f8031660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a06f19a13a60e0d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.055+0000 7f94f37fe640 1 -- 172.21.15.129:0/1129985523 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8040070 con 0x5613a7111730 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.055+0000 7f94f37fe640 1 -- 172.21.15.129:0/1129985523 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f8005020 con 0x5613a7111730 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.055+0000 7f94f37fe640 1 -- 172.21.15.129:0/1129985523 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8035040 con 0x5613a7111730 2024-05-20T18:07:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.056+0000 7f95163fb740 1 -- 172.21.15.129:0/1129985523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7111730 msgr2=0x5613a74d20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.056+0000 7f95163fb740 1 --2- 172.21.15.129:0/1129985523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7111730 0x5613a74d20a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f94f8005940 tx=0x7f94f8031660 comp rx=0 tx=0).stop 2024-05-20T18:07:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.056+0000 7f95163fb740 1 -- 172.21.15.129:0/1129985523 shutdown_connections 2024-05-20T18:07:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.056+0000 7f95163fb740 1 --2- 172.21.15.129:0/1129985523 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7111730 0x5613a74d20a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.056+0000 7f95163fb740 1 --2- 172.21.15.129:0/1129985523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7109550 0x5613a74d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.056+0000 7f95163fb740 1 --2- 172.21.15.129:0/1129985523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74d1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.056+0000 7f95163fb740 1 -- 172.21.15.129:0/1129985523 >> 172.21.15.129:0/1129985523 conn(0x5613a7126560 msgr2=0x5613a74cddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.056+0000 7f95163fb740 1 -- 172.21.15.129:0/1129985523 shutdown_connections 2024-05-20T18:07:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.056+0000 7f95163fb740 1 -- 172.21.15.129:0/1129985523 wait complete. 2024-05-20T18:07:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.056+0000 7f95163fb740 1 -- start start 2024-05-20T18:07:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.056+0000 7f95163fb740 1 Processor -- start 2024-05-20T18:07:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f95163fb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a74dd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f95163fb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74ddc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f95163fb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74dbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f95163fb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5613a74de910 con 0x5613a7111730 2024-05-20T18:07:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f95163fb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5613a73f1b40 con 0x5613a7132410 2024-05-20T18:07:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f95163fb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5613a74dea90 con 0x5613a7109550 2024-05-20T18:07:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f95163fb740 10 client.0 _ll_register_callbacks cb 0x5613a73e60f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950ca0e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74ddc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950ca0e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74ddc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36934/0 (socket says 172.21.15.129:36934) 2024-05-20T18:07:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950ca0e640 1 -- 172.21.15.129:0/4019419303 learned_addr learned my addr 172.21.15.129:0/4019419303 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950d20f640 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a74dd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950da10640 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74dbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950ca0e640 1 -- 172.21.15.129:0/4019419303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 msgr2=0x5613a74dd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950ca0e640 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a74dd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950ca0e640 1 -- 172.21.15.129:0/4019419303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 msgr2=0x5613a74dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950ca0e640 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74dbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950ca0e640 1 -- 172.21.15.129:0/4019419303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613a7272f70 con 0x5613a7111730 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950d20f640 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a74dd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950da10640 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74dbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f950ca0e640 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74ddc50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f94f802a9a0 tx=0x7f94f8005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d60a8d6d45bc04 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f94f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f94f0ff9640 1 -- 172.21.15.129:0/4019419303 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8040070 con 0x5613a7111730 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f94f0ff9640 1 -- 172.21.15.129:0/4019419303 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f94f80053d0 con 0x5613a7111730 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.057+0000 7f95163fb740 1 -- 172.21.15.129:0/4019419303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94f8005020 con 0x5613a7111730 2024-05-20T18:07:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f94f0ff9640 1 -- 172.21.15.129:0/4019419303 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f94f8056050 con 0x5613a7111730 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f95163fb740 10 client.16959 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f95163fb740 1 -- 172.21.15.129:0/4019419303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5613a710a170 con 0x5613a7111730 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f94f0ff9640 1 -- 172.21.15.129:0/4019419303 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f8005020 con 0x5613a7111730 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f94f0ff9640 1 -- 172.21.15.129:0/4019419303 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f94f8003860 con 0x5613a7111730 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f95163fb740 10 client.16959 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f95163fb740 10 client.16959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f95163fb740 20 client.16959 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f95163fb740 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5613a70f11f0 0x5613a74edf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f95163fb740 4 client.16959 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f95163fb740 1 -- 172.21.15.129:0/4019419303 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613a74440c0 con 0x5613a70f11f0 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.058+0000 7f950d20f640 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5613a70f11f0 0x5613a74edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.059+0000 7f94f0ff9640 10 client.16959 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.059+0000 7f950d20f640 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5613a70f11f0 0x5613a74edf80 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd995d28490376e1 server_cookie=fecc6b0741df9e2a in_seq=0 out_seq=0 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f94f0ff9640 1 -- 172.21.15.129:0/4019419303 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5613a74440c0 con 0x5613a70f11f0 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f94f0ff9640 10 client.16959 handle_command_reply: tid=0 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 1 client.16959 shutdown 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 1 -- 172.21.15.129:0/4019419303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 msgr2=0x5613a74ddc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74ddc50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f94f802a9a0 tx=0x7f94f8005e20 comp rx=0 tx=0).stop 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 1 -- 172.21.15.129:0/4019419303 shutdown_connections 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5613a70f11f0 0x5613a74edf80 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74dbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74ddc50 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f94f0ff9640 0 client.16959 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 1 --2- 172.21.15.129:0/4019419303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a74dd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 1 -- 172.21.15.129:0/4019419303 >> 172.21.15.129:0/4019419303 conn(0x5613a7126560 msgr2=0x5613a74d3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 1 -- 172.21.15.129:0/4019419303 shutdown_connections 2024-05-20T18:07:59.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 1 -- 172.21.15.129:0/4019419303 wait complete. 2024-05-20T18:07:59.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.060+0000 7f95163fb740 20 client.16959 trim_cache size 0 max 16384 2024-05-20T18:07:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f95163fb740 1 Processor -- start 2024-05-20T18:07:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f95163fb740 1 -- start start 2024-05-20T18:07:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f95163fb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a74d1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f95163fb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f95163fb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f95163fb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f95081acb70 con 0x5613a7111730 2024-05-20T18:07:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f95163fb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f95081af3d0 con 0x5613a7132410 2024-05-20T18:07:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f95163fb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f95081ad1c0 con 0x5613a7109550 2024-05-20T18:07:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f950ca0e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f950ca0e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74f0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36944/0 (socket says 172.21.15.129:36944) 2024-05-20T18:07:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f950ca0e640 1 -- 172.21.15.129:0/983181583 learned_addr learned my addr 172.21.15.129:0/983181583 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:59.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f950d20f640 1 --2- 172.21.15.129:0/983181583 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a74d1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f950da10640 1 --2- 172.21.15.129:0/983181583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f950ca0e640 1 -- 172.21.15.129:0/983181583 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 msgr2=0x5613a74d1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f950ca0e640 1 --2- 172.21.15.129:0/983181583 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a74d1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f950ca0e640 1 -- 172.21.15.129:0/983181583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 msgr2=0x5613a74f14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f950ca0e640 1 --2- 172.21.15.129:0/983181583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74f14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.078+0000 7f950ca0e640 1 -- 172.21.15.129:0/983181583 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613a74f3f50 con 0x5613a7111730 2024-05-20T18:07:59.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.079+0000 7f950ca0e640 1 --2- 172.21.15.129:0/983181583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74f0fb0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f94f8003e20 tx=0x7f94f802ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160894ec42b06a1e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.079+0000 7f94f2ffd640 1 -- 172.21.15.129:0/983181583 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8040040 con 0x5613a7111730 2024-05-20T18:07:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.079+0000 7f94f2ffd640 1 -- 172.21.15.129:0/983181583 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f804d070 con 0x5613a7111730 2024-05-20T18:07:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.079+0000 7f94f2ffd640 1 -- 172.21.15.129:0/983181583 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8035040 con 0x5613a7111730 2024-05-20T18:07:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.079+0000 7f95163fb740 1 -- 172.21.15.129:0/983181583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 msgr2=0x5613a74f0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.079+0000 7f95163fb740 1 --2- 172.21.15.129:0/983181583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74f0fb0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f94f8003e20 tx=0x7f94f802ab80 comp rx=0 tx=0).stop 2024-05-20T18:07:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 -- 172.21.15.129:0/983181583 shutdown_connections 2024-05-20T18:07:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 --2- 172.21.15.129:0/983181583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74f14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 --2- 172.21.15.129:0/983181583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a74f0fb0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 --2- 172.21.15.129:0/983181583 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a74d1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 -- 172.21.15.129:0/983181583 >> 172.21.15.129:0/983181583 conn(0x5613a7126560 msgr2=0x5613a74d3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 -- 172.21.15.129:0/983181583 shutdown_connections 2024-05-20T18:07:59.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 -- 172.21.15.129:0/983181583 wait complete. 2024-05-20T18:07:59.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 -- start start 2024-05-20T18:07:59.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 Processor -- start 2024-05-20T18:07:59.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a754e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a754eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74f0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f95081ad890 con 0x5613a7111730 2024-05-20T18:07:59.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f95081aed00 con 0x5613a7132410 2024-05-20T18:07:59.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.080+0000 7f95163fb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f95081adf60 con 0x5613a7109550 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f95163fb740 10 client.0 _ll_register_callbacks cb 0x7f94f80315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950ca0e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a754eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950ca0e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a754eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36958/0 (socket says 172.21.15.129:36958) 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950ca0e640 1 -- 172.21.15.129:0/473441328 learned_addr learned my addr 172.21.15.129:0/473441328 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950d20f640 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a754e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950da10640 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74f0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950ca0e640 1 -- 172.21.15.129:0/473441328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 msgr2=0x5613a754e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950ca0e640 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a754e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950ca0e640 1 -- 172.21.15.129:0/473441328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 msgr2=0x5613a74f0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950ca0e640 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74f0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950ca0e640 1 -- 172.21.15.129:0/473441328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613a74f0b70 con 0x5613a7111730 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950da10640 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74f0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:07:59.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f950ca0e640 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a754eae0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f94f8000c00 tx=0x7f94f80045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e739c8536a0c97 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f94f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f94f0ff9640 1 -- 172.21.15.129:0/473441328 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8035040 con 0x5613a7111730 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f94f0ff9640 1 -- 172.21.15.129:0/473441328 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f94f80037f0 con 0x5613a7111730 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f95163fb740 1 -- 172.21.15.129:0/473441328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94f804d070 con 0x5613a7111730 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f94f0ff9640 1 -- 172.21.15.129:0/473441328 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f94f806f050 con 0x5613a7111730 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f95163fb740 10 client.16965 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.081+0000 7f95163fb740 1 -- 172.21.15.129:0/473441328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5613a7555c80 con 0x5613a7111730 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.082+0000 7f94f0ff9640 1 -- 172.21.15.129:0/473441328 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f804d070 con 0x5613a7111730 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.082+0000 7f94f0ff9640 1 -- 172.21.15.129:0/473441328 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f94f806a030 con 0x5613a7111730 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.082+0000 7f95163fb740 10 client.16965 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.082+0000 7f95163fb740 10 client.16965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:59.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.082+0000 7f95163fb740 20 client.16965 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.082+0000 7f95163fb740 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5613a70c6630 0x5613a75583a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.082+0000 7f95163fb740 4 client.16965 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.082+0000 7f95163fb740 1 -- 172.21.15.129:0/473441328 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5613a743e410 con 0x5613a70c6630 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.082+0000 7f950d20f640 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5613a70c6630 0x5613a75583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.082+0000 7f94f0ff9640 10 client.16965 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.082+0000 7f950d20f640 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5613a70c6630 0x5613a75583a0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cef5d0691cb75e21 server_cookie=ba056a9e8b22b951 in_seq=0 out_seq=0 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.083+0000 7f94f0ff9640 1 -- 172.21.15.129:0/473441328 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5613a743e410 con 0x5613a70c6630 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.083+0000 7f94f0ff9640 10 client.16965 handle_command_reply: tid=0 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 1 client.16965 shutdown 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 1 -- 172.21.15.129:0/473441328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 msgr2=0x5613a754eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a754eae0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f94f8000c00 tx=0x7f94f80045b0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 1 -- 172.21.15.129:0/473441328 shutdown_connections 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5613a70c6630 0x5613a75583a0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613a7132410 0x5613a74f0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5613a7111730 0x5613a754eae0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 1 --2- 172.21.15.129:0/473441328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5613a7109550 0x5613a754e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 1 -- 172.21.15.129:0/473441328 >> 172.21.15.129:0/473441328 conn(0x5613a7126560 msgr2=0x5613a74d3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f94f0ff9640 0 client.16965 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:59.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 1 -- 172.21.15.129:0/473441328 shutdown_connections 2024-05-20T18:07:59.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 1 -- 172.21.15.129:0/473441328 wait complete. 2024-05-20T18:07:59.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f95163fb740 20 client.16965 trim_cache size 0 max 16384 2024-05-20T18:07:59.088 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:07:59.088 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:07:59.089 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "850bf4d0-8223-4691-9b42-eb9ad04b07be", 2024-05-20T18:07:59.089 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:07:59.089 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:07:59.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f950f473640 1 -- 172.21.15.129:0/2221705311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f94f40a2d80 msgr2=0x7f94f40a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f950f473640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f94f40a2d80 0x7f94f40a5290 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9500009440 tx=0x7f950000ab20 comp rx=0 tx=0).stop 2024-05-20T18:07:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f950f473640 1 -- 172.21.15.129:0/2221705311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9508103c30 msgr2=0x7f950819f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.084+0000 7f950f473640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9508103c30 0x7f950819f570 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f950400d950 tx=0x7f950400de10 comp rx=0 tx=0).stop 2024-05-20T18:07:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.085+0000 7f950f473640 1 -- 172.21.15.129:0/2221705311 shutdown_connections 2024-05-20T18:07:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.085+0000 7f950f473640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f94f40a2d80 0x7f94f40a5290 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.085+0000 7f950f473640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f950810d040 0x7f95081996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.085+0000 7f950f473640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f950810a690 0x7f950819fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.085+0000 7f950f473640 1 --2- 172.21.15.129:0/2221705311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9508103c30 0x7f950819f570 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.085+0000 7f950f473640 1 -- 172.21.15.129:0/2221705311 >> 172.21.15.129:0/2221705311 conn(0x7f9508077130 msgr2=0x7f9508100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.085+0000 7f950f473640 1 -- 172.21.15.129:0/2221705311 shutdown_connections 2024-05-20T18:07:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.085+0000 7f950f473640 1 -- 172.21.15.129:0/2221705311 wait complete. 2024-05-20T18:07:59.094 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:07:59.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:07:59.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.210+0000 7f2149af7640 1 -- 172.21.15.129:0/2590004516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214410d520 msgr2=0x7f214410f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.210+0000 7f2149af7640 1 --2- 172.21.15.129:0/2590004516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214410d520 0x7f214410f940 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f213c009ca0 tx=0x7f213c02a8c0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.210+0000 7f2149af7640 1 -- 172.21.15.129:0/2590004516 shutdown_connections 2024-05-20T18:07:59.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.210+0000 7f2149af7640 1 --2- 172.21.15.129:0/2590004516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2144101020 0x7f2144101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.210+0000 7f2149af7640 1 --2- 172.21.15.129:0/2590004516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f214410d520 0x7f214410f940 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.210+0000 7f2149af7640 1 --2- 172.21.15.129:0/2590004516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2144108b70 0x7f2144100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.210+0000 7f2149af7640 1 -- 172.21.15.129:0/2590004516 >> 172.21.15.129:0/2590004516 conn(0x7f21440768d0 msgr2=0x7f2144076d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.210+0000 7f2149af7640 1 -- 172.21.15.129:0/2590004516 shutdown_connections 2024-05-20T18:07:59.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.210+0000 7f2149af7640 1 -- 172.21.15.129:0/2590004516 wait complete. 2024-05-20T18:07:59.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.211+0000 7f2149af7640 1 Processor -- start 2024-05-20T18:07:59.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.211+0000 7f2149af7640 1 -- start start 2024-05-20T18:07:59.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.211+0000 7f2149af7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2144101020 0x7f2144199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.211+0000 7f2149af7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2144108b70 0x7f2144199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.211+0000 7f2149af7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f214410d520 0x7f214419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.211+0000 7f2149af7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2144112e80 con 0x7f2144101020 2024-05-20T18:07:59.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.211+0000 7f2149af7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2144112d00 con 0x7f214410d520 2024-05-20T18:07:59.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.211+0000 7f2149af7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2144113000 con 0x7f2144108b70 2024-05-20T18:07:59.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.211+0000 7f21437fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2144101020 0x7f2144199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.211+0000 7f21437fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2144101020 0x7f2144199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36984/0 (socket says 172.21.15.129:36984) 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.211+0000 7f21437fe640 1 -- 172.21.15.129:0/997916880 learned_addr learned my addr 172.21.15.129:0/997916880 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f2142ffd640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2144108b70 0x7f2144199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f2143fff640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f214410d520 0x7f214419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f21437fe640 1 -- 172.21.15.129:0/997916880 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2144108b70 msgr2=0x7f2144199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f21437fe640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2144108b70 0x7f2144199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f21437fe640 1 -- 172.21.15.129:0/997916880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f214410d520 msgr2=0x7f214419df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f21437fe640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f214410d520 0x7f214419df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f21437fe640 1 -- 172.21.15.129:0/997916880 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f214419e5c0 con 0x7f2144101020 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f2143fff640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f214410d520 0x7f214419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f21437fe640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2144101020 0x7f2144199760 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f2134005d00 tx=0x7f213400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f2140ff9640 1 -- 172.21.15.129:0/997916880 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2134017070 con 0x7f2144101020 2024-05-20T18:07:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f2149af7640 1 -- 172.21.15.129:0/997916880 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f214419e8b0 con 0x7f2144101020 2024-05-20T18:07:59.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f2140ff9640 1 -- 172.21.15.129:0/997916880 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2134009c30 con 0x7f2144101020 2024-05-20T18:07:59.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f2140ff9640 1 -- 172.21.15.129:0/997916880 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f213400f620 con 0x7f2144101020 2024-05-20T18:07:59.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.212+0000 7f2149af7640 1 -- 172.21.15.129:0/997916880 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21441a6230 con 0x7f2144101020 2024-05-20T18:07:59.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.215+0000 7f2140ff9640 1 -- 172.21.15.129:0/997916880 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f213401a070 con 0x7f2144101020 2024-05-20T18:07:59.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.215+0000 7f215099d740 1 Processor -- start 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.215+0000 7f215099d740 1 -- start start 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.215+0000 7f215099d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b785978390 0x55b785d16d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.215+0000 7f215099d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b78594f4d0 0x55b785d17270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.215+0000 7f215099d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d177b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.215+0000 7f215099d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b785c7cd00 con 0x55b7859576b0 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.215+0000 7f215099d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b785c244c0 con 0x55b78594f4d0 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.215+0000 7f215099d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b785c78cc0 con 0x55b785978390 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.215+0000 7f2142ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d177b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2140ff9640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f212c0a2d00 0x7f212c0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2142ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d177b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36998/0 (socket says 172.21.15.129:36998) 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2142ffd640 1 -- 172.21.15.129:0/693523890 learned_addr learned my addr 172.21.15.129:0/693523890 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f21437fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b78594f4d0 0x55b785d17270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2143fff640 1 --2- 172.21.15.129:0/693523890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b785978390 0x55b785d16d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2143fff640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f212c0a2d00 0x7f212c0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2142ffd640 1 -- 172.21.15.129:0/693523890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b785978390 msgr2=0x55b785d16d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2142ffd640 1 --2- 172.21.15.129:0/693523890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b785978390 0x55b785d16d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2142ffd640 1 -- 172.21.15.129:0/693523890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b78594f4d0 msgr2=0x55b785d17270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2142ffd640 1 --2- 172.21.15.129:0/693523890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b78594f4d0 0x55b785d17270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2142ffd640 1 -- 172.21.15.129:0/693523890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b785a5a7d0 con 0x55b7859576b0 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2140ff9640 1 -- 172.21.15.129:0/997916880 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f21340ceef0 con 0x7f2144101020 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2142ffd640 1 --2- 172.21.15.129:0/693523890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d177b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2138009c60 tx=0x7f2138011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee45799e6d5d3ea server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2143fff640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f212c0a2d00 0x7f212c0a5210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f213c009a60 tx=0x7f213c0095c0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2129ffb640 1 -- 172.21.15.129:0/693523890 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2138004230 con 0x55b7859576b0 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2129ffb640 1 -- 172.21.15.129:0/693523890 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21380043d0 con 0x55b7859576b0 2024-05-20T18:07:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.216+0000 7f2129ffb640 1 -- 172.21.15.129:0/693523890 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2138004db0 con 0x55b7859576b0 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.217+0000 7f215099d740 1 -- 172.21.15.129:0/693523890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 msgr2=0x55b785d177b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.217+0000 7f215099d740 1 --2- 172.21.15.129:0/693523890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d177b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2138009c60 tx=0x7f2138011810 comp rx=0 tx=0).stop 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.217+0000 7f215099d740 1 -- 172.21.15.129:0/693523890 shutdown_connections 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.217+0000 7f215099d740 1 --2- 172.21.15.129:0/693523890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b78594f4d0 0x55b785d17270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.217+0000 7f215099d740 1 --2- 172.21.15.129:0/693523890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d177b0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.217+0000 7f215099d740 1 --2- 172.21.15.129:0/693523890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b785978390 0x55b785d16d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.217+0000 7f215099d740 1 -- 172.21.15.129:0/693523890 >> 172.21.15.129:0/693523890 conn(0x55b78596c4e0 msgr2=0x55b785d134e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.217+0000 7f215099d740 1 -- 172.21.15.129:0/693523890 shutdown_connections 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.217+0000 7f215099d740 1 -- 172.21.15.129:0/693523890 wait complete. 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f215099d740 1 -- start start 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f215099d740 1 Processor -- start 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f215099d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b78594f4d0 0x55b785d22e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f215099d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d23360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f215099d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b785978390 0x55b785d212c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f215099d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b785d23fc0 con 0x55b7859576b0 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f215099d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b785c37b90 con 0x55b785978390 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f215099d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b785d24140 con 0x55b78594f4d0 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f215099d740 10 client.0 _ll_register_callbacks cb 0x55b785c73da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f2142ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d23360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f2142ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d23360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37004/0 (socket says 172.21.15.129:37004) 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f2142ffd640 1 -- 172.21.15.129:0/3811629449 learned_addr learned my addr 172.21.15.129:0/3811629449 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f2143fff640 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b785978390 0x55b785d212c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.218+0000 7f21437fe640 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b78594f4d0 0x55b785d22e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2142ffd640 1 -- 172.21.15.129:0/3811629449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b78594f4d0 msgr2=0x55b785d22e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2142ffd640 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b78594f4d0 0x55b785d22e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2142ffd640 1 -- 172.21.15.129:0/3811629449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b785978390 msgr2=0x55b785d212c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2142ffd640 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b785978390 0x55b785d212c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2142ffd640 1 -- 172.21.15.129:0/3811629449 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7859903f0 con 0x55b7859576b0 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f21437fe640 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b78594f4d0 0x55b785d22e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2142ffd640 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d23360 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f21380058c0 tx=0x7f2138019b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e4e5192367afdc server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2122ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2122ffd640 1 -- 172.21.15.129:0/3811629449 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2138028020 con 0x55b7859576b0 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2122ffd640 1 -- 172.21.15.129:0/3811629449 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f2138024620 con 0x55b7859576b0 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f215099d740 1 -- 172.21.15.129:0/3811629449 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21380043d0 con 0x55b7859576b0 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2122ffd640 1 -- 172.21.15.129:0/3811629449 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2138033050 con 0x55b7859576b0 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f215099d740 10 client.16977 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f215099d740 1 -- 172.21.15.129:0/3811629449 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b7859500f0 con 0x55b7859576b0 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2122ffd640 1 -- 172.21.15.129:0/3811629449 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21380043d0 con 0x55b7859576b0 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f2122ffd640 1 -- 172.21.15.129:0/3811629449 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f2138018070 con 0x55b7859576b0 2024-05-20T18:07:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f215099d740 10 client.16977 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.219+0000 7f215099d740 10 client.16977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.220+0000 7f215099d740 20 client.16977 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.220+0000 7f215099d740 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b785936f60 0x55b785d33aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.220+0000 7f215099d740 4 client.16977 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.220+0000 7f215099d740 1 -- 172.21.15.129:0/3811629449 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b785c8a430 con 0x55b785936f60 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.220+0000 7f21437fe640 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b785936f60 0x55b785d33aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.220+0000 7f2122ffd640 10 client.16977 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.220+0000 7f21437fe640 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b785936f60 0x55b785d33aa0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7bcd2dfdc80b079 server_cookie=4c2ff16d0e67e541 in_seq=0 out_seq=0 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f2122ffd640 1 -- 172.21.15.129:0/3811629449 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b785c8a430 con 0x55b785936f60 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f2122ffd640 10 client.16977 handle_command_reply: tid=0 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 1 client.16977 shutdown 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 1 -- 172.21.15.129:0/3811629449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 msgr2=0x55b785d23360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d23360 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f21380058c0 tx=0x7f2138019b20 comp rx=0 tx=0).stop 2024-05-20T18:07:59.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 1 -- 172.21.15.129:0/3811629449 shutdown_connections 2024-05-20T18:07:59.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b785936f60 0x55b785d33aa0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b785978390 0x55b785d212c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f2122ffd640 0 client.16977 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:59.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d23360 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 1 --2- 172.21.15.129:0/3811629449 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b78594f4d0 0x55b785d22e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 1 -- 172.21.15.129:0/3811629449 >> 172.21.15.129:0/3811629449 conn(0x55b78596c4e0 msgr2=0x55b785d18a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 1 -- 172.21.15.129:0/3811629449 shutdown_connections 2024-05-20T18:07:59.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 1 -- 172.21.15.129:0/3811629449 wait complete. 2024-05-20T18:07:59.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.221+0000 7f215099d740 20 client.16977 trim_cache size 0 max 16384 2024-05-20T18:07:59.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f215099d740 1 Processor -- start 2024-05-20T18:07:59.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f215099d740 1 -- start start 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f215099d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b785936f60 0x55b785d21260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f215099d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b78594f4d0 0x55b785d39130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f215099d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b7859576b0 0x55b785d39670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f215099d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f21441acae0 con 0x55b78594f4d0 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f215099d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21441af340 con 0x55b785936f60 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f215099d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f21441ad130 con 0x55b7859576b0 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f2142ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b78594f4d0 0x55b785d39130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f2142ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b78594f4d0 0x55b785d39130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37018/0 (socket says 172.21.15.129:37018) 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f2142ffd640 1 -- 172.21.15.129:0/1550824947 learned_addr learned my addr 172.21.15.129:0/1550824947 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f21437fe640 1 --2- 172.21.15.129:0/1550824947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b785936f60 0x55b785d21260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f2142ffd640 1 -- 172.21.15.129:0/1550824947 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b7859576b0 msgr2=0x55b785d39670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f2142ffd640 1 --2- 172.21.15.129:0/1550824947 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b7859576b0 0x55b785d39670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f2142ffd640 1 -- 172.21.15.129:0/1550824947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b785936f60 msgr2=0x55b785d21260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.239+0000 7f2142ffd640 1 --2- 172.21.15.129:0/1550824947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b785936f60 0x55b785d21260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.240+0000 7f2142ffd640 1 -- 172.21.15.129:0/1550824947 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b785d22780 con 0x55b78594f4d0 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.240+0000 7f2142ffd640 1 --2- 172.21.15.129:0/1550824947 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b78594f4d0 0x55b785d39130 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2138006010 tx=0x7f2138018300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c8aaeb677b544d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.240+0000 7f21297fa640 1 -- 172.21.15.129:0/1550824947 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2138028070 con 0x55b78594f4d0 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.240+0000 7f21297fa640 1 -- 172.21.15.129:0/1550824947 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f213801f040 con 0x55b78594f4d0 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.240+0000 7f21297fa640 1 -- 172.21.15.129:0/1550824947 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2138018b80 con 0x55b78594f4d0 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.240+0000 7f215099d740 1 -- 172.21.15.129:0/1550824947 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b78594f4d0 msgr2=0x55b785d39130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 --2- 172.21.15.129:0/1550824947 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b78594f4d0 0x55b785d39130 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2138006010 tx=0x7f2138018300 comp rx=0 tx=0).stop 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 -- 172.21.15.129:0/1550824947 shutdown_connections 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 --2- 172.21.15.129:0/1550824947 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b7859576b0 0x55b785d39670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 --2- 172.21.15.129:0/1550824947 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b78594f4d0 0x55b785d39130 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 --2- 172.21.15.129:0/1550824947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b785936f60 0x55b785d21260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 -- 172.21.15.129:0/1550824947 >> 172.21.15.129:0/1550824947 conn(0x55b78596c4e0 msgr2=0x55b785d20b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 -- 172.21.15.129:0/1550824947 shutdown_connections 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 -- 172.21.15.129:0/1550824947 wait complete. 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 -- start start 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 Processor -- start 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b785936f60 0x55b785d26630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b78594f4d0 0x55b785d26b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d99ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f21441ad800 con 0x55b7859576b0 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21441aec70 con 0x55b78594f4d0 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.241+0000 7f215099d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f21441aded0 con 0x55b785936f60 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f215099d740 10 client.0 _ll_register_callbacks cb 0x7f213409b1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2143fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d99ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2143fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d99ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37026/0 (socket says 172.21.15.129:37026) 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2143fff640 1 -- 172.21.15.129:0/504271433 learned_addr learned my addr 172.21.15.129:0/504271433 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2142ffd640 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b78594f4d0 0x55b785d26b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f21437fe640 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b785936f60 0x55b785d26630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2143fff640 1 -- 172.21.15.129:0/504271433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b785936f60 msgr2=0x55b785d26630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2143fff640 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b785936f60 0x55b785d26630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2143fff640 1 -- 172.21.15.129:0/504271433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b78594f4d0 msgr2=0x55b785d26b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2143fff640 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b78594f4d0 0x55b785d26b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2143fff640 1 -- 172.21.15.129:0/504271433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b785d9a220 con 0x55b7859576b0 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2143fff640 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d99ce0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f213c009ca0 tx=0x7f213c009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3c7107a1541d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2122ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2122ffd640 1 -- 172.21.15.129:0/504271433 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f213c051070 con 0x55b7859576b0 2024-05-20T18:07:59.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2122ffd640 1 -- 172.21.15.129:0/504271433 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f213c031720 con 0x55b7859576b0 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f215099d740 1 -- 172.21.15.129:0/504271433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f213801f040 con 0x55b7859576b0 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f2122ffd640 1 -- 172.21.15.129:0/504271433 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f213c05d050 con 0x55b7859576b0 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f215099d740 10 client.16983 fetch_fsmap learned FSMap version 26 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.242+0000 7f215099d740 1 -- 172.21.15.129:0/504271433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b785d9b380 con 0x55b7859576b0 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.243+0000 7f2122ffd640 1 -- 172.21.15.129:0/504271433 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f213801f040 con 0x55b7859576b0 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.243+0000 7f2122ffd640 1 -- 172.21.15.129:0/504271433 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f213c02f870 con 0x55b7859576b0 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.243+0000 7f215099d740 10 client.16983 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.243+0000 7f215099d740 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.243+0000 7f215099d740 20 client.16983 populate_metadata read hostname 'smithi129' 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.243+0000 7f215099d740 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b785978390 0x55b785d9db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.243+0000 7f215099d740 4 client.16983 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.243+0000 7f215099d740 1 -- 172.21.15.129:0/504271433 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b785c84780 con 0x55b785978390 2024-05-20T18:07:59.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.243+0000 7f21437fe640 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b785978390 0x55b785d9db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f21437fe640 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b785978390 0x55b785d9db70 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c20c524e65c928 server_cookie=bac483920c248af8 in_seq=0 out_seq=0 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f2122ffd640 10 client.16983 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f2122ffd640 1 -- 172.21.15.129:0/504271433 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b785c84780 con 0x55b785978390 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f2122ffd640 10 client.16983 handle_command_reply: tid=0 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 1 client.16983 shutdown 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 1 -- 172.21.15.129:0/504271433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 msgr2=0x55b785d99ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d99ce0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f213c009ca0 tx=0x7f213c009360 comp rx=0 tx=0).stop 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 1 -- 172.21.15.129:0/504271433 shutdown_connections 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b785978390 0x55b785d9db70 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b78594f4d0 0x55b785d26b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7859576b0 0x55b785d99ce0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f2122ffd640 0 client.16983 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 1 --2- 172.21.15.129:0/504271433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b785936f60 0x55b785d26630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 1 -- 172.21.15.129:0/504271433 >> 172.21.15.129:0/504271433 conn(0x55b78596c4e0 msgr2=0x55b785d18f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 1 -- 172.21.15.129:0/504271433 shutdown_connections 2024-05-20T18:07:59.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 1 -- 172.21.15.129:0/504271433 wait complete. 2024-05-20T18:07:59.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.244+0000 7f215099d740 20 client.16983 trim_cache size 0 max 16384 2024-05-20T18:07:59.249 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:07:59.249 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:07:59.249 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:07:59.249 INFO:teuthology.orchestra.run.smithi129.stdout: "850bf4d0-8223-4691-9b42-eb9ad04b07be": { 2024-05-20T18:07:59.249 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:07:59.250 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "850bf4d0-8223-4691-9b42-eb9ad04b07be", 2024-05-20T18:07:59.250 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:07:59.250 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:07:59.250 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:07:59.250 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:07:59.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 -- 172.21.15.129:0/997916880 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f212c0a2d00 msgr2=0x7f212c0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f212c0a2d00 0x7f212c0a5210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f213c009a60 tx=0x7f213c0095c0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 -- 172.21.15.129:0/997916880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2144101020 msgr2=0x7f2144199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:07:59.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2144101020 0x7f2144199760 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f2134005d00 tx=0x7f213400dde0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 -- 172.21.15.129:0/997916880 shutdown_connections 2024-05-20T18:07:59.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f212c0a2d00 0x7f212c0a5210 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f214410d520 0x7f214419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2144108b70 0x7f2144199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 --2- 172.21.15.129:0/997916880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2144101020 0x7f2144199760 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:07:59.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 -- 172.21.15.129:0/997916880 >> 172.21.15.129:0/997916880 conn(0x7f21440768d0 msgr2=0x7f21441094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:07:59.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 -- 172.21.15.129:0/997916880 shutdown_connections 2024-05-20T18:07:59.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:07:59.245+0000 7f2149af7640 1 -- 172.21.15.129:0/997916880 wait complete. 2024-05-20T18:07:59.254 INFO:tasks.cephfs.filesystem:scrub status for tag:850bf4d0-8223-4691-9b42-eb9ad04b07be - {'path': '/', 'tag': '850bf4d0-8223-4691-9b42-eb9ad04b07be', 'options': 'recursive,force'} 2024-05-20T18:08:00.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:59 smithi204 ceph-mon[37076]: pgmap v291: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 341 B/s wr, 1 op/s 2024-05-20T18:08:00.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:59 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:08:00.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:59 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:00.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:59 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:08:00.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:07:59 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:00.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:59 smithi154 ceph-mon[36482]: pgmap v291: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 341 B/s wr, 1 op/s 2024-05-20T18:08:00.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:59 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:08:00.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:59 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:00.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:59 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:08:00.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:07:59 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:00.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:59 smithi129 ceph-mon[32029]: pgmap v291: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 341 B/s wr, 1 op/s 2024-05-20T18:08:00.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:59 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:08:00.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:59 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:00.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:59 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:08:00.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:07:59 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:01.182 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:00 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:08:01.293 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:00 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:08:01.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:00 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:08:02.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:01 smithi204 ceph-mon[37076]: pgmap v292: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:02.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:01 smithi154 ceph-mon[36482]: pgmap v292: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:02.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:01 smithi129 ceph-mon[32029]: pgmap v292: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:04.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:03 smithi204 ceph-mon[37076]: pgmap v293: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:04.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:03 smithi154 ceph-mon[36482]: pgmap v293: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:04.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:03 smithi129 ceph-mon[32029]: pgmap v293: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:05.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:04 smithi154 ceph-mon[36482]: pgmap v294: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-05-20T18:08:05.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:04 smithi129 ceph-mon[32029]: pgmap v294: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-05-20T18:08:05.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:04 smithi204 ceph-mon[37076]: pgmap v294: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-05-20T18:08:07.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:07 smithi154 ceph-mon[36482]: pgmap v295: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-20T18:08:07.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:07 smithi129 ceph-mon[32029]: pgmap v295: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-20T18:08:07.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:07 smithi204 ceph-mon[37076]: pgmap v295: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-20T18:08:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.687+0000 7fa02c32f640 1 -- 172.21.15.129:0/2234465500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa024107a40 msgr2=0x7fa0241015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.687+0000 7fa02c32f640 1 --2- 172.21.15.129:0/2234465500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa024107a40 0x7fa0241015d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa020009900 tx=0x7fa02002a410 comp rx=0 tx=0).stop 2024-05-20T18:08:08.687 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.687+0000 7fa02c32f640 1 -- 172.21.15.129:0/2234465500 shutdown_connections 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.687+0000 7fa02c32f640 1 --2- 172.21.15.129:0/2234465500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa02410c3a0 0x7fa02410e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.687+0000 7fa02c32f640 1 --2- 172.21.15.129:0/2234465500 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa024101b10 0x7fa024101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.687+0000 7fa02c32f640 1 --2- 172.21.15.129:0/2234465500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa024107a40 0x7fa0241015d0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.687+0000 7fa02c32f640 1 -- 172.21.15.129:0/2234465500 >> 172.21.15.129:0/2234465500 conn(0x7fa0240fd210 msgr2=0x7fa0240ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.687+0000 7fa02c32f640 1 -- 172.21.15.129:0/2234465500 shutdown_connections 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.687+0000 7fa02c32f640 1 -- 172.21.15.129:0/2234465500 wait complete. 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.687+0000 7fa02c32f640 1 Processor -- start 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02c32f640 1 -- start start 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02c32f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa024101b10 0x7fa02419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02c32f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa024107a40 0x7fa02419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02c32f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa02410c3a0 0x7fa02419b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.688 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02c32f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa024111d50 con 0x7fa02410c3a0 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02c32f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa024111bd0 con 0x7fa024101b10 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02c32f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa024111ed0 con 0x7fa024107a40 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02a8cc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa02410c3a0 0x7fa02419b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02a8cc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa02410c3a0 0x7fa02419b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60138/0 (socket says 172.21.15.129:60138) 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02a8cc640 1 -- 172.21.15.129:0/3757081649 learned_addr learned my addr 172.21.15.129:0/3757081649 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa0298ca640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa024107a40 0x7fa02419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa024101b10 0x7fa02419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02a8cc640 1 -- 172.21.15.129:0/3757081649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa024107a40 msgr2=0x7fa02419abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02a8cc640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa024107a40 0x7fa02419abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02a8cc640 1 -- 172.21.15.129:0/3757081649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa024101b10 msgr2=0x7fa02419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02a8cc640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa024101b10 0x7fa02419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02a8cc640 1 -- 172.21.15.129:0/3757081649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0241a59b0 con 0x7fa02410c3a0 2024-05-20T18:08:08.689 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa024101b10 0x7fa02419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:08.690 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.688+0000 7fa02a8cc640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa02410c3a0 0x7fa02419b110 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009fd0 tx=0x7fa01c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:08.690 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.689+0000 7fa01b7fe640 1 -- 172.21.15.129:0/3757081649 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c017070 con 0x7fa02410c3a0 2024-05-20T18:08:08.690 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.689+0000 7fa01b7fe640 1 -- 172.21.15.129:0/3757081649 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa01c003e50 con 0x7fa02410c3a0 2024-05-20T18:08:08.690 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.689+0000 7fa02c32f640 1 -- 172.21.15.129:0/3757081649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0241a5ca0 con 0x7fa02410c3a0 2024-05-20T18:08:08.690 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.689+0000 7fa01b7fe640 1 -- 172.21.15.129:0/3757081649 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c0106c0 con 0x7fa02410c3a0 2024-05-20T18:08:08.690 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.689+0000 7fa02c32f640 1 -- 172.21.15.129:0/3757081649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0241a61a0 con 0x7fa02410c3a0 2024-05-20T18:08:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.691+0000 7fa01b7fe640 1 -- 172.21.15.129:0/3757081649 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa01c003970 con 0x7fa02410c3a0 2024-05-20T18:08:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.691+0000 7fa0331d1740 1 Processor -- start 2024-05-20T18:08:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa0331d1740 1 -- start start 2024-05-20T18:08:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6dd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f6dd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f31d4d0 0x55e05f6ddb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa0331d1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e05f642340 con 0x55e05f3152f0 2024-05-20T18:08:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa0331d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e05f5ea290 con 0x55e05f33e1b0 2024-05-20T18:08:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa0331d1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e05f63e300 con 0x55e05f31d4d0 2024-05-20T18:08:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa01b7fe640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0100a2cf0 0x7fa0100a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa02a0cb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f6dd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa02a0cb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f6dd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60144/0 (socket says 172.21.15.129:60144) 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3447039471 learned_addr learned my addr 172.21.15.129:0/3447039471 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa02a8cc640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0100a2cf0 0x7fa0100a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa0298ca640 1 --2- 172.21.15.129:0/3447039471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6dd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa01b7fe640 1 -- 172.21.15.129:0/3757081649 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fa01c0cf2b0 con 0x7fa02410c3a0 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.692+0000 7fa0298ca640 1 --2- 172.21.15.129:0/3447039471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f31d4d0 0x55e05f6ddb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.693+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3447039471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f31d4d0 msgr2=0x55e05f6ddb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.693+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/3447039471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f31d4d0 0x55e05f6ddb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.693+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3447039471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 msgr2=0x55e05f6dd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.693+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/3447039471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6dd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.693+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3447039471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e05f60e8e0 con 0x55e05f3152f0 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.693+0000 7fa0298ca640 1 --2- 172.21.15.129:0/3447039471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f31d4d0 0x55e05f6ddb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.693+0000 7fa02a8cc640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0100a2cf0 0x7fa0100a5200 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa01c09b490 tx=0x7fa01c004020 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.693+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/3447039471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f6dd5c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa020005ab0 tx=0x7fa020009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b606b84f0e447bbd server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.693+0000 7fa018bfe640 1 -- 172.21.15.129:0/3447039471 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02003f070 con 0x55e05f3152f0 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.693+0000 7fa018bfe640 1 -- 172.21.15.129:0/3447039471 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa020004030 con 0x55e05f3152f0 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.693+0000 7fa018bfe640 1 -- 172.21.15.129:0/3447039471 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020035070 con 0x55e05f3152f0 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.694+0000 7fa0331d1740 1 -- 172.21.15.129:0/3447039471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 msgr2=0x55e05f6dd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.694+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3447039471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f6dd5c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fa020005ab0 tx=0x7fa020009680 comp rx=0 tx=0).stop 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.694+0000 7fa0331d1740 1 -- 172.21.15.129:0/3447039471 shutdown_connections 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.694+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3447039471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f31d4d0 0x55e05f6ddb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.694+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3447039471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f6dd5c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.694+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3447039471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6dd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.694+0000 7fa0331d1740 1 -- 172.21.15.129:0/3447039471 >> 172.21.15.129:0/3447039471 conn(0x55e05f332300 msgr2=0x55e05f6d9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.694+0000 7fa0331d1740 1 -- 172.21.15.129:0/3447039471 shutdown_connections 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.694+0000 7fa0331d1740 1 -- 172.21.15.129:0/3447039471 wait complete. 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.694+0000 7fa0331d1740 1 -- start start 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.694+0000 7fa0331d1740 1 Processor -- start 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f3152f0 0x55e05f6e76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6ed630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa0331d1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e05f6ea150 con 0x55e05f31d4d0 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa0331d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e05f5fd790 con 0x55e05f33e1b0 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa0331d1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e05f6ea2d0 con 0x55e05f3152f0 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa0331d1740 10 client.0 _ll_register_callbacks cb 0x55e05f638a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa02a0cb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa02a0cb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60156/0 (socket says 172.21.15.129:60156) 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3324087096 learned_addr learned my addr 172.21.15.129:0/3324087096 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa0298ca640 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f3152f0 0x55e05f6e76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa0298ca640 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6ed630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3324087096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f3152f0 msgr2=0x55e05f6e76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f3152f0 0x55e05f6e76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3324087096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 msgr2=0x55e05f6ed630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6ed630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3324087096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e05f356210 con 0x55e05f31d4d0 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa0298ca640 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6ed630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e7c10 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa020031550 tx=0x7fa020004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f707698c358764 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.695+0000 7fa00dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa00dffb640 1 -- 172.21.15.129:0/3324087096 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020035030 con 0x55e05f31d4d0 2024-05-20T18:08:08.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa00dffb640 1 -- 172.21.15.129:0/3324087096 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fa020005480 con 0x55e05f31d4d0 2024-05-20T18:08:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa0331d1740 1 -- 172.21.15.129:0/3324087096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa020004030 con 0x55e05f31d4d0 2024-05-20T18:08:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa00dffb640 1 -- 172.21.15.129:0/3324087096 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa020055050 con 0x55e05f31d4d0 2024-05-20T18:08:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa0331d1740 10 client.16995 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa0331d1740 1 -- 172.21.15.129:0/3324087096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e05f422620 con 0x55e05f31d4d0 2024-05-20T18:08:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa00dffb640 1 -- 172.21.15.129:0/3324087096 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa020004030 con 0x55e05f31d4d0 2024-05-20T18:08:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa00dffb640 1 -- 172.21.15.129:0/3324087096 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fa0200035f0 con 0x55e05f31d4d0 2024-05-20T18:08:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa0331d1740 10 client.16995 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa0331d1740 10 client.16995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:08.702 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa0331d1740 20 client.16995 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e05f2fcd80 0x55e05f6f9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa0331d1740 4 client.16995 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:08:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa0331d1740 1 -- 172.21.15.129:0/3324087096 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e05f64fa70 con 0x55e05f2fcd80 2024-05-20T18:08:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.696+0000 7fa0298ca640 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e05f2fcd80 0x55e05f6f9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.697+0000 7fa00dffb640 10 client.16995 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.697+0000 7fa0298ca640 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e05f2fcd80 0x55e05f6f9720 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c69f62cf79a61ab5 server_cookie=7655b5fd10e8d3c2 in_seq=0 out_seq=0 2024-05-20T18:08:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa00dffb640 1 -- 172.21.15.129:0/3324087096 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e05f64fa70 con 0x55e05f2fcd80 2024-05-20T18:08:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa00dffb640 10 client.16995 handle_command_reply: tid=0 2024-05-20T18:08:08.703 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 1 client.16995 shutdown 2024-05-20T18:08:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 1 -- 172.21.15.129:0/3324087096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 msgr2=0x55e05f6e7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e7c10 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa020031550 tx=0x7fa020004260 comp rx=0 tx=0).stop 2024-05-20T18:08:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 1 -- 172.21.15.129:0/3324087096 shutdown_connections 2024-05-20T18:08:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e05f2fcd80 0x55e05f6f9720 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6ed630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e7c10 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3324087096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f3152f0 0x55e05f6e76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa00dffb640 0 client.16995 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 1 -- 172.21.15.129:0/3324087096 >> 172.21.15.129:0/3324087096 conn(0x55e05f332300 msgr2=0x55e05f6deda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:08.704 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 1 -- 172.21.15.129:0/3324087096 shutdown_connections 2024-05-20T18:08:08.705 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 1 -- 172.21.15.129:0/3324087096 wait complete. 2024-05-20T18:08:08.705 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.698+0000 7fa0331d1740 20 client.16995 trim_cache size 0 max 16384 2024-05-20T18:08:08.715 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.713+0000 7fa0331d1740 1 Processor -- start 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.713+0000 7fa0331d1740 1 -- start start 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.713+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f3152f0 0x55e05f6e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.713+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.713+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6ed5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.713+0000 7fa0331d1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241acb10 con 0x55e05f31d4d0 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.713+0000 7fa0331d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241af370 con 0x55e05f33e1b0 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.713+0000 7fa0331d1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241ad160 con 0x55e05f3152f0 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.713+0000 7fa02a0cb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.713+0000 7fa02a0cb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60172/0 (socket says 172.21.15.129:60172) 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.713+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3481885161 learned_addr learned my addr 172.21.15.129:0/3481885161 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.714+0000 7fa0298ca640 1 --2- 172.21.15.129:0/3481885161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6ed5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.714+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3481885161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f3152f0 msgr2=0x55e05f6e67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:08.716 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.714+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/3481885161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f3152f0 0x55e05f6e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.714+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3481885161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 msgr2=0x55e05f6ed5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.714+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/3481885161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6ed5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.714+0000 7fa02a0cb640 1 -- 172.21.15.129:0/3481885161 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e05f6e2d50 con 0x55e05f31d4d0 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.714+0000 7fa0298ca640 1 --2- 172.21.15.129:0/3481885161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6ed5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.714+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/3481885161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e9310 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa020003a40 tx=0x7fa020000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ecbe5ea5dcbf5a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.714+0000 7fa00ffff640 1 -- 172.21.15.129:0/3481885161 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02005e070 con 0x55e05f31d4d0 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.714+0000 7fa00ffff640 1 -- 172.21.15.129:0/3481885161 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02004c070 con 0x55e05f31d4d0 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.714+0000 7fa00ffff640 1 -- 172.21.15.129:0/3481885161 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020035070 con 0x55e05f31d4d0 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 -- 172.21.15.129:0/3481885161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 msgr2=0x55e05f6e9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3481885161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e9310 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa020003a40 tx=0x7fa020000bc0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 -- 172.21.15.129:0/3481885161 shutdown_connections 2024-05-20T18:08:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3481885161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f33e1b0 0x55e05f6ed5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3481885161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f31d4d0 0x55e05f6e9310 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 --2- 172.21.15.129:0/3481885161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f3152f0 0x55e05f6e67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 -- 172.21.15.129:0/3481885161 >> 172.21.15.129:0/3481885161 conn(0x55e05f332300 msgr2=0x55e05f6e8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 -- 172.21.15.129:0/3481885161 shutdown_connections 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 -- 172.21.15.129:0/3481885161 wait complete. 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 -- start start 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 Processor -- start 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f75e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f31d4d0 0x55e05f75e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f33e1b0 0x55e05f6eee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241ad830 con 0x55e05f3152f0 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241aeca0 con 0x55e05f31d4d0 2024-05-20T18:08:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241adf00 con 0x55e05f33e1b0 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0331d1740 10 client.0 _ll_register_callbacks cb 0x7fa0140096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0298ca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f75e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0298ca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f75e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60184/0 (socket says 172.21.15.129:60184) 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.715+0000 7fa0298ca640 1 -- 172.21.15.129:0/1362090657 learned_addr learned my addr 172.21.15.129:0/1362090657 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f31d4d0 0x55e05f75e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa0298ca640 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f33e1b0 0x55e05f6eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa0298ca640 1 -- 172.21.15.129:0/1362090657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f33e1b0 msgr2=0x55e05f6eee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa0298ca640 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f33e1b0 0x55e05f6eee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa0298ca640 1 -- 172.21.15.129:0/1362090657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f31d4d0 msgr2=0x55e05f75e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa0298ca640 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f31d4d0 0x55e05f75e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa0298ca640 1 -- 172.21.15.129:0/1362090657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e05f6ef440 con 0x55e05f3152f0 2024-05-20T18:08:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa0298ca640 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f75e080 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa014004200 tx=0x7fa014005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fee2f995d7258 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa00dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa00dffb640 1 -- 172.21.15.129:0/1362090657 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014020940 con 0x55e05f3152f0 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa00dffb640 1 -- 172.21.15.129:0/1362090657 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fa014020c70 con 0x55e05f3152f0 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa0331d1740 1 -- 172.21.15.129:0/1362090657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa02004c070 con 0x55e05f3152f0 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa00dffb640 1 -- 172.21.15.129:0/1362090657 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa014005020 con 0x55e05f3152f0 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa0331d1740 10 client.17001 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa0331d1740 1 -- 172.21.15.129:0/1362090657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e05f765670 con 0x55e05f3152f0 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.716+0000 7fa00dffb640 1 -- 172.21.15.129:0/1362090657 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02004c070 con 0x55e05f3152f0 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.717+0000 7fa00dffb640 1 -- 172.21.15.129:0/1362090657 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fa014011440 con 0x55e05f3152f0 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.717+0000 7fa0331d1740 10 client.17001 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.717+0000 7fa0331d1740 10 client.17001 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.717+0000 7fa0331d1740 20 client.17001 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.717+0000 7fa0331d1740 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e05f2d21c0 0x55e05f767dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.717+0000 7fa0331d1740 4 client.17001 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:08:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.717+0000 7fa0331d1740 1 -- 172.21.15.129:0/1362090657 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e05f649dc0 con 0x55e05f2d21c0 2024-05-20T18:08:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.717+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e05f2d21c0 0x55e05f767dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.717+0000 7fa00dffb640 10 client.17001 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.717+0000 7fa02a0cb640 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e05f2d21c0 0x55e05f767dd0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f8e97ce36e7c4a server_cookie=9bc1730b03cc4d7d in_seq=0 out_seq=0 2024-05-20T18:08:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa00dffb640 1 -- 172.21.15.129:0/1362090657 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e05f649dc0 con 0x55e05f2d21c0 2024-05-20T18:08:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa00dffb640 10 client.17001 handle_command_reply: tid=0 2024-05-20T18:08:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 1 client.17001 shutdown 2024-05-20T18:08:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 1 -- 172.21.15.129:0/1362090657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 msgr2=0x55e05f75e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f75e080 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa014004200 tx=0x7fa014005f90 comp rx=0 tx=0).stop 2024-05-20T18:08:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 1 -- 172.21.15.129:0/1362090657 shutdown_connections 2024-05-20T18:08:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e05f2d21c0 0x55e05f767dd0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e05f31d4d0 0x55e05f75e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa00dffb640 0 client.17001 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e05f33e1b0 0x55e05f6eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 1 --2- 172.21.15.129:0/1362090657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e05f3152f0 0x55e05f75e080 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 1 -- 172.21.15.129:0/1362090657 >> 172.21.15.129:0/1362090657 conn(0x55e05f332300 msgr2=0x55e05f6e6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 1 -- 172.21.15.129:0/1362090657 shutdown_connections 2024-05-20T18:08:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 1 -- 172.21.15.129:0/1362090657 wait complete. 2024-05-20T18:08:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.726+0000 7fa0331d1740 20 client.17001 trim_cache size 0 max 16384 2024-05-20T18:08:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:08:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:08:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:08:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:08:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 -- 172.21.15.129:0/3757081649 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0100a2cf0 msgr2=0x7fa0100a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0100a2cf0 0x7fa0100a5200 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa01c09b490 tx=0x7fa01c004020 comp rx=0 tx=0).stop 2024-05-20T18:08:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 -- 172.21.15.129:0/3757081649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa02410c3a0 msgr2=0x7fa02419b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa02410c3a0 0x7fa02419b110 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009fd0 tx=0x7fa01c00b260 comp rx=0 tx=0).stop 2024-05-20T18:08:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 -- 172.21.15.129:0/3757081649 shutdown_connections 2024-05-20T18:08:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa0100a2cf0 0x7fa0100a5200 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa02410c3a0 0x7fa02419b110 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa024107a40 0x7fa02419abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 --2- 172.21.15.129:0/3757081649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa024101b10 0x7fa02419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 -- 172.21.15.129:0/3757081649 >> 172.21.15.129:0/3757081649 conn(0x7fa0240fd210 msgr2=0x7fa0240fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 -- 172.21.15.129:0/3757081649 shutdown_connections 2024-05-20T18:08:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:08.727+0000 7fa02c32f640 1 -- 172.21.15.129:0/3757081649 wait complete. 2024-05-20T18:08:09.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:09 smithi154 ceph-mon[36482]: pgmap v296: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-20T18:08:09.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:09 smithi129 ceph-mon[32029]: pgmap v296: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-20T18:08:09.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:09 smithi204 ceph-mon[37076]: pgmap v296: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-20T18:08:10.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:10 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:08:10.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:10 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:08:10.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:10 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:08:11.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:11 smithi129 ceph-mon[32029]: pgmap v297: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 46 KiB/s wr, 23 op/s 2024-05-20T18:08:11.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:11 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:08:11.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:11 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:08:11.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:11 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:08:11.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:11 smithi204 ceph-mon[37076]: pgmap v297: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 46 KiB/s wr, 23 op/s 2024-05-20T18:08:11.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:11 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:08:11.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:11 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:08:11.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:11 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:08:11.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:11 smithi154 ceph-mon[36482]: pgmap v297: 97 pgs: 97 active+clean; 52 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 46 KiB/s wr, 23 op/s 2024-05-20T18:08:11.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:11 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:08:11.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:11 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:08:11.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:11 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:08:12.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:12 smithi204 ceph-mon[37076]: osdmap e101: 12 total, 12 up, 12 in 2024-05-20T18:08:12.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:12 smithi154 ceph-mon[36482]: osdmap e101: 12 total, 12 up, 12 in 2024-05-20T18:08:12.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:12 smithi129 ceph-mon[32029]: osdmap e101: 12 total, 12 up, 12 in 2024-05-20T18:08:13.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:13 smithi204 ceph-mon[37076]: pgmap v299: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 KiB/s wr, 23 op/s 2024-05-20T18:08:13.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:13 smithi154 ceph-mon[36482]: pgmap v299: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 KiB/s wr, 23 op/s 2024-05-20T18:08:13.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:13 smithi129 ceph-mon[32029]: pgmap v299: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 51 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 KiB/s wr, 23 op/s 2024-05-20T18:08:15.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:15 smithi204 ceph-mon[37076]: pgmap v300: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 6 op/s 2024-05-20T18:08:15.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:15 smithi154 ceph-mon[36482]: pgmap v300: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 6 op/s 2024-05-20T18:08:15.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:15 smithi129 ceph-mon[32029]: pgmap v300: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 50 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 6 op/s 2024-05-20T18:08:17.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:17 smithi204 ceph-mon[37076]: pgmap v301: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-05-20T18:08:17.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:17 smithi154 ceph-mon[36482]: pgmap v301: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-05-20T18:08:17.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:17 smithi129 ceph-mon[32029]: pgmap v301: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-05-20T18:08:19.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:19 smithi204 ceph-mon[37076]: pgmap v302: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-05-20T18:08:19.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:19 smithi154 ceph-mon[36482]: pgmap v302: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-05-20T18:08:19.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:19 smithi129 ceph-mon[32029]: pgmap v302: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-05-20T18:08:21.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:21 smithi154 ceph-mon[36482]: pgmap v303: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-05-20T18:08:21.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:21 smithi154 ceph-mon[36482]: osdmap e102: 12 total, 12 up, 12 in 2024-05-20T18:08:21.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:21 smithi129 ceph-mon[32029]: pgmap v303: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-05-20T18:08:21.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:21 smithi129 ceph-mon[32029]: osdmap e102: 12 total, 12 up, 12 in 2024-05-20T18:08:21.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:21 smithi204 ceph-mon[37076]: pgmap v303: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-05-20T18:08:21.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:21 smithi204 ceph-mon[37076]: osdmap e102: 12 total, 12 up, 12 in 2024-05-20T18:08:21.853 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.854+0000 7ffa94dd5640 1 -- 172.21.15.129:0/2216373073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa90108af0 msgr2=0x7ffa90102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:21.853 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.854+0000 7ffa94dd5640 1 --2- 172.21.15.129:0/2216373073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa90108af0 0x7ffa90102690 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7ffa80009900 tx=0x7ffa8002a410 comp rx=0 tx=0).stop 2024-05-20T18:08:21.853 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.855+0000 7ffa94dd5640 1 -- 172.21.15.129:0/2216373073 shutdown_connections 2024-05-20T18:08:21.853 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.855+0000 7ffa94dd5640 1 --2- 172.21.15.129:0/2216373073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa90108f90 0x7ffa90102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.853 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.855+0000 7ffa94dd5640 1 --2- 172.21.15.129:0/2216373073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa90103240 0x7ffa9010f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.853 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.855+0000 7ffa94dd5640 1 --2- 172.21.15.129:0/2216373073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa90108af0 0x7ffa90102690 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.853 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.855+0000 7ffa94dd5640 1 -- 172.21.15.129:0/2216373073 >> 172.21.15.129:0/2216373073 conn(0x7ffa900fe1b0 msgr2=0x7ffa90100600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:21.853 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.855+0000 7ffa94dd5640 1 -- 172.21.15.129:0/2216373073 shutdown_connections 2024-05-20T18:08:21.853 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.855+0000 7ffa94dd5640 1 -- 172.21.15.129:0/2216373073 wait complete. 2024-05-20T18:08:21.853 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.855+0000 7ffa94dd5640 1 Processor -- start 2024-05-20T18:08:21.854 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.855+0000 7ffa94dd5640 1 -- start start 2024-05-20T18:08:21.855 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.855+0000 7ffa94dd5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa90103240 0x7ffa9019f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.855 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa94dd5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa90108af0 0x7ffa9019fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.855 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa94dd5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa90108f90 0x7ffa90199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa94dd5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa90112e30 con 0x7ffa90108af0 2024-05-20T18:08:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa94dd5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa90112cb0 con 0x7ffa90108f90 2024-05-20T18:08:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa94dd5640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffa90112fb0 con 0x7ffa90103240 2024-05-20T18:08:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa90108af0 0x7ffa9019fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa90108af0 0x7ffa9019fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57494/0 (socket says 172.21.15.129:57494) 2024-05-20T18:08:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/1324787721 learned_addr learned my addr 172.21.15.129:0/1324787721 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa90108f90 0x7ffa90199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8e59c640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa90103240 0x7ffa9019f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/1324787721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa90103240 msgr2=0x7ffa9019f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:21.856 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa90103240 0x7ffa9019f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.857 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/1324787721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa90108f90 msgr2=0x7ffa90199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:21.857 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa90108f90 0x7ffa90199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.857 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/1324787721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa90199d80 con 0x7ffa90108af0 2024-05-20T18:08:21.857 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa90108f90 0x7ffa90199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:08:21.857 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa90108af0 0x7ffa9019fa10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ffa84005fd0 tx=0x7ffa8400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:21.857 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa7f7fe640 1 -- 172.21.15.129:0/1324787721 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84017070 con 0x7ffa90108af0 2024-05-20T18:08:21.857 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa7f7fe640 1 -- 172.21.15.129:0/1324787721 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa84009c30 con 0x7ffa90108af0 2024-05-20T18:08:21.857 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa94dd5640 1 -- 172.21.15.129:0/1324787721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa9019a010 con 0x7ffa90108af0 2024-05-20T18:08:21.857 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa7f7fe640 1 -- 172.21.15.129:0/1324787721 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84003e40 con 0x7ffa90108af0 2024-05-20T18:08:21.857 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.856+0000 7ffa94dd5640 1 -- 172.21.15.129:0/1324787721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa901a6270 con 0x7ffa90108af0 2024-05-20T18:08:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.859+0000 7ffa7f7fe640 1 -- 172.21.15.129:0/1324787721 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ffa84003fe0 con 0x7ffa90108af0 2024-05-20T18:08:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.859+0000 7ffa9bc7f740 1 Processor -- start 2024-05-20T18:08:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.859+0000 7ffa9bc7f740 1 -- start start 2024-05-20T18:08:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.859+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f9f1b0 0x55674533e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.859+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 0x55674533e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.859+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f7e4d0 0x55674533eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5567452a3340 con 0x556744f9f1b0 2024-05-20T18:08:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55674524b290 con 0x556744f7e4d0 2024-05-20T18:08:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55674529f300 con 0x556744f762f0 2024-05-20T18:08:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f9f1b0 0x55674533e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa7f7fe640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa780a2cf0 0x7ffa780a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f9f1b0 0x55674533e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57508/0 (socket says 172.21.15.129:57508) 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8ed9d640 1 -- 172.21.15.129:0/3887477083 learned_addr learned my addr 172.21.15.129:0/3887477083 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa780a2cf0 0x7ffa780a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8e59c640 1 --2- 172.21.15.129:0/3887477083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 0x55674533e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa7f7fe640 1 -- 172.21.15.129:0/1324787721 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7ffa840cf120 con 0x7ffa90108af0 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/3887477083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f7e4d0 0x55674533eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8ed9d640 1 -- 172.21.15.129:0/3887477083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 msgr2=0x55674533e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/3887477083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 0x55674533e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8ed9d640 1 -- 172.21.15.129:0/3887477083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f7e4d0 msgr2=0x55674533eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/3887477083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f7e4d0 0x55674533eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.866 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8ed9d640 1 -- 172.21.15.129:0/3887477083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55674526f8e0 con 0x556744f9f1b0 2024-05-20T18:08:21.867 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/3887477083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f7e4d0 0x55674533eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:21.867 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/3887477083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f9f1b0 0x55674533e080 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ffa88014a90 tx=0x7ffa88014f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb92dc83bf4ecc49 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:21.867 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa7cff9640 1 -- 172.21.15.129:0/3887477083 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa88014530 con 0x556744f9f1b0 2024-05-20T18:08:21.867 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa7cff9640 1 -- 172.21.15.129:0/3887477083 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa88023070 con 0x556744f9f1b0 2024-05-20T18:08:21.867 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa7cff9640 1 -- 172.21.15.129:0/3887477083 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8800ce10 con 0x556744f9f1b0 2024-05-20T18:08:21.867 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.860+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa780a2cf0 0x7ffa780a5200 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffa88022000 tx=0x7ffa88020000 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:21.867 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.861+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/3887477083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f9f1b0 msgr2=0x55674533e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:21.867 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.861+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/3887477083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f9f1b0 0x55674533e080 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ffa88014a90 tx=0x7ffa88014f50 comp rx=0 tx=0).stop 2024-05-20T18:08:21.867 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.861+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/3887477083 shutdown_connections 2024-05-20T18:08:21.867 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.861+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/3887477083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f7e4d0 0x55674533eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.861+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/3887477083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 0x55674533e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.861+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/3887477083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f9f1b0 0x55674533e080 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.861+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/3887477083 >> 172.21.15.129:0/3887477083 conn(0x556744f93300 msgr2=0x55674533a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.861+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/3887477083 shutdown_connections 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.861+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/3887477083 wait complete. 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa9bc7f740 1 -- start start 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa9bc7f740 1 Processor -- start 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f762f0 0x55674534a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x55674534a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f9f1b0 0x556745348590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55674534b1b0 con 0x556744f7e4d0 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55674525e790 con 0x556744f762f0 2024-05-20T18:08:21.868 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55674534b330 con 0x556744f9f1b0 2024-05-20T18:08:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa9bc7f740 10 client.0 _ll_register_callbacks cb 0x556745299a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa8dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x55674534a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa8dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x55674534a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57520/0 (socket says 172.21.15.129:57520) 2024-05-20T18:08:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/1517574700 learned_addr learned my addr 172.21.15.129:0/1517574700 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f9f1b0 0x556745348590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa8e59c640 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f762f0 0x55674534a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/1517574700 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f9f1b0 msgr2=0x556745348590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:21.869 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f9f1b0 0x556745348590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/1517574700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f762f0 msgr2=0x55674534a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f762f0 0x55674534a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/1517574700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556744fb7210 con 0x556744f7e4d0 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.862+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f9f1b0 0x556745348590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x55674534a600 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ffa840cce30 tx=0x7ffa840042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da0c7da96bf98d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa75ffb640 1 -- 172.21.15.129:0/1517574700 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84097040 con 0x556744f7e4d0 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa75ffb640 1 -- 172.21.15.129:0/1517574700 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7ffa840a3de0 con 0x556744f7e4d0 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/1517574700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa88023070 con 0x556744f7e4d0 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa75ffb640 1 -- 172.21.15.129:0/1517574700 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ffa84009de0 con 0x556744f7e4d0 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa9bc7f740 10 client.17013 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/1517574700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556745083620 con 0x556744f7e4d0 2024-05-20T18:08:21.870 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa75ffb640 1 -- 172.21.15.129:0/1517574700 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa88023070 con 0x556744f7e4d0 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa75ffb640 1 -- 172.21.15.129:0/1517574700 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7ffa84084b40 con 0x556744f7e4d0 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa9bc7f740 10 client.17013 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa9bc7f740 10 client.17013 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa9bc7f740 20 client.17013 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556744f5dd80 0x55674535a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa9bc7f740 4 client.17013 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.863+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/1517574700 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567452b0a70 con 0x556744f5dd80 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.864+0000 7ffa8e59c640 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556744f5dd80 0x55674535a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.864+0000 7ffa75ffb640 10 client.17013 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.864+0000 7ffa8e59c640 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556744f5dd80 0x55674535a880 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ee7d417908e057b server_cookie=a5565c777c4a97d4 in_seq=0 out_seq=0 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa75ffb640 1 -- 172.21.15.129:0/1517574700 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5567452b0a70 con 0x556744f5dd80 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa75ffb640 10 client.17013 handle_command_reply: tid=0 2024-05-20T18:08:21.871 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 1 client.17013 shutdown 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/1517574700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 msgr2=0x55674534a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x55674534a600 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ffa840cce30 tx=0x7ffa840042d0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/1517574700 shutdown_connections 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556744f5dd80 0x55674535a880 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f9f1b0 0x556745348590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x55674534a600 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/1517574700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f762f0 0x55674534a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa75ffb640 0 client.17013 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/1517574700 >> 172.21.15.129:0/1517574700 conn(0x556744f93300 msgr2=0x55674533de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/1517574700 shutdown_connections 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/1517574700 wait complete. 2024-05-20T18:08:21.872 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.865+0000 7ffa9bc7f740 20 client.17013 trim_cache size 0 max 16384 2024-05-20T18:08:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.880+0000 7ffa9bc7f740 1 Processor -- start 2024-05-20T18:08:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.880+0000 7ffa9bc7f740 1 -- start start 2024-05-20T18:08:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 0x5567453474d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.879 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f9f1b0 0x556745360010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901acca0 con 0x556744f7e4d0 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901af500 con 0x556744f9f1b0 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901ad2f0 con 0x556744f762f0 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa8dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa8dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57522/0 (socket says 172.21.15.129:57522) 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/2486633828 learned_addr learned my addr 172.21.15.129:0/2486633828 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/2486633828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f9f1b0 0x556745360010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/2486633828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 msgr2=0x5567453474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/2486633828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 0x5567453474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/2486633828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f9f1b0 msgr2=0x556745360010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/2486633828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f9f1b0 0x556745360010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/2486633828 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556745346ae0 con 0x556744f7e4d0 2024-05-20T18:08:21.880 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/2486633828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453423e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ffa84097a60 tx=0x7ffa84004400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c73f4768f9dee3d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa77fff640 1 -- 172.21.15.129:0/2486633828 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84097040 con 0x556744f7e4d0 2024-05-20T18:08:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa77fff640 1 -- 172.21.15.129:0/2486633828 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa8409b600 con 0x556744f7e4d0 2024-05-20T18:08:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.881+0000 7ffa77fff640 1 -- 172.21.15.129:0/2486633828 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8409abf0 con 0x556744f7e4d0 2024-05-20T18:08:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.882+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/2486633828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 msgr2=0x5567453423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.882+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/2486633828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453423e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ffa84097a60 tx=0x7ffa84004400 comp rx=0 tx=0).stop 2024-05-20T18:08:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.882+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/2486633828 shutdown_connections 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.882+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/2486633828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f9f1b0 0x556745360010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.882+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/2486633828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453423e0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.882+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/2486633828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 0x5567453474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.882+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/2486633828 >> 172.21.15.129:0/2486633828 conn(0x556744f93300 msgr2=0x556745341cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.882+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/2486633828 shutdown_connections 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.882+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/2486633828 wait complete. 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.882+0000 7ffa9bc7f740 1 -- start start 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa9bc7f740 1 Processor -- start 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 0x5567453ba0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453ba630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa9bc7f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f9f1b0 0x55674534fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901ad9c0 con 0x556744f7e4d0 2024-05-20T18:08:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901aee30 con 0x556744f9f1b0 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa9bc7f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901ae090 con 0x556744f762f0 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa9bc7f740 10 client.0 _ll_register_callbacks cb 0x7ffa8400df10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453ba630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453ba630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57532/0 (socket says 172.21.15.129:57532) 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/544122757 learned_addr learned my addr 172.21.15.129:0/544122757 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f9f1b0 0x55674534fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8e59c640 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 0x5567453ba0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/544122757 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 msgr2=0x5567453ba0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 0x5567453ba0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/544122757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f9f1b0 msgr2=0x55674534fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f9f1b0 0x55674534fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8dd9b640 1 -- 172.21.15.129:0/544122757 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556745350250 con 0x556744f7e4d0 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8ed9d640 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f9f1b0 0x55674534fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa8dd9b640 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453ba630 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ffa84009610 tx=0x7ffa84097950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbcca3e12106e393 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.883+0000 7ffa75ffb640 1 -- 172.21.15.129:0/544122757 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84064070 con 0x556744f7e4d0 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa75ffb640 1 -- 172.21.15.129:0/544122757 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7ffa8409aa50 con 0x556744f7e4d0 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/544122757 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa8409b600 con 0x556744f7e4d0 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa75ffb640 1 -- 172.21.15.129:0/544122757 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ffa84097040 con 0x556744f7e4d0 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa9bc7f740 10 client.17019 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/544122757 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556745350250 con 0x556744f7e4d0 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa75ffb640 1 -- 172.21.15.129:0/544122757 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa8409b600 con 0x556744f7e4d0 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa75ffb640 1 -- 172.21.15.129:0/544122757 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7ffa84097350 con 0x556744f7e4d0 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa9bc7f740 10 client.17019 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa9bc7f740 10 client.17019 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa9bc7f740 20 client.17019 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556744f331c0 0x5567453c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa9bc7f740 4 client.17019 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:08:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/544122757 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5567452aadc0 con 0x556744f331c0 2024-05-20T18:08:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.884+0000 7ffa8e59c640 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556744f331c0 0x5567453c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.885+0000 7ffa75ffb640 10 client.17019 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.885+0000 7ffa8e59c640 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556744f331c0 0x5567453c4b40 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e645bfaf4143b3ec server_cookie=558de4c6d71754e1 in_seq=0 out_seq=0 2024-05-20T18:08:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.892+0000 7ffa75ffb640 1 -- 172.21.15.129:0/544122757 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5567452aadc0 con 0x556744f331c0 2024-05-20T18:08:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.892+0000 7ffa75ffb640 10 client.17019 handle_command_reply: tid=0 2024-05-20T18:08:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.892+0000 7ffa9bc7f740 1 client.17019 shutdown 2024-05-20T18:08:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/544122757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 msgr2=0x5567453ba630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453ba630 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ffa84009610 tx=0x7ffa84097950 comp rx=0 tx=0).stop 2024-05-20T18:08:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/544122757 shutdown_connections 2024-05-20T18:08:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x556744f331c0 0x5567453c4b40 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556744f9f1b0 0x55674534fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa75ffb640 0 client.17019 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556744f7e4d0 0x5567453ba630 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.891 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa9bc7f740 1 --2- 172.21.15.129:0/544122757 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556744f762f0 0x5567453ba0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/544122757 >> 172.21.15.129:0/544122757 conn(0x556744f93300 msgr2=0x5567453483c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/544122757 shutdown_connections 2024-05-20T18:08:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa9bc7f740 1 -- 172.21.15.129:0/544122757 wait complete. 2024-05-20T18:08:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa9bc7f740 20 client.17019 trim_cache size 0 max 16384 2024-05-20T18:08:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:08:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:08:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:08:21.892 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:08:21.893 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 -- 172.21.15.129:0/1324787721 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa780a2cf0 msgr2=0x7ffa780a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:21.893 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa780a2cf0 0x7ffa780a5200 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffa88022000 tx=0x7ffa88020000 comp rx=0 tx=0).stop 2024-05-20T18:08:21.893 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 -- 172.21.15.129:0/1324787721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa90108af0 msgr2=0x7ffa9019fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:21.893 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa90108af0 0x7ffa9019fa10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ffa84005fd0 tx=0x7ffa8400dde0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.893 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 -- 172.21.15.129:0/1324787721 shutdown_connections 2024-05-20T18:08:21.893 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffa780a2cf0 0x7ffa780a5200 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.893 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa90108f90 0x7ffa90199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa90108af0 0x7ffa9019fa10 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 --2- 172.21.15.129:0/1324787721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffa90103240 0x7ffa9019f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 -- 172.21.15.129:0/1324787721 >> 172.21.15.129:0/1324787721 conn(0x7ffa900fe1b0 msgr2=0x7ffa900fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 -- 172.21.15.129:0/1324787721 shutdown_connections 2024-05-20T18:08:21.894 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:21.893+0000 7ffa94dd5640 1 -- 172.21.15.129:0/1324787721 wait complete. 2024-05-20T18:08:23.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:23 smithi154 ceph-mon[36482]: pgmap v305: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-05-20T18:08:23.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:23 smithi129 ceph-mon[32029]: pgmap v305: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-05-20T18:08:23.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:23 smithi204 ceph-mon[37076]: pgmap v305: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-05-20T18:08:25.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:25 smithi154 ceph-mon[36482]: pgmap v306: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-05-20T18:08:25.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:25 smithi129 ceph-mon[32029]: pgmap v306: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-05-20T18:08:25.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:25 smithi204 ceph-mon[37076]: pgmap v306: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-05-20T18:08:27.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:27 smithi154 ceph-mon[36482]: pgmap v307: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-05-20T18:08:27.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:27 smithi129 ceph-mon[32029]: pgmap v307: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-05-20T18:08:27.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:27 smithi204 ceph-mon[37076]: pgmap v307: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-05-20T18:08:29.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:08:29.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:08:29.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.389+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4169533429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1cc103c20 msgr2=0x7fe1cc104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.389+0000 7fe1d26e9640 1 --2- 172.21.15.129:0/4169533429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1cc103c20 0x7fe1cc104030 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009ca0 tx=0x7fe1bc02a8c0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.389+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4169533429 shutdown_connections 2024-05-20T18:08:29.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.389+0000 7fe1d26e9640 1 --2- 172.21.15.129:0/4169533429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1cc10d070 0x7fe1cc10f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.389+0000 7fe1d26e9640 1 --2- 172.21.15.129:0/4169533429 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1cc10a680 0x7fe1cc10caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.389+0000 7fe1d26e9640 1 --2- 172.21.15.129:0/4169533429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1cc103c20 0x7fe1cc104030 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.389+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4169533429 >> 172.21.15.129:0/4169533429 conn(0x7fe1cc0770c0 msgr2=0x7fe1cc077500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.389+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4169533429 shutdown_connections 2024-05-20T18:08:29.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.389+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4169533429 wait complete. 2024-05-20T18:08:29.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.389+0000 7fe1d26e9640 1 Processor -- start 2024-05-20T18:08:29.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.389+0000 7fe1d26e9640 1 -- start start 2024-05-20T18:08:29.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1d26e9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1cc103c20 0x7fe1cc1997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1d26e9640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1cc10a680 0x7fe1cc199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1d26e9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1cc10d070 0x7fe1cc19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1d26e9640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc112de0 con 0x7fe1cc10a680 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1d26e9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc112c60 con 0x7fe1cc10d070 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1d26e9640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc112f60 con 0x7fe1cc103c20 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1cb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1cc10a680 0x7fe1cc199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1cb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1cc10a680 0x7fe1cc199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57552/0 (socket says 172.21.15.129:57552) 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1cb7fe640 1 -- 172.21.15.129:0/4198852016 learned_addr learned my addr 172.21.15.129:0/4198852016 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1d0c86640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1cc10d070 0x7fe1cc19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1cc103c20 0x7fe1cc1997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1cb7fe640 1 -- 172.21.15.129:0/4198852016 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1cc103c20 msgr2=0x7fe1cc1997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1cc103c20 0x7fe1cc1997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1cb7fe640 1 -- 172.21.15.129:0/4198852016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1cc10d070 msgr2=0x7fe1cc19dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1cc10d070 0x7fe1cc19dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1cb7fe640 1 -- 172.21.15.129:0/4198852016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1cc19e620 con 0x7fe1cc10a680 2024-05-20T18:08:29.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1cc103c20 0x7fe1cc1997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1d0c86640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1cc10d070 0x7fe1cc19dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1cc10a680 0x7fe1cc199d20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000d770 tx=0x7fe1c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.390+0000 7fe1c97fa640 1 -- 172.21.15.129:0/4198852016 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0017070 con 0x7fe1cc10a680 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.391+0000 7fe1c97fa640 1 -- 172.21.15.129:0/4198852016 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1c0009c30 con 0x7fe1cc10a680 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.391+0000 7fe1c97fa640 1 -- 172.21.15.129:0/4198852016 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c000f620 con 0x7fe1cc10a680 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.391+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4198852016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1cc19e910 con 0x7fe1cc10a680 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.391+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4198852016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1cc1a6290 con 0x7fe1cc10a680 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.393+0000 7fe1c97fa640 1 -- 172.21.15.129:0/4198852016 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fe1c001a070 con 0x7fe1cc10a680 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.393+0000 7fe1d9591740 1 Processor -- start 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.393+0000 7fe1d9591740 1 -- start start 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.393+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434f4390 0x557a43892d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.393+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43893270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434d36b0 0x557a438937b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1d9591740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557a437f8d00 con 0x557a434cb4d0 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1d9591740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557a437a04c0 con 0x557a434d36b0 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1d9591740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557a437f4cc0 con 0x557a434f4390 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1cbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43893270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1c97fa640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe1b40a2d80 0x7fe1b40a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1cbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43893270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57566/0 (socket says 172.21.15.129:57566) 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1cbfff640 1 -- 172.21.15.129:0/3793833180 learned_addr learned my addr 172.21.15.129:0/3793833180 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/3793833180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434d36b0 0x557a438937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1d0c86640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe1b40a2d80 0x7fe1b40a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1d0c86640 1 --2- 172.21.15.129:0/3793833180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434f4390 0x557a43892d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1c97fa640 1 -- 172.21.15.129:0/4198852016 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fe1c00cefd0 con 0x7fe1cc10a680 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1cbfff640 1 -- 172.21.15.129:0/3793833180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434f4390 msgr2=0x557a43892d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/3793833180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434f4390 0x557a43892d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1cbfff640 1 -- 172.21.15.129:0/3793833180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434d36b0 msgr2=0x557a438937b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/3793833180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434d36b0 0x557a438937b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1cbfff640 1 -- 172.21.15.129:0/3793833180 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a435d67d0 con 0x557a434cb4d0 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/3793833180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434d36b0 0x557a438937b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/3793833180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43893270 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009490 tx=0x7fe1bc009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed94596e16a8e9a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1d0c86640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe1b40a2d80 0x7fe1b40a5290 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x557a4388fea0 tx=0x7fe1c4009530 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.394+0000 7fe1b27fc640 1 -- 172.21.15.129:0/3793833180 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc031db0 con 0x557a434cb4d0 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.395+0000 7fe1b27fc640 1 -- 172.21.15.129:0/3793833180 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1bc036070 con 0x557a434cb4d0 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.395+0000 7fe1b27fc640 1 -- 172.21.15.129:0/3793833180 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc004330 con 0x557a434cb4d0 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.395+0000 7fe1d9591740 1 -- 172.21.15.129:0/3793833180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 msgr2=0x557a43893270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.395+0000 7fe1d9591740 1 --2- 172.21.15.129:0/3793833180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43893270 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009490 tx=0x7fe1bc009800 comp rx=0 tx=0).stop 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.395+0000 7fe1d9591740 1 -- 172.21.15.129:0/3793833180 shutdown_connections 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.395+0000 7fe1d9591740 1 --2- 172.21.15.129:0/3793833180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434d36b0 0x557a438937b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.395+0000 7fe1d9591740 1 --2- 172.21.15.129:0/3793833180 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43893270 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.395+0000 7fe1d9591740 1 --2- 172.21.15.129:0/3793833180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434f4390 0x557a43892d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.395+0000 7fe1d9591740 1 -- 172.21.15.129:0/3793833180 >> 172.21.15.129:0/3793833180 conn(0x557a434e84e0 msgr2=0x557a4388f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.395+0000 7fe1d9591740 1 -- 172.21.15.129:0/3793833180 shutdown_connections 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1d9591740 1 -- 172.21.15.129:0/3793833180 wait complete. 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1d9591740 1 -- start start 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1d9591740 1 Processor -- start 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434cb4d0 0x557a4389eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434d36b0 0x557a4389f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a4389d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1d9591740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557a4389ffc0 con 0x557a434d36b0 2024-05-20T18:08:29.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1d9591740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557a437b3b90 con 0x557a434f4390 2024-05-20T18:08:29.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1d9591740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557a438a0140 con 0x557a434cb4d0 2024-05-20T18:08:29.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1d9591740 10 client.0 _ll_register_callbacks cb 0x557a437efda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:29.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1cb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434d36b0 0x557a4389f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1cb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434d36b0 0x557a4389f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57572/0 (socket says 172.21.15.129:57572) 2024-05-20T18:08:29.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1cb7fe640 1 -- 172.21.15.129:0/2182807501 learned_addr learned my addr 172.21.15.129:0/2182807501 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:29.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1d0c86640 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a4389d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.396+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434cb4d0 0x557a4389eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1cb7fe640 1 -- 172.21.15.129:0/2182807501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434cb4d0 msgr2=0x557a4389eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434cb4d0 0x557a4389eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1cb7fe640 1 -- 172.21.15.129:0/2182807501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 msgr2=0x557a4389d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a4389d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1cb7fe640 1 -- 172.21.15.129:0/2182807501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a4350c3f0 con 0x557a434d36b0 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434d36b0 0x557a4389f420 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fe1c009b1b0 tx=0x7fe1c0021c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59624efe24d8735a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2182807501 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0021040 con 0x557a434d36b0 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2182807501 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fe1c0011030 con 0x557a434d36b0 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1d9591740 1 -- 172.21.15.129:0/2182807501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1bc036070 con 0x557a434d36b0 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2182807501 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe1c0009de0 con 0x557a434d36b0 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1d9591740 10 client.17031 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1d9591740 1 -- 172.21.15.129:0/2182807501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557a434cc0f0 con 0x557a434d36b0 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2182807501 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1bc036070 con 0x557a434d36b0 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.397+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2182807501 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fe1c0021ca0 con 0x557a434d36b0 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.398+0000 7fe1d9591740 10 client.17031 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:29.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.398+0000 7fe1d9591740 10 client.17031 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.398+0000 7fe1d9591740 20 client.17031 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.398+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a434b2f60 0x557a438afb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.398+0000 7fe1d9591740 4 client.17031 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.398+0000 7fe1d9591740 1 -- 172.21.15.129:0/2182807501 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a43806430 con 0x557a434b2f60 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.398+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a434b2f60 0x557a438afb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.398+0000 7fe1ab7fe640 10 client.17031 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.398+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a434b2f60 0x557a438afb00 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23930ef5892a9b41 server_cookie=7522a5ca96ddd66 in_seq=0 out_seq=0 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2182807501 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557a43806430 con 0x557a434b2f60 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1ab7fe640 10 client.17031 handle_command_reply: tid=0 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 1 client.17031 shutdown 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 1 -- 172.21.15.129:0/2182807501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434d36b0 msgr2=0x557a4389f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434d36b0 0x557a4389f420 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fe1c009b1b0 tx=0x7fe1c0021c30 comp rx=0 tx=0).stop 2024-05-20T18:08:29.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 1 -- 172.21.15.129:0/2182807501 shutdown_connections 2024-05-20T18:08:29.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a434b2f60 0x557a438afb00 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a4389d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434d36b0 0x557a4389f420 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1ab7fe640 0 client.17031 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:29.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2182807501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434cb4d0 0x557a4389eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 1 -- 172.21.15.129:0/2182807501 >> 172.21.15.129:0/2182807501 conn(0x557a434e84e0 msgr2=0x557a43894a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 1 -- 172.21.15.129:0/2182807501 shutdown_connections 2024-05-20T18:08:29.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 1 -- 172.21.15.129:0/2182807501 wait complete. 2024-05-20T18:08:29.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.399+0000 7fe1d9591740 20 client.17031 trim_cache size 0 max 16384 2024-05-20T18:08:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.417+0000 7fe1d9591740 1 Processor -- start 2024-05-20T18:08:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.417+0000 7fe1d9591740 1 -- start start 2024-05-20T18:08:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.417+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43894eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.417+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434d36b0 0x557a438b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.417+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a438b5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.417+0000 7fe1d9591740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1acc80 con 0x557a434cb4d0 2024-05-20T18:08:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.417+0000 7fe1d9591740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1af4e0 con 0x557a434f4390 2024-05-20T18:08:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.417+0000 7fe1d9591740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1ad2d0 con 0x557a434d36b0 2024-05-20T18:08:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.417+0000 7fe1cbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43894eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.417+0000 7fe1cbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43894eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57574/0 (socket says 172.21.15.129:57574) 2024-05-20T18:08:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1cbfff640 1 -- 172.21.15.129:0/3755434545 learned_addr learned my addr 172.21.15.129:0/3755434545 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1d0c86640 1 --2- 172.21.15.129:0/3755434545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a438b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/3755434545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434d36b0 0x557a438b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1cbfff640 1 -- 172.21.15.129:0/3755434545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434d36b0 msgr2=0x557a438b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/3755434545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434d36b0 0x557a438b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1cbfff640 1 -- 172.21.15.129:0/3755434545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 msgr2=0x557a438b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/3755434545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a438b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1cbfff640 1 -- 172.21.15.129:0/3755434545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a43880990 con 0x557a434cb4d0 2024-05-20T18:08:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1d0c86640 1 --2- 172.21.15.129:0/3755434545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a438b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/3755434545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43894eb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009870 tx=0x7fe1bc031a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc21a7a4ba8704ed server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1b1ffb640 1 -- 172.21.15.129:0/3755434545 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc055070 con 0x557a434cb4d0 2024-05-20T18:08:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1b1ffb640 1 -- 172.21.15.129:0/3755434545 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1bc036050 con 0x557a434cb4d0 2024-05-20T18:08:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.418+0000 7fe1b1ffb640 1 -- 172.21.15.129:0/3755434545 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc049040 con 0x557a434cb4d0 2024-05-20T18:08:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.419+0000 7fe1d9591740 1 -- 172.21.15.129:0/3755434545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 msgr2=0x557a43894eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.419+0000 7fe1d9591740 1 --2- 172.21.15.129:0/3755434545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43894eb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009870 tx=0x7fe1bc031a10 comp rx=0 tx=0).stop 2024-05-20T18:08:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.419+0000 7fe1d9591740 1 -- 172.21.15.129:0/3755434545 shutdown_connections 2024-05-20T18:08:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.419+0000 7fe1d9591740 1 --2- 172.21.15.129:0/3755434545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a438b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.419+0000 7fe1d9591740 1 --2- 172.21.15.129:0/3755434545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434d36b0 0x557a438b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.419+0000 7fe1d9591740 1 --2- 172.21.15.129:0/3755434545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a43894eb0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.419+0000 7fe1d9591740 1 -- 172.21.15.129:0/3755434545 >> 172.21.15.129:0/3755434545 conn(0x557a434e84e0 msgr2=0x557a438934f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.419+0000 7fe1d9591740 1 -- 172.21.15.129:0/3755434545 shutdown_connections 2024-05-20T18:08:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.419+0000 7fe1d9591740 1 -- 172.21.15.129:0/3755434545 wait complete. 2024-05-20T18:08:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.419+0000 7fe1d9591740 1 -- start start 2024-05-20T18:08:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.419+0000 7fe1d9591740 1 Processor -- start 2024-05-20T18:08:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a438a9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434d36b0 0x557a438a6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d9591740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a438a7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d9591740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1ad9a0 con 0x557a434cb4d0 2024-05-20T18:08:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d9591740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1aee10 con 0x557a434f4390 2024-05-20T18:08:29.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d9591740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe1cc1ae070 con 0x557a434d36b0 2024-05-20T18:08:29.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d9591740 10 client.0 _ll_register_callbacks cb 0x7fe1c0017f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:29.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1cbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a438a9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1cbfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a438a9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57590/0 (socket says 172.21.15.129:57590) 2024-05-20T18:08:29.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1cbfff640 1 -- 172.21.15.129:0/2245031957 learned_addr learned my addr 172.21.15.129:0/2245031957 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:29.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434d36b0 0x557a438a6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d0c86640 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a438a7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1cbfff640 1 -- 172.21.15.129:0/2245031957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434d36b0 msgr2=0x557a438a6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434d36b0 0x557a438a6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1cbfff640 1 -- 172.21.15.129:0/2245031957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 msgr2=0x557a438a7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a438a7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1cbfff640 1 -- 172.21.15.129:0/2245031957 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a438a7970 con 0x557a434cb4d0 2024-05-20T18:08:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d0c86640 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a438a7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1cbfff640 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a438a9b80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc02a900 tx=0x7fe1bc02f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc9e9541ac5fa8 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2245031957 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc055070 con 0x557a434cb4d0 2024-05-20T18:08:29.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2245031957 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fe1bc031650 con 0x557a434cb4d0 2024-05-20T18:08:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d9591740 1 -- 172.21.15.129:0/2245031957 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1bc036050 con 0x557a434cb4d0 2024-05-20T18:08:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2245031957 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe1bc004b40 con 0x557a434cb4d0 2024-05-20T18:08:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d9591740 10 client.17037 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.420+0000 7fe1d9591740 1 -- 172.21.15.129:0/2245031957 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557a43917a40 con 0x557a434cb4d0 2024-05-20T18:08:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.421+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2245031957 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1bc036050 con 0x557a434cb4d0 2024-05-20T18:08:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.421+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2245031957 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fe1bc0045b0 con 0x557a434cb4d0 2024-05-20T18:08:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.421+0000 7fe1d9591740 10 client.17037 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.421+0000 7fe1d9591740 10 client.17037 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:29.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.421+0000 7fe1d9591740 20 client.17037 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:29.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.421+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a434883a0 0x557a43919d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.421+0000 7fe1d9591740 4 client.17037 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:08:29.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.421+0000 7fe1d9591740 1 -- 172.21.15.129:0/2245031957 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557a43800780 con 0x557a434883a0 2024-05-20T18:08:29.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.421+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a434883a0 0x557a43919d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.421+0000 7fe1ab7fe640 10 client.17037 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:29.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.421+0000 7fe1cb7fe640 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a434883a0 0x557a43919d20 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e4ecbec5521bb13 server_cookie=47b57cac19435ddf in_seq=0 out_seq=0 2024-05-20T18:08:29.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1ab7fe640 1 -- 172.21.15.129:0/2245031957 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557a43800780 con 0x557a434883a0 2024-05-20T18:08:29.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1ab7fe640 10 client.17037 handle_command_reply: tid=0 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 1 client.17037 shutdown 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 1 -- 172.21.15.129:0/2245031957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 msgr2=0x557a438a9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a438a9b80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc02a900 tx=0x7fe1bc02f430 comp rx=0 tx=0).stop 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 1 -- 172.21.15.129:0/2245031957 shutdown_connections 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x557a434883a0 0x557a43919d20 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a434f4390 0x557a438a7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557a434d36b0 0x557a438a6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 1 --2- 172.21.15.129:0/2245031957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a434cb4d0 0x557a438a9b80 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1ab7fe640 0 client.17037 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 1 -- 172.21.15.129:0/2245031957 >> 172.21.15.129:0/2245031957 conn(0x557a434e84e0 msgr2=0x557a438934f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 1 -- 172.21.15.129:0/2245031957 shutdown_connections 2024-05-20T18:08:29.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 1 -- 172.21.15.129:0/2245031957 wait complete. 2024-05-20T18:08:29.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d9591740 20 client.17037 trim_cache size 0 max 16384 2024-05-20T18:08:29.430 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:08:29.430 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:08:29.430 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:08:29.430 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:08:29.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4198852016 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe1b40a2d80 msgr2=0x7fe1b40a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d26e9640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe1b40a2d80 0x7fe1b40a5290 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x557a4388fea0 tx=0x7fe1c4009530 comp rx=0 tx=0).stop 2024-05-20T18:08:29.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4198852016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1cc10a680 msgr2=0x7fe1cc199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.422+0000 7fe1d26e9640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1cc10a680 0x7fe1cc199d20 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000d770 tx=0x7fe1c000dc30 comp rx=0 tx=0).stop 2024-05-20T18:08:29.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.423+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4198852016 shutdown_connections 2024-05-20T18:08:29.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.423+0000 7fe1d26e9640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe1b40a2d80 0x7fe1b40a5290 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.423+0000 7fe1d26e9640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1cc10d070 0x7fe1cc19dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.423+0000 7fe1d26e9640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1cc10a680 0x7fe1cc199d20 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.423+0000 7fe1d26e9640 1 --2- 172.21.15.129:0/4198852016 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe1cc103c20 0x7fe1cc1997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.423+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4198852016 >> 172.21.15.129:0/4198852016 conn(0x7fe1cc0770c0 msgr2=0x7fe1cc100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.423+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4198852016 shutdown_connections 2024-05-20T18:08:29.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.423+0000 7fe1d26e9640 1 -- 172.21.15.129:0/4198852016 wait complete. 2024-05-20T18:08:29.435 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:08:29.435 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:08:29.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.549+0000 7fba6e606640 1 -- 172.21.15.129:0/1716411881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba6810c340 msgr2=0x7fba6810e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.549+0000 7fba6e606640 1 --2- 172.21.15.129:0/1716411881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba6810c340 0x7fba6810e760 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fba60009ca0 tx=0x7fba6002a8c0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.549+0000 7fba6e606640 1 -- 172.21.15.129:0/1716411881 shutdown_connections 2024-05-20T18:08:29.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.549+0000 7fba6e606640 1 --2- 172.21.15.129:0/1716411881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba6810c340 0x7fba6810e760 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.549+0000 7fba6e606640 1 --2- 172.21.15.129:0/1716411881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba68101ab0 0x7fba68101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.549+0000 7fba6e606640 1 --2- 172.21.15.129:0/1716411881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba681079e0 0x7fba68101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.549+0000 7fba6e606640 1 -- 172.21.15.129:0/1716411881 >> 172.21.15.129:0/1716411881 conn(0x7fba680fd030 msgr2=0x7fba680ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.549+0000 7fba6e606640 1 -- 172.21.15.129:0/1716411881 shutdown_connections 2024-05-20T18:08:29.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.549+0000 7fba6e606640 1 -- 172.21.15.129:0/1716411881 wait complete. 2024-05-20T18:08:29.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.550+0000 7fba6e606640 1 Processor -- start 2024-05-20T18:08:29.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.550+0000 7fba6e606640 1 -- start start 2024-05-20T18:08:29.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.550+0000 7fba6e606640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba68101ab0 0x7fba681985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.550+0000 7fba6e606640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba681079e0 0x7fba68198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.550+0000 7fba6e606640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba6810c340 0x7fba6819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.550+0000 7fba6e606640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fba68111cf0 con 0x7fba681079e0 2024-05-20T18:08:29.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.550+0000 7fba6e606640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fba68111b70 con 0x7fba68101ab0 2024-05-20T18:08:29.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.550+0000 7fba6e606640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fba68111e70 con 0x7fba6810c340 2024-05-20T18:08:29.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba677fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba681079e0 0x7fba68198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba677fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba681079e0 0x7fba68198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57598/0 (socket says 172.21.15.129:57598) 2024-05-20T18:08:29.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba677fe640 1 -- 172.21.15.129:0/1380947144 learned_addr learned my addr 172.21.15.129:0/1380947144 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba67fff640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba68101ab0 0x7fba681985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba6cba3640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba6810c340 0x7fba6819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba677fe640 1 -- 172.21.15.129:0/1380947144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba6810c340 msgr2=0x7fba6819ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba677fe640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba6810c340 0x7fba6819ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba677fe640 1 -- 172.21.15.129:0/1380947144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba68101ab0 msgr2=0x7fba681985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba677fe640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba68101ab0 0x7fba681985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba677fe640 1 -- 172.21.15.129:0/1380947144 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6819d3d0 con 0x7fba681079e0 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba677fe640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba681079e0 0x7fba68198b50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fba5c00c9a0 tx=0x7fba5c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba657fa640 1 -- 172.21.15.129:0/1380947144 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c007c70 con 0x7fba681079e0 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba657fa640 1 -- 172.21.15.129:0/1380947144 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba5c007e10 con 0x7fba681079e0 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba657fa640 1 -- 172.21.15.129:0/1380947144 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c018960 con 0x7fba681079e0 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba6e606640 1 -- 172.21.15.129:0/1380947144 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba6819d6c0 con 0x7fba681079e0 2024-05-20T18:08:29.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.551+0000 7fba6e606640 1 -- 172.21.15.129:0/1380947144 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba6819dc10 con 0x7fba681079e0 2024-05-20T18:08:29.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.552+0000 7fba6e606640 1 -- 172.21.15.129:0/1380947144 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba68069140 con 0x7fba681079e0 2024-05-20T18:08:29.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.554+0000 7fba657fa640 1 -- 172.21.15.129:0/1380947144 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fba5c018b00 con 0x7fba681079e0 2024-05-20T18:08:29.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.554+0000 7fba657fa640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fba500a2d00 0x7fba500a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.554+0000 7fba67fff640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fba500a2d00 0x7fba500a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.554+0000 7fba657fa640 1 -- 172.21.15.129:0/1380947144 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fba5c0cf4b0 con 0x7fba681079e0 2024-05-20T18:08:29.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.555+0000 7fba67fff640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fba500a2d00 0x7fba500a5210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fba681029e0 tx=0x7fba58005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.559+0000 7fba657fa640 1 -- 172.21.15.129:0/1380947144 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fba5c014030 con 0x7fba681079e0 2024-05-20T18:08:29.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:29 smithi154 ceph-mon[36482]: pgmap v308: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-05-20T18:08:29.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:29 smithi129 ceph-mon[32029]: pgmap v308: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-05-20T18:08:29.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:29 smithi204 ceph-mon[37076]: pgmap v308: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-05-20T18:08:29.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.773+0000 7fba6e606640 1 -- 172.21.15.129:0/1380947144 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba68199940 con 0x7fba681079e0 2024-05-20T18:08:29.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.774+0000 7fba657fa640 1 -- 172.21.15.129:0/1380947144 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11829 (secure 0 0 0) 0x7fba5c09c090 con 0x7fba681079e0 2024-05-20T18:08:29.772 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:08:29.772 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":26255,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/2724397229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":2724397229},{"type":"v1","addr":"172.21.15.154:6837","nonce":2724397229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:06:10.247052+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:08:29.772 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-05-20T18:08:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 -- 172.21.15.129:0/1380947144 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fba500a2d00 msgr2=0x7fba500a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fba500a2d00 0x7fba500a5210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fba681029e0 tx=0x7fba58005f90 comp rx=0 tx=0).stop 2024-05-20T18:08:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 -- 172.21.15.129:0/1380947144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba681079e0 msgr2=0x7fba68198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba681079e0 0x7fba68198b50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fba5c00c9a0 tx=0x7fba5c00ce60 comp rx=0 tx=0).stop 2024-05-20T18:08:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 -- 172.21.15.129:0/1380947144 shutdown_connections 2024-05-20T18:08:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fba500a2d00 0x7fba500a5210 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fba6810c340 0x7fba6819ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba681079e0 0x7fba68198b50 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 --2- 172.21.15.129:0/1380947144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba68101ab0 0x7fba681985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 -- 172.21.15.129:0/1380947144 >> 172.21.15.129:0/1380947144 conn(0x7fba680fd030 msgr2=0x7fba6810d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 -- 172.21.15.129:0/1380947144 shutdown_connections 2024-05-20T18:08:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.776+0000 7fba6e606640 1 -- 172.21.15.129:0/1380947144 wait complete. 2024-05-20T18:08:29.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:08:29.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.902+0000 7fc030537640 1 -- 172.21.15.129:0/590854013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc02810c330 msgr2=0x7fc02810e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.902+0000 7fc030537640 1 --2- 172.21.15.129:0/590854013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc02810c330 0x7fc02810e750 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc018009900 tx=0x7fc01802a410 comp rx=0 tx=0).stop 2024-05-20T18:08:29.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.902+0000 7fc030537640 1 -- 172.21.15.129:0/590854013 shutdown_connections 2024-05-20T18:08:29.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.902+0000 7fc030537640 1 --2- 172.21.15.129:0/590854013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc02810c330 0x7fc02810e750 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.902+0000 7fc030537640 1 --2- 172.21.15.129:0/590854013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc028101aa0 0x7fc028101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.902+0000 7fc030537640 1 --2- 172.21.15.129:0/590854013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0281079d0 0x7fc028101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.902+0000 7fc030537640 1 -- 172.21.15.129:0/590854013 >> 172.21.15.129:0/590854013 conn(0x7fc0280770b0 msgr2=0x7fc0280774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.902+0000 7fc030537640 1 -- 172.21.15.129:0/590854013 shutdown_connections 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.902+0000 7fc030537640 1 -- 172.21.15.129:0/590854013 wait complete. 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc030537640 1 Processor -- start 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc030537640 1 -- start start 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc030537640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc028101aa0 0x7fc0281996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc030537640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0281079d0 0x7fc028199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc030537640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc02810c330 0x7fc02819df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc030537640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc028111d00 con 0x7fc0281079d0 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc030537640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc028111b80 con 0x7fc028101aa0 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc030537640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc028111e80 con 0x7fc02810c330 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc02dad2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0281079d0 0x7fc028199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc02dad2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0281079d0 0x7fc028199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33394/0 (socket says 172.21.15.129:33394) 2024-05-20T18:08:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc02dad2640 1 -- 172.21.15.129:0/3623098445 learned_addr learned my addr 172.21.15.129:0/3623098445 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc02e2d3640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc028101aa0 0x7fc0281996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.903+0000 7fc02ead4640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc02810c330 0x7fc02819df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.904+0000 7fc02dad2640 1 -- 172.21.15.129:0/3623098445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc02810c330 msgr2=0x7fc02819df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.904+0000 7fc02dad2640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc02810c330 0x7fc02819df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.904+0000 7fc02dad2640 1 -- 172.21.15.129:0/3623098445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc028101aa0 msgr2=0x7fc0281996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.904+0000 7fc02dad2640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc028101aa0 0x7fc0281996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.904+0000 7fc02dad2640 1 -- 172.21.15.129:0/3623098445 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02819e550 con 0x7fc0281079d0 2024-05-20T18:08:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.904+0000 7fc02dad2640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0281079d0 0x7fc028199c50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00ead0 tx=0x7fc01c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.904+0000 7fc0277fe640 1 -- 172.21.15.129:0/3623098445 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c00cde0 con 0x7fc0281079d0 2024-05-20T18:08:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.904+0000 7fc030537640 1 -- 172.21.15.129:0/3623098445 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc02819e840 con 0x7fc0281079d0 2024-05-20T18:08:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.904+0000 7fc0277fe640 1 -- 172.21.15.129:0/3623098445 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc01c003e50 con 0x7fc0281079d0 2024-05-20T18:08:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.904+0000 7fc0277fe640 1 -- 172.21.15.129:0/3623098445 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c0116b0 con 0x7fc0281079d0 2024-05-20T18:08:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.904+0000 7fc030537640 1 -- 172.21.15.129:0/3623098445 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc02819ed40 con 0x7fc0281079d0 2024-05-20T18:08:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc0277fe640 1 -- 172.21.15.129:0/3623098445 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc01c003970 con 0x7fc0281079d0 2024-05-20T18:08:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc0373db740 1 Processor -- start 2024-05-20T18:08:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc0373db740 1 -- start start 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc0373db740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9988390 0x55f4c9d26bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc0373db740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d270f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc0373db740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d27630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc0373db740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f4c9c8cc40 con 0x55f4c99676b0 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc0373db740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f4c9c34580 con 0x55f4c9988390 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc0373db740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f4c9c88c00 con 0x55f4c995f4d0 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc02dad2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d27630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc02dad2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d27630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33398/0 (socket says 172.21.15.129:33398) 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc02dad2640 1 -- 172.21.15.129:0/2153853131 learned_addr learned my addr 172.21.15.129:0/2153853131 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc02ead4640 1 --2- 172.21.15.129:0/2153853131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9988390 0x55f4c9d26bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.907+0000 7fc02e2d3640 1 --2- 172.21.15.129:0/2153853131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d270f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc0277fe640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0100a2d00 0x7fc0100a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc02dad2640 1 -- 172.21.15.129:0/2153853131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 msgr2=0x55f4c9d270f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc02dad2640 1 --2- 172.21.15.129:0/2153853131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d270f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc02dad2640 1 -- 172.21.15.129:0/2153853131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9988390 msgr2=0x55f4c9d26bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc02dad2640 1 --2- 172.21.15.129:0/2153853131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9988390 0x55f4c9d26bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc0277fe640 1 -- 172.21.15.129:0/3623098445 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fc01c0cfa40 con 0x7fc0281079d0 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc02dad2640 1 -- 172.21.15.129:0/2153853131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4c99a03f0 con 0x55f4c99676b0 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc02ead4640 1 --2- 172.21.15.129:0/2153853131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9988390 0x55f4c9d26bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc02ead4640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0100a2d00 0x7fc0100a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc02dad2640 1 --2- 172.21.15.129:0/2153853131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d27630 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc01c0cd0c0 tx=0x7fc01c0046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dca0ff6e2965f62 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc024bfe640 1 -- 172.21.15.129:0/2153853131 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c09ba60 con 0x55f4c99676b0 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc024bfe640 1 -- 172.21.15.129:0/2153853131 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc01c09c620 con 0x55f4c99676b0 2024-05-20T18:08:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.908+0000 7fc024bfe640 1 -- 172.21.15.129:0/2153853131 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c09b880 con 0x55f4c99676b0 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.909+0000 7fc0373db740 1 -- 172.21.15.129:0/2153853131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 msgr2=0x55f4c9d27630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.909+0000 7fc0373db740 1 --2- 172.21.15.129:0/2153853131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d27630 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc01c0cd0c0 tx=0x7fc01c0046e0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.909+0000 7fc02ead4640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0100a2d00 0x7fc0100a5210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x55f4c9d23ec0 tx=0x7fc018009260 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.909+0000 7fc0373db740 1 -- 172.21.15.129:0/2153853131 shutdown_connections 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.909+0000 7fc0373db740 1 --2- 172.21.15.129:0/2153853131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d27630 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.909+0000 7fc0373db740 1 --2- 172.21.15.129:0/2153853131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d270f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.909+0000 7fc0373db740 1 --2- 172.21.15.129:0/2153853131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9988390 0x55f4c9d26bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.909+0000 7fc0373db740 1 -- 172.21.15.129:0/2153853131 >> 172.21.15.129:0/2153853131 conn(0x55f4c997c4e0 msgr2=0x55f4c9d234d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.909+0000 7fc0373db740 1 -- 172.21.15.129:0/2153853131 shutdown_connections 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.909+0000 7fc0373db740 1 -- 172.21.15.129:0/2153853131 wait complete. 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc0373db740 1 -- start start 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc0373db740 1 Processor -- start 2024-05-20T18:08:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc0373db740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c995f4d0 0x55f4c9d35780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc0373db740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c99676b0 0x55f4c9d35cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc0373db740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9988390 0x55f4c9d3b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc0373db740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f4c9d38420 con 0x55f4c995f4d0 2024-05-20T18:08:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc0373db740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f4c9c47c40 con 0x55f4c9988390 2024-05-20T18:08:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc0373db740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f4c9d385a0 con 0x55f4c99676b0 2024-05-20T18:08:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc0373db740 10 client.0 _ll_register_callbacks cb 0x55f4c999fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc02e2d3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c995f4d0 0x55f4c9d35780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc02e2d3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c995f4d0 0x55f4c9d35780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33410/0 (socket says 172.21.15.129:33410) 2024-05-20T18:08:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc02e2d3640 1 -- 172.21.15.129:0/3218703181 learned_addr learned my addr 172.21.15.129:0/3218703181 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc02ead4640 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9988390 0x55f4c9d3b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc02dad2640 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c99676b0 0x55f4c9d35cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc02e2d3640 1 -- 172.21.15.129:0/3218703181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c99676b0 msgr2=0x55f4c9d35cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc02e2d3640 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c99676b0 0x55f4c9d35cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc02e2d3640 1 -- 172.21.15.129:0/3218703181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9988390 msgr2=0x55f4c9d3b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.910+0000 7fc02e2d3640 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9988390 0x55f4c9d3b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc02e2d3640 1 -- 172.21.15.129:0/3218703181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4c9a6a7d0 con 0x55f4c995f4d0 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc02e2d3640 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c995f4d0 0x55f4c9d35780 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc0200126e0 tx=0x7fc020012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612a3974b19ce516 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc00dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc00dffb640 1 -- 172.21.15.129:0/3218703181 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc020012190 con 0x55f4c995f4d0 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc00dffb640 1 -- 172.21.15.129:0/3218703181 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fc02000f920 con 0x55f4c995f4d0 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc0373db740 1 -- 172.21.15.129:0/3218703181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc01c09c620 con 0x55f4c995f4d0 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc00dffb640 1 -- 172.21.15.129:0/3218703181 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc02000fb20 con 0x55f4c995f4d0 2024-05-20T18:08:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc0373db740 10 client.17055 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc0373db740 1 -- 172.21.15.129:0/3218703181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f4c9d44bb0 con 0x55f4c995f4d0 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc00dffb640 1 -- 172.21.15.129:0/3218703181 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc01c09c620 con 0x55f4c995f4d0 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc00dffb640 1 -- 172.21.15.129:0/3218703181 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fc020022030 con 0x55f4c995f4d0 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc0373db740 10 client.17055 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.911+0000 7fc0373db740 10 client.17055 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.912+0000 7fc0373db740 20 client.17055 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.912+0000 7fc0373db740 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f4c9946f60 0x55f4c9d48180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.912+0000 7fc0373db740 4 client.17055 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.912+0000 7fc0373db740 1 -- 172.21.15.129:0/3218703181 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4c9c9a370 con 0x55f4c9946f60 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.912+0000 7fc02dad2640 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f4c9946f60 0x55f4c9d48180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.912+0000 7fc02dad2640 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f4c9946f60 0x55f4c9d48180 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a385baa2b348466 server_cookie=f2c9b4e365e1fc3 in_seq=0 out_seq=0 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.912+0000 7fc00dffb640 10 client.17055 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc00dffb640 1 -- 172.21.15.129:0/3218703181 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f4c9c9a370 con 0x55f4c9946f60 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc00dffb640 10 client.17055 handle_command_reply: tid=0 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 1 client.17055 shutdown 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 1 -- 172.21.15.129:0/3218703181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c995f4d0 msgr2=0x55f4c9d35780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c995f4d0 0x55f4c9d35780 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc0200126e0 tx=0x7fc020012ba0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 1 -- 172.21.15.129:0/3218703181 shutdown_connections 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f4c9946f60 0x55f4c9d48180 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9988390 0x55f4c9d3b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c99676b0 0x55f4c9d35cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc00dffb640 0 client.17055 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 1 --2- 172.21.15.129:0/3218703181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c995f4d0 0x55f4c9d35780 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 1 -- 172.21.15.129:0/3218703181 >> 172.21.15.129:0/3218703181 conn(0x55f4c997c4e0 msgr2=0x55f4c9d28a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 1 -- 172.21.15.129:0/3218703181 shutdown_connections 2024-05-20T18:08:29.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 1 -- 172.21.15.129:0/3218703181 wait complete. 2024-05-20T18:08:29.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.913+0000 7fc0373db740 20 client.17055 trim_cache size 0 max 16384 2024-05-20T18:08:29.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc0373db740 1 Processor -- start 2024-05-20T18:08:29.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc0373db740 1 -- start start 2024-05-20T18:08:29.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc0373db740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9946f60 0x55f4c9d34870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc0373db740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d3b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc0373db740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d373a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc0373db740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0281aa780 con 0x55f4c99676b0 2024-05-20T18:08:29.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc0373db740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0281acfe0 con 0x55f4c9946f60 2024-05-20T18:08:29.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc0373db740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc0281aadd0 con 0x55f4c995f4d0 2024-05-20T18:08:29.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc02ead4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d373a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc02ead4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d373a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33416/0 (socket says 172.21.15.129:33416) 2024-05-20T18:08:29.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc02ead4640 1 -- 172.21.15.129:0/3424222506 learned_addr learned my addr 172.21.15.129:0/3424222506 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc02e2d3640 1 --2- 172.21.15.129:0/3424222506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9946f60 0x55f4c9d34870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc02dad2640 1 --2- 172.21.15.129:0/3424222506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d3b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc02ead4640 1 -- 172.21.15.129:0/3424222506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 msgr2=0x55f4c9d3b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc02ead4640 1 --2- 172.21.15.129:0/3424222506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d3b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc02ead4640 1 -- 172.21.15.129:0/3424222506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9946f60 msgr2=0x55f4c9d34870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc02ead4640 1 --2- 172.21.15.129:0/3424222506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9946f60 0x55f4c9d34870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc02ead4640 1 -- 172.21.15.129:0/3424222506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4c9d29170 con 0x55f4c99676b0 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.927+0000 7fc02dad2640 1 --2- 172.21.15.129:0/3424222506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d3b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.928+0000 7fc02ead4640 1 --2- 172.21.15.129:0/3424222506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d373a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc01802a930 tx=0x7fc01803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38dcd2e76fee04f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.928+0000 7fc00ffff640 1 -- 172.21.15.129:0/3424222506 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01804e070 con 0x55f4c99676b0 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.928+0000 7fc00ffff640 1 -- 172.21.15.129:0/3424222506 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc018041070 con 0x55f4c99676b0 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.928+0000 7fc00ffff640 1 -- 172.21.15.129:0/3424222506 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc018035040 con 0x55f4c99676b0 2024-05-20T18:08:29.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.928+0000 7fc0373db740 1 -- 172.21.15.129:0/3424222506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 msgr2=0x55f4c9d373a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.928+0000 7fc0373db740 1 --2- 172.21.15.129:0/3424222506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d373a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc01802a930 tx=0x7fc01803f000 comp rx=0 tx=0).stop 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.928+0000 7fc0373db740 1 -- 172.21.15.129:0/3424222506 shutdown_connections 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.928+0000 7fc0373db740 1 --2- 172.21.15.129:0/3424222506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9d373a0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 --2- 172.21.15.129:0/3424222506 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d3b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 --2- 172.21.15.129:0/3424222506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9946f60 0x55f4c9d34870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 -- 172.21.15.129:0/3424222506 >> 172.21.15.129:0/3424222506 conn(0x55f4c997c4e0 msgr2=0x55f4c9d36c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 -- 172.21.15.129:0/3424222506 shutdown_connections 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 -- 172.21.15.129:0/3424222506 wait complete. 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 -- start start 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 Processor -- start 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9946f60 0x55f4c9d3d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d3d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9dae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0281ab4a0 con 0x55f4c99676b0 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0281ac910 con 0x55f4c9946f60 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc0281abb70 con 0x55f4c995f4d0 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc0373db740 10 client.0 _ll_register_callbacks cb 0x7fc020009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc02ead4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9dae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc02ead4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9dae4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33430/0 (socket says 172.21.15.129:33430) 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc02ead4640 1 -- 172.21.15.129:0/1749525905 learned_addr learned my addr 172.21.15.129:0/1749525905 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc02dad2640 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d3d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.929+0000 7fc02e2d3640 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9946f60 0x55f4c9d3d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc02ead4640 1 -- 172.21.15.129:0/1749525905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 msgr2=0x55f4c9d3d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc02ead4640 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d3d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc02ead4640 1 -- 172.21.15.129:0/1749525905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9946f60 msgr2=0x55f4c9d3d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:29.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc02ead4640 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9946f60 0x55f4c9d3d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc02ead4640 1 -- 172.21.15.129:0/1749525905 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4c9daea20 con 0x55f4c99676b0 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc02ead4640 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9dae4e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc018006010 tx=0x7fc018031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8404f2deb1b862c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc00dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc00dffb640 1 -- 172.21.15.129:0/1749525905 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01805b070 con 0x55f4c99676b0 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc00dffb640 1 -- 172.21.15.129:0/1749525905 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fc018003a00 con 0x55f4c99676b0 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc0373db740 1 -- 172.21.15.129:0/1749525905 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc018041070 con 0x55f4c99676b0 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc00dffb640 1 -- 172.21.15.129:0/1749525905 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc018067050 con 0x55f4c99676b0 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc0373db740 10 client.17061 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc0373db740 1 -- 172.21.15.129:0/1749525905 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f4c9dafb10 con 0x55f4c99676b0 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc00dffb640 1 -- 172.21.15.129:0/1749525905 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc018041070 con 0x55f4c99676b0 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc00dffb640 1 -- 172.21.15.129:0/1749525905 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fc018053570 con 0x55f4c99676b0 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc0373db740 10 client.17061 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:29.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc0373db740 10 client.17061 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.930+0000 7fc0373db740 20 client.17061 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.931+0000 7fc0373db740 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f4c9988390 0x55f4c9db2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.931+0000 7fc0373db740 4 client.17061 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.931+0000 7fc0373db740 1 -- 172.21.15.129:0/1749525905 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4c9c946c0 con 0x55f4c9988390 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.931+0000 7fc02e2d3640 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f4c9988390 0x55f4c9db2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.931+0000 7fc00dffb640 10 client.17061 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.931+0000 7fc02e2d3640 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f4c9988390 0x55f4c9db2240 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=368fdbc9630e6003 server_cookie=95bd1e448f2a3270 in_seq=0 out_seq=0 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.931+0000 7fc00dffb640 1 -- 172.21.15.129:0/1749525905 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4c9c946c0 con 0x55f4c9988390 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.931+0000 7fc00dffb640 10 client.17061 handle_command_reply: tid=0 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.931+0000 7fc0373db740 1 client.17061 shutdown 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.931+0000 7fc0373db740 1 -- 172.21.15.129:0/1749525905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 msgr2=0x55f4c9dae4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.931+0000 7fc0373db740 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9dae4e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc018006010 tx=0x7fc018031b60 comp rx=0 tx=0).stop 2024-05-20T18:08:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc0373db740 1 -- 172.21.15.129:0/1749525905 shutdown_connections 2024-05-20T18:08:29.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc0373db740 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f4c9988390 0x55f4c9db2240 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc0373db740 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f4c99676b0 0x55f4c9dae4e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc0373db740 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f4c995f4d0 0x55f4c9d3d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc00dffb640 0 client.17061 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:29.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc0373db740 1 --2- 172.21.15.129:0/1749525905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4c9946f60 0x55f4c9d3d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc0373db740 1 -- 172.21.15.129:0/1749525905 >> 172.21.15.129:0/1749525905 conn(0x55f4c997c4e0 msgr2=0x55f4c9d37570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc0373db740 1 -- 172.21.15.129:0/1749525905 shutdown_connections 2024-05-20T18:08:29.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc0373db740 1 -- 172.21.15.129:0/1749525905 wait complete. 2024-05-20T18:08:29.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc0373db740 20 client.17061 trim_cache size 0 max 16384 2024-05-20T18:08:29.936 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:08:29.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 -- 172.21.15.129:0/3623098445 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0100a2d00 msgr2=0x7fc0100a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0100a2d00 0x7fc0100a5210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x55f4c9d23ec0 tx=0x7fc018009260 comp rx=0 tx=0).stop 2024-05-20T18:08:29.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 -- 172.21.15.129:0/3623098445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0281079d0 msgr2=0x7fc028199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:29.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0281079d0 0x7fc028199c50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00ead0 tx=0x7fc01c00ef90 comp rx=0 tx=0).stop 2024-05-20T18:08:29.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 -- 172.21.15.129:0/3623098445 shutdown_connections 2024-05-20T18:08:29.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0100a2d00 0x7fc0100a5210 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc02810c330 0x7fc02819df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0281079d0 0x7fc028199c50 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 --2- 172.21.15.129:0/3623098445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc028101aa0 0x7fc0281996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 -- 172.21.15.129:0/3623098445 >> 172.21.15.129:0/3623098445 conn(0x7fc0280770b0 msgr2=0x7fc028068e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 -- 172.21.15.129:0/3623098445 shutdown_connections 2024-05-20T18:08:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:29.932+0000 7fc030537640 1 -- 172.21.15.129:0/3623098445 wait complete. 2024-05-20T18:08:30.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:30 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1380947144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:08:30.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:30 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1380947144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:08:30.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:30 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1380947144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:08:30.941 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:08:30.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:08:30.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:08:31.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.073+0000 7f340f81b640 1 -- 172.21.15.129:0/1040841546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34081016d0 msgr2=0x7f340810f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.073+0000 7f340f81b640 1 --2- 172.21.15.129:0/1040841546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34081016d0 0x7f340810f520 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f3400009ca0 tx=0x7f340002a8c0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.073+0000 7f340f81b640 1 -- 172.21.15.129:0/1040841546 shutdown_connections 2024-05-20T18:08:31.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.073+0000 7f340f81b640 1 --2- 172.21.15.129:0/1040841546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3408108f40 0x7f3408101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.073+0000 7f340f81b640 1 --2- 172.21.15.129:0/1040841546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34081016d0 0x7f340810f520 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.073+0000 7f340f81b640 1 --2- 172.21.15.129:0/1040841546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3408108aa0 0x7f3408100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.073+0000 7f340f81b640 1 -- 172.21.15.129:0/1040841546 >> 172.21.15.129:0/1040841546 conn(0x7f3408076830 msgr2=0x7f3408076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.073+0000 7f340f81b640 1 -- 172.21.15.129:0/1040841546 shutdown_connections 2024-05-20T18:08:31.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.073+0000 7f340f81b640 1 -- 172.21.15.129:0/1040841546 wait complete. 2024-05-20T18:08:31.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.074+0000 7f340f81b640 1 Processor -- start 2024-05-20T18:08:31.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.074+0000 7f340f81b640 1 -- start start 2024-05-20T18:08:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.074+0000 7f340f81b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34081016d0 0x7f34081997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.074+0000 7f340f81b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3408108aa0 0x7f3408199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.074+0000 7f340f81b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3408108f40 0x7f340819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.074+0000 7f340f81b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3408112de0 con 0x7f34081016d0 2024-05-20T18:08:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.074+0000 7f340f81b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3408112c60 con 0x7f3408108aa0 2024-05-20T18:08:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.074+0000 7f340f81b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3408112f60 con 0x7f3408108f40 2024-05-20T18:08:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.074+0000 7f340d5b7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34081016d0 0x7f34081997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.074+0000 7f340d5b7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34081016d0 0x7f34081997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33452/0 (socket says 172.21.15.129:33452) 2024-05-20T18:08:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.074+0000 7f340d5b7640 1 -- 172.21.15.129:0/3250139231 learned_addr learned my addr 172.21.15.129:0/3250139231 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f340cdb6640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3408108aa0 0x7f3408199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f340ddb8640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3408108f40 0x7f340819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f340d5b7640 1 -- 172.21.15.129:0/3250139231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3408108f40 msgr2=0x7f340819dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f340d5b7640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3408108f40 0x7f340819dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f340d5b7640 1 -- 172.21.15.129:0/3250139231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3408108aa0 msgr2=0x7f3408199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f340d5b7640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3408108aa0 0x7f3408199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f340d5b7640 1 -- 172.21.15.129:0/3250139231 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340819e600 con 0x7f34081016d0 2024-05-20T18:08:31.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f340d5b7640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34081016d0 0x7f34081997a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f340400d950 tx=0x7f340400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f33fe7fc640 1 -- 172.21.15.129:0/3250139231 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404017030 con 0x7f34081016d0 2024-05-20T18:08:31.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f33fe7fc640 1 -- 172.21.15.129:0/3250139231 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3404004020 con 0x7f34081016d0 2024-05-20T18:08:31.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f33fe7fc640 1 -- 172.21.15.129:0/3250139231 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34040047a0 con 0x7f34081016d0 2024-05-20T18:08:31.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f340f81b640 1 -- 172.21.15.129:0/3250139231 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f340819e8f0 con 0x7f34081016d0 2024-05-20T18:08:31.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.075+0000 7f340f81b640 1 -- 172.21.15.129:0/3250139231 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f340819edf0 con 0x7f34081016d0 2024-05-20T18:08:31.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.078+0000 7f33fe7fc640 1 -- 172.21.15.129:0/3250139231 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f34040041d0 con 0x7f34081016d0 2024-05-20T18:08:31.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.078+0000 7f34167a6740 1 Processor -- start 2024-05-20T18:08:31.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.078+0000 7f34167a6740 1 -- start start 2024-05-20T18:08:31.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.078+0000 7f34167a6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7f2410 0x562d7cb91620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f34167a6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7c9550 0x562d7cb91b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f34167a6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb920a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f34167a6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562d7caf6990 con 0x562d7c7d1730 2024-05-20T18:08:31.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f34167a6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562d7ca9e5b0 con 0x562d7c7c9550 2024-05-20T18:08:31.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f34167a6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x562d7caf2950 con 0x562d7c7f2410 2024-05-20T18:08:31.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340cdb6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb920a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340cdb6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb920a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33458/0 (socket says 172.21.15.129:33458) 2024-05-20T18:08:31.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340cdb6640 1 -- 172.21.15.129:0/184731017 learned_addr learned my addr 172.21.15.129:0/184731017 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:31.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340ddb8640 1 --2- 172.21.15.129:0/184731017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7f2410 0x562d7cb91620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340d5b7640 1 --2- 172.21.15.129:0/184731017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7c9550 0x562d7cb91b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f33fe7fc640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f33f40a2d80 0x7f33f40a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340cdb6640 1 -- 172.21.15.129:0/184731017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7f2410 msgr2=0x562d7cb91620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340cdb6640 1 --2- 172.21.15.129:0/184731017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7f2410 0x562d7cb91620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340cdb6640 1 -- 172.21.15.129:0/184731017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7c9550 msgr2=0x562d7cb91b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340cdb6640 1 --2- 172.21.15.129:0/184731017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7c9550 0x562d7cb91b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340cdb6640 1 -- 172.21.15.129:0/184731017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d7c8d27e0 con 0x562d7c7d1730 2024-05-20T18:08:31.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340ddb8640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f33f40a2d80 0x7f33f40a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f33fe7fc640 1 -- 172.21.15.129:0/3250139231 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f34040ced60 con 0x7f34081016d0 2024-05-20T18:08:31.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.079+0000 7f340cdb6640 1 --2- 172.21.15.129:0/184731017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb920a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009c20 tx=0x7f33f8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed415913857c7fa3 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.080+0000 7f33f37fe640 1 -- 172.21.15.129:0/184731017 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8004050 con 0x562d7c7d1730 2024-05-20T18:08:31.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.080+0000 7f33f37fe640 1 -- 172.21.15.129:0/184731017 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33f8005500 con 0x562d7c7d1730 2024-05-20T18:08:31.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.080+0000 7f33f37fe640 1 -- 172.21.15.129:0/184731017 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8004d60 con 0x562d7c7d1730 2024-05-20T18:08:31.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.080+0000 7f34167a6740 1 -- 172.21.15.129:0/184731017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 msgr2=0x562d7cb920a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.080+0000 7f340ddb8640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f33f40a2d80 0x7f33f40a5290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3400005e30 tx=0x7f34000095c0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.080+0000 7f34167a6740 1 --2- 172.21.15.129:0/184731017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb920a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f33f8009c20 tx=0x7f33f8011810 comp rx=0 tx=0).stop 2024-05-20T18:08:31.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.081+0000 7f34167a6740 1 -- 172.21.15.129:0/184731017 shutdown_connections 2024-05-20T18:08:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.081+0000 7f34167a6740 1 --2- 172.21.15.129:0/184731017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7c9550 0x562d7cb91b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.081+0000 7f34167a6740 1 --2- 172.21.15.129:0/184731017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb920a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.081+0000 7f34167a6740 1 --2- 172.21.15.129:0/184731017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7f2410 0x562d7cb91620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.081+0000 7f34167a6740 1 -- 172.21.15.129:0/184731017 >> 172.21.15.129:0/184731017 conn(0x562d7c7e6560 msgr2=0x562d7cb8ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.081+0000 7f34167a6740 1 -- 172.21.15.129:0/184731017 shutdown_connections 2024-05-20T18:08:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.081+0000 7f34167a6740 1 -- 172.21.15.129:0/184731017 wait complete. 2024-05-20T18:08:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.081+0000 7f34167a6740 1 -- start start 2024-05-20T18:08:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.081+0000 7f34167a6740 1 Processor -- start 2024-05-20T18:08:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.081+0000 7f34167a6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7c9550 0x562d7cb9bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f34167a6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7d1730 0x562d7cb9c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f34167a6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7f2410 0x562d7cba1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f34167a6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562d7cb9e970 con 0x562d7c7f2410 2024-05-20T18:08:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f34167a6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562d7cab1b40 con 0x562d7c7d1730 2024-05-20T18:08:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f34167a6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x562d7cb9eaf0 con 0x562d7c7c9550 2024-05-20T18:08:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f34167a6740 10 client.0 _ll_register_callbacks cb 0x562d7caa60f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340ddb8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7f2410 0x562d7cba1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340ddb8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7f2410 0x562d7cba1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33474/0 (socket says 172.21.15.129:33474) 2024-05-20T18:08:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340ddb8640 1 -- 172.21.15.129:0/1807482345 learned_addr learned my addr 172.21.15.129:0/1807482345 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340d5b7640 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7c9550 0x562d7cb9bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340cdb6640 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7d1730 0x562d7cb9c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340ddb8640 1 -- 172.21.15.129:0/1807482345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7c9550 msgr2=0x562d7cb9bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340ddb8640 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7c9550 0x562d7cb9bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340ddb8640 1 -- 172.21.15.129:0/1807482345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7d1730 msgr2=0x562d7cb9c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340ddb8640 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7d1730 0x562d7cb9c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340ddb8640 1 -- 172.21.15.129:0/1807482345 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d7c932f70 con 0x562d7c7f2410 2024-05-20T18:08:31.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340cdb6640 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7d1730 0x562d7cb9c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:08:31.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f340ddb8640 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7f2410 0x562d7cba1ed0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f3400035000 tx=0x7f340002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a445557d76cca478 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f33f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:31.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f33f0ff9640 1 -- 172.21.15.129:0/1807482345 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400004450 con 0x562d7c7f2410 2024-05-20T18:08:31.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f33f0ff9640 1 -- 172.21.15.129:0/1807482345 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f34000045f0 con 0x562d7c7f2410 2024-05-20T18:08:31.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f34167a6740 1 -- 172.21.15.129:0/1807482345 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33f8005500 con 0x562d7c7f2410 2024-05-20T18:08:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f33f0ff9640 1 -- 172.21.15.129:0/1807482345 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f34000047f0 con 0x562d7c7f2410 2024-05-20T18:08:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.082+0000 7f34167a6740 10 client.17073 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.083+0000 7f34167a6740 1 -- 172.21.15.129:0/1807482345 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562d7c7ca170 con 0x562d7c7f2410 2024-05-20T18:08:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.083+0000 7f33f0ff9640 1 -- 172.21.15.129:0/1807482345 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33f8005500 con 0x562d7c7f2410 2024-05-20T18:08:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.083+0000 7f33f0ff9640 1 -- 172.21.15.129:0/1807482345 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f3400030820 con 0x562d7c7f2410 2024-05-20T18:08:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.083+0000 7f34167a6740 10 client.17073 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.083+0000 7f34167a6740 10 client.17073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.083+0000 7f34167a6740 20 client.17073 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.083+0000 7f34167a6740 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562d7c7b11f0 0x562d7cbae130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.083+0000 7f34167a6740 4 client.17073 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:08:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.083+0000 7f34167a6740 1 -- 172.21.15.129:0/1807482345 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d7cb040c0 con 0x562d7c7b11f0 2024-05-20T18:08:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.083+0000 7f340d5b7640 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562d7c7b11f0 0x562d7cbae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.084+0000 7f33f0ff9640 10 client.17073 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.084+0000 7f340d5b7640 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562d7c7b11f0 0x562d7cbae130 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dce5ed6802f64b23 server_cookie=f63d79b1a30ed086 in_seq=0 out_seq=0 2024-05-20T18:08:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.084+0000 7f33f0ff9640 1 -- 172.21.15.129:0/1807482345 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562d7cb040c0 con 0x562d7c7b11f0 2024-05-20T18:08:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.084+0000 7f33f0ff9640 10 client.17073 handle_command_reply: tid=0 2024-05-20T18:08:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.084+0000 7f34167a6740 1 client.17073 shutdown 2024-05-20T18:08:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f34167a6740 1 -- 172.21.15.129:0/1807482345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7f2410 msgr2=0x562d7cba1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f34167a6740 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7f2410 0x562d7cba1ed0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f3400035000 tx=0x7f340002a8c0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f34167a6740 1 -- 172.21.15.129:0/1807482345 shutdown_connections 2024-05-20T18:08:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f34167a6740 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562d7c7b11f0 0x562d7cbae130 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f34167a6740 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7d1730 0x562d7cb9c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f34167a6740 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7f2410 0x562d7cba1ed0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f33f0ff9640 0 client.17073 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:31.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f34167a6740 1 --2- 172.21.15.129:0/1807482345 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7c9550 0x562d7cb9bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f34167a6740 1 -- 172.21.15.129:0/1807482345 >> 172.21.15.129:0/1807482345 conn(0x562d7c7e6560 msgr2=0x562d7cb93340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f34167a6740 1 -- 172.21.15.129:0/1807482345 shutdown_connections 2024-05-20T18:08:31.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f34167a6740 1 -- 172.21.15.129:0/1807482345 wait complete. 2024-05-20T18:08:31.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.085+0000 7f34167a6740 20 client.17073 trim_cache size 0 max 16384 2024-05-20T18:08:31.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f34167a6740 1 Processor -- start 2024-05-20T18:08:31.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f34167a6740 1 -- start start 2024-05-20T18:08:31.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f34167a6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7c9550 0x562d7cb9d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f34167a6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb9de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f34167a6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7f2410 0x562d7cba1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f34167a6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f34081acc40 con 0x562d7c7d1730 2024-05-20T18:08:31.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f34167a6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f34081af4a0 con 0x562d7c7f2410 2024-05-20T18:08:31.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f34167a6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f34081ad290 con 0x562d7c7c9550 2024-05-20T18:08:31.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f340cdb6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb9de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f340cdb6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb9de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33478/0 (socket says 172.21.15.129:33478) 2024-05-20T18:08:31.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f340cdb6640 1 -- 172.21.15.129:0/2313021778 learned_addr learned my addr 172.21.15.129:0/2313021778 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:31.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.103+0000 7f340ddb8640 1 --2- 172.21.15.129:0/2313021778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7f2410 0x562d7cba1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.104+0000 7f340cdb6640 1 -- 172.21.15.129:0/2313021778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7c9550 msgr2=0x562d7cb9d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.104+0000 7f340d5b7640 1 --2- 172.21.15.129:0/2313021778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7c9550 0x562d7cb9d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.104+0000 7f340cdb6640 1 --2- 172.21.15.129:0/2313021778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7c9550 0x562d7cb9d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.104+0000 7f340cdb6640 1 -- 172.21.15.129:0/2313021778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7f2410 msgr2=0x562d7cba1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.104+0000 7f340cdb6640 1 --2- 172.21.15.129:0/2313021778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7f2410 0x562d7cba1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.104+0000 7f340cdb6640 1 -- 172.21.15.129:0/2313021778 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d7cb9aab0 con 0x562d7c7d1730 2024-05-20T18:08:31.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.104+0000 7f340ddb8640 1 --2- 172.21.15.129:0/2313021778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7f2410 0x562d7cba1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:08:31.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.104+0000 7f340cdb6640 1 --2- 172.21.15.129:0/2313021778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb9de70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f33f8011d10 tx=0x7f33f8009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8775e5977c933ac9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.104+0000 7f33f2ffd640 1 -- 172.21.15.129:0/2313021778 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f80052c0 con 0x562d7c7d1730 2024-05-20T18:08:31.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.104+0000 7f33f2ffd640 1 -- 172.21.15.129:0/2313021778 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33f8005460 con 0x562d7c7d1730 2024-05-20T18:08:31.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.104+0000 7f33f2ffd640 1 -- 172.21.15.129:0/2313021778 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8019790 con 0x562d7c7d1730 2024-05-20T18:08:31.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.105+0000 7f34167a6740 1 -- 172.21.15.129:0/2313021778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 msgr2=0x562d7cb9de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.105+0000 7f34167a6740 1 --2- 172.21.15.129:0/2313021778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb9de70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f33f8011d10 tx=0x7f33f8009930 comp rx=0 tx=0).stop 2024-05-20T18:08:31.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.105+0000 7f34167a6740 1 -- 172.21.15.129:0/2313021778 shutdown_connections 2024-05-20T18:08:31.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.105+0000 7f34167a6740 1 --2- 172.21.15.129:0/2313021778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7f2410 0x562d7cba1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.105+0000 7f34167a6740 1 --2- 172.21.15.129:0/2313021778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cb9de70 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.105+0000 7f34167a6740 1 --2- 172.21.15.129:0/2313021778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7c9550 0x562d7cb9d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.105+0000 7f34167a6740 1 -- 172.21.15.129:0/2313021778 >> 172.21.15.129:0/2313021778 conn(0x562d7c7e6560 msgr2=0x562d7cb9d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.105+0000 7f34167a6740 1 -- 172.21.15.129:0/2313021778 shutdown_connections 2024-05-20T18:08:31.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.105+0000 7f34167a6740 1 -- 172.21.15.129:0/2313021778 wait complete. 2024-05-20T18:08:31.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.105+0000 7f34167a6740 1 -- start start 2024-05-20T18:08:31.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.105+0000 7f34167a6740 1 Processor -- start 2024-05-20T18:08:31.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f34167a6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7c9550 0x562d7cba4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f34167a6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cba46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f34167a6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7f2410 0x562d7cba4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f34167a6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f34081ad960 con 0x562d7c7d1730 2024-05-20T18:08:31.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f34167a6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f34081aedd0 con 0x562d7c7c9550 2024-05-20T18:08:31.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f34167a6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f34081ae030 con 0x562d7c7f2410 2024-05-20T18:08:31.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f34167a6740 10 client.0 _ll_register_callbacks cb 0x7f3400003a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:31.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f340cdb6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cba46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f340cdb6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cba46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33482/0 (socket says 172.21.15.129:33482) 2024-05-20T18:08:31.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f340cdb6640 1 -- 172.21.15.129:0/3217274274 learned_addr learned my addr 172.21.15.129:0/3217274274 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:31.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f340d5b7640 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7c9550 0x562d7cba4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f340cdb6640 1 -- 172.21.15.129:0/3217274274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7f2410 msgr2=0x562d7cba4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f340cdb6640 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7f2410 0x562d7cba4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f340cdb6640 1 -- 172.21.15.129:0/3217274274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7c9550 msgr2=0x562d7cba4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f340cdb6640 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7c9550 0x562d7cba4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f340cdb6640 1 -- 172.21.15.129:0/3217274274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d7cc15600 con 0x562d7c7d1730 2024-05-20T18:08:31.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f340d5b7640 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7c9550 0x562d7cba4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:31.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f340cdb6640 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cba46c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f33f8011850 tx=0x7f33f8003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da8dc8bfbaccb23 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.106+0000 7f33f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:31.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f33f0ff9640 1 -- 172.21.15.129:0/3217274274 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f803b060 con 0x562d7c7d1730 2024-05-20T18:08:31.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f33f0ff9640 1 -- 172.21.15.129:0/3217274274 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f33f8003ba0 con 0x562d7c7d1730 2024-05-20T18:08:31.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f34167a6740 1 -- 172.21.15.129:0/3217274274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33f8005460 con 0x562d7c7d1730 2024-05-20T18:08:31.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f33f0ff9640 1 -- 172.21.15.129:0/3217274274 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f33f8005610 con 0x562d7c7d1730 2024-05-20T18:08:31.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f34167a6740 10 client.17079 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:31.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f34167a6740 1 -- 172.21.15.129:0/3217274274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562d7cc168a0 con 0x562d7c7d1730 2024-05-20T18:08:31.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f33f0ff9640 1 -- 172.21.15.129:0/3217274274 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33f8005460 con 0x562d7c7d1730 2024-05-20T18:08:31.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f33f0ff9640 1 -- 172.21.15.129:0/3217274274 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f33f80223f0 con 0x562d7c7d1730 2024-05-20T18:08:31.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f34167a6740 10 client.17079 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:31.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f34167a6740 10 client.17079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:31.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f34167a6740 20 client.17079 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:31.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f34167a6740 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562d7c786630 0x562d7cc18cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f34167a6740 4 client.17079 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:08:31.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f34167a6740 1 -- 172.21.15.129:0/3217274274 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562d7cafe410 con 0x562d7c786630 2024-05-20T18:08:31.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.107+0000 7f340d5b7640 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562d7c786630 0x562d7cc18cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.108+0000 7f33f0ff9640 10 client.17079 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:31.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.108+0000 7f340d5b7640 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562d7c786630 0x562d7cc18cd0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa3fea758b77b94b server_cookie=63c393ceeb2d13c4 in_seq=0 out_seq=0 2024-05-20T18:08:31.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f33f0ff9640 1 -- 172.21.15.129:0/3217274274 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562d7cafe410 con 0x562d7c786630 2024-05-20T18:08:31.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f33f0ff9640 10 client.17079 handle_command_reply: tid=0 2024-05-20T18:08:31.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 1 client.17079 shutdown 2024-05-20T18:08:31.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 1 -- 172.21.15.129:0/3217274274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 msgr2=0x562d7cba46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cba46c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f33f8011850 tx=0x7f33f8003820 comp rx=0 tx=0).stop 2024-05-20T18:08:31.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 1 -- 172.21.15.129:0/3217274274 shutdown_connections 2024-05-20T18:08:31.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x562d7c786630 0x562d7cc18cd0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562d7c7f2410 0x562d7cba4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562d7c7d1730 0x562d7cba46c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f33f0ff9640 0 client.17079 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:31.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 1 --2- 172.21.15.129:0/3217274274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562d7c7c9550 0x562d7cba4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 1 -- 172.21.15.129:0/3217274274 >> 172.21.15.129:0/3217274274 conn(0x562d7c7e6560 msgr2=0x562d7cb91de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 1 -- 172.21.15.129:0/3217274274 shutdown_connections 2024-05-20T18:08:31.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 1 -- 172.21.15.129:0/3217274274 wait complete. 2024-05-20T18:08:31.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.109+0000 7f34167a6740 20 client.17079 trim_cache size 0 max 16384 2024-05-20T18:08:31.115 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:08:31.115 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:08:31.115 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "5dcbd937-b6be-4d86-8afe-1fe96d197e51", 2024-05-20T18:08:31.116 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:08:31.116 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:08:31.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 -- 172.21.15.129:0/3250139231 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f33f40a2d80 msgr2=0x7f33f40a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f33f40a2d80 0x7f33f40a5290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3400005e30 tx=0x7f34000095c0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 -- 172.21.15.129:0/3250139231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34081016d0 msgr2=0x7f34081997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34081016d0 0x7f34081997a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f340400d950 tx=0x7f340400de10 comp rx=0 tx=0).stop 2024-05-20T18:08:31.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 -- 172.21.15.129:0/3250139231 shutdown_connections 2024-05-20T18:08:31.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f33f40a2d80 0x7f33f40a5290 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3408108aa0 0x7f3408199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3408108f40 0x7f340819dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 --2- 172.21.15.129:0/3250139231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34081016d0 0x7f34081997a0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 -- 172.21.15.129:0/3250139231 >> 172.21.15.129:0/3250139231 conn(0x7f3408076830 msgr2=0x7f34080ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 -- 172.21.15.129:0/3250139231 shutdown_connections 2024-05-20T18:08:31.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.110+0000 7f340f81b640 1 -- 172.21.15.129:0/3250139231 wait complete. 2024-05-20T18:08:31.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:08:31.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:08:31.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.234+0000 7f8a02a99640 1 -- 172.21.15.129:0/3748744336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89fc10d040 msgr2=0x7f89fc10f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.234+0000 7f8a02a99640 1 --2- 172.21.15.129:0/3748744336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89fc10d040 0x7f89fc10f530 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009900 tx=0x7f89f402a510 comp rx=0 tx=0).stop 2024-05-20T18:08:31.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.234+0000 7f8a02a99640 1 -- 172.21.15.129:0/3748744336 shutdown_connections 2024-05-20T18:08:31.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.234+0000 7f8a02a99640 1 --2- 172.21.15.129:0/3748744336 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89fc10d040 0x7f89fc10f530 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.234+0000 7f8a02a99640 1 --2- 172.21.15.129:0/3748744336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89fc10a690 0x7f89fc10cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.234+0000 7f8a02a99640 1 --2- 172.21.15.129:0/3748744336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89fc103c30 0x7f89fc104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.234+0000 7f8a02a99640 1 -- 172.21.15.129:0/3748744336 >> 172.21.15.129:0/3748744336 conn(0x7f89fc077100 msgr2=0x7f89fc077540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.235+0000 7f8a02a99640 1 -- 172.21.15.129:0/3748744336 shutdown_connections 2024-05-20T18:08:31.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.235+0000 7f8a02a99640 1 -- 172.21.15.129:0/3748744336 wait complete. 2024-05-20T18:08:31.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.235+0000 7f8a02a99640 1 Processor -- start 2024-05-20T18:08:31.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.235+0000 7f8a02a99640 1 -- start start 2024-05-20T18:08:31.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.235+0000 7f8a02a99640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89fc103c30 0x7f89fc199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.235+0000 7f8a02a99640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89fc10a690 0x7f89fc199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.235+0000 7f8a02a99640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89fc10d040 0x7f89fc19df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.235+0000 7f8a02a99640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc112df0 con 0x7f89fc103c30 2024-05-20T18:08:31.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.235+0000 7f8a02a99640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc112c70 con 0x7f89fc10a690 2024-05-20T18:08:31.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.235+0000 7f8a02a99640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc112f70 con 0x7f89fc10d040 2024-05-20T18:08:31.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a00835640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89fc103c30 0x7f89fc199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a00835640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89fc103c30 0x7f89fc199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33486/0 (socket says 172.21.15.129:33486) 2024-05-20T18:08:31.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a00835640 1 -- 172.21.15.129:0/2832263041 learned_addr learned my addr 172.21.15.129:0/2832263041 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:31.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f89f3fff640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89fc10a690 0x7f89fc199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a01036640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89fc10d040 0x7f89fc19df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a00835640 1 -- 172.21.15.129:0/2832263041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89fc10d040 msgr2=0x7f89fc19df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a00835640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89fc10d040 0x7f89fc19df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a00835640 1 -- 172.21.15.129:0/2832263041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89fc10a690 msgr2=0x7f89fc199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a00835640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89fc10a690 0x7f89fc199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a00835640 1 -- 172.21.15.129:0/2832263041 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89fc19e5b0 con 0x7f89fc103c30 2024-05-20T18:08:31.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a01036640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89fc10d040 0x7f89fc19df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:31.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a00835640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89fc103c30 0x7f89fc199750 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f89f800db80 tx=0x7f89f800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f89f1ffb640 1 -- 172.21.15.129:0/2832263041 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8017070 con 0x7f89fc103c30 2024-05-20T18:08:31.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f89f1ffb640 1 -- 172.21.15.129:0/2832263041 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f8003b70 con 0x7f89fc103c30 2024-05-20T18:08:31.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a02a99640 1 -- 172.21.15.129:0/2832263041 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89fc19e8a0 con 0x7f89fc103c30 2024-05-20T18:08:31.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f89f1ffb640 1 -- 172.21.15.129:0/2832263041 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f80106c0 con 0x7f89fc103c30 2024-05-20T18:08:31.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.236+0000 7f8a02a99640 1 -- 172.21.15.129:0/2832263041 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89fc1a6220 con 0x7f89fc103c30 2024-05-20T18:08:31.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.239+0000 7f89f1ffb640 1 -- 172.21.15.129:0/2832263041 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f89f8004020 con 0x7f89fc103c30 2024-05-20T18:08:31.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.239+0000 7f8a09a1e740 1 Processor -- start 2024-05-20T18:08:31.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.239+0000 7f8a09a1e740 1 -- start start 2024-05-20T18:08:31.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.239+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748f0b390 0x55a7492a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.239+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748ee24d0 0x55a7492aa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748eea6b0 0x55a7492aa7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a74920fd00 con 0x55a748f0b390 2024-05-20T18:08:31.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a7491b74c0 con 0x55a748eea6b0 2024-05-20T18:08:31.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a74920bcc0 con 0x55a748ee24d0 2024-05-20T18:08:31.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f89f1ffb640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f89e40a2d80 0x7f89e40a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a01036640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748f0b390 0x55a7492a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a01036640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748f0b390 0x55a7492a9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33494/0 (socket says 172.21.15.129:33494) 2024-05-20T18:08:31.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a01036640 1 -- 172.21.15.129:0/1628783615 learned_addr learned my addr 172.21.15.129:0/1628783615 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:31.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a00835640 1 --2- 172.21.15.129:0/1628783615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748ee24d0 0x55a7492aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a01036640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f89e40a2d80 0x7f89e40a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f89f3fff640 1 --2- 172.21.15.129:0/1628783615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748eea6b0 0x55a7492aa7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f89f1ffb640 1 -- 172.21.15.129:0/2832263041 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f89f80cf310 con 0x7f89fc103c30 2024-05-20T18:08:31.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a01036640 1 -- 172.21.15.129:0/1628783615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748ee24d0 msgr2=0x55a7492aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a01036640 1 --2- 172.21.15.129:0/1628783615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748ee24d0 0x55a7492aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a01036640 1 -- 172.21.15.129:0/1628783615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748eea6b0 msgr2=0x55a7492aa7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a01036640 1 --2- 172.21.15.129:0/1628783615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748eea6b0 0x55a7492aa7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a01036640 1 -- 172.21.15.129:0/1628783615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a748fed7d0 con 0x55a748f0b390 2024-05-20T18:08:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a01036640 1 --2- 172.21.15.129:0/1628783615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748f0b390 0x55a7492a9d30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f89f402ab00 tx=0x7f89f4031b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82658da5a8d8f3a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f8a01036640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f89e40a2d80 0x7f89e40a5290 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f89f4004040 tx=0x7f89f4045000 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.240+0000 7f89e2ffd640 1 -- 172.21.15.129:0/1628783615 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f404d070 con 0x55a748f0b390 2024-05-20T18:08:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.241+0000 7f89e2ffd640 1 -- 172.21.15.129:0/1628783615 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f4004430 con 0x55a748f0b390 2024-05-20T18:08:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.241+0000 7f89e2ffd640 1 -- 172.21.15.129:0/1628783615 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4035040 con 0x55a748f0b390 2024-05-20T18:08:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.241+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1628783615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748f0b390 msgr2=0x55a7492a9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.241+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/1628783615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748f0b390 0x55a7492a9d30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f89f402ab00 tx=0x7f89f4031b40 comp rx=0 tx=0).stop 2024-05-20T18:08:31.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.241+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1628783615 shutdown_connections 2024-05-20T18:08:31.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.241+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/1628783615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748eea6b0 0x55a7492aa7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.241+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/1628783615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748ee24d0 0x55a7492aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.241+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/1628783615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748f0b390 0x55a7492a9d30 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.241+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1628783615 >> 172.21.15.129:0/1628783615 conn(0x55a748eff4e0 msgr2=0x55a7492a64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.241+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1628783615 shutdown_connections 2024-05-20T18:08:31.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.241+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1628783615 wait complete. 2024-05-20T18:08:31.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.242+0000 7f8a09a1e740 1 -- start start 2024-05-20T18:08:31.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.242+0000 7f8a09a1e740 1 Processor -- start 2024-05-20T18:08:31.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.242+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.242+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748eea6b0 0x55a7492b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.242+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748f0b390 0x55a7492b4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.242+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a7492b6fc0 con 0x55a748ee24d0 2024-05-20T18:08:31.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.242+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a7491cab90 con 0x55a748eea6b0 2024-05-20T18:08:31.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.242+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a7492b7140 con 0x55a748f0b390 2024-05-20T18:08:31.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.242+0000 7f8a09a1e740 10 client.0 _ll_register_callbacks cb 0x55a749206da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:31.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.242+0000 7f8a00835640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a00835640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33498/0 (socket says 172.21.15.129:33498) 2024-05-20T18:08:31.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a00835640 1 -- 172.21.15.129:0/4099178433 learned_addr learned my addr 172.21.15.129:0/4099178433 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:31.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f89f3fff640 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748eea6b0 0x55a7492b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a01036640 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748f0b390 0x55a7492b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a00835640 1 -- 172.21.15.129:0/4099178433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748f0b390 msgr2=0x55a7492b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a00835640 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748f0b390 0x55a7492b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a00835640 1 -- 172.21.15.129:0/4099178433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748eea6b0 msgr2=0x55a7492b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a00835640 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748eea6b0 0x55a7492b63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a00835640 1 -- 172.21.15.129:0/4099178433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a748f233f0 con 0x55a748ee24d0 2024-05-20T18:08:31.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a00835640 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b5e60 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f89f800db50 tx=0x7f89f806a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a0fd0a760640e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f89dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:31.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f89dbfff640 1 -- 172.21.15.129:0/4099178433 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f809be90 con 0x55a748ee24d0 2024-05-20T18:08:31.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f89dbfff640 1 -- 172.21.15.129:0/4099178433 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f89f8003d20 con 0x55a748ee24d0 2024-05-20T18:08:31.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a09a1e740 1 -- 172.21.15.129:0/4099178433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89f4004430 con 0x55a748ee24d0 2024-05-20T18:08:31.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f89dbfff640 1 -- 172.21.15.129:0/4099178433 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f89f80a4400 con 0x55a748ee24d0 2024-05-20T18:08:31.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a09a1e740 10 client.17091 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:31.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f8a09a1e740 1 -- 172.21.15.129:0/4099178433 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a748ee30f0 con 0x55a748ee24d0 2024-05-20T18:08:31.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.243+0000 7f89dbfff640 1 -- 172.21.15.129:0/4099178433 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f4004430 con 0x55a748ee24d0 2024-05-20T18:08:31.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.244+0000 7f89dbfff640 1 -- 172.21.15.129:0/4099178433 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f89f8097e00 con 0x55a748ee24d0 2024-05-20T18:08:31.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.244+0000 7f8a09a1e740 10 client.17091 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:31.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.244+0000 7f8a09a1e740 10 client.17091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:31.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.244+0000 7f8a09a1e740 20 client.17091 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:31.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.244+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a748ec9f60 0x55a7492c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.244+0000 7f8a09a1e740 4 client.17091 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:08:31.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.244+0000 7f8a09a1e740 1 -- 172.21.15.129:0/4099178433 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a74921d430 con 0x55a748ec9f60 2024-05-20T18:08:31.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.244+0000 7f89f3fff640 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a748ec9f60 0x55a7492c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.244+0000 7f89dbfff640 10 client.17091 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:31.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.244+0000 7f89f3fff640 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a748ec9f60 0x55a7492c6b60 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a480f64e8b54141f server_cookie=cde18f54ef8f513e in_seq=0 out_seq=0 2024-05-20T18:08:31.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f89dbfff640 1 -- 172.21.15.129:0/4099178433 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a74921d430 con 0x55a748ec9f60 2024-05-20T18:08:31.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f89dbfff640 10 client.17091 handle_command_reply: tid=0 2024-05-20T18:08:31.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f8a09a1e740 1 client.17091 shutdown 2024-05-20T18:08:31.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f8a09a1e740 1 -- 172.21.15.129:0/4099178433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 msgr2=0x55a7492b5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b5e60 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f89f800db50 tx=0x7f89f806a6c0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f8a09a1e740 1 -- 172.21.15.129:0/4099178433 shutdown_connections 2024-05-20T18:08:31.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a748ec9f60 0x55a7492c6b60 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748eea6b0 0x55a7492b63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748f0b390 0x55a7492b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/4099178433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b5e60 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f89dbfff640 0 client.17091 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:31.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f8a09a1e740 1 -- 172.21.15.129:0/4099178433 >> 172.21.15.129:0/4099178433 conn(0x55a748eff4e0 msgr2=0x55a7492aba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f8a09a1e740 1 -- 172.21.15.129:0/4099178433 shutdown_connections 2024-05-20T18:08:31.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.245+0000 7f8a09a1e740 1 -- 172.21.15.129:0/4099178433 wait complete. 2024-05-20T18:08:31.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.246+0000 7f8a09a1e740 20 client.17091 trim_cache size 0 max 16384 2024-05-20T18:08:31.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.263+0000 7f8a09a1e740 1 Processor -- start 2024-05-20T18:08:31.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.263+0000 7f8a09a1e740 1 -- start start 2024-05-20T18:08:31.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.263+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.263+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748eea6b0 0x55a7492ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.263+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 0x55a7492bc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.263+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1acba0 con 0x55a748ee24d0 2024-05-20T18:08:31.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.263+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1af400 con 0x55a748f0b390 2024-05-20T18:08:31.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.263+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1ad1f0 con 0x55a748eea6b0 2024-05-20T18:08:31.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f8a00835640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f8a00835640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33512/0 (socket says 172.21.15.129:33512) 2024-05-20T18:08:31.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f8a00835640 1 -- 172.21.15.129:0/3247448975 learned_addr learned my addr 172.21.15.129:0/3247448975 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:31.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f8a01036640 1 --2- 172.21.15.129:0/3247448975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 0x55a7492bc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f8a00835640 1 -- 172.21.15.129:0/3247448975 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748eea6b0 msgr2=0x55a7492ac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f8a00835640 1 --2- 172.21.15.129:0/3247448975 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748eea6b0 0x55a7492ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f8a00835640 1 -- 172.21.15.129:0/3247448975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 msgr2=0x55a7492bc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f8a00835640 1 --2- 172.21.15.129:0/3247448975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 0x55a7492bc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f8a00835640 1 -- 172.21.15.129:0/3247448975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a749297990 con 0x55a748ee24d0 2024-05-20T18:08:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f8a01036640 1 --2- 172.21.15.129:0/3247448975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 0x55a7492bc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f8a00835640 1 --2- 172.21.15.129:0/3247448975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b3670 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f89f809b490 tx=0x7f89f800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b4adf1433ffa0d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f89e27fc640 1 -- 172.21.15.129:0/3247448975 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f809fb40 con 0x55a748ee24d0 2024-05-20T18:08:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f89e27fc640 1 -- 172.21.15.129:0/3247448975 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f809b790 con 0x55a748ee24d0 2024-05-20T18:08:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.264+0000 7f89e27fc640 1 -- 172.21.15.129:0/3247448975 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f809ae90 con 0x55a748ee24d0 2024-05-20T18:08:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.265+0000 7f8a09a1e740 1 -- 172.21.15.129:0/3247448975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 msgr2=0x55a7492b3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.265+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/3247448975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b3670 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f89f809b490 tx=0x7f89f800be60 comp rx=0 tx=0).stop 2024-05-20T18:08:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.265+0000 7f8a09a1e740 1 -- 172.21.15.129:0/3247448975 shutdown_connections 2024-05-20T18:08:31.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.265+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/3247448975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 0x55a7492bc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.265+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/3247448975 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748eea6b0 0x55a7492ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.265+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/3247448975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748ee24d0 0x55a7492b3670 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.265+0000 7f8a09a1e740 1 -- 172.21.15.129:0/3247448975 >> 172.21.15.129:0/3247448975 conn(0x55a748eff4e0 msgr2=0x55a7492ab9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.265+0000 7f8a09a1e740 1 -- 172.21.15.129:0/3247448975 shutdown_connections 2024-05-20T18:08:31.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.265+0000 7f8a09a1e740 1 -- 172.21.15.129:0/3247448975 wait complete. 2024-05-20T18:08:31.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.265+0000 7f8a09a1e740 1 -- start start 2024-05-20T18:08:31.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.265+0000 7f8a09a1e740 1 Processor -- start 2024-05-20T18:08:31.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748ee24d0 0x55a7492bcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748eea6b0 0x55a7492b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f8a09a1e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 0x55a7492b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1ad8c0 con 0x55a748eea6b0 2024-05-20T18:08:31.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1aed30 con 0x55a748f0b390 2024-05-20T18:08:31.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f8a09a1e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1adf90 con 0x55a748ee24d0 2024-05-20T18:08:31.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f8a09a1e740 10 client.0 _ll_register_callbacks cb 0x7f89f809b560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:31.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f89f3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748eea6b0 0x55a7492b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f89f3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748eea6b0 0x55a7492b7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33518/0 (socket says 172.21.15.129:33518) 2024-05-20T18:08:31.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f89f3fff640 1 -- 172.21.15.129:0/1210896253 learned_addr learned my addr 172.21.15.129:0/1210896253 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:31.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f8a00835640 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748ee24d0 0x55a7492bcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f8a01036640 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 0x55a7492b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f89f3fff640 1 -- 172.21.15.129:0/1210896253 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748ee24d0 msgr2=0x55a7492bcaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f89f3fff640 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748ee24d0 0x55a7492bcaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f89f3fff640 1 -- 172.21.15.129:0/1210896253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 msgr2=0x55a7492b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:31.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f89f3fff640 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 0x55a7492b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f89f3fff640 1 -- 172.21.15.129:0/1210896253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7492b8c00 con 0x55a748eea6b0 2024-05-20T18:08:31.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f8a01036640 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 0x55a7492b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:31.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f89f3fff640 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748eea6b0 0x55a7492b7df0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x55a7492c71d0 tx=0x7f89ec00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf16cd65e7b562af server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:31.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f89dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:31.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.266+0000 7f89dbfff640 1 -- 172.21.15.129:0/1210896253 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec021020 con 0x55a748eea6b0 2024-05-20T18:08:31.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f89dbfff640 1 -- 172.21.15.129:0/1210896253 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f89ec01d420 con 0x55a748eea6b0 2024-05-20T18:08:31.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1210896253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89f809b790 con 0x55a748eea6b0 2024-05-20T18:08:31.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f89dbfff640 1 -- 172.21.15.129:0/1210896253 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f89ec02c050 con 0x55a748eea6b0 2024-05-20T18:08:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f8a09a1e740 10 client.17097 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1210896253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a74932ee40 con 0x55a748eea6b0 2024-05-20T18:08:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f89dbfff640 1 -- 172.21.15.129:0/1210896253 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f809b790 con 0x55a748eea6b0 2024-05-20T18:08:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f89dbfff640 1 -- 172.21.15.129:0/1210896253 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f89ec011b90 con 0x55a748eea6b0 2024-05-20T18:08:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f8a09a1e740 10 client.17097 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f8a09a1e740 10 client.17097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f8a09a1e740 20 client.17097 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a748e9f3a0 0x55a749331650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f8a09a1e740 4 client.17097 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:08:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1210896253 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a7492c1380 con 0x55a748e9f3a0 2024-05-20T18:08:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.267+0000 7f8a00835640 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a748e9f3a0 0x55a749331650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:31.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f89dbfff640 10 client.17097 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:31.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a00835640 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a748e9f3a0 0x55a749331650 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5a785a81bd55844 server_cookie=46466337551f2194 in_seq=0 out_seq=0 2024-05-20T18:08:31.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f89dbfff640 1 -- 172.21.15.129:0/1210896253 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a7492c1380 con 0x55a748e9f3a0 2024-05-20T18:08:31.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f89dbfff640 10 client.17097 handle_command_reply: tid=0 2024-05-20T18:08:31.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 1 client.17097 shutdown 2024-05-20T18:08:31.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1210896253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748eea6b0 msgr2=0x55a7492b7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748eea6b0 0x55a7492b7df0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x55a7492c71d0 tx=0x7f89ec00be80 comp rx=0 tx=0).stop 2024-05-20T18:08:31.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1210896253 shutdown_connections 2024-05-20T18:08:31.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55a748e9f3a0 0x55a749331650 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a748f0b390 0x55a7492b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a748eea6b0 0x55a7492b7df0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f89dbfff640 0 client.17097 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:31.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 1 --2- 172.21.15.129:0/1210896253 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a748ee24d0 0x55a7492bcaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1210896253 >> 172.21.15.129:0/1210896253 conn(0x55a748eff4e0 msgr2=0x55a7492b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1210896253 shutdown_connections 2024-05-20T18:08:31.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 1 -- 172.21.15.129:0/1210896253 wait complete. 2024-05-20T18:08:31.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.268+0000 7f8a09a1e740 20 client.17097 trim_cache size 0 max 16384 2024-05-20T18:08:31.275 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:08:31.275 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:08:31.276 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:08:31.276 INFO:teuthology.orchestra.run.smithi129.stdout: "5dcbd937-b6be-4d86-8afe-1fe96d197e51": { 2024-05-20T18:08:31.276 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:08:31.276 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "5dcbd937-b6be-4d86-8afe-1fe96d197e51", 2024-05-20T18:08:31.276 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:08:31.276 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:08:31.276 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:08:31.276 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:08:31.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 -- 172.21.15.129:0/2832263041 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f89e40a2d80 msgr2=0x7f89e40a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f89e40a2d80 0x7f89e40a5290 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f89f4004040 tx=0x7f89f4045000 comp rx=0 tx=0).stop 2024-05-20T18:08:31.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 -- 172.21.15.129:0/2832263041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89fc103c30 msgr2=0x7f89fc199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:31.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89fc103c30 0x7f89fc199750 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f89f800db80 tx=0x7f89f800b630 comp rx=0 tx=0).stop 2024-05-20T18:08:31.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 -- 172.21.15.129:0/2832263041 shutdown_connections 2024-05-20T18:08:31.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f89e40a2d80 0x7f89e40a5290 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89fc10a690 0x7f89fc199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f89fc10d040 0x7f89fc19df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 --2- 172.21.15.129:0/2832263041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89fc103c30 0x7f89fc199750 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:31.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 -- 172.21.15.129:0/2832263041 >> 172.21.15.129:0/2832263041 conn(0x7f89fc077100 msgr2=0x7f89fc1001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:31.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 -- 172.21.15.129:0/2832263041 shutdown_connections 2024-05-20T18:08:31.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:08:31.269+0000 7f8a02a99640 1 -- 172.21.15.129:0/2832263041 wait complete. 2024-05-20T18:08:31.279 INFO:tasks.cephfs.filesystem:scrub status for tag:5dcbd937-b6be-4d86-8afe-1fe96d197e51 - {'path': '/', 'tag': '5dcbd937-b6be-4d86-8afe-1fe96d197e51', 'options': 'recursive,force'} 2024-05-20T18:08:31.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:31 smithi154 ceph-mon[36482]: pgmap v309: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 409 B/s wr, 2 op/s 2024-05-20T18:08:31.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:31 smithi129 ceph-mon[32029]: pgmap v309: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 409 B/s wr, 2 op/s 2024-05-20T18:08:31.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:31 smithi204 ceph-mon[37076]: pgmap v309: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 409 B/s wr, 2 op/s 2024-05-20T18:08:32.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:32 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:08:32.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:32 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:32.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:32 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:08:32.643 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:32 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:32.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:32 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:08:32.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:32 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:32.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:32 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:08:32.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:32 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:32.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:32 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:08:32.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:32 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:32.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:32 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:08:32.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:32 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:08:33.580 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:33 smithi204 ceph-mon[37076]: pgmap v310: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 346 B/s wr, 2 op/s 2024-05-20T18:08:33.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:33 smithi154 ceph-mon[36482]: pgmap v310: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 346 B/s wr, 2 op/s 2024-05-20T18:08:33.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:33 smithi129 ceph-mon[32029]: pgmap v310: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 346 B/s wr, 2 op/s 2024-05-20T18:08:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.004+0000 7f6e0ea44640 1 -- 172.21.15.129:0/990145399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e08103250 msgr2=0x7f6e0810f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.004+0000 7f6e0ea44640 1 --2- 172.21.15.129:0/990145399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e08103250 0x7f6e0810f620 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009ca0 tx=0x7f6e0002a8c0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.004+0000 7f6e0ea44640 1 -- 172.21.15.129:0/990145399 shutdown_connections 2024-05-20T18:08:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.004+0000 7f6e0ea44640 1 --2- 172.21.15.129:0/990145399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e08103250 0x7f6e0810f620 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.004+0000 7f6e0ea44640 1 --2- 172.21.15.129:0/990145399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e08108fa0 0x7f6e08102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.004+0000 7f6e0ea44640 1 --2- 172.21.15.129:0/990145399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e08108b00 0x7f6e081026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.004+0000 7f6e0ea44640 1 -- 172.21.15.129:0/990145399 >> 172.21.15.129:0/990145399 conn(0x7f6e080fe1e0 msgr2=0x7f6e08100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.004+0000 7f6e0ea44640 1 -- 172.21.15.129:0/990145399 shutdown_connections 2024-05-20T18:08:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.004+0000 7f6e0ea44640 1 -- 172.21.15.129:0/990145399 wait complete. 2024-05-20T18:08:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.004+0000 7f6e0ea44640 1 Processor -- start 2024-05-20T18:08:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.004+0000 7f6e0ea44640 1 -- start start 2024-05-20T18:08:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e0ea44640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e08103250 0x7f6e081a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e0ea44640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e08108b00 0x7f6e081a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e0ea44640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e08108fa0 0x7f6e0819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e0ea44640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08076360 con 0x7f6e08103250 2024-05-20T18:08:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e0ea44640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e080761e0 con 0x7f6e08108fa0 2024-05-20T18:08:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e0ea44640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6e080764e0 con 0x7f6e08108b00 2024-05-20T18:08:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e07fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e08103250 0x7f6e081a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e07fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e08103250 0x7f6e081a1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33536/0 (socket says 172.21.15.129:33536) 2024-05-20T18:08:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e07fff640 1 -- 172.21.15.129:0/1401985182 learned_addr learned my addr 172.21.15.129:0/1401985182 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e0cfe1640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e08108fa0 0x7f6e0819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e077fe640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e08108b00 0x7f6e081a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e07fff640 1 -- 172.21.15.129:0/1401985182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e08108b00 msgr2=0x7f6e081a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e07fff640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e08108b00 0x7f6e081a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e07fff640 1 -- 172.21.15.129:0/1401985182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e08108fa0 msgr2=0x7f6e0819b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e07fff640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e08108fa0 0x7f6e0819b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e07fff640 1 -- 172.21.15.129:0/1401985182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0819be30 con 0x7f6e08103250 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e0cfe1640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e08108fa0 0x7f6e0819b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.005+0000 7f6e077fe640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e08108b00 0x7f6e081a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.006+0000 7f6e07fff640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e08103250 0x7f6e081a1590 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f6df800e9c0 tx=0x7f6df800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.006+0000 7f6e057fa640 1 -- 172.21.15.129:0/1401985182 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df800cd80 con 0x7f6e08103250 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.006+0000 7f6e057fa640 1 -- 172.21.15.129:0/1401985182 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6df8005500 con 0x7f6e08103250 2024-05-20T18:08:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.006+0000 7f6e0ea44640 1 -- 172.21.15.129:0/1401985182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0819c120 con 0x7f6e08103250 2024-05-20T18:08:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.006+0000 7f6e057fa640 1 -- 172.21.15.129:0/1401985182 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df8010690 con 0x7f6e08103250 2024-05-20T18:08:35.007 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.006+0000 7f6e0ea44640 1 -- 172.21.15.129:0/1401985182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e081a83e0 con 0x7f6e08103250 2024-05-20T18:08:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.008+0000 7f6e057fa640 1 -- 172.21.15.129:0/1401985182 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f6df80046c0 con 0x7f6e08103250 2024-05-20T18:08:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.008+0000 7f6e159ca740 1 Processor -- start 2024-05-20T18:08:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.008+0000 7f6e159ca740 1 -- start start 2024-05-20T18:08:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5821b0 0x55f21b921080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5592f0 0x55f21b9215c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b921b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e159ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f21b886340 con 0x55f21b5614d0 2024-05-20T18:08:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e159ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f21b82e290 con 0x55f21b5821b0 2024-05-20T18:08:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e159ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f21b882300 con 0x55f21b5592f0 2024-05-20T18:08:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e057fa640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6df00a2cf0 0x7f6df00a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e077fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b921b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e077fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b921b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33542/0 (socket says 172.21.15.129:33542) 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e077fe640 1 -- 172.21.15.129:0/1764668852 learned_addr learned my addr 172.21.15.129:0/1764668852 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e0cfe1640 1 --2- 172.21.15.129:0/1764668852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5821b0 0x55f21b921080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e0cfe1640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6df00a2cf0 0x7f6df00a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e057fa640 1 -- 172.21.15.129:0/1401985182 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f6df80cf980 con 0x7f6e08103250 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e07fff640 1 --2- 172.21.15.129:0/1764668852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5592f0 0x55f21b9215c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e077fe640 1 -- 172.21.15.129:0/1764668852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5592f0 msgr2=0x55f21b9215c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e077fe640 1 --2- 172.21.15.129:0/1764668852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5592f0 0x55f21b9215c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e077fe640 1 -- 172.21.15.129:0/1764668852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5821b0 msgr2=0x55f21b921080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e077fe640 1 --2- 172.21.15.129:0/1764668852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5821b0 0x55f21b921080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.009+0000 7f6e077fe640 1 -- 172.21.15.129:0/1764668852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f21b8528e0 con 0x55f21b5614d0 2024-05-20T18:08:35.013 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.010+0000 7f6e077fe640 1 --2- 172.21.15.129:0/1764668852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b921b00 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00bae0 tx=0x7f6dfc00d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af4cc3f9f47c5545 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.010+0000 7f6e0cfe1640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6df00a2cf0 0x7f6df00a5200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009bf0 tx=0x7f6e00005fd0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.010+0000 7f6dee7fc640 1 -- 172.21.15.129:0/1764668852 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc004020 con 0x55f21b5614d0 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.010+0000 7f6dee7fc640 1 -- 172.21.15.129:0/1764668852 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dfc0041c0 con 0x55f21b5614d0 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.010+0000 7f6dee7fc640 1 -- 172.21.15.129:0/1764668852 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc00f7a0 con 0x55f21b5614d0 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.011+0000 7f6e159ca740 1 -- 172.21.15.129:0/1764668852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 msgr2=0x55f21b921b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.011+0000 7f6e159ca740 1 --2- 172.21.15.129:0/1764668852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b921b00 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00bae0 tx=0x7f6dfc00d340 comp rx=0 tx=0).stop 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.011+0000 7f6e159ca740 1 -- 172.21.15.129:0/1764668852 shutdown_connections 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.011+0000 7f6e159ca740 1 --2- 172.21.15.129:0/1764668852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b921b00 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.011+0000 7f6e159ca740 1 --2- 172.21.15.129:0/1764668852 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5592f0 0x55f21b9215c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.011+0000 7f6e159ca740 1 --2- 172.21.15.129:0/1764668852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5821b0 0x55f21b921080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.011+0000 7f6e159ca740 1 -- 172.21.15.129:0/1764668852 >> 172.21.15.129:0/1764668852 conn(0x55f21b576300 msgr2=0x55f21b91d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.011+0000 7f6e159ca740 1 -- 172.21.15.129:0/1764668852 shutdown_connections 2024-05-20T18:08:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.011+0000 7f6e159ca740 1 -- 172.21.15.129:0/1764668852 wait complete. 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.011+0000 7f6e159ca740 1 -- start start 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.011+0000 7f6e159ca740 1 Processor -- start 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5592f0 0x55f21b92b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b92bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5821b0 0x55f21b9316a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e159ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f21b92e1b0 con 0x55f21b5614d0 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e159ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f21b841790 con 0x55f21b5592f0 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e159ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f21b92e330 con 0x55f21b5821b0 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e159ca740 10 client.0 _ll_register_callbacks cb 0x55f21b87ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e077fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b92bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e077fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b92bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33544/0 (socket says 172.21.15.129:33544) 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e077fe640 1 -- 172.21.15.129:0/367951698 learned_addr learned my addr 172.21.15.129:0/367951698 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e07fff640 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5592f0 0x55f21b92b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e0cfe1640 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5821b0 0x55f21b9316a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e077fe640 1 -- 172.21.15.129:0/367951698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5821b0 msgr2=0x55f21b9316a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e077fe640 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5821b0 0x55f21b9316a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e077fe640 1 -- 172.21.15.129:0/367951698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5592f0 msgr2=0x55f21b92b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e077fe640 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5592f0 0x55f21b92b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e077fe640 1 -- 172.21.15.129:0/367951698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f21b59a210 con 0x55f21b5614d0 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e07fff640 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5592f0 0x55f21b92b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6e077fe640 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b92bc80 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc003ea0 tx=0x7f6dfc004030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e45a4b5a36c8861 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.012+0000 7f6de77fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6de77fe640 1 -- 172.21.15.129:0/367951698 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc025020 con 0x55f21b5614d0 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6e159ca740 1 -- 172.21.15.129:0/367951698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dfc0041c0 con 0x55f21b5614d0 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6de77fe640 1 -- 172.21.15.129:0/367951698 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f6dfc0035f0 con 0x55f21b5614d0 2024-05-20T18:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6de77fe640 1 -- 172.21.15.129:0/367951698 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6dfc030050 con 0x55f21b5614d0 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6e159ca740 10 client.17109 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6e159ca740 1 -- 172.21.15.129:0/367951698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f21b666620 con 0x55f21b5614d0 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6de77fe640 1 -- 172.21.15.129:0/367951698 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dfc0041c0 con 0x55f21b5614d0 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6de77fe640 1 -- 172.21.15.129:0/367951698 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f6dfc00fcd0 con 0x55f21b5614d0 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6e159ca740 10 client.17109 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6e159ca740 10 client.17109 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6e159ca740 20 client.17109 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6e159ca740 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f21b540d80 0x55f21b93d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6e159ca740 4 client.17109 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6e159ca740 1 -- 172.21.15.129:0/367951698 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f21b893a70 con 0x55f21b540d80 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.013+0000 7f6e07fff640 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f21b540d80 0x55f21b93d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.014+0000 7f6de77fe640 10 client.17109 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.014+0000 7f6e07fff640 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f21b540d80 0x55f21b93d820 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c06de17fda4d61d server_cookie=9f2d093e8ec25808 in_seq=0 out_seq=0 2024-05-20T18:08:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6de77fe640 1 -- 172.21.15.129:0/367951698 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f21b893a70 con 0x55f21b540d80 2024-05-20T18:08:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6de77fe640 10 client.17109 handle_command_reply: tid=0 2024-05-20T18:08:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 1 client.17109 shutdown 2024-05-20T18:08:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 1 -- 172.21.15.129:0/367951698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 msgr2=0x55f21b92bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b92bc80 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc003ea0 tx=0x7f6dfc004030 comp rx=0 tx=0).stop 2024-05-20T18:08:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 1 -- 172.21.15.129:0/367951698 shutdown_connections 2024-05-20T18:08:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f21b540d80 0x55f21b93d820 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5821b0 0x55f21b9316a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6de77fe640 0 client.17109 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5614d0 0x55f21b92bc80 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 1 --2- 172.21.15.129:0/367951698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5592f0 0x55f21b92b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 1 -- 172.21.15.129:0/367951698 >> 172.21.15.129:0/367951698 conn(0x55f21b576300 msgr2=0x55f21b922da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 1 -- 172.21.15.129:0/367951698 shutdown_connections 2024-05-20T18:08:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 1 -- 172.21.15.129:0/367951698 wait complete. 2024-05-20T18:08:35.020 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.015+0000 7f6e159ca740 20 client.17109 trim_cache size 0 max 16384 2024-05-20T18:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.030+0000 7f6e159ca740 1 Processor -- start 2024-05-20T18:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.030+0000 7f6e159ca740 1 -- start start 2024-05-20T18:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.030+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5592f0 0x55f21b921640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.030+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5614d0 0x55f21b9315a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.030+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5821b0 0x55f21b931ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.030+0000 7f6e159ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081aed50 con 0x55f21b5592f0 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.030+0000 7f6e159ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081b15b0 con 0x55f21b5821b0 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.030+0000 7f6e159ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081af3a0 con 0x55f21b5614d0 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.030+0000 7f6e07fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5592f0 0x55f21b921640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.031+0000 7f6e07fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5592f0 0x55f21b921640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33546/0 (socket says 172.21.15.129:33546) 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.031+0000 7f6e07fff640 1 -- 172.21.15.129:0/3358437314 learned_addr learned my addr 172.21.15.129:0/3358437314 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.031+0000 7f6e07fff640 1 -- 172.21.15.129:0/3358437314 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5614d0 msgr2=0x55f21b9315a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.031+0000 7f6e07fff640 1 --2- 172.21.15.129:0/3358437314 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5614d0 0x55f21b9315a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.031+0000 7f6e07fff640 1 -- 172.21.15.129:0/3358437314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5821b0 msgr2=0x55f21b931ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.031+0000 7f6e07fff640 1 --2- 172.21.15.129:0/3358437314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5821b0 0x55f21b931ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.031+0000 7f6e07fff640 1 -- 172.21.15.129:0/3358437314 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f21b93dc80 con 0x55f21b5592f0 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.031+0000 7f6e07fff640 1 --2- 172.21.15.129:0/3358437314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5592f0 0x55f21b921640 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f6df80049d0 tx=0x7f6df80bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cef0b69f015b69 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.031+0000 7f6dedffb640 1 -- 172.21.15.129:0/3358437314 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df80a0210 con 0x55f21b5592f0 2024-05-20T18:08:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.031+0000 7f6dedffb640 1 -- 172.21.15.129:0/3358437314 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6df80a4ad0 con 0x55f21b5592f0 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.031+0000 7f6dedffb640 1 -- 172.21.15.129:0/3358437314 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df80a48a0 con 0x55f21b5592f0 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.032+0000 7f6e159ca740 1 -- 172.21.15.129:0/3358437314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5592f0 msgr2=0x55f21b921640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.032+0000 7f6e159ca740 1 --2- 172.21.15.129:0/3358437314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5592f0 0x55f21b921640 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f6df80049d0 tx=0x7f6df80bc040 comp rx=0 tx=0).stop 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.032+0000 7f6e159ca740 1 -- 172.21.15.129:0/3358437314 shutdown_connections 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.032+0000 7f6e159ca740 1 --2- 172.21.15.129:0/3358437314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5821b0 0x55f21b931ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.032+0000 7f6e159ca740 1 --2- 172.21.15.129:0/3358437314 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5614d0 0x55f21b9315a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.032+0000 7f6e159ca740 1 --2- 172.21.15.129:0/3358437314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5592f0 0x55f21b921640 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.032+0000 7f6e159ca740 1 -- 172.21.15.129:0/3358437314 >> 172.21.15.129:0/3358437314 conn(0x55f21b576300 msgr2=0x55f21b920f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.032+0000 7f6e159ca740 1 -- 172.21.15.129:0/3358437314 shutdown_connections 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.032+0000 7f6e159ca740 1 -- 172.21.15.129:0/3358437314 wait complete. 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.032+0000 7f6e159ca740 1 -- start start 2024-05-20T18:08:35.032 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.032+0000 7f6e159ca740 1 Processor -- start 2024-05-20T18:08:35.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5592f0 0x55f21b932eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5614d0 0x55f21b9333f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e159ca740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5821b0 0x55f21b9a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e159ca740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081afa70 con 0x55f21b5821b0 2024-05-20T18:08:35.035 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e159ca740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081b0ee0 con 0x55f21b5614d0 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e159ca740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6e081b0140 con 0x55f21b5592f0 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e159ca740 10 client.0 _ll_register_callbacks cb 0x7f6df8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e0cfe1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5821b0 0x55f21b9a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e0cfe1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5821b0 0x55f21b9a3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33550/0 (socket says 172.21.15.129:33550) 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e0cfe1640 1 -- 172.21.15.129:0/393333210 learned_addr learned my addr 172.21.15.129:0/393333210 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e07fff640 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5592f0 0x55f21b932eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e077fe640 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5614d0 0x55f21b9333f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e0cfe1640 1 -- 172.21.15.129:0/393333210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5592f0 msgr2=0x55f21b932eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e0cfe1640 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5592f0 0x55f21b932eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e0cfe1640 1 -- 172.21.15.129:0/393333210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5614d0 msgr2=0x55f21b9333f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e0cfe1640 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5614d0 0x55f21b9333f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e0cfe1640 1 -- 172.21.15.129:0/393333210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f21b9a3f90 con 0x55f21b5821b0 2024-05-20T18:08:35.036 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6e0cfe1640 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5821b0 0x55f21b9a3a50 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009ca0 tx=0x7f6e00005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b245adef30285823 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6de77fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.033+0000 7f6de77fe640 1 -- 172.21.15.129:0/393333210 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e00051070 con 0x55f21b5821b0 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6de77fe640 1 -- 172.21.15.129:0/393333210 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f6e0002fdf0 con 0x55f21b5821b0 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6e159ca740 1 -- 172.21.15.129:0/393333210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6df80a4ad0 con 0x55f21b5821b0 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6de77fe640 1 -- 172.21.15.129:0/393333210 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6e000585e0 con 0x55f21b5821b0 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6e159ca740 10 client.17115 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6e159ca740 1 -- 172.21.15.129:0/393333210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f21b9a5130 con 0x55f21b5821b0 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6de77fe640 1 -- 172.21.15.129:0/393333210 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6df80a4ad0 con 0x55f21b5821b0 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6de77fe640 1 -- 172.21.15.129:0/393333210 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f6e000305c0 con 0x55f21b5821b0 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6e159ca740 10 client.17115 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6e159ca740 10 client.17115 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6e159ca740 20 client.17115 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6e159ca740 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f21b5161c0 0x55f21b9a7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:35.037 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6e159ca740 4 client.17115 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:08:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6e159ca740 1 -- 172.21.15.129:0/393333210 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f21b88ddc0 con 0x55f21b5161c0 2024-05-20T18:08:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.034+0000 7f6e07fff640 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f21b5161c0 0x55f21b9a7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.035+0000 7f6de77fe640 10 client.17115 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.035+0000 7f6e07fff640 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f21b5161c0 0x55f21b9a7920 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6482e80b0a966386 server_cookie=363537a566dc813e in_seq=0 out_seq=0 2024-05-20T18:08:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.039+0000 7f6de77fe640 1 -- 172.21.15.129:0/393333210 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f21b88ddc0 con 0x55f21b5161c0 2024-05-20T18:08:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.039+0000 7f6de77fe640 10 client.17115 handle_command_reply: tid=0 2024-05-20T18:08:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.039+0000 7f6e159ca740 1 client.17115 shutdown 2024-05-20T18:08:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e159ca740 1 -- 172.21.15.129:0/393333210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5821b0 msgr2=0x55f21b9a3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e159ca740 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5821b0 0x55f21b9a3a50 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009ca0 tx=0x7f6e00005df0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e159ca740 1 -- 172.21.15.129:0/393333210 shutdown_connections 2024-05-20T18:08:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e159ca740 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55f21b5161c0 0x55f21b9a7920 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e159ca740 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21b5614d0 0x55f21b9333f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e159ca740 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f21b5821b0 0x55f21b9a3a50 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e159ca740 1 --2- 172.21.15.129:0/393333210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f21b5592f0 0x55f21b932eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e159ca740 1 -- 172.21.15.129:0/393333210 >> 172.21.15.129:0/393333210 conn(0x55f21b576300 msgr2=0x55f21b9252d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6de77fe640 0 client.17115 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e159ca740 1 -- 172.21.15.129:0/393333210 shutdown_connections 2024-05-20T18:08:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e159ca740 1 -- 172.21.15.129:0/393333210 wait complete. 2024-05-20T18:08:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e159ca740 20 client.17115 trim_cache size 0 max 16384 2024-05-20T18:08:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:08:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:08:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:08:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:08:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e0ea44640 1 -- 172.21.15.129:0/1401985182 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6df00a2cf0 msgr2=0x7f6df00a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e0ea44640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6df00a2cf0 0x7f6df00a5200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009bf0 tx=0x7f6e00005fd0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e0ea44640 1 -- 172.21.15.129:0/1401985182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e08103250 msgr2=0x7f6e081a1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e0ea44640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e08103250 0x7f6e081a1590 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f6df800e9c0 tx=0x7f6df800ee80 comp rx=0 tx=0).stop 2024-05-20T18:08:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e0ea44640 1 -- 172.21.15.129:0/1401985182 shutdown_connections 2024-05-20T18:08:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e0ea44640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6df00a2cf0 0x7f6df00a5200 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e0ea44640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e08108fa0 0x7f6e0819b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e0ea44640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e08108b00 0x7f6e081a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e0ea44640 1 --2- 172.21.15.129:0/1401985182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e08103250 0x7f6e081a1590 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.040+0000 7f6e0ea44640 1 -- 172.21.15.129:0/1401985182 >> 172.21.15.129:0/1401985182 conn(0x7f6e080fe1e0 msgr2=0x7f6e08104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.041+0000 7f6e0ea44640 1 -- 172.21.15.129:0/1401985182 shutdown_connections 2024-05-20T18:08:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:35.041+0000 7f6e0ea44640 1 -- 172.21.15.129:0/1401985182 wait complete. 2024-05-20T18:08:35.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:35 smithi154 ceph-mon[36482]: pgmap v311: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 341 B/s wr, 2 op/s 2024-05-20T18:08:35.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:35 smithi129 ceph-mon[32029]: pgmap v311: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 341 B/s wr, 2 op/s 2024-05-20T18:08:35.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:35 smithi204 ceph-mon[37076]: pgmap v311: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 341 B/s wr, 2 op/s 2024-05-20T18:08:36.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:36 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:08:36.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:36 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:08:36.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:36 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:08:37.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:37 smithi154 ceph-mon[36482]: pgmap v312: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-20T18:08:37.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:37 smithi129 ceph-mon[32029]: pgmap v312: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-20T18:08:37.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:37 smithi204 ceph-mon[37076]: pgmap v312: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-20T18:08:39.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:39 smithi154 ceph-mon[36482]: pgmap v313: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 4 op/s 2024-05-20T18:08:39.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:39 smithi129 ceph-mon[32029]: pgmap v313: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 4 op/s 2024-05-20T18:08:39.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:39 smithi204 ceph-mon[37076]: pgmap v313: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 4 op/s 2024-05-20T18:08:41.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:41 smithi154 ceph-mon[36482]: pgmap v314: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-20T18:08:41.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:41 smithi129 ceph-mon[32029]: pgmap v314: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-20T18:08:41.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:41 smithi204 ceph-mon[37076]: pgmap v314: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-20T18:08:43.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:43 smithi154 ceph-mon[36482]: pgmap v315: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:43.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:43 smithi129 ceph-mon[32029]: pgmap v315: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:43.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:43 smithi204 ceph-mon[37076]: pgmap v315: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:45.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:45 smithi154 ceph-mon[36482]: pgmap v316: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:45.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:45 smithi129 ceph-mon[32029]: pgmap v316: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:45.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:45 smithi204 ceph-mon[37076]: pgmap v316: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:47.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:47 smithi154 ceph-mon[36482]: pgmap v317: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:47.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:47 smithi129 ceph-mon[32029]: pgmap v317: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:47.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:47 smithi204 ceph-mon[37076]: pgmap v317: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:08:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.150+0000 7f28fa110640 1 -- 172.21.15.129:0/4230039165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f410d090 msgr2=0x7f28f410f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:48.148 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.150+0000 7f28fa110640 1 --2- 172.21.15.129:0/4230039165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f410d090 0x7f28f410f580 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009ca0 tx=0x7f28ec02a9e0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.150+0000 7f28fa110640 1 -- 172.21.15.129:0/4230039165 shutdown_connections 2024-05-20T18:08:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.150+0000 7f28fa110640 1 --2- 172.21.15.129:0/4230039165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f410d090 0x7f28f410f580 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.150+0000 7f28fa110640 1 --2- 172.21.15.129:0/4230039165 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f410a6e0 0x7f28f410cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.150+0000 7f28fa110640 1 --2- 172.21.15.129:0/4230039165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f4103c80 0x7f28f4104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.150+0000 7f28fa110640 1 -- 172.21.15.129:0/4230039165 >> 172.21.15.129:0/4230039165 conn(0x7f28f4077160 msgr2=0x7f28f40775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.150+0000 7f28fa110640 1 -- 172.21.15.129:0/4230039165 shutdown_connections 2024-05-20T18:08:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.150+0000 7f28fa110640 1 -- 172.21.15.129:0/4230039165 wait complete. 2024-05-20T18:08:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.151+0000 7f28fa110640 1 Processor -- start 2024-05-20T18:08:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.151+0000 7f28fa110640 1 -- start start 2024-05-20T18:08:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28fa110640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f4103c80 0x7f28f4199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28fa110640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f410a6e0 0x7f28f4199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28fa110640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f410d090 0x7f28f419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28fa110640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f28f4112e40 con 0x7f28f4103c80 2024-05-20T18:08:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28fa110640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28f4112cc0 con 0x7f28f410a6e0 2024-05-20T18:08:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28fa110640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f28f4112fc0 con 0x7f28f410d090 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f4103c80 0x7f28f4199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f4103c80 0x7f28f4199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51600/0 (socket says 172.21.15.129:51600) 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f37fe640 1 -- 172.21.15.129:0/1885913845 learned_addr learned my addr 172.21.15.129:0/1885913845 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f2ffd640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f410a6e0 0x7f28f4199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f3fff640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f410d090 0x7f28f419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f37fe640 1 -- 172.21.15.129:0/1885913845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f410d090 msgr2=0x7f28f419df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f37fe640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f410d090 0x7f28f419df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f37fe640 1 -- 172.21.15.129:0/1885913845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f410a6e0 msgr2=0x7f28f4199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f37fe640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f410a6e0 0x7f28f4199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f37fe640 1 -- 172.21.15.129:0/1885913845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f419e5a0 con 0x7f28f4103c80 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f37fe640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f4103c80 0x7f28f4199740 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f28e400d950 tx=0x7f28e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f0ff9640 1 -- 172.21.15.129:0/1885913845 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4017070 con 0x7f28f4103c80 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f0ff9640 1 -- 172.21.15.129:0/1885913845 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e4009c30 con 0x7f28f4103c80 2024-05-20T18:08:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28fa110640 1 -- 172.21.15.129:0/1885913845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f419e890 con 0x7f28f4103c80 2024-05-20T18:08:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28f0ff9640 1 -- 172.21.15.129:0/1885913845 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e400f6b0 con 0x7f28f4103c80 2024-05-20T18:08:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.152+0000 7f28fa110640 1 -- 172.21.15.129:0/1885913845 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f41a6210 con 0x7f28f4103c80 2024-05-20T18:08:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.155+0000 7f28f0ff9640 1 -- 172.21.15.129:0/1885913845 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f28e40109d0 con 0x7f28f4103c80 2024-05-20T18:08:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.155+0000 7f2900fbc740 1 Processor -- start 2024-05-20T18:08:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.155+0000 7f2900fbc740 1 -- start start 2024-05-20T18:08:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f0ff9640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f28dc0a2cf0 0x7f28dc0a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e09080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a412f0 0x55dbe7e095c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a494d0 0x55dbe7e09b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f2900fbc740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dbe7d6e340 con 0x55dbe7a494d0 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f2900fbc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dbe7d16290 con 0x55dbe7a412f0 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f2900fbc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55dbe7d6a300 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f3fff640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f28dc0a2cf0 0x7f28dc0a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f0ff9640 1 -- 172.21.15.129:0/1885913845 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f28e40cf120 con 0x7f28f4103c80 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a494d0 0x55dbe7e09b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a494d0 0x55dbe7e09b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51612/0 (socket says 172.21.15.129:51612) 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f3fff640 1 -- 172.21.15.129:0/737567599 learned_addr learned my addr 172.21.15.129:0/737567599 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f37fe640 1 --2- 172.21.15.129:0/737567599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e09080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f2ffd640 1 --2- 172.21.15.129:0/737567599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a412f0 0x55dbe7e095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f37fe640 1 -- 172.21.15.129:0/737567599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a412f0 msgr2=0x55dbe7e095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f37fe640 1 --2- 172.21.15.129:0/737567599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a412f0 0x55dbe7e095c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f37fe640 1 -- 172.21.15.129:0/737567599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a494d0 msgr2=0x55dbe7e09b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f37fe640 1 --2- 172.21.15.129:0/737567599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a494d0 0x55dbe7e09b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f37fe640 1 -- 172.21.15.129:0/737567599 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbe7d3a8e0 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f3fff640 1 --2- 172.21.15.129:0/737567599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a494d0 0x55dbe7e09b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.156+0000 7f28f3fff640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f28dc0a2cf0 0x7f28dc0a5200 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009a60 tx=0x7f28ec0364d0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.157+0000 7f28f37fe640 1 --2- 172.21.15.129:0/737567599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e09080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f28e40cc830 tx=0x7f28e4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=287cd3501437b1f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.157+0000 7f28d9ffb640 1 -- 172.21.15.129:0/737567599 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4020070 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.157+0000 7f28d9ffb640 1 -- 172.21.15.129:0/737567599 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e4009de0 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.157+0000 7f28d9ffb640 1 -- 172.21.15.129:0/737567599 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4097040 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.158+0000 7f2900fbc740 1 -- 172.21.15.129:0/737567599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a6a1b0 msgr2=0x55dbe7e09080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.158+0000 7f2900fbc740 1 --2- 172.21.15.129:0/737567599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e09080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f28e40cc830 tx=0x7f28e4005f90 comp rx=0 tx=0).stop 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.158+0000 7f2900fbc740 1 -- 172.21.15.129:0/737567599 shutdown_connections 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.158+0000 7f2900fbc740 1 --2- 172.21.15.129:0/737567599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a412f0 0x55dbe7e095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.158+0000 7f2900fbc740 1 --2- 172.21.15.129:0/737567599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a494d0 0x55dbe7e09b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.158+0000 7f2900fbc740 1 --2- 172.21.15.129:0/737567599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e09080 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.158+0000 7f2900fbc740 1 -- 172.21.15.129:0/737567599 >> 172.21.15.129:0/737567599 conn(0x55dbe7a5e300 msgr2=0x55dbe7e05830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.158+0000 7f2900fbc740 1 -- 172.21.15.129:0/737567599 shutdown_connections 2024-05-20T18:08:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.158+0000 7f2900fbc740 1 -- 172.21.15.129:0/737567599 wait complete. 2024-05-20T18:08:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.158+0000 7f2900fbc740 1 -- start start 2024-05-20T18:08:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.158+0000 7f2900fbc740 1 Processor -- start 2024-05-20T18:08:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e15140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e15680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e13650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f2900fbc740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dbe7e160f0 con 0x55dbe7a412f0 2024-05-20T18:08:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f2900fbc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dbe7d29790 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f2900fbc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55dbe7e16270 con 0x55dbe7a494d0 2024-05-20T18:08:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f2900fbc740 10 client.0 _ll_register_callbacks cb 0x55dbe7d64a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e15140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e15140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51626/0 (socket says 172.21.15.129:51626) 2024-05-20T18:08:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28f37fe640 1 -- 172.21.15.129:0/1657430944 learned_addr learned my addr 172.21.15.129:0/1657430944 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28f2ffd640 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e15680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28f3fff640 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e13650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28f37fe640 1 -- 172.21.15.129:0/1657430944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 msgr2=0x55dbe7e15680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28f37fe640 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e15680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28f37fe640 1 -- 172.21.15.129:0/1657430944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a6a1b0 msgr2=0x55dbe7e13650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28f37fe640 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e13650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28f37fe640 1 -- 172.21.15.129:0/1657430944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbe7a82210 con 0x55dbe7a412f0 2024-05-20T18:08:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28f37fe640 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e15140 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f28e40cc8e0 tx=0x7f28e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463b8a8d451d8265 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1657430944 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4020050 con 0x55dbe7a412f0 2024-05-20T18:08:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1657430944 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f28e40afb20 con 0x55dbe7a412f0 2024-05-20T18:08:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.159+0000 7f2900fbc740 1 -- 172.21.15.129:0/1657430944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28e4009de0 con 0x55dbe7a412f0 2024-05-20T18:08:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1657430944 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f28e4097040 con 0x55dbe7a412f0 2024-05-20T18:08:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f2900fbc740 10 client.17127 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f2900fbc740 1 -- 172.21.15.129:0/1657430944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dbe7b4e620 con 0x55dbe7a412f0 2024-05-20T18:08:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1657430944 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e4009de0 con 0x55dbe7a412f0 2024-05-20T18:08:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1657430944 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f28e40b10e0 con 0x55dbe7a412f0 2024-05-20T18:08:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f2900fbc740 10 client.17127 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f2900fbc740 10 client.17127 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f2900fbc740 20 client.17127 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dbe7a28d80 0x55dbe7e257a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f2900fbc740 4 client.17127 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:08:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f2900fbc740 1 -- 172.21.15.129:0/1657430944 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbe7d7ba70 con 0x55dbe7a28d80 2024-05-20T18:08:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.160+0000 7f28f2ffd640 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dbe7a28d80 0x55dbe7e257a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.161+0000 7f28d2ffd640 10 client.17127 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.161+0000 7f28f2ffd640 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dbe7a28d80 0x55dbe7e257a0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dd80c454ebcd582 server_cookie=157b3d27c5eb7d40 in_seq=0 out_seq=0 2024-05-20T18:08:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1657430944 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55dbe7d7ba70 con 0x55dbe7a28d80 2024-05-20T18:08:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f28d2ffd640 10 client.17127 handle_command_reply: tid=0 2024-05-20T18:08:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 1 client.17127 shutdown 2024-05-20T18:08:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 1 -- 172.21.15.129:0/1657430944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 msgr2=0x55dbe7e15140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e15140 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f28e40cc8e0 tx=0x7f28e4005880 comp rx=0 tx=0).stop 2024-05-20T18:08:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 1 -- 172.21.15.129:0/1657430944 shutdown_connections 2024-05-20T18:08:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dbe7a28d80 0x55dbe7e257a0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e13650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e15680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f28d2ffd640 0 client.17127 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1657430944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e15140 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 1 -- 172.21.15.129:0/1657430944 >> 172.21.15.129:0/1657430944 conn(0x55dbe7a5e300 msgr2=0x55dbe7e08e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 1 -- 172.21.15.129:0/1657430944 shutdown_connections 2024-05-20T18:08:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 1 -- 172.21.15.129:0/1657430944 wait complete. 2024-05-20T18:08:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.162+0000 7f2900fbc740 20 client.17127 trim_cache size 0 max 16384 2024-05-20T18:08:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.177+0000 7f2900fbc740 1 Processor -- start 2024-05-20T18:08:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.177+0000 7f2900fbc740 1 -- start start 2024-05-20T18:08:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.177+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e09520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.177+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e287d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.177+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e28d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.177+0000 7f2900fbc740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41acba0 con 0x55dbe7a412f0 2024-05-20T18:08:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.177+0000 7f2900fbc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41af400 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.177+0000 7f2900fbc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41ad1f0 con 0x55dbe7a494d0 2024-05-20T18:08:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e09520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e09520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51630/0 (socket says 172.21.15.129:51630) 2024-05-20T18:08:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28f37fe640 1 -- 172.21.15.129:0/4082548634 learned_addr learned my addr 172.21.15.129:0/4082548634 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28f2ffd640 1 --2- 172.21.15.129:0/4082548634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e287d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28f3fff640 1 --2- 172.21.15.129:0/4082548634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e28d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28f37fe640 1 -- 172.21.15.129:0/4082548634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 msgr2=0x55dbe7e287d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28f37fe640 1 --2- 172.21.15.129:0/4082548634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e287d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28f37fe640 1 -- 172.21.15.129:0/4082548634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a6a1b0 msgr2=0x55dbe7e28d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28f37fe640 1 --2- 172.21.15.129:0/4082548634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e28d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28f37fe640 1 -- 172.21.15.129:0/4082548634 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbe7e063e0 con 0x55dbe7a412f0 2024-05-20T18:08:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28f37fe640 1 --2- 172.21.15.129:0/4082548634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e09520 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f28e40998b0 tx=0x7f28e4099b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853cced8a16ee84b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28d97fa640 1 -- 172.21.15.129:0/4082548634 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4020070 con 0x55dbe7a412f0 2024-05-20T18:08:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28d97fa640 1 -- 172.21.15.129:0/4082548634 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e409f9b0 con 0x55dbe7a412f0 2024-05-20T18:08:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.178+0000 7f28d97fa640 1 -- 172.21.15.129:0/4082548634 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4097de0 con 0x55dbe7a412f0 2024-05-20T18:08:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.179+0000 7f2900fbc740 1 -- 172.21.15.129:0/4082548634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 msgr2=0x55dbe7e09520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.179+0000 7f2900fbc740 1 --2- 172.21.15.129:0/4082548634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e09520 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f28e40998b0 tx=0x7f28e4099b10 comp rx=0 tx=0).stop 2024-05-20T18:08:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.179+0000 7f2900fbc740 1 -- 172.21.15.129:0/4082548634 shutdown_connections 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.179+0000 7f2900fbc740 1 --2- 172.21.15.129:0/4082548634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e28d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.179+0000 7f2900fbc740 1 --2- 172.21.15.129:0/4082548634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e287d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.179+0000 7f2900fbc740 1 --2- 172.21.15.129:0/4082548634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a412f0 0x55dbe7e09520 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.179+0000 7f2900fbc740 1 -- 172.21.15.129:0/4082548634 >> 172.21.15.129:0/4082548634 conn(0x55dbe7a5e300 msgr2=0x55dbe7e08e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.179+0000 7f2900fbc740 1 -- 172.21.15.129:0/4082548634 shutdown_connections 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.179+0000 7f2900fbc740 1 -- 172.21.15.129:0/4082548634 wait complete. 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.179+0000 7f2900fbc740 1 -- start start 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f2900fbc740 1 Processor -- start 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a412f0 0x55dbe7e85680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e85bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f2900fbc740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e7fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f2900fbc740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41ad8c0 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f2900fbc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41aed30 con 0x55dbe7a412f0 2024-05-20T18:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f2900fbc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41adf90 con 0x55dbe7a494d0 2024-05-20T18:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f2900fbc740 10 client.0 _ll_register_callbacks cb 0x7f28ec009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f28f3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e7fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f28f3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e7fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51642/0 (socket says 172.21.15.129:51642) 2024-05-20T18:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f28f3fff640 1 -- 172.21.15.129:0/1484945101 learned_addr learned my addr 172.21.15.129:0/1484945101 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f28f37fe640 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a412f0 0x55dbe7e85680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f28f2ffd640 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e85bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f28f3fff640 1 -- 172.21.15.129:0/1484945101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 msgr2=0x55dbe7e85bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f28f3fff640 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e85bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f28f3fff640 1 -- 172.21.15.129:0/1484945101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a412f0 msgr2=0x55dbe7e85680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f28f3fff640 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a412f0 0x55dbe7e85680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.180+0000 7f28f3fff640 1 -- 172.21.15.129:0/1484945101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbe7e80080 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f28f3fff640 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e7fa70 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f28ec02ec20 tx=0x7f28ec048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606e9368f566c0dd server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f28d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1484945101 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec033050 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1484945101 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f28ec02fbc0 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f2900fbc740 1 -- 172.21.15.129:0/1484945101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28e409f9b0 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1484945101 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f28ec060050 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f2900fbc740 10 client.17133 fetch_fsmap learned FSMap version 26 2024-05-20T18:08:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f2900fbc740 1 -- 172.21.15.129:0/1484945101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dbe7e8cd30 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1484945101 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e409f9b0 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1484945101 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f28ec048580 con 0x55dbe7a6a1b0 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f2900fbc740 10 client.17133 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f2900fbc740 10 client.17133 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f2900fbc740 20 client.17133 populate_metadata read hostname 'smithi129' 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dbe79fe1c0 0x55dbe7e8f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f2900fbc740 4 client.17133 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.181+0000 7f2900fbc740 1 -- 172.21.15.129:0/1484945101 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dbe7d75dc0 con 0x55dbe79fe1c0 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.182+0000 7f28f37fe640 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dbe79fe1c0 0x55dbe7e8f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.182+0000 7f28d2ffd640 10 client.17133 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.182+0000 7f28f37fe640 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dbe79fe1c0 0x55dbe7e8f540 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5fd63aea544d9fd server_cookie=7abcce62983217c3 in_seq=0 out_seq=0 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f28d2ffd640 1 -- 172.21.15.129:0/1484945101 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dbe7d75dc0 con 0x55dbe79fe1c0 2024-05-20T18:08:48.199 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f28d2ffd640 10 client.17133 handle_command_reply: tid=0 2024-05-20T18:08:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 1 client.17133 shutdown 2024-05-20T18:08:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 1 -- 172.21.15.129:0/1484945101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a6a1b0 msgr2=0x55dbe7e7fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e7fa70 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f28ec02ec20 tx=0x7f28ec048000 comp rx=0 tx=0).stop 2024-05-20T18:08:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 1 -- 172.21.15.129:0/1484945101 shutdown_connections 2024-05-20T18:08:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55dbe79fe1c0 0x55dbe7e8f540 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.200 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbe7a6a1b0 0x55dbe7e7fa70 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55dbe7a494d0 0x55dbe7e85bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f28d2ffd640 0 client.17133 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:08:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 1 --2- 172.21.15.129:0/1484945101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dbe7a412f0 0x55dbe7e85680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 1 -- 172.21.15.129:0/1484945101 >> 172.21.15.129:0/1484945101 conn(0x55dbe7a5e300 msgr2=0x55dbe7e0cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 1 -- 172.21.15.129:0/1484945101 shutdown_connections 2024-05-20T18:08:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 1 -- 172.21.15.129:0/1484945101 wait complete. 2024-05-20T18:08:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.188+0000 7f2900fbc740 20 client.17133 trim_cache size 0 max 16384 2024-05-20T18:08:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:08:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:08:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:08:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:08:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 -- 172.21.15.129:0/1885913845 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f28dc0a2cf0 msgr2=0x7f28dc0a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f28dc0a2cf0 0x7f28dc0a5200 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009a60 tx=0x7f28ec0364d0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 -- 172.21.15.129:0/1885913845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f4103c80 msgr2=0x7f28f4199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:08:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f4103c80 0x7f28f4199740 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f28e400d950 tx=0x7f28e400de10 comp rx=0 tx=0).stop 2024-05-20T18:08:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 -- 172.21.15.129:0/1885913845 shutdown_connections 2024-05-20T18:08:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f28dc0a2cf0 0x7f28dc0a5200 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28f410a6e0 0x7f28f4199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f410d090 0x7f28f419df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 --2- 172.21.15.129:0/1885913845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28f4103c80 0x7f28f4199740 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:08:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 -- 172.21.15.129:0/1885913845 >> 172.21.15.129:0/1885913845 conn(0x7f28f4077160 msgr2=0x7f28f4100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:08:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 -- 172.21.15.129:0/1885913845 shutdown_connections 2024-05-20T18:08:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:08:48.189+0000 7f28fa110640 1 -- 172.21.15.129:0/1885913845 wait complete. 2024-05-20T18:08:49.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:49 smithi154 ceph-mon[36482]: pgmap v318: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:49.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:49 smithi129 ceph-mon[32029]: pgmap v318: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:49.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:49 smithi204 ceph-mon[37076]: pgmap v318: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:51.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:51 smithi154 ceph-mon[36482]: pgmap v319: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-20T18:08:51.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:51 smithi129 ceph-mon[32029]: pgmap v319: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-20T18:08:51.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:51 smithi204 ceph-mon[37076]: pgmap v319: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-20T18:08:53.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:53 smithi154 ceph-mon[36482]: pgmap v320: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:53.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:53 smithi129 ceph-mon[32029]: pgmap v320: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:53.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:53 smithi204 ceph-mon[37076]: pgmap v320: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:55.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:55 smithi154 ceph-mon[36482]: pgmap v321: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:55.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:55 smithi129 ceph-mon[32029]: pgmap v321: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:55.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:55 smithi204 ceph-mon[37076]: pgmap v321: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:57.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:57 smithi154 ceph-mon[36482]: pgmap v322: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:57.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:57 smithi129 ceph-mon[32029]: pgmap v322: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:57.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:57 smithi204 ceph-mon[37076]: pgmap v322: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:59.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:08:59 smithi154 ceph-mon[36482]: pgmap v323: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:59.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:08:59 smithi129 ceph-mon[32029]: pgmap v323: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:08:59.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:08:59 smithi204 ceph-mon[37076]: pgmap v323: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:01.281 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:09:01.281 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:09:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.300+0000 7f591a777640 1 -- 172.21.15.129:0/3751828895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5914103c20 msgr2=0x7f5914104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:01.298 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.300+0000 7f591a777640 1 --2- 172.21.15.129:0/3751828895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5914103c20 0x7f5914104030 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f5904009900 tx=0x7f590402a410 comp rx=0 tx=0).stop 2024-05-20T18:09:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.300+0000 7f591a777640 1 -- 172.21.15.129:0/3751828895 shutdown_connections 2024-05-20T18:09:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.300+0000 7f591a777640 1 --2- 172.21.15.129:0/3751828895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f591410d070 0x7f591410f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.300+0000 7f591a777640 1 --2- 172.21.15.129:0/3751828895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591410a680 0x7f591410caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.300+0000 7f591a777640 1 --2- 172.21.15.129:0/3751828895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5914103c20 0x7f5914104030 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.300+0000 7f591a777640 1 -- 172.21.15.129:0/3751828895 >> 172.21.15.129:0/3751828895 conn(0x7f5914077160 msgr2=0x7f59140775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.300+0000 7f591a777640 1 -- 172.21.15.129:0/3751828895 shutdown_connections 2024-05-20T18:09:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.300+0000 7f591a777640 1 -- 172.21.15.129:0/3751828895 wait complete. 2024-05-20T18:09:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.301+0000 7f591a777640 1 Processor -- start 2024-05-20T18:09:01.299 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.301+0000 7f591a777640 1 -- start start 2024-05-20T18:09:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.301+0000 7f591a777640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5914103c20 0x7f59141014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.301+0000 7f591a777640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591410a680 0x7f5914101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.301+0000 7f591a777640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f591410d070 0x7f5914101f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.301+0000 7f591a777640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5914112e70 con 0x7f5914103c20 2024-05-20T18:09:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.301+0000 7f591a777640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5914112cf0 con 0x7f591410d070 2024-05-20T18:09:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.301+0000 7f591a777640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5914112ff0 con 0x7f591410a680 2024-05-20T18:09:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f5913fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5914103c20 0x7f59141014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f5913fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5914103c20 0x7f59141014e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58812/0 (socket says 172.21.15.129:58812) 2024-05-20T18:09:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f5913fff640 1 -- 172.21.15.129:0/231983447 learned_addr learned my addr 172.21.15.129:0/231983447 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f5918d14640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f591410d070 0x7f5914101f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f59137fe640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591410a680 0x7f5914101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f5913fff640 1 -- 172.21.15.129:0/231983447 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591410a680 msgr2=0x7f5914101a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f5913fff640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591410a680 0x7f5914101a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f5913fff640 1 -- 172.21.15.129:0/231983447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f591410d070 msgr2=0x7f5914101f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f5913fff640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f591410d070 0x7f5914101f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f5913fff640 1 -- 172.21.15.129:0/231983447 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59141a7b30 con 0x7f5914103c20 2024-05-20T18:09:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f5913fff640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5914103c20 0x7f59141014e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f5904004770 tx=0x7f5904003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:01.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f59117fa640 1 -- 172.21.15.129:0/231983447 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590403f070 con 0x7f5914103c20 2024-05-20T18:09:01.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f59117fa640 1 -- 172.21.15.129:0/231983447 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5904031ca0 con 0x7f5914103c20 2024-05-20T18:09:01.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f591a777640 1 -- 172.21.15.129:0/231983447 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59141a7dc0 con 0x7f5914103c20 2024-05-20T18:09:01.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.302+0000 7f59117fa640 1 -- 172.21.15.129:0/231983447 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904035030 con 0x7f5914103c20 2024-05-20T18:09:01.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.303+0000 7f591a777640 1 -- 172.21.15.129:0/231983447 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59141a81f0 con 0x7f5914103c20 2024-05-20T18:09:01.305 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.305+0000 7f59117fa640 1 -- 172.21.15.129:0/231983447 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5904042070 con 0x7f5914103c20 2024-05-20T18:09:01.311 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.305+0000 7f592161d740 1 Processor -- start 2024-05-20T18:09:01.311 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f592161d740 1 -- start start 2024-05-20T18:09:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f592161d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a27080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f592161d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a275c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f592161d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a27b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f592161d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561d8198c340 con 0x561d816674d0 2024-05-20T18:09:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f592161d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561d81934290 con 0x561d8165f2f0 2024-05-20T18:09:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f592161d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561d81988300 con 0x561d816881b0 2024-05-20T18:09:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f59117fa640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58fc0a2cf0 0x7f58fc0a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f59137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a27b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f59137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a27b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58824/0 (socket says 172.21.15.129:58824) 2024-05-20T18:09:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f5918d14640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58fc0a2cf0 0x7f58fc0a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.312 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f59137fe640 1 -- 172.21.15.129:0/3180125904 learned_addr learned my addr 172.21.15.129:0/3180125904 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f59117fa640 1 -- 172.21.15.129:0/231983447 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f59040efa50 con 0x7f5914103c20 2024-05-20T18:09:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f5913fff640 1 --2- 172.21.15.129:0/3180125904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a275c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f5918d14640 1 --2- 172.21.15.129:0/3180125904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a27080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f59137fe640 1 -- 172.21.15.129:0/3180125904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 msgr2=0x561d81a27080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f59137fe640 1 --2- 172.21.15.129:0/3180125904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a27080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f59137fe640 1 -- 172.21.15.129:0/3180125904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 msgr2=0x561d81a275c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f59137fe640 1 --2- 172.21.15.129:0/3180125904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a275c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.306+0000 7f59137fe640 1 -- 172.21.15.129:0/3180125904 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d819588e0 con 0x561d816674d0 2024-05-20T18:09:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.307+0000 7f5918d14640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58fc0a2cf0 0x7f58fc0a5200 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f590c005fd0 tx=0x7f590c011b40 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.307+0000 7f59137fe640 1 --2- 172.21.15.129:0/3180125904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a27b00 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f5908009ca0 tx=0x7f5908011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d4f02930741893 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:01.313 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.307+0000 7f58fa7fc640 1 -- 172.21.15.129:0/3180125904 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5908019620 con 0x561d816674d0 2024-05-20T18:09:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.307+0000 7f58fa7fc640 1 -- 172.21.15.129:0/3180125904 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5908019ca0 con 0x561d816674d0 2024-05-20T18:09:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.307+0000 7f58fa7fc640 1 -- 172.21.15.129:0/3180125904 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5908018b20 con 0x561d816674d0 2024-05-20T18:09:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.308+0000 7f592161d740 1 -- 172.21.15.129:0/3180125904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 msgr2=0x561d81a27b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.308+0000 7f592161d740 1 --2- 172.21.15.129:0/3180125904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a27b00 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f5908009ca0 tx=0x7f5908011810 comp rx=0 tx=0).stop 2024-05-20T18:09:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.308+0000 7f592161d740 1 -- 172.21.15.129:0/3180125904 shutdown_connections 2024-05-20T18:09:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.308+0000 7f592161d740 1 --2- 172.21.15.129:0/3180125904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a275c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.308+0000 7f592161d740 1 --2- 172.21.15.129:0/3180125904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a27b00 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.308+0000 7f592161d740 1 --2- 172.21.15.129:0/3180125904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a27080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.308+0000 7f592161d740 1 -- 172.21.15.129:0/3180125904 >> 172.21.15.129:0/3180125904 conn(0x561d8167c300 msgr2=0x561d81a23830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.308+0000 7f592161d740 1 -- 172.21.15.129:0/3180125904 shutdown_connections 2024-05-20T18:09:01.314 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.308+0000 7f592161d740 1 -- 172.21.15.129:0/3180125904 wait complete. 2024-05-20T18:09:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.308+0000 7f592161d740 1 -- start start 2024-05-20T18:09:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f592161d740 1 Processor -- start 2024-05-20T18:09:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f592161d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a31740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f592161d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a31c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f592161d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a376a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f592161d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561d81a341b0 con 0x561d816674d0 2024-05-20T18:09:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f592161d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561d81947790 con 0x561d8165f2f0 2024-05-20T18:09:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f592161d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561d81a34330 con 0x561d816881b0 2024-05-20T18:09:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f592161d740 10 client.0 _ll_register_callbacks cb 0x561d81982a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f59137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a31c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.315 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f59137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a31c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58826/0 (socket says 172.21.15.129:58826) 2024-05-20T18:09:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f59137fe640 1 -- 172.21.15.129:0/4249035793 learned_addr learned my addr 172.21.15.129:0/4249035793 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f5913fff640 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a31740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.309+0000 7f5918d14640 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f59137fe640 1 -- 172.21.15.129:0/4249035793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 msgr2=0x561d81a376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f59137fe640 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a376a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f59137fe640 1 -- 172.21.15.129:0/4249035793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 msgr2=0x561d81a31740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f59137fe640 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a31740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f59137fe640 1 -- 172.21.15.129:0/4249035793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d816a0210 con 0x561d816674d0 2024-05-20T18:09:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f5913fff640 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a31740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:09:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f59137fe640 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a31c80 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f59080187c0 tx=0x7f5908009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e469c33d257e6310 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:01.316 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f58f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:01.317 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f58f37fe640 1 -- 172.21.15.129:0/4249035793 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5908029020 con 0x561d816674d0 2024-05-20T18:09:01.317 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f58f37fe640 1 -- 172.21.15.129:0/4249035793 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f5908018860 con 0x561d816674d0 2024-05-20T18:09:01.317 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f592161d740 1 -- 172.21.15.129:0/4249035793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5908019ca0 con 0x561d816674d0 2024-05-20T18:09:01.317 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f58f37fe640 1 -- 172.21.15.129:0/4249035793 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5908018a60 con 0x561d816674d0 2024-05-20T18:09:01.317 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f592161d740 10 client.17145 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:01.317 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.310+0000 7f592161d740 1 -- 172.21.15.129:0/4249035793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561d8176c620 con 0x561d816674d0 2024-05-20T18:09:01.317 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.311+0000 7f58f37fe640 1 -- 172.21.15.129:0/4249035793 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5908019ca0 con 0x561d816674d0 2024-05-20T18:09:01.317 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.311+0000 7f58f37fe640 1 -- 172.21.15.129:0/4249035793 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f59080047e0 con 0x561d816674d0 2024-05-20T18:09:01.317 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.311+0000 7f592161d740 10 client.17145 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:01.317 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.311+0000 7f592161d740 10 client.17145 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:01.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.311+0000 7f592161d740 20 client.17145 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:01.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.311+0000 7f592161d740 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561d81646d80 0x561d81a438d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.311+0000 7f592161d740 4 client.17145 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:01.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.311+0000 7f592161d740 1 -- 172.21.15.129:0/4249035793 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d81999a70 con 0x561d81646d80 2024-05-20T18:09:01.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.311+0000 7f5913fff640 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561d81646d80 0x561d81a438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.312+0000 7f58f37fe640 10 client.17145 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:01.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.312+0000 7f5913fff640 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561d81646d80 0x561d81a438d0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67d8d7c3fb195d6c server_cookie=1c036082535d0b9c in_seq=0 out_seq=0 2024-05-20T18:09:01.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.312+0000 7f58f37fe640 1 -- 172.21.15.129:0/4249035793 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561d81999a70 con 0x561d81646d80 2024-05-20T18:09:01.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.312+0000 7f58f37fe640 10 client.17145 handle_command_reply: tid=0 2024-05-20T18:09:01.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 1 client.17145 shutdown 2024-05-20T18:09:01.318 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 1 -- 172.21.15.129:0/4249035793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 msgr2=0x561d81a31c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a31c80 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f59080187c0 tx=0x7f5908009620 comp rx=0 tx=0).stop 2024-05-20T18:09:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 1 -- 172.21.15.129:0/4249035793 shutdown_connections 2024-05-20T18:09:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561d81646d80 0x561d81a438d0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a376a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a31c80 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f58f37fe640 0 client.17145 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 1 --2- 172.21.15.129:0/4249035793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a31740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 1 -- 172.21.15.129:0/4249035793 >> 172.21.15.129:0/4249035793 conn(0x561d8167c300 msgr2=0x561d81a28da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 1 -- 172.21.15.129:0/4249035793 shutdown_connections 2024-05-20T18:09:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 1 -- 172.21.15.129:0/4249035793 wait complete. 2024-05-20T18:09:01.319 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.313+0000 7f592161d740 20 client.17145 trim_cache size 0 max 16384 2024-05-20T18:09:01.328 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.328+0000 7f592161d740 1 Processor -- start 2024-05-20T18:09:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.328+0000 7f592161d740 1 -- start start 2024-05-20T18:09:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.328+0000 7f592161d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d8165f2f0 0x561d81a310e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.328+0000 7f592161d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d816674d0 0x561d81a29440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.328+0000 7f592161d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816881b0 0x561d81a48f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.328+0000 7f592161d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59141aed10 con 0x561d816881b0 2024-05-20T18:09:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.328+0000 7f592161d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59141b1570 con 0x561d816674d0 2024-05-20T18:09:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.328+0000 7f592161d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f59141af360 con 0x561d8165f2f0 2024-05-20T18:09:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f5918d14640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816881b0 0x561d81a48f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f5918d14640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816881b0 0x561d81a48f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58834/0 (socket says 172.21.15.129:58834) 2024-05-20T18:09:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f5918d14640 1 -- 172.21.15.129:0/2666307337 learned_addr learned my addr 172.21.15.129:0/2666307337 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f5913fff640 1 --2- 172.21.15.129:0/2666307337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d8165f2f0 0x561d81a310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f59137fe640 1 --2- 172.21.15.129:0/2666307337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d816674d0 0x561d81a29440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f5918d14640 1 -- 172.21.15.129:0/2666307337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d8165f2f0 msgr2=0x561d81a310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f5918d14640 1 --2- 172.21.15.129:0/2666307337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d8165f2f0 0x561d81a310e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f5918d14640 1 -- 172.21.15.129:0/2666307337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d816674d0 msgr2=0x561d81a29440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f5918d14640 1 --2- 172.21.15.129:0/2666307337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d816674d0 0x561d81a29440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f5918d14640 1 -- 172.21.15.129:0/2666307337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d81a24940 con 0x561d816881b0 2024-05-20T18:09:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f5918d14640 1 --2- 172.21.15.129:0/2666307337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816881b0 0x561d81a48f60 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f590c00faa0 tx=0x7f590c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505c77028ba20740 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f58f9ffb640 1 -- 172.21.15.129:0/2666307337 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c004b20 con 0x561d816881b0 2024-05-20T18:09:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f58f9ffb640 1 -- 172.21.15.129:0/2666307337 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f590c004cc0 con 0x561d816881b0 2024-05-20T18:09:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.329+0000 7f58f9ffb640 1 -- 172.21.15.129:0/2666307337 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c005460 con 0x561d816881b0 2024-05-20T18:09:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.330+0000 7f592161d740 1 -- 172.21.15.129:0/2666307337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816881b0 msgr2=0x561d81a48f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.330+0000 7f592161d740 1 --2- 172.21.15.129:0/2666307337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816881b0 0x561d81a48f60 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f590c00faa0 tx=0x7f590c004ab0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.330+0000 7f592161d740 1 -- 172.21.15.129:0/2666307337 shutdown_connections 2024-05-20T18:09:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.330+0000 7f592161d740 1 --2- 172.21.15.129:0/2666307337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d816674d0 0x561d81a29440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.330+0000 7f592161d740 1 --2- 172.21.15.129:0/2666307337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816881b0 0x561d81a48f60 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.330+0000 7f592161d740 1 --2- 172.21.15.129:0/2666307337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d8165f2f0 0x561d81a310e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.330+0000 7f592161d740 1 -- 172.21.15.129:0/2666307337 >> 172.21.15.129:0/2666307337 conn(0x561d8167c300 msgr2=0x561d81a28d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.330+0000 7f592161d740 1 -- 172.21.15.129:0/2666307337 shutdown_connections 2024-05-20T18:09:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.330+0000 7f592161d740 1 -- 172.21.15.129:0/2666307337 wait complete. 2024-05-20T18:09:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.331+0000 7f592161d740 1 -- start start 2024-05-20T18:09:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.331+0000 7f592161d740 1 Processor -- start 2024-05-20T18:09:01.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.331+0000 7f592161d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a38210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.338 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.331+0000 7f592161d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a38750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.331+0000 7f592161d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a38c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.331+0000 7f592161d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59141afa30 con 0x561d816674d0 2024-05-20T18:09:01.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.331+0000 7f592161d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59141b0ea0 con 0x561d8165f2f0 2024-05-20T18:09:01.339 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.331+0000 7f592161d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f59141b0100 con 0x561d816881b0 2024-05-20T18:09:01.340 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.331+0000 7f592161d740 10 client.0 _ll_register_callbacks cb 0x7f59040bd270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:01.340 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.331+0000 7f59137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a38750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.340 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f59137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a38750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58842/0 (socket says 172.21.15.129:58842) 2024-05-20T18:09:01.340 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f59137fe640 1 -- 172.21.15.129:0/237911579 learned_addr learned my addr 172.21.15.129:0/237911579 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:01.340 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f5918d14640 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a38c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.340 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f5913fff640 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a38210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.340 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f59137fe640 1 -- 172.21.15.129:0/237911579 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 msgr2=0x561d81a38c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f59137fe640 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a38c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f59137fe640 1 -- 172.21.15.129:0/237911579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 msgr2=0x561d81a38210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f59137fe640 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a38210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f59137fe640 1 -- 172.21.15.129:0/237911579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d81a43d30 con 0x561d816674d0 2024-05-20T18:09:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f59137fe640 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a38750 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f5908011850 tx=0x7f59080053c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10bba5e15c20aa80 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f58f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f58f37fe640 1 -- 172.21.15.129:0/237911579 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5908029040 con 0x561d816674d0 2024-05-20T18:09:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f592161d740 1 -- 172.21.15.129:0/237911579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f590c004cc0 con 0x561d816674d0 2024-05-20T18:09:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f58f37fe640 1 -- 172.21.15.129:0/237911579 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f5908018e00 con 0x561d816674d0 2024-05-20T18:09:01.341 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f58f37fe640 1 -- 172.21.15.129:0/237911579 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f59080035f0 con 0x561d816674d0 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f592161d740 10 client.17151 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.332+0000 7f592161d740 1 -- 172.21.15.129:0/237911579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561d81aaadc0 con 0x561d816674d0 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.333+0000 7f58f37fe640 1 -- 172.21.15.129:0/237911579 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f590c004cc0 con 0x561d816674d0 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.333+0000 7f58f37fe640 1 -- 172.21.15.129:0/237911579 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f5908004500 con 0x561d816674d0 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.333+0000 7f592161d740 10 client.17151 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.333+0000 7f592161d740 10 client.17151 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.333+0000 7f592161d740 20 client.17151 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.333+0000 7f592161d740 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561d8161c1c0 0x561d81aad4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.333+0000 7f592161d740 4 client.17151 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.333+0000 7f592161d740 1 -- 172.21.15.129:0/237911579 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561d81993dc0 con 0x561d8161c1c0 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.333+0000 7f5913fff640 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561d8161c1c0 0x561d81aad4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.342 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.334+0000 7f58f37fe640 10 client.17151 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:01.334+0000 7f5913fff640 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561d8161c1c0 0x561d81aad4e0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a4051878050443f server_cookie=36848e1417060900 in_seq=0 out_seq=0 2024-05-20T18:09:01.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.423+0000 7f72ac302640 1 -- 172.21.15.129:0/3422524900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a4101ad0 msgr2=0x7f72a4101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.423+0000 7f72ac302640 1 --2- 172.21.15.129:0/3422524900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a4101ad0 0x7f72a4101ee0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f7298009900 tx=0x7f729802a410 comp rx=0 tx=0).stop 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.424+0000 7f72ac302640 1 -- 172.21.15.129:0/3422524900 shutdown_connections 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.424+0000 7f72ac302640 1 --2- 172.21.15.129:0/3422524900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72a410c360 0x7f72a410e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.424+0000 7f72ac302640 1 --2- 172.21.15.129:0/3422524900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a4101ad0 0x7f72a4101ee0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.424+0000 7f72ac302640 1 --2- 172.21.15.129:0/3422524900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a4107a00 0x7f72a4101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.424+0000 7f72ac302640 1 -- 172.21.15.129:0/3422524900 >> 172.21.15.129:0/3422524900 conn(0x7f72a4077100 msgr2=0x7f72a4077540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.424+0000 7f72ac302640 1 -- 172.21.15.129:0/3422524900 shutdown_connections 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.424+0000 7f72ac302640 1 -- 172.21.15.129:0/3422524900 wait complete. 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.424+0000 7f72ac302640 1 Processor -- start 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.424+0000 7f72ac302640 1 -- start start 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.424+0000 7f72ac302640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a4101ad0 0x7f72a41997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.424+0000 7f72ac302640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72a4107a00 0x7f72a4199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72ac302640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a410c360 0x7f72a419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72ac302640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72a4111d10 con 0x7f72a4101ad0 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72ac302640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f72a4111b90 con 0x7f72a4107a00 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72ac302640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f72a4111e90 con 0x7f72a410c360 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72aa09e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a4101ad0 0x7f72a41997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72aa09e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a4101ad0 0x7f72a41997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58854/0 (socket says 172.21.15.129:58854) 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72aa09e640 1 -- 172.21.15.129:0/2591045042 learned_addr learned my addr 172.21.15.129:0/2591045042 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72a989d640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72a4107a00 0x7f72a4199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72aa89f640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a410c360 0x7f72a419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72aa09e640 1 -- 172.21.15.129:0/2591045042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a410c360 msgr2=0x7f72a419dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72aa09e640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a410c360 0x7f72a419dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72aa09e640 1 -- 172.21.15.129:0/2591045042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72a4107a00 msgr2=0x7f72a4199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72aa09e640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72a4107a00 0x7f72a4199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72aa09e640 1 -- 172.21.15.129:0/2591045042 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a419e620 con 0x7f72a4101ad0 2024-05-20T18:09:01.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72a989d640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72a4107a00 0x7f72a4199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:09:01.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72aa09e640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a4101ad0 0x7f72a41997c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f729400ea50 tx=0x7f729400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:01.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72a37fe640 1 -- 172.21.15.129:0/2591045042 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729400cdc0 con 0x7f72a4101ad0 2024-05-20T18:09:01.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72ac302640 1 -- 172.21.15.129:0/2591045042 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a419e910 con 0x7f72a4101ad0 2024-05-20T18:09:01.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72a37fe640 1 -- 172.21.15.129:0/2591045042 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7294005020 con 0x7f72a4101ad0 2024-05-20T18:09:01.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72a37fe640 1 -- 172.21.15.129:0/2591045042 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7294011690 con 0x7f72a4101ad0 2024-05-20T18:09:01.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.425+0000 7f72ac302640 1 -- 172.21.15.129:0/2591045042 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a41a6290 con 0x7f72a4101ad0 2024-05-20T18:09:01.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72a37fe640 1 -- 172.21.15.129:0/2591045042 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7294005540 con 0x7f72a4101ad0 2024-05-20T18:09:01.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72b31af740 1 Processor -- start 2024-05-20T18:09:01.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72b31af740 1 -- start start 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72b31af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135dcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72b31af740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639132154d0 0x5639135dd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72b31af740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56391321d6b0 0x5639135dd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72b31af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563913542d00 con 0x56391323e390 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72b31af740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5639134ea4c0 con 0x56391321d6b0 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72b31af740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56391353ecc0 con 0x5639132154d0 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72aa89f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135dcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72aa89f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135dcd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58858/0 (socket says 172.21.15.129:58858) 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.428+0000 7f72aa89f640 1 -- 172.21.15.129:0/3793961177 learned_addr learned my addr 172.21.15.129:0/3793961177 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72a989d640 1 --2- 172.21.15.129:0/3793961177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56391321d6b0 0x5639135dd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72aa09e640 1 --2- 172.21.15.129:0/3793961177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639132154d0 0x5639135dd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72aa89f640 1 -- 172.21.15.129:0/3793961177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639132154d0 msgr2=0x5639135dd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72aa89f640 1 --2- 172.21.15.129:0/3793961177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639132154d0 0x5639135dd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72aa89f640 1 -- 172.21.15.129:0/3793961177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56391321d6b0 msgr2=0x5639135dd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72aa89f640 1 --2- 172.21.15.129:0/3793961177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56391321d6b0 0x5639135dd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72aa89f640 1 -- 172.21.15.129:0/3793961177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639133207d0 con 0x56391323e390 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72a37fe640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f728c0a2d00 0x7f728c0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72aa09e640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f728c0a2d00 0x7f728c0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72aa89f640 1 --2- 172.21.15.129:0/3793961177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135dcd30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f729c00bc60 tx=0x7f729c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6077fb4f4abfd0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72a37fe640 1 -- 172.21.15.129:0/2591045042 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f72940cf890 con 0x7f72a4101ad0 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72a0bfe640 1 -- 172.21.15.129:0/3793961177 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c015070 con 0x56391323e390 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72a0bfe640 1 -- 172.21.15.129:0/3793961177 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c01a940 con 0x56391323e390 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.429+0000 7f72a0bfe640 1 -- 172.21.15.129:0/3793961177 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c011a40 con 0x56391323e390 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.430+0000 7f72b31af740 1 -- 172.21.15.129:0/3793961177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 msgr2=0x5639135dcd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.430+0000 7f72aa09e640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f728c0a2d00 0x7f728c0a5210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x5639135da820 tx=0x7f72940054b0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:01.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.430+0000 7f72b31af740 1 --2- 172.21.15.129:0/3793961177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135dcd30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f729c00bc60 tx=0x7f729c00f730 comp rx=0 tx=0).stop 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.430+0000 7f72b31af740 1 -- 172.21.15.129:0/3793961177 shutdown_connections 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.430+0000 7f72b31af740 1 --2- 172.21.15.129:0/3793961177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56391321d6b0 0x5639135dd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.430+0000 7f72b31af740 1 --2- 172.21.15.129:0/3793961177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5639132154d0 0x5639135dd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.430+0000 7f72b31af740 1 --2- 172.21.15.129:0/3793961177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135dcd30 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.430+0000 7f72b31af740 1 -- 172.21.15.129:0/3793961177 >> 172.21.15.129:0/3793961177 conn(0x5639132324e0 msgr2=0x5639135d94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.430+0000 7f72b31af740 1 -- 172.21.15.129:0/3793961177 shutdown_connections 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.430+0000 7f72b31af740 1 -- 172.21.15.129:0/3793961177 wait complete. 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.430+0000 7f72b31af740 1 -- start start 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72b31af740 1 Processor -- start 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72b31af740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639132154d0 0x5639135e8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72b31af740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391321d6b0 0x5639135e93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72b31af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135e73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72b31af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5639135e9fc0 con 0x56391323e390 2024-05-20T18:09:01.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72b31af740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5639134fdb90 con 0x5639132154d0 2024-05-20T18:09:01.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72b31af740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5639135ea140 con 0x56391321d6b0 2024-05-20T18:09:01.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72b31af740 10 client.0 _ll_register_callbacks cb 0x563913539da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:01.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72aa89f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135e73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72aa89f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135e73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58874/0 (socket says 172.21.15.129:58874) 2024-05-20T18:09:01.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72aa89f640 1 -- 172.21.15.129:0/1116930297 learned_addr learned my addr 172.21.15.129:0/1116930297 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:01.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72aa89f640 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639132154d0 0x5639135e8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72a989d640 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391321d6b0 0x5639135e93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72aa89f640 1 -- 172.21.15.129:0/1116930297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391321d6b0 msgr2=0x5639135e93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72aa89f640 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391321d6b0 0x5639135e93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72aa89f640 1 -- 172.21.15.129:0/1116930297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639132154d0 msgr2=0x5639135e8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72aa89f640 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639132154d0 0x5639135e8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.431+0000 7f72aa89f640 1 -- 172.21.15.129:0/1116930297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639132563f0 con 0x56391323e390 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f72aa89f640 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135e73c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x5639135e84d0 tx=0x7f729c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb96f6fe5a1399a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f7289ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f7289ffb640 1 -- 172.21.15.129:0/1116930297 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c015040 con 0x56391323e390 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f7289ffb640 1 -- 172.21.15.129:0/1116930297 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f729c01a660 con 0x56391323e390 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f72b31af740 1 -- 172.21.15.129:0/1116930297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f729c01a940 con 0x56391323e390 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f7289ffb640 1 -- 172.21.15.129:0/1116930297 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f729c005030 con 0x56391323e390 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f72b31af740 10 client.17163 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f72b31af740 1 -- 172.21.15.129:0/1116930297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5639132160f0 con 0x56391323e390 2024-05-20T18:09:01.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f7289ffb640 1 -- 172.21.15.129:0/1116930297 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c01a940 con 0x56391323e390 2024-05-20T18:09:01.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f7289ffb640 1 -- 172.21.15.129:0/1116930297 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f729c029060 con 0x56391323e390 2024-05-20T18:09:01.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f72b31af740 10 client.17163 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:01.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f72b31af740 10 client.17163 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:01.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f72b31af740 20 client.17163 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:01.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f72b31af740 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639131fcf60 0x5639135f9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:01.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f72b31af740 4 client.17163 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:01.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f72b31af740 1 -- 172.21.15.129:0/1116930297 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563913550430 con 0x5639131fcf60 2024-05-20T18:09:01.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.432+0000 7f72a989d640 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639131fcf60 0x5639135f9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:01.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.433+0000 7f7289ffb640 10 client.17163 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:01.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:01.433+0000 7f72a989d640 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639131fcf60 0x5639135f9ab0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bffdfca277cc56a server_cookie=8da11c60e21a33dc in_seq=0 out_seq=0 2024-05-20T18:09:01.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:01 smithi154 ceph-mon[36482]: pgmap v324: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:01.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:01 smithi129 ceph-mon[32029]: pgmap v324: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:01.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:01 smithi204 ceph-mon[37076]: pgmap v324: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:03.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:03 smithi154 ceph-mon[36482]: pgmap v325: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T18:09:03.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:03 smithi129 ceph-mon[32029]: pgmap v325: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T18:09:03.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:03 smithi204 ceph-mon[37076]: pgmap v325: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-05-20T18:09:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.248+0000 7f58f37fe640 1 -- 172.21.15.129:0/237911579 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561d81993dc0 con 0x561d8161c1c0 2024-05-20T18:09:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f58f37fe640 10 client.17151 handle_command_reply: tid=0 2024-05-20T18:09:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 1 client.17151 shutdown 2024-05-20T18:09:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 1 -- 172.21.15.129:0/237911579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 msgr2=0x561d81a38750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a38750 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f5908011850 tx=0x7f59080053c0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 1 -- 172.21.15.129:0/237911579 shutdown_connections 2024-05-20T18:09:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x561d8161c1c0 0x561d81aad4e0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d816881b0 0x561d81a38c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f7289ffb640 1 -- 172.21.15.129:0/1116930297 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563913550430 con 0x5639131fcf60 2024-05-20T18:09:05.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f7289ffb640 10 client.17163 handle_command_reply: tid=0 2024-05-20T18:09:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561d816674d0 0x561d81a38750 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 1 --2- 172.21.15.129:0/237911579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d8165f2f0 0x561d81a38210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f58f37fe640 0 client.17151 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 1 -- 172.21.15.129:0/237911579 >> 172.21.15.129:0/237911579 conn(0x561d8167c300 msgr2=0x561d81a26f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:05.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f72b31af740 1 client.17163 shutdown 2024-05-20T18:09:05.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 1 -- 172.21.15.129:0/237911579 shutdown_connections 2024-05-20T18:09:05.249 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 1 -- 172.21.15.129:0/237911579 wait complete. 2024-05-20T18:09:05.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f72b31af740 1 -- 172.21.15.129:0/1116930297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 msgr2=0x5639135e73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f72b31af740 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135e73c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x5639135e84d0 tx=0x7f729c0117b0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f592161d740 20 client.17151 trim_cache size 0 max 16384 2024-05-20T18:09:05.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f72b31af740 1 -- 172.21.15.129:0/1116930297 shutdown_connections 2024-05-20T18:09:05.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f72b31af740 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639131fcf60 0x5639135f9ab0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f72b31af740 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135e73c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f72b31af740 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391321d6b0 0x5639135e93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f7289ffb640 0 client.17163 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:05.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f72b31af740 1 --2- 172.21.15.129:0/1116930297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639132154d0 0x5639135e8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.249+0000 7f72b31af740 1 -- 172.21.15.129:0/1116930297 >> 172.21.15.129:0/1116930297 conn(0x5639132324e0 msgr2=0x5639135dea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:05.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f72b31af740 1 -- 172.21.15.129:0/1116930297 shutdown_connections 2024-05-20T18:09:05.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f72b31af740 1 -- 172.21.15.129:0/1116930297 wait complete. 2024-05-20T18:09:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:09:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:09:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:09:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:09:05.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f72b31af740 20 client.17163 trim_cache size 0 max 16384 2024-05-20T18:09:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 -- 172.21.15.129:0/231983447 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58fc0a2cf0 msgr2=0x7f58fc0a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58fc0a2cf0 0x7f58fc0a5200 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f590c005fd0 tx=0x7f590c011b40 comp rx=0 tx=0).stop 2024-05-20T18:09:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 -- 172.21.15.129:0/231983447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5914103c20 msgr2=0x7f59141014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5914103c20 0x7f59141014e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f5904004770 tx=0x7f5904003c10 comp rx=0 tx=0).stop 2024-05-20T18:09:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 -- 172.21.15.129:0/231983447 shutdown_connections 2024-05-20T18:09:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f58fc0a2cf0 0x7f58fc0a5200 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f591410d070 0x7f5914101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f591410a680 0x7f5914101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 --2- 172.21.15.129:0/231983447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5914103c20 0x7f59141014e0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 -- 172.21.15.129:0/231983447 >> 172.21.15.129:0/231983447 conn(0x7f5914077160 msgr2=0x7f5914100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:05.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 -- 172.21.15.129:0/231983447 shutdown_connections 2024-05-20T18:09:05.254 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:05.250+0000 7f591a777640 1 -- 172.21.15.129:0/231983447 wait complete. 2024-05-20T18:09:05.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72b31af740 1 Processor -- start 2024-05-20T18:09:05.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72b31af740 1 -- start start 2024-05-20T18:09:05.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72b31af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639132154d0 0x5639135dd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:05.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72b31af740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56391321d6b0 0x5639135e6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:05.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72b31af740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391323e390 0x5639135e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:05.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72b31af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72a41acbf0 con 0x5639132154d0 2024-05-20T18:09:05.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72b31af740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f72a41af450 con 0x56391321d6b0 2024-05-20T18:09:05.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72b31af740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f72a41ad240 con 0x56391323e390 2024-05-20T18:09:05.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72aa89f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639132154d0 0x5639135dd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72aa89f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639132154d0 0x5639135dd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58886/0 (socket says 172.21.15.129:58886) 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72aa89f640 1 -- 172.21.15.129:0/3007437513 learned_addr learned my addr 172.21.15.129:0/3007437513 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.270+0000 7f72a989d640 1 --2- 172.21.15.129:0/3007437513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56391321d6b0 0x5639135e6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.271+0000 7f72aa89f640 1 --2- 172.21.15.129:0/3007437513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391323e390 0x5639135e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.271+0000 7f72aa89f640 1 -- 172.21.15.129:0/3007437513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391323e390 msgr2=0x5639135e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.271+0000 7f72aa89f640 1 --2- 172.21.15.129:0/3007437513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391323e390 0x5639135e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.271+0000 7f72aa89f640 1 -- 172.21.15.129:0/3007437513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56391321d6b0 msgr2=0x5639135e6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.271+0000 7f72aa89f640 1 --2- 172.21.15.129:0/3007437513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56391321d6b0 0x5639135e6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.271+0000 7f72aa89f640 1 -- 172.21.15.129:0/3007437513 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639135d4bd0 con 0x5639132154d0 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.271+0000 7f72aa89f640 1 --2- 172.21.15.129:0/3007437513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639132154d0 0x5639135dd6c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f729c00b480 tx=0x7f729c00b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b4bd602d2995f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.271+0000 7f728bfff640 1 -- 172.21.15.129:0/3007437513 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c015040 con 0x5639132154d0 2024-05-20T18:09:05.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.271+0000 7f728bfff640 1 -- 172.21.15.129:0/3007437513 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c00bd50 con 0x5639132154d0 2024-05-20T18:09:05.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.271+0000 7f728bfff640 1 -- 172.21.15.129:0/3007437513 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c005550 con 0x5639132154d0 2024-05-20T18:09:05.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.272+0000 7f72b31af740 1 -- 172.21.15.129:0/3007437513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639132154d0 msgr2=0x5639135dd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.272+0000 7f72b31af740 1 --2- 172.21.15.129:0/3007437513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639132154d0 0x5639135dd6c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f729c00b480 tx=0x7f729c00b560 comp rx=0 tx=0).stop 2024-05-20T18:09:05.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.272+0000 7f72b31af740 1 -- 172.21.15.129:0/3007437513 shutdown_connections 2024-05-20T18:09:05.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.272+0000 7f72b31af740 1 --2- 172.21.15.129:0/3007437513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56391321d6b0 0x5639135e6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.272+0000 7f72b31af740 1 --2- 172.21.15.129:0/3007437513 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391323e390 0x5639135e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.272+0000 7f72b31af740 1 --2- 172.21.15.129:0/3007437513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5639132154d0 0x5639135dd6c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.272+0000 7f72b31af740 1 -- 172.21.15.129:0/3007437513 >> 172.21.15.129:0/3007437513 conn(0x5639132324e0 msgr2=0x5639135def80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:05.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.272+0000 7f72b31af740 1 -- 172.21.15.129:0/3007437513 shutdown_connections 2024-05-20T18:09:05.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.272+0000 7f72b31af740 1 -- 172.21.15.129:0/3007437513 wait complete. 2024-05-20T18:09:05.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.273+0000 7f72b31af740 1 -- start start 2024-05-20T18:09:05.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.273+0000 7f72b31af740 1 Processor -- start 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.273+0000 7f72b31af740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639132154d0 0x5639135f2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.273+0000 7f72b31af740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391321d6b0 0x5639135f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.273+0000 7f72b31af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.273+0000 7f72b31af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72a41ad910 con 0x56391323e390 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.273+0000 7f72b31af740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f72a41aed80 con 0x5639132154d0 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.273+0000 7f72b31af740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f72a41adfe0 con 0x56391321d6b0 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.273+0000 7f72b31af740 10 client.0 _ll_register_callbacks cb 0x7f7298009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.273+0000 7f72aa89f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f72aa89f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135f31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58898/0 (socket says 172.21.15.129:58898) 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f72aa89f640 1 -- 172.21.15.129:0/3656213696 learned_addr learned my addr 172.21.15.129:0/3656213696 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f72aa89f640 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639132154d0 0x5639135f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f72a989d640 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391321d6b0 0x5639135f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f72aa89f640 1 -- 172.21.15.129:0/3656213696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391321d6b0 msgr2=0x5639135f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f72aa89f640 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391321d6b0 0x5639135f2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f72aa89f640 1 -- 172.21.15.129:0/3656213696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639132154d0 msgr2=0x5639135f2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f72aa89f640 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639132154d0 0x5639135f2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f72aa89f640 1 -- 172.21.15.129:0/3656213696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56391365fcb0 con 0x56391323e390 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f72aa89f640 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135f31b0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f729c005230 tx=0x7f729c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64a71a849b0dc62 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f7289ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f7289ffb640 1 -- 172.21.15.129:0/3656213696 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c04c060 con 0x56391323e390 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f7289ffb640 1 -- 172.21.15.129:0/3656213696 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f729c004b30 con 0x56391323e390 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.274+0000 7f72b31af740 1 -- 172.21.15.129:0/3656213696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f729c00bd50 con 0x56391323e390 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.275+0000 7f7289ffb640 1 -- 172.21.15.129:0/3656213696 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f729c015030 con 0x56391323e390 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.275+0000 7f72b31af740 10 client.17169 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.275+0000 7f72b31af740 1 -- 172.21.15.129:0/3656213696 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563913660f00 con 0x56391323e390 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.275+0000 7f7289ffb640 1 -- 172.21.15.129:0/3656213696 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c00bd50 con 0x56391323e390 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.275+0000 7f7289ffb640 1 -- 172.21.15.129:0/3656213696 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f729c029070 con 0x56391323e390 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.275+0000 7f72b31af740 10 client.17169 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.275+0000 7f72b31af740 10 client.17169 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.275+0000 7f72b31af740 20 client.17169 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.275+0000 7f72b31af740 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639131d23a0 0x563913663300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.275+0000 7f72b31af740 4 client.17169 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.275+0000 7f72b31af740 1 -- 172.21.15.129:0/3656213696 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56391354a780 con 0x5639131d23a0 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.276+0000 7f72a989d640 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639131d23a0 0x563913663300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.276+0000 7f7289ffb640 10 client.17169 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.276+0000 7f72a989d640 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639131d23a0 0x563913663300 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6988b80377b0b002 server_cookie=f380280c778dc505 in_seq=0 out_seq=0 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.276+0000 7f7289ffb640 1 -- 172.21.15.129:0/3656213696 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56391354a780 con 0x5639131d23a0 2024-05-20T18:09:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.276+0000 7f7289ffb640 10 client.17169 handle_command_reply: tid=0 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.276+0000 7f72b31af740 1 client.17169 shutdown 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.276+0000 7f72b31af740 1 -- 172.21.15.129:0/3656213696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 msgr2=0x5639135f31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.276+0000 7f72b31af740 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135f31b0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f729c005230 tx=0x7f729c004a40 comp rx=0 tx=0).stop 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72b31af740 1 -- 172.21.15.129:0/3656213696 shutdown_connections 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72b31af740 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5639131d23a0 0x563913663300 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72b31af740 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56391323e390 0x5639135f31b0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72b31af740 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56391321d6b0 0x5639135f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f7289ffb640 0 client.17169 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72b31af740 1 --2- 172.21.15.129:0/3656213696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5639132154d0 0x5639135f2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72b31af740 1 -- 172.21.15.129:0/3656213696 >> 172.21.15.129:0/3656213696 conn(0x5639132324e0 msgr2=0x5639135def80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72b31af740 1 -- 172.21.15.129:0/3656213696 shutdown_connections 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72b31af740 1 -- 172.21.15.129:0/3656213696 wait complete. 2024-05-20T18:09:05.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72b31af740 20 client.17169 trim_cache size 0 max 16384 2024-05-20T18:09:05.281 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:09:05.281 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:09:05.281 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:09:05.281 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:09:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72ac302640 1 -- 172.21.15.129:0/2591045042 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f728c0a2d00 msgr2=0x7f728c0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72ac302640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f728c0a2d00 0x7f728c0a5210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x5639135da820 tx=0x7f72940054b0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72ac302640 1 -- 172.21.15.129:0/2591045042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a4101ad0 msgr2=0x7f72a41997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72ac302640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a4101ad0 0x7f72a41997c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f729400ea50 tx=0x7f729400ef10 comp rx=0 tx=0).stop 2024-05-20T18:09:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72ac302640 1 -- 172.21.15.129:0/2591045042 shutdown_connections 2024-05-20T18:09:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72ac302640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f728c0a2d00 0x7f728c0a5210 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72ac302640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72a4107a00 0x7f72a4199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72ac302640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a410c360 0x7f72a419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72ac302640 1 --2- 172.21.15.129:0/2591045042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a4101ad0 0x7f72a41997c0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.277+0000 7f72ac302640 1 -- 172.21.15.129:0/2591045042 >> 172.21.15.129:0/2591045042 conn(0x7f72a4077100 msgr2=0x7f72a4073d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:05.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.278+0000 7f72ac302640 1 -- 172.21.15.129:0/2591045042 shutdown_connections 2024-05-20T18:09:05.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.278+0000 7f72ac302640 1 -- 172.21.15.129:0/2591045042 wait complete. 2024-05-20T18:09:05.291 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:09:05.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:09:05.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.504+0000 7feb9c957640 1 -- 172.21.15.129:0/3847889066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb98109060 msgr2=0x7feb98109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.504+0000 7feb9c957640 1 --2- 172.21.15.129:0/3847889066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb98109060 0x7feb98109470 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7feb90009ca0 tx=0x7feb9002a970 comp rx=0 tx=0).stop 2024-05-20T18:09:05.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.506+0000 7feb9c957640 1 -- 172.21.15.129:0/3847889066 shutdown_connections 2024-05-20T18:09:05.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.506+0000 7feb9c957640 1 --2- 172.21.15.129:0/3847889066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb98119770 0x7feb9811bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.506+0000 7feb9c957640 1 --2- 172.21.15.129:0/3847889066 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb9811c0d0 0x7feb9811e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.506+0000 7feb9c957640 1 --2- 172.21.15.129:0/3847889066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb98109060 0x7feb98109470 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.506+0000 7feb9c957640 1 -- 172.21.15.129:0/3847889066 >> 172.21.15.129:0/3847889066 conn(0x7feb9806d760 msgr2=0x7feb9806dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:05.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.506+0000 7feb9c957640 1 -- 172.21.15.129:0/3847889066 shutdown_connections 2024-05-20T18:09:05.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.508+0000 7feb9c957640 1 -- 172.21.15.129:0/3847889066 wait complete. 2024-05-20T18:09:05.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.509+0000 7feb9c957640 1 Processor -- start 2024-05-20T18:09:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.510+0000 7feb9c957640 1 -- start start 2024-05-20T18:09:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.510+0000 7feb9c957640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb98109060 0x7feb981a8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.510+0000 7feb9c957640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb98119770 0x7feb981a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.510+0000 7feb9c957640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb9811c0d0 0x7feb981ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.510+0000 7feb9c957640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feb98121c80 con 0x7feb9811c0d0 2024-05-20T18:09:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.510+0000 7feb9c957640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feb98121b00 con 0x7feb98109060 2024-05-20T18:09:05.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.510+0000 7feb9c957640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feb98121e00 con 0x7feb98119770 2024-05-20T18:09:05.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.511+0000 7feb96d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb9811c0d0 0x7feb981ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.511+0000 7feb96d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb9811c0d0 0x7feb981ace40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58924/0 (socket says 172.21.15.129:58924) 2024-05-20T18:09:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.511+0000 7feb96d9d640 1 -- 172.21.15.129:0/2569331524 learned_addr learned my addr 172.21.15.129:0/2569331524 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.512+0000 7feb96d9d640 1 -- 172.21.15.129:0/2569331524 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb98119770 msgr2=0x7feb981a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.512+0000 7feb96d9d640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb98119770 0x7feb981a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.512+0000 7feb96d9d640 1 -- 172.21.15.129:0/2569331524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb98109060 msgr2=0x7feb981a8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.512+0000 7feb96d9d640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb98109060 0x7feb981a8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.512+0000 7feb96d9d640 1 -- 172.21.15.129:0/2569331524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb981ad480 con 0x7feb9811c0d0 2024-05-20T18:09:05.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.512+0000 7feb96d9d640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb9811c0d0 0x7feb981ace40 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7feb8800b3f0 tx=0x7feb8800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:05.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.517+0000 7feb877fe640 1 -- 172.21.15.129:0/2569331524 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb88015020 con 0x7feb9811c0d0 2024-05-20T18:09:05.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.517+0000 7feb877fe640 1 -- 172.21.15.129:0/2569331524 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb88011d10 con 0x7feb9811c0d0 2024-05-20T18:09:05.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.517+0000 7feb877fe640 1 -- 172.21.15.129:0/2569331524 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8800f2e0 con 0x7feb9811c0d0 2024-05-20T18:09:05.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.517+0000 7feb9c957640 1 -- 172.21.15.129:0/2569331524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb981ad770 con 0x7feb9811c0d0 2024-05-20T18:09:05.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.517+0000 7feb9c957640 1 -- 172.21.15.129:0/2569331524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb981adc70 con 0x7feb9811c0d0 2024-05-20T18:09:05.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.519+0000 7feb9c957640 1 -- 172.21.15.129:0/2569331524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb98068cf0 con 0x7feb9811c0d0 2024-05-20T18:09:05.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.523+0000 7feb877fe640 1 -- 172.21.15.129:0/2569331524 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7feb88018020 con 0x7feb9811c0d0 2024-05-20T18:09:05.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.524+0000 7feb877fe640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7feb800a2f00 0x7feb800a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:05.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.524+0000 7feb9659c640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7feb800a2f00 0x7feb800a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:05.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.525+0000 7feb877fe640 1 -- 172.21.15.129:0/2569331524 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7feb880cf2d0 con 0x7feb9811c0d0 2024-05-20T18:09:05.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.525+0000 7feb9659c640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7feb800a2f00 0x7feb800a5410 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb900094e0 tx=0x7feb90000f40 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:05.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.532+0000 7feb877fe640 1 -- 172.21.15.129:0/2569331524 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7feb981adc70 con 0x7feb9811c0d0 2024-05-20T18:09:05.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:05 smithi154 ceph-mon[36482]: pgmap v326: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-05-20T18:09:05.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:05 smithi129 ceph-mon[32029]: pgmap v326: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-05-20T18:09:05.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:05 smithi204 ceph-mon[37076]: pgmap v326: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 21 KiB/s wr, 16 op/s 2024-05-20T18:09:05.961 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:09:05.961 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":26255,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/2724397229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":2724397229},{"type":"v1","addr":"172.21.15.154:6837","nonce":2724397229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:06:10.247052+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24511":{"gid":24511,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/2880425505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2880425505},{"type":"v1","addr":"172.21.15.204:6839","nonce":2880425505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:09:05.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.961+0000 7feb9c957640 1 -- 172.21.15.129:0/2569331524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb981a9970 con 0x7feb9811c0d0 2024-05-20T18:09:05.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.962+0000 7feb877fe640 1 -- 172.21.15.129:0/2569331524 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11829 (secure 0 0 0) 0x7feb8809bfb0 con 0x7feb9811c0d0 2024-05-20T18:09:05.961 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-05-20T18:09:05.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 -- 172.21.15.129:0/2569331524 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7feb800a2f00 msgr2=0x7feb800a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7feb800a2f00 0x7feb800a5410 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb900094e0 tx=0x7feb90000f40 comp rx=0 tx=0).stop 2024-05-20T18:09:05.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 -- 172.21.15.129:0/2569331524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb9811c0d0 msgr2=0x7feb981ace40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:05.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb9811c0d0 0x7feb981ace40 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7feb8800b3f0 tx=0x7feb8800b8b0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 -- 172.21.15.129:0/2569331524 shutdown_connections 2024-05-20T18:09:05.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7feb800a2f00 0x7feb800a5410 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb9811c0d0 0x7feb981ace40 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb98119770 0x7feb981a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 --2- 172.21.15.129:0/2569331524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb98109060 0x7feb981a8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:05.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 -- 172.21.15.129:0/2569331524 >> 172.21.15.129:0/2569331524 conn(0x7feb9806d760 msgr2=0x7feb98119b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:05.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 -- 172.21.15.129:0/2569331524 shutdown_connections 2024-05-20T18:09:05.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:05.966+0000 7feb9c957640 1 -- 172.21.15.129:0/2569331524 wait complete. 2024-05-20T18:09:05.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:09:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.166+0000 7fe77ee6c640 1 -- 172.21.15.129:0/169516860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7781071a0 msgr2=0x7fe778100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.166+0000 7fe77ee6c640 1 --2- 172.21.15.129:0/169516860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7781071a0 0x7fe778100e30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe76c008410 tx=0x7fe76c02e560 comp rx=0 tx=0).stop 2024-05-20T18:09:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.166+0000 7fe77ee6c640 1 -- 172.21.15.129:0/169516860 shutdown_connections 2024-05-20T18:09:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.166+0000 7fe77ee6c640 1 --2- 172.21.15.129:0/169516860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7781014a0 0x7fe77810d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.166+0000 7fe77ee6c640 1 --2- 172.21.15.129:0/169516860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7781071a0 0x7fe778100e30 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.166+0000 7fe77ee6c640 1 --2- 172.21.15.129:0/169516860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe778106d00 0x7fe7781008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.166+0000 7fe77ee6c640 1 -- 172.21.15.129:0/169516860 >> 172.21.15.129:0/169516860 conn(0x7fe7780763f0 msgr2=0x7fe778076830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:06.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.166+0000 7fe77ee6c640 1 -- 172.21.15.129:0/169516860 shutdown_connections 2024-05-20T18:09:06.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.166+0000 7fe77ee6c640 1 -- 172.21.15.129:0/169516860 wait complete. 2024-05-20T18:09:06.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.167+0000 7fe77ee6c640 1 Processor -- start 2024-05-20T18:09:06.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.168+0000 7fe77ee6c640 1 -- start start 2024-05-20T18:09:06.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.168+0000 7fe77ee6c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7781014a0 0x7fe778073400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.168+0000 7fe77ee6c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe778106d00 0x7fe77806f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.169+0000 7fe77ee6c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7781071a0 0x7fe77806fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.169+0000 7fe777fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe778106d00 0x7fe77806f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.169+0000 7fe777fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe778106d00 0x7fe77806f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:39170/0 (socket says 172.21.15.129:39170) 2024-05-20T18:09:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.169+0000 7fe777fff640 1 -- 172.21.15.129:0/2429772474 learned_addr learned my addr 172.21.15.129:0/2429772474 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.169+0000 7fe77ee6c640 1 -- 172.21.15.129:0/2429772474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe77806da70 con 0x7fe7781071a0 2024-05-20T18:09:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.169+0000 7fe77ee6c640 1 -- 172.21.15.129:0/2429772474 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe77806d8f0 con 0x7fe778106d00 2024-05-20T18:09:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.169+0000 7fe77d409640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7781071a0 0x7fe77806fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.169+0000 7fe77ee6c640 1 -- 172.21.15.129:0/2429772474 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe77806dbf0 con 0x7fe7781014a0 2024-05-20T18:09:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.170+0000 7fe77d409640 1 -- 172.21.15.129:0/2429772474 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7781014a0 msgr2=0x7fe778073400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.170+0000 7fe77d409640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7781014a0 0x7fe778073400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.170+0000 7fe77d409640 1 -- 172.21.15.129:0/2429772474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe778106d00 msgr2=0x7fe77806f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.170+0000 7fe77d409640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe778106d00 0x7fe77806f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.170+0000 7fe77d409640 1 -- 172.21.15.129:0/2429772474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe778070000 con 0x7fe7781071a0 2024-05-20T18:09:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.170+0000 7fe777fff640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe778106d00 0x7fe77806f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:09:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.170+0000 7fe77d409640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7781071a0 0x7fe77806fac0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe76800b410 tx=0x7fe76800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.172+0000 7fe775ffb640 1 -- 172.21.15.129:0/2429772474 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe768015020 con 0x7fe7781071a0 2024-05-20T18:09:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.172+0000 7fe775ffb640 1 -- 172.21.15.129:0/2429772474 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe768011d10 con 0x7fe7781071a0 2024-05-20T18:09:06.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.172+0000 7fe775ffb640 1 -- 172.21.15.129:0/2429772474 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76800f2e0 con 0x7fe7781071a0 2024-05-20T18:09:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.172+0000 7fe77ee6c640 1 -- 172.21.15.129:0/2429772474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7780702f0 con 0x7fe7781071a0 2024-05-20T18:09:06.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.172+0000 7fe77ee6c640 1 -- 172.21.15.129:0/2429772474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7781a3ea0 con 0x7fe7781071a0 2024-05-20T18:09:06.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.176+0000 7fe785deb740 1 Processor -- start 2024-05-20T18:09:06.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.176+0000 7fe785deb740 1 -- start start 2024-05-20T18:09:06.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.177+0000 7fe785deb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6fa390 0x5605bda99010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.177+0000 7fe785deb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bda99550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.177+0000 7fe785deb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 0x5605bda99a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.177+0000 7fe785deb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5605bd9fec40 con 0x5605bd6d14d0 2024-05-20T18:09:06.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.177+0000 7fe785deb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5605bd9a6580 con 0x5605bd6d96b0 2024-05-20T18:09:06.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.177+0000 7fe785deb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5605bd9fac00 con 0x5605bd6fa390 2024-05-20T18:09:06.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.177+0000 7fe77cc08640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bda99550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:06.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.177+0000 7fe77cc08640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bda99550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58956/0 (socket says 172.21.15.129:58956) 2024-05-20T18:09:06.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.177+0000 7fe77cc08640 1 -- 172.21.15.129:0/4224224810 learned_addr learned my addr 172.21.15.129:0/4224224810 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:06.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.178+0000 7fe77cc08640 1 -- 172.21.15.129:0/4224224810 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6fa390 msgr2=0x5605bda99010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:09:06.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.178+0000 7fe77cc08640 1 --2- 172.21.15.129:0/4224224810 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6fa390 0x5605bda99010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.178+0000 7fe77cc08640 1 -- 172.21.15.129:0/4224224810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 msgr2=0x5605bda99a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:09:06.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.178+0000 7fe77cc08640 1 --2- 172.21.15.129:0/4224224810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 0x5605bda99a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.178+0000 7fe77cc08640 1 -- 172.21.15.129:0/4224224810 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605bd7123f0 con 0x5605bd6d14d0 2024-05-20T18:09:06.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.179+0000 7fe77cc08640 1 --2- 172.21.15.129:0/4224224810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bda99550 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe77001ae40 tx=0x7fe77001af30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e2114eaa344ad7d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:06.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.181+0000 7fe775ffb640 1 -- 172.21.15.129:0/2429772474 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fe768018020 con 0x7fe7781071a0 2024-05-20T18:09:06.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.181+0000 7fe775ffb640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe7600a2ed0 0x7fe7600a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.182+0000 7fe775ffb640 1 -- 172.21.15.129:0/2429772474 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7fe7680ceb50 con 0x7fe7781071a0 2024-05-20T18:09:06.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.183+0000 7fe75e7fc640 1 -- 172.21.15.129:0/4224224810 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77001fd10 con 0x5605bd6d14d0 2024-05-20T18:09:06.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.183+0000 7fe75e7fc640 1 -- 172.21.15.129:0/4224224810 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7700208f0 con 0x5605bd6d14d0 2024-05-20T18:09:06.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.183+0000 7fe75e7fc640 1 -- 172.21.15.129:0/4224224810 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77001d910 con 0x5605bd6d14d0 2024-05-20T18:09:06.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.184+0000 7fe777fff640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe7600a2ed0 0x7fe7600a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:06.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.185+0000 7fe777fff640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe7600a2ed0 0x7fe7600a53e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe76c02eb50 tx=0x7fe76c0359e0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:06.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.185+0000 7fe785deb740 1 -- 172.21.15.129:0/4224224810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 msgr2=0x5605bda99550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:06.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.185+0000 7fe785deb740 1 --2- 172.21.15.129:0/4224224810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bda99550 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe77001ae40 tx=0x7fe77001af30 comp rx=0 tx=0).stop 2024-05-20T18:09:06.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.185+0000 7fe785deb740 1 -- 172.21.15.129:0/4224224810 shutdown_connections 2024-05-20T18:09:06.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.185+0000 7fe785deb740 1 --2- 172.21.15.129:0/4224224810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 0x5605bda99a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.185+0000 7fe785deb740 1 --2- 172.21.15.129:0/4224224810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bda99550 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.185+0000 7fe785deb740 1 --2- 172.21.15.129:0/4224224810 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6fa390 0x5605bda99010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.185+0000 7fe785deb740 1 -- 172.21.15.129:0/4224224810 >> 172.21.15.129:0/4224224810 conn(0x5605bd6ee4e0 msgr2=0x5605bda957c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:06.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.186+0000 7fe785deb740 1 -- 172.21.15.129:0/4224224810 shutdown_connections 2024-05-20T18:09:06.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.186+0000 7fe785deb740 1 -- 172.21.15.129:0/4224224810 wait complete. 2024-05-20T18:09:06.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.186+0000 7fe785deb740 1 -- start start 2024-05-20T18:09:06.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.186+0000 7fe785deb740 1 Processor -- start 2024-05-20T18:09:06.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.187+0000 7fe785deb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaa3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.187+0000 7fe785deb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6d96b0 0x5605bdaa3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.187+0000 7fe785deb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6fa390 0x5605bdaaf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.187+0000 7fe785deb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5605bdaa6270 con 0x5605bd6d14d0 2024-05-20T18:09:06.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.187+0000 7fe77cc08640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaa3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:06.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.187+0000 7fe77cc08640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaa3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58970/0 (socket says 172.21.15.129:58970) 2024-05-20T18:09:06.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.187+0000 7fe77cc08640 1 -- 172.21.15.129:0/585499560 learned_addr learned my addr 172.21.15.129:0/585499560 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:06.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.187+0000 7fe785deb740 1 -- 172.21.15.129:0/585499560 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5605bd9b9c40 con 0x5605bd6fa390 2024-05-20T18:09:06.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.187+0000 7fe785deb740 1 -- 172.21.15.129:0/585499560 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5605bdaa63f0 con 0x5605bd6d96b0 2024-05-20T18:09:06.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.188+0000 7fe785deb740 10 client.0 _ll_register_callbacks cb 0x5605bd711be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.188+0000 7fe777fff640 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6d96b0 0x5605bdaa3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.188+0000 7fe77cc08640 1 -- 172.21.15.129:0/585499560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6d96b0 msgr2=0x5605bdaa3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.188+0000 7fe77cc08640 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6d96b0 0x5605bdaa3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.188+0000 7fe77cc08640 1 -- 172.21.15.129:0/585499560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6fa390 msgr2=0x5605bdaaf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.188+0000 7fe77cc08640 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6fa390 0x5605bdaaf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.188+0000 7fe77cc08640 1 -- 172.21.15.129:0/585499560 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605bd7dc7d0 con 0x5605bd6d14d0 2024-05-20T18:09:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.188+0000 7fe77cc08640 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaa3540 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x5605bda96b00 tx=0x7fe770000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc6bc9879f654ca server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.188+0000 7fe757fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.188+0000 7fe757fff640 1 -- 172.21.15.129:0/585499560 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe770034070 con 0x5605bd6d14d0 2024-05-20T18:09:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.189+0000 7fe757fff640 1 -- 172.21.15.129:0/585499560 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7fe770003970 con 0x5605bd6d14d0 2024-05-20T18:09:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.189+0000 7fe785deb740 1 -- 172.21.15.129:0/585499560 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7700208f0 con 0x5605bd6d14d0 2024-05-20T18:09:06.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.189+0000 7fe757fff640 1 -- 172.21.15.129:0/585499560 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe770040050 con 0x5605bd6d14d0 2024-05-20T18:09:06.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.189+0000 7fe785deb740 10 client.17187 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:06.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.189+0000 7fe785deb740 1 -- 172.21.15.129:0/585499560 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5605bdab2680 con 0x5605bd6d14d0 2024-05-20T18:09:06.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.189+0000 7fe757fff640 1 -- 172.21.15.129:0/585499560 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7700208f0 con 0x5605bd6d14d0 2024-05-20T18:09:06.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.190+0000 7fe757fff640 1 -- 172.21.15.129:0/585499560 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fe77001d5b0 con 0x5605bd6d14d0 2024-05-20T18:09:06.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.190+0000 7fe785deb740 10 client.17187 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:06.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.190+0000 7fe785deb740 10 client.17187 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:06.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.190+0000 7fe785deb740 20 client.17187 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:06.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.190+0000 7fe785deb740 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5605bd6b8f60 0x5605bdab5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.190+0000 7fe785deb740 4 client.17187 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:06.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.190+0000 7fe785deb740 1 -- 172.21.15.129:0/585499560 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605bda0c370 con 0x5605bd6b8f60 2024-05-20T18:09:06.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.191+0000 7fe777fff640 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5605bd6b8f60 0x5605bdab5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.193+0000 7fe777fff640 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5605bd6b8f60 0x5605bdab5f00 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e77035fb525537 server_cookie=1f69975a1cf97189 in_seq=0 out_seq=0 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.193+0000 7fe757fff640 10 client.17187 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.194+0000 7fe757fff640 1 -- 172.21.15.129:0/585499560 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5605bda0c370 con 0x5605bd6b8f60 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.194+0000 7fe757fff640 10 client.17187 handle_command_reply: tid=0 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.194+0000 7fe785deb740 1 client.17187 shutdown 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.195+0000 7fe785deb740 1 -- 172.21.15.129:0/585499560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 msgr2=0x5605bdaa3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.195+0000 7fe785deb740 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaa3540 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x5605bda96b00 tx=0x7fe770000f40 comp rx=0 tx=0).stop 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.195+0000 7fe785deb740 1 -- 172.21.15.129:0/585499560 shutdown_connections 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.195+0000 7fe785deb740 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5605bd6b8f60 0x5605bdab5f00 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.195+0000 7fe785deb740 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6fa390 0x5605bdaaf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.195+0000 7fe785deb740 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6d96b0 0x5605bdaa3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.195+0000 7fe785deb740 1 --2- 172.21.15.129:0/585499560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaa3540 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.195+0000 7fe785deb740 1 -- 172.21.15.129:0/585499560 >> 172.21.15.129:0/585499560 conn(0x5605bd6ee4e0 msgr2=0x5605bda98de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:06.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.195+0000 7fe757fff640 0 client.17187 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:06.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.195+0000 7fe785deb740 1 -- 172.21.15.129:0/585499560 shutdown_connections 2024-05-20T18:09:06.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.195+0000 7fe785deb740 1 -- 172.21.15.129:0/585499560 wait complete. 2024-05-20T18:09:06.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.196+0000 7fe785deb740 20 client.17187 trim_cache size 0 max 16384 2024-05-20T18:09:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.221+0000 7fe785deb740 1 Processor -- start 2024-05-20T18:09:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.221+0000 7fe785deb740 1 -- start start 2024-05-20T18:09:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.222+0000 7fe785deb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6d14d0 0x5605bda9d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.222+0000 7fe785deb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 0x5605bdaaf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.222+0000 7fe785deb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6fa390 0x5605bdabb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.222+0000 7fe785deb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe7781aaa50 con 0x5605bd6fa390 2024-05-20T18:09:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.222+0000 7fe785deb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe7781ad2b0 con 0x5605bd6d96b0 2024-05-20T18:09:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.222+0000 7fe785deb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe7781ab0a0 con 0x5605bd6d14d0 2024-05-20T18:09:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.222+0000 7fe77d409640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6fa390 0x5605bdabb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.222+0000 7fe77d409640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6fa390 0x5605bdabb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58978/0 (socket says 172.21.15.129:58978) 2024-05-20T18:09:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.222+0000 7fe77d409640 1 -- 172.21.15.129:0/1838205215 learned_addr learned my addr 172.21.15.129:0/1838205215 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.223+0000 7fe77d409640 1 -- 172.21.15.129:0/1838205215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6d14d0 msgr2=0x5605bda9d3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:09:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.223+0000 7fe77d409640 1 --2- 172.21.15.129:0/1838205215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6d14d0 0x5605bda9d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.223+0000 7fe77d409640 1 -- 172.21.15.129:0/1838205215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 msgr2=0x5605bdaaf300 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:09:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.223+0000 7fe77d409640 1 --2- 172.21.15.129:0/1838205215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 0x5605bdaaf300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.223+0000 7fe77d409640 1 -- 172.21.15.129:0/1838205215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605bdaa63e0 con 0x5605bd6fa390 2024-05-20T18:09:06.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.223+0000 7fe77d409640 1 --2- 172.21.15.129:0/1838205215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6fa390 0x5605bdabb700 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fe76809b040 tx=0x7fe76809abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed59743f6285698 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:06.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.224+0000 7fe75dffb640 1 -- 172.21.15.129:0/1838205215 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76809b0b0 con 0x5605bd6fa390 2024-05-20T18:09:06.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.224+0000 7fe75dffb640 1 -- 172.21.15.129:0/1838205215 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe76809f460 con 0x5605bd6fa390 2024-05-20T18:09:06.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.224+0000 7fe75dffb640 1 -- 172.21.15.129:0/1838205215 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe768097720 con 0x5605bd6fa390 2024-05-20T18:09:06.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.225+0000 7fe785deb740 1 -- 172.21.15.129:0/1838205215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6fa390 msgr2=0x5605bdabb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:06.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.225+0000 7fe785deb740 1 --2- 172.21.15.129:0/1838205215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6fa390 0x5605bdabb700 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fe76809b040 tx=0x7fe76809abf0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.227+0000 7fe785deb740 1 -- 172.21.15.129:0/1838205215 shutdown_connections 2024-05-20T18:09:06.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.227+0000 7fe785deb740 1 --2- 172.21.15.129:0/1838205215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 0x5605bdaaf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.227+0000 7fe785deb740 1 --2- 172.21.15.129:0/1838205215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6fa390 0x5605bdabb700 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.227+0000 7fe785deb740 1 --2- 172.21.15.129:0/1838205215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6d14d0 0x5605bda9d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.227+0000 7fe785deb740 1 -- 172.21.15.129:0/1838205215 >> 172.21.15.129:0/1838205215 conn(0x5605bd6ee4e0 msgr2=0x5605bda9ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:06.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.232+0000 7fe785deb740 1 -- 172.21.15.129:0/1838205215 shutdown_connections 2024-05-20T18:09:06.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.235+0000 7fe785deb740 1 -- 172.21.15.129:0/1838205215 wait complete. 2024-05-20T18:09:06.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.236+0000 7fe785deb740 1 -- start start 2024-05-20T18:09:06.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.236+0000 7fe785deb740 1 Processor -- start 2024-05-20T18:09:06.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.237+0000 7fe785deb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaab690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.237+0000 7fe785deb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 0x5605bdaabbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.237+0000 7fe785deb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6fa390 0x5605bdaa6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.237+0000 7fe785deb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe7781ab770 con 0x5605bd6d14d0 2024-05-20T18:09:06.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.237+0000 7fe785deb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe7781acbe0 con 0x5605bd6d96b0 2024-05-20T18:09:06.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.237+0000 7fe785deb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe7781abe40 con 0x5605bd6fa390 2024-05-20T18:09:06.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.237+0000 7fe785deb740 10 client.0 _ll_register_callbacks cb 0x7fe77001ac90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:06.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.238+0000 7fe77cc08640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:06.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.238+0000 7fe77cc08640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaab690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58990/0 (socket says 172.21.15.129:58990) 2024-05-20T18:09:06.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.238+0000 7fe77cc08640 1 -- 172.21.15.129:0/3020619358 learned_addr learned my addr 172.21.15.129:0/3020619358 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:06.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.238+0000 7fe77d409640 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6fa390 0x5605bdaa6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:06.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.238+0000 7fe77cc08640 1 -- 172.21.15.129:0/3020619358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6fa390 msgr2=0x5605bdaa6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:06.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.238+0000 7fe77cc08640 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6fa390 0x5605bdaa6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.238+0000 7fe77cc08640 1 -- 172.21.15.129:0/3020619358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 msgr2=0x5605bdaabbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:06.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.238+0000 7fe77cc08640 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 0x5605bdaabbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.238+0000 7fe77cc08640 1 -- 172.21.15.129:0/3020619358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605bdaa6d30 con 0x5605bd6d14d0 2024-05-20T18:09:06.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.239+0000 7fe77cc08640 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaab690 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe770006010 tx=0x7fe7700206d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287e96b87bb8a7ab server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:06.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.239+0000 7fe7577fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:06.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.239+0000 7fe7577fe640 1 -- 172.21.15.129:0/3020619358 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe770034040 con 0x5605bd6d14d0 2024-05-20T18:09:06.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.239+0000 7fe785deb740 1 -- 172.21.15.129:0/3020619358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe76809f460 con 0x5605bd6d14d0 2024-05-20T18:09:06.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.239+0000 7fe7577fe640 1 -- 172.21.15.129:0/3020619358 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7fe770020880 con 0x5605bd6d14d0 2024-05-20T18:09:06.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.240+0000 7fe7577fe640 1 -- 172.21.15.129:0/3020619358 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe77001aa10 con 0x5605bd6d14d0 2024-05-20T18:09:06.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.240+0000 7fe785deb740 10 client.17193 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:06.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.240+0000 7fe785deb740 1 -- 172.21.15.129:0/3020619358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5605bdb1cbe0 con 0x5605bd6d14d0 2024-05-20T18:09:06.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.241+0000 7fe7577fe640 1 -- 172.21.15.129:0/3020619358 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe76809f460 con 0x5605bd6d14d0 2024-05-20T18:09:06.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.241+0000 7fe7577fe640 1 -- 172.21.15.129:0/3020619358 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fe770004030 con 0x5605bd6d14d0 2024-05-20T18:09:06.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.241+0000 7fe785deb740 10 client.17193 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:06.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.241+0000 7fe785deb740 10 client.17193 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:06.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.241+0000 7fe785deb740 20 client.17193 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:06.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.241+0000 7fe785deb740 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5605bd68e3a0 0x5605bdb1f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:06.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.241+0000 7fe785deb740 4 client.17193 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:09:06.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.241+0000 7fe785deb740 1 -- 172.21.15.129:0/3020619358 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5605bda066c0 con 0x5605bd68e3a0 2024-05-20T18:09:06.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.242+0000 7fe777fff640 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5605bd68e3a0 0x5605bdb1f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:06.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.244+0000 7fe777fff640 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5605bd68e3a0 0x5605bdb1f330 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4f88bc99f63df47 server_cookie=23f8351268e83565 in_seq=0 out_seq=0 2024-05-20T18:09:06.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.244+0000 7fe7577fe640 10 client.17193 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:06.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.244+0000 7fe7577fe640 1 -- 172.21.15.129:0/3020619358 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5605bda066c0 con 0x5605bd68e3a0 2024-05-20T18:09:06.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.244+0000 7fe7577fe640 10 client.17193 handle_command_reply: tid=0 2024-05-20T18:09:06.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.244+0000 7fe785deb740 1 client.17193 shutdown 2024-05-20T18:09:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.245+0000 7fe785deb740 1 -- 172.21.15.129:0/3020619358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 msgr2=0x5605bdaab690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.245+0000 7fe785deb740 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaab690 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe770006010 tx=0x7fe7700206d0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.245+0000 7fe785deb740 1 -- 172.21.15.129:0/3020619358 shutdown_connections 2024-05-20T18:09:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.245+0000 7fe785deb740 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5605bd68e3a0 0x5605bdb1f330 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.245+0000 7fe785deb740 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605bd6d96b0 0x5605bdaabbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.245+0000 7fe785deb740 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5605bd6fa390 0x5605bdaa6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.245+0000 7fe785deb740 1 --2- 172.21.15.129:0/3020619358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5605bd6d14d0 0x5605bdaab690 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.245+0000 7fe785deb740 1 -- 172.21.15.129:0/3020619358 >> 172.21.15.129:0/3020619358 conn(0x5605bd6ee4e0 msgr2=0x5605bda95180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.245+0000 7fe7577fe640 0 client.17193 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.245+0000 7fe785deb740 1 -- 172.21.15.129:0/3020619358 shutdown_connections 2024-05-20T18:09:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.245+0000 7fe785deb740 1 -- 172.21.15.129:0/3020619358 wait complete. 2024-05-20T18:09:06.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.246+0000 7fe785deb740 20 client.17193 trim_cache size 0 max 16384 2024-05-20T18:09:06.245 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:09:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.251+0000 7fe75f7fe640 1 -- 172.21.15.129:0/2429772474 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe7600a2ed0 msgr2=0x7fe7600a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.251+0000 7fe75f7fe640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe7600a2ed0 0x7fe7600a53e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe76c02eb50 tx=0x7fe76c0359e0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.251+0000 7fe75f7fe640 1 -- 172.21.15.129:0/2429772474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7781071a0 msgr2=0x7fe77806fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:06.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.251+0000 7fe75f7fe640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7781071a0 0x7fe77806fac0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe76800b410 tx=0x7fe76800b8d0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.252+0000 7fe75f7fe640 1 -- 172.21.15.129:0/2429772474 shutdown_connections 2024-05-20T18:09:06.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.252+0000 7fe75f7fe640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe7600a2ed0 0x7fe7600a53e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.253+0000 7fe75f7fe640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe778106d00 0x7fe77806f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.253+0000 7fe75f7fe640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7781071a0 0x7fe77806fac0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.253+0000 7fe75f7fe640 1 --2- 172.21.15.129:0/2429772474 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7781014a0 0x7fe778073400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:06.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.253+0000 7fe75f7fe640 1 -- 172.21.15.129:0/2429772474 >> 172.21.15.129:0/2429772474 conn(0x7fe7780763f0 msgr2=0x7fe7780ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:06.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.254+0000 7fe75f7fe640 1 -- 172.21.15.129:0/2429772474 shutdown_connections 2024-05-20T18:09:06.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:06.254+0000 7fe75f7fe640 1 -- 172.21.15.129:0/2429772474 wait complete. 2024-05-20T18:09:06.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:06 smithi154 ceph-mon[36482]: osdmap e103: 12 total, 12 up, 12 in 2024-05-20T18:09:06.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:06 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2569331524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:09:06.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:06 smithi129 ceph-mon[32029]: osdmap e103: 12 total, 12 up, 12 in 2024-05-20T18:09:06.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:06 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2569331524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:09:06.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:06 smithi204 ceph-mon[37076]: osdmap e103: 12 total, 12 up, 12 in 2024-05-20T18:09:06.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:06 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2569331524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:09:06.679 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: *** Caught signal (Segmentation fault) ** 2024-05-20T18:09:06.679 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: in thread 7f8c48b66700 thread_name:md_log_replay 2024-05-20T18:09:06.679 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:09:06.679 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c59f28d20] 2024-05-20T18:09:06.679 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x33) [0x558ad4f0eb97] 2024-05-20T18:09:06.679 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x558ad4f0f02e] 2024-05-20T18:09:06.679 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x558ad4f096a5] 2024-05-20T18:09:06.679 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x558ad4bcb3c3] 2024-05-20T18:09:06.679 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 6: (MDLog::_replay_thread()+0x1f38) [0x558ad4f6f69c] 2024-05-20T18:09:06.679 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 7: (MDLog::ReplayThread::entry()+0x11) [0x558ad4f78ae7] 2024-05-20T18:09:06.680 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 8: (Thread::entry_wrapper()+0x43) [0x7f8c5b3796f1] 2024-05-20T18:09:06.680 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 9: (Thread::_entry_func(void*)+0xd) [0x7f8c5b37970d] 2024-05-20T18:09:06.680 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8c59f1e1ca] 2024-05-20T18:09:06.680 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 11: clone() 2024-05-20T18:09:06.680 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 2024-05-20T18:09:06.241+0000 7f8c48b66700 -1 *** Caught signal (Segmentation fault) ** 2024-05-20T18:09:06.680 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: in thread 7f8c48b66700 thread_name:md_log_replay 2024-05-20T18:09:06.680 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 2024-05-20T18:09:06.680 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:09:06.680 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c59f28d20] 2024-05-20T18:09:06.680 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x33) [0x558ad4f0eb97] 2024-05-20T18:09:06.681 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x558ad4f0f02e] 2024-05-20T18:09:06.681 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x558ad4f096a5] 2024-05-20T18:09:06.681 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x558ad4bcb3c3] 2024-05-20T18:09:06.681 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 6: (MDLog::_replay_thread()+0x1f38) [0x558ad4f6f69c] 2024-05-20T18:09:06.681 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 7: (MDLog::ReplayThread::entry()+0x11) [0x558ad4f78ae7] 2024-05-20T18:09:06.681 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 8: (Thread::entry_wrapper()+0x43) [0x7f8c5b3796f1] 2024-05-20T18:09:06.681 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 9: (Thread::_entry_func(void*)+0xd) [0x7f8c5b37970d] 2024-05-20T18:09:06.681 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8c59f1e1ca] 2024-05-20T18:09:06.681 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 11: clone() 2024-05-20T18:09:06.681 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 0> 2024-05-20T18:09:06.241+0000 7f8c48b66700 -1 *** Caught signal (Segmentation fault) ** 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: in thread 7f8c48b66700 thread_name:md_log_replay 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c59f28d20] 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x33) [0x558ad4f0eb97] 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x558ad4f0f02e] 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x558ad4f096a5] 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x558ad4bcb3c3] 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 6: (MDLog::_replay_thread()+0x1f38) [0x558ad4f6f69c] 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 7: (MDLog::ReplayThread::entry()+0x11) [0x558ad4f78ae7] 2024-05-20T18:09:06.682 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 8: (Thread::entry_wrapper()+0x43) [0x7f8c5b3796f1] 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 9: (Thread::_entry_func(void*)+0xd) [0x7f8c5b37970d] 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8c59f1e1ca] 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 11: clone() 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 0> 2024-05-20T18:09:06.241+0000 7f8c48b66700 -1 *** Caught signal (Segmentation fault) ** 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: in thread 7f8c48b66700 thread_name:md_log_replay 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: ceph version 19.0.0-2744-gd2b507c7 (d2b507c724ef6c8295ee0a35d185cdab167fd61d) squid (dev) 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c59f28d20] 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x33) [0x558ad4f0eb97] 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x558ad4f0f02e] 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x558ad4f096a5] 2024-05-20T18:09:06.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x558ad4bcb3c3] 2024-05-20T18:09:06.684 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 6: (MDLog::_replay_thread()+0x1f38) [0x558ad4f6f69c] 2024-05-20T18:09:06.684 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 7: (MDLog::ReplayThread::entry()+0x11) [0x558ad4f78ae7] 2024-05-20T18:09:06.684 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 8: (Thread::entry_wrapper()+0x43) [0x7f8c5b3796f1] 2024-05-20T18:09:06.684 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 9: (Thread::_entry_func(void*)+0xd) [0x7f8c5b37970d] 2024-05-20T18:09:06.684 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8c59f1e1ca] 2024-05-20T18:09:06.684 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 11: clone() 2024-05-20T18:09:06.684 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-20T18:09:06.684 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[68061]: 2024-05-20T18:09:06.684 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:06 smithi204 podman[72352]: 2024-05-20 18:09:06.346215593 +0000 UTC m=+0.040838422 container died ab802f7dcb52ddf9da091ffff945a0e4c44a3a4c7a60abe4a635b16d9c9ea829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-20T18:09:07.271 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:09:07.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:09:07.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:09:07.382 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:07 smithi204 podman[72352]: 2024-05-20 18:09:07.133991931 +0000 UTC m=+0.828614733 container remove ab802f7dcb52ddf9da091ffff945a0e4c44a3a4c7a60abe4a635b16d9c9ea829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-20T18:09:07.383 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:07 smithi204 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.i.service: Main process exited, code=exited, status=139/n/a 2024-05-20T18:09:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.403+0000 7f361999d640 1 -- 172.21.15.129:0/4227953712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3614108b50 msgr2=0x7f3614100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.403+0000 7f361999d640 1 --2- 172.21.15.129:0/4227953712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3614108b50 0x7f3614100be0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3604009ca0 tx=0x7f360402a8c0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.403+0000 7f361999d640 1 -- 172.21.15.129:0/4227953712 shutdown_connections 2024-05-20T18:09:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.403+0000 7f361999d640 1 --2- 172.21.15.129:0/4227953712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3614108ff0 0x7f3614101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.403+0000 7f361999d640 1 --2- 172.21.15.129:0/4227953712 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3614101790 0x7f361410f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.403+0000 7f361999d640 1 --2- 172.21.15.129:0/4227953712 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3614108b50 0x7f3614100be0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.403+0000 7f361999d640 1 -- 172.21.15.129:0/4227953712 >> 172.21.15.129:0/4227953712 conn(0x7f3614076830 msgr2=0x7f3614076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.403+0000 7f361999d640 1 -- 172.21.15.129:0/4227953712 shutdown_connections 2024-05-20T18:09:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.403+0000 7f361999d640 1 -- 172.21.15.129:0/4227953712 wait complete. 2024-05-20T18:09:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.403+0000 7f361999d640 1 Processor -- start 2024-05-20T18:09:07.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f361999d640 1 -- start start 2024-05-20T18:09:07.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f361999d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3614101790 0x7f361419f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f361999d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3614108b50 0x7f361419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f361999d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3614108ff0 0x7f36141996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f361999d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3614112e90 con 0x7f3614108ff0 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f361999d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3614112d10 con 0x7f3614108b50 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f361999d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3614113010 con 0x7f3614101790 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f36137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3614108ff0 0x7f36141996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f36137fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3614108ff0 0x7f36141996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59016/0 (socket says 172.21.15.129:59016) 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f36137fe640 1 -- 172.21.15.129:0/3777478366 learned_addr learned my addr 172.21.15.129:0/3777478366 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f36127fc640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3614108b50 0x7f361419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f3612ffd640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3614101790 0x7f361419f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f36137fe640 1 -- 172.21.15.129:0/3777478366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3614101790 msgr2=0x7f361419f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f36137fe640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3614101790 0x7f361419f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f36137fe640 1 -- 172.21.15.129:0/3777478366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3614108b50 msgr2=0x7f361419fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f36137fe640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3614108b50 0x7f361419fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f36137fe640 1 -- 172.21.15.129:0/3777478366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3614199e00 con 0x7f3614108ff0 2024-05-20T18:09:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.404+0000 7f36137fe640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3614108ff0 0x7f36141996d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f360c00e970 tx=0x7f360c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.405+0000 7f361899b640 1 -- 172.21.15.129:0/3777478366 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c00ccc0 con 0x7f3614108ff0 2024-05-20T18:09:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.405+0000 7f361899b640 1 -- 172.21.15.129:0/3777478366 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f360c00ce60 con 0x7f3614108ff0 2024-05-20T18:09:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.405+0000 7f361899b640 1 -- 172.21.15.129:0/3777478366 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c010490 con 0x7f3614108ff0 2024-05-20T18:09:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.405+0000 7f361999d640 1 -- 172.21.15.129:0/3777478366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361419a090 con 0x7f3614108ff0 2024-05-20T18:09:07.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.405+0000 7f361999d640 1 -- 172.21.15.129:0/3777478366 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36141a62f0 con 0x7f3614108ff0 2024-05-20T18:09:07.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.407+0000 7f3620849740 1 Processor -- start 2024-05-20T18:09:07.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.407+0000 7f361899b640 1 -- 172.21.15.129:0/3777478366 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f360c005020 con 0x7f3614108ff0 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.407+0000 7f3620849740 1 -- start start 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3620849740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 0x56222fab1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3620849740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6e9550 0x56222fab1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3620849740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6f1730 0x56222fab20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3620849740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56222fa16990 con 0x56222f6e9550 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3620849740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56222f9be5b0 con 0x56222f6f1730 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3620849740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56222fa12950 con 0x56222f712410 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3612ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6e9550 0x56222fab1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3612ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6e9550 0x56222fab1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59020/0 (socket says 172.21.15.129:59020) 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3612ffd640 1 -- 172.21.15.129:0/2743208721 learned_addr learned my addr 172.21.15.129:0/2743208721 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f36127fc640 1 --2- 172.21.15.129:0/2743208721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 0x56222fab1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f36127fc640 1 --2- 172.21.15.129:0/2743208721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6f1730 0x56222fab20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f361899b640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f35fc0a2d00 0x7f35fc0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3612ffd640 1 -- 172.21.15.129:0/2743208721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 msgr2=0x56222fab1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3612ffd640 1 --2- 172.21.15.129:0/2743208721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 0x56222fab1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3612ffd640 1 -- 172.21.15.129:0/2743208721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6f1730 msgr2=0x56222fab20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3612ffd640 1 --2- 172.21.15.129:0/2743208721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6f1730 0x56222fab20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3612ffd640 1 -- 172.21.15.129:0/2743208721 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56222f7f27e0 con 0x56222f6e9550 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f36127fc640 1 --2- 172.21.15.129:0/2743208721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 0x56222fab1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f36137fe640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f35fc0a2d00 0x7f35fc0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f361899b640 1 -- 172.21.15.129:0/3777478366 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7f360c0cfa90 con 0x7f3614108ff0 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f3612ffd640 1 --2- 172.21.15.129:0/2743208721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6e9550 0x56222fab1b60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3604009ca0 tx=0x7f360402f410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491529117423251f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.408+0000 7f35f9ffb640 1 -- 172.21.15.129:0/2743208721 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604031db0 con 0x56222f6e9550 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f35f9ffb640 1 -- 172.21.15.129:0/2743208721 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3604036050 con 0x56222f6e9550 2024-05-20T18:09:07.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f35f9ffb640 1 -- 172.21.15.129:0/2743208721 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36040042a0 con 0x56222f6e9550 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f3620849740 1 -- 172.21.15.129:0/2743208721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6e9550 msgr2=0x56222fab1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f3620849740 1 --2- 172.21.15.129:0/2743208721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6e9550 0x56222fab1b60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3604009ca0 tx=0x7f360402f410 comp rx=0 tx=0).stop 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f36137fe640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f35fc0a2d00 0x7f35fc0a5210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f360c0cd220 tx=0x7f360c0bc040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f3620849740 1 -- 172.21.15.129:0/2743208721 shutdown_connections 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f3620849740 1 --2- 172.21.15.129:0/2743208721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6f1730 0x56222fab20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f3620849740 1 --2- 172.21.15.129:0/2743208721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6e9550 0x56222fab1b60 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f3620849740 1 --2- 172.21.15.129:0/2743208721 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 0x56222fab1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f3620849740 1 -- 172.21.15.129:0/2743208721 >> 172.21.15.129:0/2743208721 conn(0x56222f706560 msgr2=0x56222faaddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f3620849740 1 -- 172.21.15.129:0/2743208721 shutdown_connections 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.409+0000 7f3620849740 1 -- 172.21.15.129:0/2743208721 wait complete. 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.410+0000 7f3620849740 1 -- start start 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.410+0000 7f3620849740 1 Processor -- start 2024-05-20T18:09:07.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.410+0000 7f3620849740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fabd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.410+0000 7f3620849740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 0x56222fabdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.410+0000 7f3620849740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 0x56222fabbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.410+0000 7f3620849740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56222fabe790 con 0x56222f712410 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.410+0000 7f3620849740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56222f9d1b40 con 0x56222f6e9550 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.410+0000 7f3620849740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56222fabe910 con 0x56222f6f1730 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.410+0000 7f3620849740 10 client.0 _ll_register_callbacks cb 0x56222f9c60f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.410+0000 7f36127fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 0x56222fabbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f36127fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 0x56222fabbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59024/0 (socket says 172.21.15.129:59024) 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f3612ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 0x56222fabdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f36127fc640 1 -- 172.21.15.129:0/2507770132 learned_addr learned my addr 172.21.15.129:0/2507770132 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f36127fc640 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fabd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f3612ffd640 1 -- 172.21.15.129:0/2507770132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 msgr2=0x56222fabd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f3612ffd640 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fabd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f3612ffd640 1 -- 172.21.15.129:0/2507770132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 msgr2=0x56222fabbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f3612ffd640 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 0x56222fabbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f3612ffd640 1 -- 172.21.15.129:0/2507770132 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56222f852f70 con 0x56222f6f1730 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f36127fc640 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 0x56222fabbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f3612ffd640 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 0x56222fabdbe0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x56222fab2be0 tx=0x7f36040095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5f83d1a41095312 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f35f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f35f2ffd640 1 -- 172.21.15.129:0/2507770132 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604053060 con 0x56222f6f1730 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f35f2ffd640 1 -- 172.21.15.129:0/2507770132 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7f3604003d80 con 0x56222f6f1730 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f3620849740 1 -- 172.21.15.129:0/2507770132 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3604036050 con 0x56222f6f1730 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f35f2ffd640 1 -- 172.21.15.129:0/2507770132 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3604009360 con 0x56222f6f1730 2024-05-20T18:09:07.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f3620849740 10 client.26788 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.411+0000 7f3620849740 1 -- 172.21.15.129:0/2507770132 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56222f6ea170 con 0x56222f6f1730 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.412+0000 7f35f2ffd640 1 -- 172.21.15.129:0/2507770132 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3604036050 con 0x56222f6f1730 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.412+0000 7f35f2ffd640 1 -- 172.21.15.129:0/2507770132 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f3604042070 con 0x56222f6f1730 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.412+0000 7f3620849740 10 client.26788 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.412+0000 7f3620849740 10 client.26788 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.412+0000 7f3620849740 20 client.26788 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.412+0000 7f3620849740 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56222f6d11f0 0x56222facde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.412+0000 7f3620849740 4 client.26788 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.412+0000 7f3620849740 1 -- 172.21.15.129:0/2507770132 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56222fa240c0 con 0x56222f6d11f0 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.412+0000 7f36127fc640 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56222f6d11f0 0x56222facde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.412+0000 7f35f2ffd640 10 client.26788 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.412+0000 7f36127fc640 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56222f6d11f0 0x56222facde00 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9c711acb199c0d3 server_cookie=ecd08650a905c9d1 in_seq=0 out_seq=0 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.413+0000 7f35f2ffd640 1 -- 172.21.15.129:0/2507770132 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56222fa240c0 con 0x56222f6d11f0 2024-05-20T18:09:07.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.413+0000 7f35f2ffd640 10 client.26788 handle_command_reply: tid=0 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.413+0000 7f3620849740 1 client.26788 shutdown 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f3620849740 1 -- 172.21.15.129:0/2507770132 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 msgr2=0x56222fabdbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f3620849740 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 0x56222fabdbe0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x56222fab2be0 tx=0x7f36040095c0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f3620849740 1 -- 172.21.15.129:0/2507770132 shutdown_connections 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f3620849740 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56222f6d11f0 0x56222facde00 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f3620849740 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 0x56222fabbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f3620849740 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 0x56222fabdbe0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f3620849740 1 --2- 172.21.15.129:0/2507770132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fabd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f35f2ffd640 0 client.26788 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f3620849740 1 -- 172.21.15.129:0/2507770132 >> 172.21.15.129:0/2507770132 conn(0x56222f706560 msgr2=0x56222fab13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f3620849740 1 -- 172.21.15.129:0/2507770132 shutdown_connections 2024-05-20T18:09:07.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f3620849740 1 -- 172.21.15.129:0/2507770132 wait complete. 2024-05-20T18:09:07.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.414+0000 7f3620849740 20 client.26788 trim_cache size 0 max 16384 2024-05-20T18:09:07.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3620849740 1 Processor -- start 2024-05-20T18:09:07.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3620849740 1 -- start start 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3620849740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fabb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3620849740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6f1730 0x56222fab8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3620849740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 0x56222fab92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3620849740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f36141acbd0 con 0x56222f6f1730 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3620849740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36141af430 con 0x56222f6e9550 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3620849740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f36141ad220 con 0x56222f712410 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3612ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6f1730 0x56222fab8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3612ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6f1730 0x56222fab8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59040/0 (socket says 172.21.15.129:59040) 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3612ffd640 1 -- 172.21.15.129:0/482875593 learned_addr learned my addr 172.21.15.129:0/482875593 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f36127fc640 1 --2- 172.21.15.129:0/482875593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 0x56222fab92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f36127fc640 1 --2- 172.21.15.129:0/482875593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fabb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3612ffd640 1 -- 172.21.15.129:0/482875593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 msgr2=0x56222fab92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3612ffd640 1 --2- 172.21.15.129:0/482875593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 0x56222fab92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3612ffd640 1 -- 172.21.15.129:0/482875593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 msgr2=0x56222fabb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3612ffd640 1 --2- 172.21.15.129:0/482875593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fabb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3612ffd640 1 -- 172.21.15.129:0/482875593 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56222fabe010 con 0x56222f6f1730 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f36127fc640 1 --2- 172.21.15.129:0/482875593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 0x56222fab92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.432+0000 7f3612ffd640 1 --2- 172.21.15.129:0/482875593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6f1730 0x56222fab8d80 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3604005e30 tx=0x7f3604000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768553d8fc74098b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.433+0000 7f35f97fa640 1 -- 172.21.15.129:0/482875593 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604009360 con 0x56222f6f1730 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.433+0000 7f35f97fa640 1 -- 172.21.15.129:0/482875593 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3604036070 con 0x56222f6f1730 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.433+0000 7f35f97fa640 1 -- 172.21.15.129:0/482875593 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604030a60 con 0x56222f6f1730 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.433+0000 7f3620849740 1 -- 172.21.15.129:0/482875593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6f1730 msgr2=0x56222fab8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.433+0000 7f3620849740 1 --2- 172.21.15.129:0/482875593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6f1730 0x56222fab8d80 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3604005e30 tx=0x7f3604000f40 comp rx=0 tx=0).stop 2024-05-20T18:09:07.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.433+0000 7f3620849740 1 -- 172.21.15.129:0/482875593 shutdown_connections 2024-05-20T18:09:07.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.433+0000 7f3620849740 1 --2- 172.21.15.129:0/482875593 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f712410 0x56222fab92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.433+0000 7f3620849740 1 --2- 172.21.15.129:0/482875593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f6f1730 0x56222fab8d80 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.433+0000 7f3620849740 1 --2- 172.21.15.129:0/482875593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fabb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.433+0000 7f3620849740 1 -- 172.21.15.129:0/482875593 >> 172.21.15.129:0/482875593 conn(0x56222f706560 msgr2=0x56222fab1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.434+0000 7f3620849740 1 -- 172.21.15.129:0/482875593 shutdown_connections 2024-05-20T18:09:07.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.434+0000 7f3620849740 1 -- 172.21.15.129:0/482875593 wait complete. 2024-05-20T18:09:07.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.434+0000 7f3620849740 1 -- start start 2024-05-20T18:09:07.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.434+0000 7f3620849740 1 Processor -- start 2024-05-20T18:09:07.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.434+0000 7f3620849740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fad0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.434+0000 7f3620849740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 0x56222fad08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.434+0000 7f3620849740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 0x56222fb2df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.434+0000 7f3620849740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f36141ad8f0 con 0x56222f712410 2024-05-20T18:09:07.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.434+0000 7f3620849740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36141aed60 con 0x56222f6e9550 2024-05-20T18:09:07.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.434+0000 7f3620849740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f36141adfc0 con 0x56222f6f1730 2024-05-20T18:09:07.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.434+0000 7f3620849740 10 client.0 _ll_register_callbacks cb 0x7f3604009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:07.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f36127fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 0x56222fb2df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f36127fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fad0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f36127fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fad0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:39236/0 (socket says 172.21.15.129:39236) 2024-05-20T18:09:07.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f3612ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 0x56222fad08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f36127fc640 1 -- 172.21.15.129:0/586858781 learned_addr learned my addr 172.21.15.129:0/586858781 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:07.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f36127fc640 1 -- 172.21.15.129:0/586858781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 msgr2=0x56222fad08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f36127fc640 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 0x56222fad08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f36127fc640 1 -- 172.21.15.129:0/586858781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 msgr2=0x56222fb2df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f36127fc640 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 0x56222fb2df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f36127fc640 1 -- 172.21.15.129:0/586858781 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56222fb2e500 con 0x56222f6e9550 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f3612ffd640 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 0x56222fad08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f36127fc640 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fad0360 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3608004700 tx=0x7f360800c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac433a94b26baa50 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f35f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f35f2ffd640 1 -- 172.21.15.129:0/586858781 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608011930 con 0x56222f6e9550 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f3620849740 1 -- 172.21.15.129:0/586858781 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3604036070 con 0x56222f6e9550 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f35f2ffd640 1 -- 172.21.15.129:0/586858781 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7f360801b040 con 0x56222f6e9550 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f35f2ffd640 1 -- 172.21.15.129:0/586858781 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3608034050 con 0x56222f6e9550 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f3620849740 10 client.26789 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.435+0000 7f3620849740 1 -- 172.21.15.129:0/586858781 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56222fb358b0 con 0x56222f6e9550 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.436+0000 7f35f2ffd640 1 -- 172.21.15.129:0/586858781 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3604036070 con 0x56222f6e9550 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.436+0000 7f35f2ffd640 1 -- 172.21.15.129:0/586858781 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7f360802fb00 con 0x56222f6e9550 2024-05-20T18:09:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.436+0000 7f3620849740 10 client.26789 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.436+0000 7f3620849740 10 client.26789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.436+0000 7f3620849740 20 client.26789 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.436+0000 7f3620849740 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56222f6a6630 0x56222fb38020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.436+0000 7f3620849740 4 client.26789 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.436+0000 7f3620849740 1 -- 172.21.15.129:0/586858781 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56222fa1e410 con 0x56222f6a6630 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.436+0000 7f3612ffd640 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56222f6a6630 0x56222fb38020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.436+0000 7f35f2ffd640 10 client.26789 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.436+0000 7f3612ffd640 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56222f6a6630 0x56222fb38020 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74aaf1a4b313089d server_cookie=1e8c745fb42724bd in_seq=0 out_seq=0 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f35f2ffd640 1 -- 172.21.15.129:0/586858781 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56222fa1e410 con 0x56222f6a6630 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f35f2ffd640 10 client.26789 handle_command_reply: tid=0 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 1 client.26789 shutdown 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 1 -- 172.21.15.129:0/586858781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 msgr2=0x56222fad0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fad0360 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3608004700 tx=0x7f360800c7b0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 1 -- 172.21.15.129:0/586858781 shutdown_connections 2024-05-20T18:09:07.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x56222f6a6630 0x56222fb38020 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56222f712410 0x56222fb2df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f35f2ffd640 0 client.26789 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:07.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56222f6f1730 0x56222fad08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 1 --2- 172.21.15.129:0/586858781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56222f6e9550 0x56222fad0360 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 1 -- 172.21.15.129:0/586858781 >> 172.21.15.129:0/586858781 conn(0x56222f706560 msgr2=0x56222fab1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 1 -- 172.21.15.129:0/586858781 shutdown_connections 2024-05-20T18:09:07.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 1 -- 172.21.15.129:0/586858781 wait complete. 2024-05-20T18:09:07.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.438+0000 7f3620849740 20 client.26789 trim_cache size 0 max 16384 2024-05-20T18:09:07.443 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:09:07.443 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:09:07.443 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "8d1abc27-947a-4833-a299-a316a7a12d16", 2024-05-20T18:09:07.443 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:09:07.443 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:09:07.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 -- 172.21.15.129:0/3777478366 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f35fc0a2d00 msgr2=0x7f35fc0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f35fc0a2d00 0x7f35fc0a5210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f360c0cd220 tx=0x7f360c0bc040 comp rx=0 tx=0).stop 2024-05-20T18:09:07.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 -- 172.21.15.129:0/3777478366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3614108ff0 msgr2=0x7f36141996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3614108ff0 0x7f36141996d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f360c00e970 tx=0x7f360c00ed70 comp rx=0 tx=0).stop 2024-05-20T18:09:07.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 -- 172.21.15.129:0/3777478366 shutdown_connections 2024-05-20T18:09:07.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f35fc0a2d00 0x7f35fc0a5210 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3614108b50 0x7f361419fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3614108ff0 0x7f36141996d0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 --2- 172.21.15.129:0/3777478366 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3614101790 0x7f361419f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 -- 172.21.15.129:0/3777478366 >> 172.21.15.129:0/3777478366 conn(0x7f3614076830 msgr2=0x7f3614109550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 -- 172.21.15.129:0/3777478366 shutdown_connections 2024-05-20T18:09:07.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.439+0000 7f361999d640 1 -- 172.21.15.129:0/3777478366 wait complete. 2024-05-20T18:09:07.450 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:09:07.451 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:09:07.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.578+0000 7fa1dcb85640 1 -- 172.21.15.129:0/1891242423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d81079f0 msgr2=0x7fa1d8101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.578+0000 7fa1dcb85640 1 --2- 172.21.15.129:0/1891242423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d81079f0 0x7fa1d8101cf0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8009900 tx=0x7fa1c802a410 comp rx=0 tx=0).stop 2024-05-20T18:09:07.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.578+0000 7fa1dcb85640 1 -- 172.21.15.129:0/1891242423 shutdown_connections 2024-05-20T18:09:07.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.578+0000 7fa1dcb85640 1 --2- 172.21.15.129:0/1891242423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1d810c350 0x7fa1d810e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.578+0000 7fa1dcb85640 1 --2- 172.21.15.129:0/1891242423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1d8102230 0x7fa1d8102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.578+0000 7fa1dcb85640 1 --2- 172.21.15.129:0/1891242423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d81079f0 0x7fa1d8101cf0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.578+0000 7fa1dcb85640 1 -- 172.21.15.129:0/1891242423 >> 172.21.15.129:0/1891242423 conn(0x7fa1d8076830 msgr2=0x7fa1d8076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.578+0000 7fa1dcb85640 1 -- 172.21.15.129:0/1891242423 shutdown_connections 2024-05-20T18:09:07.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.578+0000 7fa1dcb85640 1 -- 172.21.15.129:0/1891242423 wait complete. 2024-05-20T18:09:07.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1dcb85640 1 Processor -- start 2024-05-20T18:09:07.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1dcb85640 1 -- start start 2024-05-20T18:09:07.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1dcb85640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1d8102230 0x7fa1d819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1dcb85640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d81079f0 0x7fa1d819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1dcb85640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1d810c350 0x7fa1d819b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1dcb85640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d8111d00 con 0x7fa1d81079f0 2024-05-20T18:09:07.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1dcb85640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d8111b80 con 0x7fa1d8102230 2024-05-20T18:09:07.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1dcb85640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d8111e80 con 0x7fa1d810c350 2024-05-20T18:09:07.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1cdd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d81079f0 0x7fa1d819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1cdd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d81079f0 0x7fa1d819abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59064/0 (socket says 172.21.15.129:59064) 2024-05-20T18:09:07.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1cdd9b640 1 -- 172.21.15.129:0/2119348622 learned_addr learned my addr 172.21.15.129:0/2119348622 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:07.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1d659c640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1d8102230 0x7fa1d819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.579+0000 7fa1d6d9d640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1d810c350 0x7fa1d819b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.580+0000 7fa1cdd9b640 1 -- 172.21.15.129:0/2119348622 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1d810c350 msgr2=0x7fa1d819b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.580+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1d810c350 0x7fa1d819b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.580+0000 7fa1cdd9b640 1 -- 172.21.15.129:0/2119348622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1d8102230 msgr2=0x7fa1d819a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.580+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1d8102230 0x7fa1d819a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.580+0000 7fa1cdd9b640 1 -- 172.21.15.129:0/2119348622 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d81a59c0 con 0x7fa1d81079f0 2024-05-20T18:09:07.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.580+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d81079f0 0x7fa1d819abc0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005d00 tx=0x7fa1c000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.580+0000 7fa1cffff640 1 -- 172.21.15.129:0/2119348622 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0017070 con 0x7fa1d81079f0 2024-05-20T18:09:07.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.580+0000 7fa1cffff640 1 -- 172.21.15.129:0/2119348622 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1c0009c30 con 0x7fa1d81079f0 2024-05-20T18:09:07.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.580+0000 7fa1dcb85640 1 -- 172.21.15.129:0/2119348622 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d81a5cb0 con 0x7fa1d81079f0 2024-05-20T18:09:07.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.580+0000 7fa1cffff640 1 -- 172.21.15.129:0/2119348622 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c000f620 con 0x7fa1d81079f0 2024-05-20T18:09:07.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.580+0000 7fa1dcb85640 1 -- 172.21.15.129:0/2119348622 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d81a61b0 con 0x7fa1d81079f0 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1cffff640 1 -- 172.21.15.129:0/2119348622 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa1c001a070 con 0x7fa1d81079f0 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1e3a35740 1 Processor -- start 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1e3a35740 1 -- start start 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0957b390 0x55da09919d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0991a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0955a6b0 0x55da0991a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55da0987fd00 con 0x55da095524d0 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55da098274c0 con 0x55da0955a6b0 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55da0987bcc0 con 0x55da0957b390 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1d659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0991a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1d659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0991a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59072/0 (socket says 172.21.15.129:59072) 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1d659c640 1 -- 172.21.15.129:0/3810675594 learned_addr learned my addr 172.21.15.129:0/3810675594 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1d6d9d640 1 --2- 172.21.15.129:0/3810675594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0957b390 0x55da09919d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/3810675594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0955a6b0 0x55da0991a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1cffff640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa1c40a2d00 0x7fa1c40a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1d659c640 1 -- 172.21.15.129:0/3810675594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0957b390 msgr2=0x55da09919d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1d659c640 1 --2- 172.21.15.129:0/3810675594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0957b390 0x55da09919d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1d659c640 1 -- 172.21.15.129:0/3810675594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0955a6b0 msgr2=0x55da0991a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1d659c640 1 --2- 172.21.15.129:0/3810675594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0955a6b0 0x55da0991a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.583+0000 7fa1d659c640 1 -- 172.21.15.129:0/3810675594 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da0965d7d0 con 0x55da095524d0 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.584+0000 7fa1d6d9d640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa1c40a2d00 0x7fa1c40a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.584+0000 7fa1cffff640 1 -- 172.21.15.129:0/2119348622 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7fa1c00cefc0 con 0x7fa1d81079f0 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.584+0000 7fa1d659c640 1 --2- 172.21.15.129:0/3810675594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0991a270 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8009900 tx=0x7fa1c8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928f16e09613e045 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.584+0000 7fa1ccd99640 1 -- 172.21.15.129:0/3810675594 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8035030 con 0x55da095524d0 2024-05-20T18:09:07.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.584+0000 7fa1ccd99640 1 -- 172.21.15.129:0/3810675594 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1c80317b0 con 0x55da095524d0 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.584+0000 7fa1ccd99640 1 -- 172.21.15.129:0/3810675594 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c80046c0 con 0x55da095524d0 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3810675594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 msgr2=0x55da0991a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/3810675594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0991a270 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8009900 tx=0x7fa1c8005d50 comp rx=0 tx=0).stop 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1d6d9d640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa1c40a2d00 0x7fa1c40a5210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55da09916ea0 tx=0x7fa1d0004a40 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3810675594 shutdown_connections 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/3810675594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0955a6b0 0x55da0991a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/3810675594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0991a270 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/3810675594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0957b390 0x55da09919d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3810675594 >> 172.21.15.129:0/3810675594 conn(0x55da0956f4e0 msgr2=0x55da099164e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3810675594 shutdown_connections 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3810675594 wait complete. 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 -- start start 2024-05-20T18:09:07.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 Processor -- start 2024-05-20T18:09:07.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da09925de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0955a6b0 0x55da09926320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.585+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0957b390 0x55da099242f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55da09926fc0 con 0x55da095524d0 2024-05-20T18:09:07.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55da0983ab90 con 0x55da0957b390 2024-05-20T18:09:07.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55da09927140 con 0x55da0955a6b0 2024-05-20T18:09:07.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1e3a35740 10 client.0 _ll_register_callbacks cb 0x55da09876da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:07.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1d659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da09925de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1d659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da09925de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59074/0 (socket says 172.21.15.129:59074) 2024-05-20T18:09:07.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1d659c640 1 -- 172.21.15.129:0/3641879663 learned_addr learned my addr 172.21.15.129:0/3641879663 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:07.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1d6d9d640 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0957b390 0x55da099242f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0955a6b0 0x55da09926320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1d659c640 1 -- 172.21.15.129:0/3641879663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0955a6b0 msgr2=0x55da09926320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1d659c640 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0955a6b0 0x55da09926320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1d659c640 1 -- 172.21.15.129:0/3641879663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0957b390 msgr2=0x55da099242f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1d659c640 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0957b390 0x55da099242f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1d659c640 1 -- 172.21.15.129:0/3641879663 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da095933f0 con 0x55da095524d0 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0955a6b0 0x55da09926320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1d659c640 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da09925de0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa1c802a930 tx=0x7fa1c802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67391d06905e6981 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1bdffb640 1 -- 172.21.15.129:0/3641879663 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8035030 con 0x55da095524d0 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1bdffb640 1 -- 172.21.15.129:0/3641879663 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7fa1c80315b0 con 0x55da095524d0 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3641879663 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1c80317b0 con 0x55da095524d0 2024-05-20T18:09:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1bdffb640 1 -- 172.21.15.129:0/3641879663 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa1c80054f0 con 0x55da095524d0 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1e3a35740 10 client.17223 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.586+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3641879663 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55da095530f0 con 0x55da095524d0 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.587+0000 7fa1bdffb640 1 -- 172.21.15.129:0/3641879663 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1c80317b0 con 0x55da095524d0 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.587+0000 7fa1bdffb640 1 -- 172.21.15.129:0/3641879663 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fa1c8003850 con 0x55da095524d0 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.587+0000 7fa1e3a35740 10 client.17223 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.587+0000 7fa1e3a35740 10 client.17223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.587+0000 7fa1e3a35740 20 client.17223 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.587+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55da09539f60 0x55da099369d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.587+0000 7fa1e3a35740 4 client.17223 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.587+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3641879663 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da0988d430 con 0x55da09539f60 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.587+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55da09539f60 0x55da099369d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.587+0000 7fa1bdffb640 10 client.17223 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.587+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55da09539f60 0x55da099369d0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c928bb4dfda26d60 server_cookie=fb50291c2a394ba9 in_seq=0 out_seq=0 2024-05-20T18:09:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1bdffb640 1 -- 172.21.15.129:0/3641879663 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55da0988d430 con 0x55da09539f60 2024-05-20T18:09:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1bdffb640 10 client.17223 handle_command_reply: tid=0 2024-05-20T18:09:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 1 client.17223 shutdown 2024-05-20T18:09:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3641879663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 msgr2=0x55da09925de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da09925de0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa1c802a930 tx=0x7fa1c802a410 comp rx=0 tx=0).stop 2024-05-20T18:09:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3641879663 shutdown_connections 2024-05-20T18:09:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55da09539f60 0x55da099369d0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0957b390 0x55da099242f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0955a6b0 0x55da09926320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1bdffb640 0 client.17223 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:07.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/3641879663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da09925de0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3641879663 >> 172.21.15.129:0/3641879663 conn(0x55da0956f4e0 msgr2=0x55da0991ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3641879663 shutdown_connections 2024-05-20T18:09:07.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 1 -- 172.21.15.129:0/3641879663 wait complete. 2024-05-20T18:09:07.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.588+0000 7fa1e3a35740 20 client.17223 trim_cache size 0 max 16384 2024-05-20T18:09:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.606+0000 7fa1e3a35740 1 Processor -- start 2024-05-20T18:09:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.606+0000 7fa1e3a35740 1 -- start start 2024-05-20T18:09:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.606+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da09539f60 0x55da099232d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.606+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0993bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.606+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0955a6b0 0x55da0993c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.606+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d81acad0 con 0x55da095524d0 2024-05-20T18:09:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d81af330 con 0x55da09539f60 2024-05-20T18:09:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d81ad120 con 0x55da0955a6b0 2024-05-20T18:09:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1cdd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0993bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1cdd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0993bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59084/0 (socket says 172.21.15.129:59084) 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1cdd9b640 1 -- 172.21.15.129:0/4058396339 learned_addr learned my addr 172.21.15.129:0/4058396339 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1d659c640 1 --2- 172.21.15.129:0/4058396339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da09539f60 0x55da099232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1d6d9d640 1 --2- 172.21.15.129:0/4058396339 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0955a6b0 0x55da0993c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1cdd9b640 1 -- 172.21.15.129:0/4058396339 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0955a6b0 msgr2=0x55da0993c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/4058396339 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0955a6b0 0x55da0993c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1cdd9b640 1 -- 172.21.15.129:0/4058396339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da09539f60 msgr2=0x55da099232d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/4058396339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da09539f60 0x55da099232d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1cdd9b640 1 -- 172.21.15.129:0/4058396339 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da09922840 con 0x55da095524d0 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1d659c640 1 --2- 172.21.15.129:0/4058396339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da09539f60 0x55da099232d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/4058396339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0993bf70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00978d0 tx=0x7fa1c0097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65f450c77e2235f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1bffff640 1 -- 172.21.15.129:0/4058396339 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0021040 con 0x55da095524d0 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1bffff640 1 -- 172.21.15.129:0/4058396339 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1c009b470 con 0x55da095524d0 2024-05-20T18:09:07.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.607+0000 7fa1bffff640 1 -- 172.21.15.129:0/4058396339 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0097ae0 con 0x55da095524d0 2024-05-20T18:09:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.608+0000 7fa1e3a35740 1 -- 172.21.15.129:0/4058396339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 msgr2=0x55da0993bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.608+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/4058396339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0993bf70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00978d0 tx=0x7fa1c0097040 comp rx=0 tx=0).stop 2024-05-20T18:09:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.608+0000 7fa1e3a35740 1 -- 172.21.15.129:0/4058396339 shutdown_connections 2024-05-20T18:09:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.608+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/4058396339 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0955a6b0 0x55da0993c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.608+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/4058396339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da095524d0 0x55da0993bf70 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.608+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/4058396339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da09539f60 0x55da099232d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.608+0000 7fa1e3a35740 1 -- 172.21.15.129:0/4058396339 >> 172.21.15.129:0/4058396339 conn(0x55da0956f4e0 msgr2=0x55da09922bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.608+0000 7fa1e3a35740 1 -- 172.21.15.129:0/4058396339 shutdown_connections 2024-05-20T18:09:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.608+0000 7fa1e3a35740 1 -- 172.21.15.129:0/4058396339 wait complete. 2024-05-20T18:09:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.608+0000 7fa1e3a35740 1 -- start start 2024-05-20T18:09:07.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.608+0000 7fa1e3a35740 1 Processor -- start 2024-05-20T18:09:07.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da09539f60 0x55da0992c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da095524d0 0x55da0992cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1e3a35740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0955a6b0 0x55da09927b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d81ad7f0 con 0x55da09539f60 2024-05-20T18:09:07.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d81aec60 con 0x55da0955a6b0 2024-05-20T18:09:07.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1e3a35740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d81adec0 con 0x55da095524d0 2024-05-20T18:09:07.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1e3a35740 10 client.0 _ll_register_callbacks cb 0x7fa1d00098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1d659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da09539f60 0x55da0992c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1d659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da09539f60 0x55da0992c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59098/0 (socket says 172.21.15.129:59098) 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1d659c640 1 -- 172.21.15.129:0/2693781350 learned_addr learned my addr 172.21.15.129:0/2693781350 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1d6d9d640 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0955a6b0 0x55da09927b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da095524d0 0x55da0992cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1d659c640 1 -- 172.21.15.129:0/2693781350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da095524d0 msgr2=0x55da0992cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1d659c640 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da095524d0 0x55da0992cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1d659c640 1 -- 172.21.15.129:0/2693781350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0955a6b0 msgr2=0x55da09927b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1d659c640 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0955a6b0 0x55da09927b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1d659c640 1 -- 172.21.15.129:0/2693781350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da09928280 con 0x55da09539f60 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1d659c640 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da09539f60 0x55da0992c5d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8001eb0 tx=0x7fa1c8005410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf0bc60c3ff8922 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1bdffb640 1 -- 172.21.15.129:0/2693781350 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8035040 con 0x55da09539f60 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1bdffb640 1 -- 172.21.15.129:0/2693781350 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7fa1c8004ee0 con 0x55da09539f60 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1e3a35740 1 -- 172.21.15.129:0/2693781350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1c009b470 con 0x55da09539f60 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.609+0000 7fa1bdffb640 1 -- 172.21.15.129:0/2693781350 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa1c80035f0 con 0x55da09539f60 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1e3a35740 10 client.17229 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1e3a35740 1 -- 172.21.15.129:0/2693781350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55da0999e670 con 0x55da09539f60 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1bdffb640 1 -- 172.21.15.129:0/2693781350 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1c009b470 con 0x55da09539f60 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1bdffb640 1 -- 172.21.15.129:0/2693781350 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fa1c805c070 con 0x55da09539f60 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1e3a35740 10 client.17229 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1e3a35740 10 client.17229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1e3a35740 20 client.17229 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55da0957b390 0x55da099a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1e3a35740 4 client.17229 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:09:07.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1e3a35740 1 -- 172.21.15.129:0/2693781350 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55da09887780 con 0x55da0957b390 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55da0957b390 0x55da099a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1bdffb640 10 client.17229 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.610+0000 7fa1cdd9b640 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55da0957b390 0x55da099a0e20 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1c1473e49565717 server_cookie=87c874a153a2705c in_seq=0 out_seq=0 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1bdffb640 1 -- 172.21.15.129:0/2693781350 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55da09887780 con 0x55da0957b390 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1bdffb640 10 client.17229 handle_command_reply: tid=0 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 1 client.17229 shutdown 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 1 -- 172.21.15.129:0/2693781350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da09539f60 msgr2=0x55da0992c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da09539f60 0x55da0992c5d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8001eb0 tx=0x7fa1c8005410 comp rx=0 tx=0).stop 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 1 -- 172.21.15.129:0/2693781350 shutdown_connections 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55da0957b390 0x55da099a0e20 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da0955a6b0 0x55da09927b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da095524d0 0x55da0992cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1bdffb640 0 client.17229 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:07.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 1 --2- 172.21.15.129:0/2693781350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da09539f60 0x55da0992c5d0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 1 -- 172.21.15.129:0/2693781350 >> 172.21.15.129:0/2693781350 conn(0x55da0956f4e0 msgr2=0x55da0991b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 1 -- 172.21.15.129:0/2693781350 shutdown_connections 2024-05-20T18:09:07.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 1 -- 172.21.15.129:0/2693781350 wait complete. 2024-05-20T18:09:07.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1e3a35740 20 client.17229 trim_cache size 0 max 16384 2024-05-20T18:09:07.616 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:09:07.616 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:09:07.616 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:09:07.616 INFO:teuthology.orchestra.run.smithi129.stdout: "8d1abc27-947a-4833-a299-a316a7a12d16": { 2024-05-20T18:09:07.617 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:09:07.617 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "8d1abc27-947a-4833-a299-a316a7a12d16", 2024-05-20T18:09:07.617 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:09:07.617 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:09:07.617 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:09:07.617 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:09:07.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 -- 172.21.15.129:0/2119348622 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa1c40a2d00 msgr2=0x7fa1c40a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa1c40a2d00 0x7fa1c40a5210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55da09916ea0 tx=0x7fa1d0004a40 comp rx=0 tx=0).stop 2024-05-20T18:09:07.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 -- 172.21.15.129:0/2119348622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d81079f0 msgr2=0x7fa1d819abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:07.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d81079f0 0x7fa1d819abc0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005d00 tx=0x7fa1c000dde0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 -- 172.21.15.129:0/2119348622 shutdown_connections 2024-05-20T18:09:07.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa1c40a2d00 0x7fa1c40a5210 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa1d810c350 0x7fa1d819b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d81079f0 0x7fa1d819abc0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 --2- 172.21.15.129:0/2119348622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1d8102230 0x7fa1d819a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:07.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 -- 172.21.15.129:0/2119348622 >> 172.21.15.129:0/2119348622 conn(0x7fa1d8076830 msgr2=0x7fa1d8103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:07.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 -- 172.21.15.129:0/2119348622 shutdown_connections 2024-05-20T18:09:07.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:07.611+0000 7fa1dcb85640 1 -- 172.21.15.129:0/2119348622 wait complete. 2024-05-20T18:09:07.622 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:07 smithi129 ceph-mon[32029]: pgmap v328: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2024-05-20T18:09:07.623 INFO:tasks.cephfs.filesystem:scrub status for tag:8d1abc27-947a-4833-a299-a316a7a12d16 - {'path': '/', 'tag': '8d1abc27-947a-4833-a299-a316a7a12d16', 'options': 'recursive,force'} 2024-05-20T18:09:07.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:07 smithi154 ceph-mon[36482]: pgmap v328: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2024-05-20T18:09:07.678 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:07 smithi204 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-05-20T18:09:07.678 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:07 smithi204 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.i.service: Consumed 2.235s CPU time. 2024-05-20T18:09:07.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:07 smithi204 ceph-mon[37076]: pgmap v328: 97 pgs: 97 active+clean; 42 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2024-05-20T18:09:08.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:08 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:09:08.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:08 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:08.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:08 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:09:08.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:08 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:08.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:08 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:09:08.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:08 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:08.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:08 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:09:08.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:08 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:08.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:08 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:09:08.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:08 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:08.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:08 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:09:08.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:08 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:09.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:09 smithi129 ceph-mon[32029]: pgmap v329: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2024-05-20T18:09:09.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:09 smithi204 ceph-mon[37076]: pgmap v329: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2024-05-20T18:09:09.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:09 smithi154 ceph-mon[36482]: pgmap v329: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 40 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2024-05-20T18:09:11.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:11 smithi129 ceph-mon[32029]: pgmap v330: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 32 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 25 op/s 2024-05-20T18:09:11.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:11 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:09:11.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:11 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:09:11.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:11 smithi204 ceph-mon[37076]: pgmap v330: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 32 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 25 op/s 2024-05-20T18:09:11.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:11 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:09:11.679 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:11 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:09:11.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:11 smithi154 ceph-mon[36482]: pgmap v330: 97 pgs: 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 90 active+clean; 32 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 25 op/s 2024-05-20T18:09:11.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:11 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:09:11.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:11 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:09:13.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:13 smithi154 ceph-mon[36482]: pgmap v331: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 26 op/s 2024-05-20T18:09:13.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:13 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:13 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:13 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:13 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:13 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:13 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:13 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:13 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:13 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:13 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:13 smithi129 ceph-mon[32029]: pgmap v331: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 26 op/s 2024-05-20T18:09:13.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:13 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:13 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:13 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:13 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:13 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:13 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:13 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:13 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:13 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:13 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:13 smithi204 ceph-mon[37076]: pgmap v331: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 26 op/s 2024-05-20T18:09:13.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:13 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:13 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:13 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:13 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:13 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:13 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:13.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:13 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:13 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:13 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:13.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:13 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-05-20T18:09:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: Unable to set osd_memory_target on smithi154 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-20T18:09:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-05-20T18:09:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: Unable to set osd_memory_target on smithi129 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-05-20T18:09:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-05-20T18:09:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-05-20T18:09:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:09:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:09:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: pgmap v332: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 30 KiB/s wr, 9 op/s 2024-05-20T18:09:14.893 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:14.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-05-20T18:09:14.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: Unable to set osd_memory_target on smithi154 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-20T18:09:14.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-05-20T18:09:14.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: Unable to set osd_memory_target on smithi129 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-05-20T18:09:14.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.912 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-05-20T18:09:14.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-05-20T18:09:14.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:09:14.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:09:14.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: pgmap v332: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 30 KiB/s wr, 9 op/s 2024-05-20T18:09:14.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:14.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi154 to 176.2M 2024-05-20T18:09:14.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: Unable to set osd_memory_target on smithi154 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: Unable to set osd_memory_target on smithi129 to 184765952: error parsing value: Value '184765952' is below minimum 939524096 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-05-20T18:09:14.929 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:09:14.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:09:14.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: pgmap v332: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 30 KiB/s wr, 9 op/s 2024-05-20T18:09:14.930 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:09:15.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:15 smithi154 ceph-mon[36482]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-20T18:09:15.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:15 smithi154 ceph-mon[36482]: osdmap e104: 12 total, 12 up, 12 in 2024-05-20T18:09:15.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:15 smithi129 ceph-mon[32029]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-20T18:09:15.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:15 smithi129 ceph-mon[32029]: osdmap e104: 12 total, 12 up, 12 in 2024-05-20T18:09:15.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:15 smithi204 ceph-mon[37076]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-20T18:09:15.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:15 smithi204 ceph-mon[37076]: osdmap e104: 12 total, 12 up, 12 in 2024-05-20T18:09:16.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:16 smithi154 ceph-mon[36482]: pgmap v334: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 525 B/s wr, 4 op/s 2024-05-20T18:09:16.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:16 smithi129 ceph-mon[32029]: pgmap v334: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 525 B/s wr, 4 op/s 2024-05-20T18:09:16.928 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:16 smithi204 ceph-mon[37076]: pgmap v334: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 525 B/s wr, 4 op/s 2024-05-20T18:09:17.729 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:17 smithi204 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-05-20T18:09:17.729 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:17 smithi204 systemd[1]: Stopped Ceph mds.i for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T18:09:17.729 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:17 smithi204 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.i.service: Consumed 2.235s CPU time. 2024-05-20T18:09:17.729 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:17 smithi204 systemd[1]: Starting Ceph mds.i for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:09:17.729 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:17 smithi204 podman[72918]: 2024-05-20 18:09:17.5745756 +0000 UTC m=+0.017317436 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:09:17.729 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:17 smithi204 podman[72918]: 2024-05-20 18:09:17.732383624 +0000 UTC m=+0.175125447 container create 7bc558f440fca562e41b7968183dca304bb12c3a097604a3d7a68b3d6ee82f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-20T18:09:18.258 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:17 smithi204 podman[72918]: 2024-05-20 18:09:17.993839094 +0000 UTC m=+0.436580917 container init 7bc558f440fca562e41b7968183dca304bb12c3a097604a3d7a68b3d6ee82f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-05-20T18:09:18.258 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:17 smithi204 podman[72918]: 2024-05-20 18:09:17.996892066 +0000 UTC m=+0.439633898 container start 7bc558f440fca562e41b7968183dca304bb12c3a097604a3d7a68b3d6ee82f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-20T18:09:18.258 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:18 smithi204 ceph-mds[72937]: -- 172.21.15.204:0/1361180959 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55980fb21a40 con 0x55980fb99600 2024-05-20T18:09:18.258 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:18 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[72933]: starting mds.i at 2024-05-20T18:09:18.258 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:18 smithi204 bash[72918]: 7bc558f440fca562e41b7968183dca304bb12c3a097604a3d7a68b3d6ee82f19 2024-05-20T18:09:18.258 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:09:18 smithi204 systemd[1]: Started Ceph mds.i for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T18:09:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.364+0000 7fee3d621640 1 -- 172.21.15.129:0/2525197797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee3810a730 msgr2=0x7fee3810cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.364+0000 7fee3d621640 1 --2- 172.21.15.129:0/2525197797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee3810a730 0x7fee3810cb50 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009900 tx=0x7fee2c02a480 comp rx=0 tx=0).stop 2024-05-20T18:09:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.364+0000 7fee3d621640 1 -- 172.21.15.129:0/2525197797 shutdown_connections 2024-05-20T18:09:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.364+0000 7fee3d621640 1 --2- 172.21.15.129:0/2525197797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee3810d090 0x7fee3810f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.364+0000 7fee3d621640 1 --2- 172.21.15.129:0/2525197797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee3810a730 0x7fee3810cb50 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.364+0000 7fee3d621640 1 --2- 172.21.15.129:0/2525197797 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee38103cd0 0x7fee381040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.364+0000 7fee3d621640 1 -- 172.21.15.129:0/2525197797 >> 172.21.15.129:0/2525197797 conn(0x7fee380fd210 msgr2=0x7fee380ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.364+0000 7fee3d621640 1 -- 172.21.15.129:0/2525197797 shutdown_connections 2024-05-20T18:09:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.364+0000 7fee3d621640 1 -- 172.21.15.129:0/2525197797 wait complete. 2024-05-20T18:09:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee3d621640 1 Processor -- start 2024-05-20T18:09:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee3d621640 1 -- start start 2024-05-20T18:09:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee3d621640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee38103cd0 0x7fee38101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee3d621640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee3810a730 0x7fee38101ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee3d621640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee3810d090 0x7fee38102010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee3d621640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fee38112e20 con 0x7fee3810a730 2024-05-20T18:09:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee3d621640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fee38112ca0 con 0x7fee38103cd0 2024-05-20T18:09:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee3d621640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fee38112fa0 con 0x7fee3810d090 2024-05-20T18:09:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee367fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee3810a730 0x7fee38101ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee367fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee3810a730 0x7fee38101ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45696/0 (socket says 172.21.15.129:45696) 2024-05-20T18:09:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee367fc640 1 -- 172.21.15.129:0/971349564 learned_addr learned my addr 172.21.15.129:0/971349564 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee36ffd640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee38103cd0 0x7fee38101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee377fe640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee3810d090 0x7fee38102010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee367fc640 1 -- 172.21.15.129:0/971349564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee3810d090 msgr2=0x7fee38102010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee367fc640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee3810d090 0x7fee38102010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee367fc640 1 -- 172.21.15.129:0/971349564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee38103cd0 msgr2=0x7fee38101590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee367fc640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee38103cd0 0x7fee38101590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee367fc640 1 -- 172.21.15.129:0/971349564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee381a7b80 con 0x7fee3810a730 2024-05-20T18:09:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.365+0000 7fee36ffd640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee38103cd0 0x7fee38101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:09:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.366+0000 7fee367fc640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee3810a730 0x7fee38101ad0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fee2c006010 tx=0x7fee2c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.366+0000 7fee1ffff640 1 -- 172.21.15.129:0/971349564 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c03f030 con 0x7fee3810a730 2024-05-20T18:09:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.366+0000 7fee3d621640 1 -- 172.21.15.129:0/971349564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee381a7e10 con 0x7fee3810a730 2024-05-20T18:09:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.366+0000 7fee1ffff640 1 -- 172.21.15.129:0/971349564 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee2c003ce0 con 0x7fee3810a730 2024-05-20T18:09:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.366+0000 7fee1ffff640 1 -- 172.21.15.129:0/971349564 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c035070 con 0x7fee3810a730 2024-05-20T18:09:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.366+0000 7fee3d621640 1 -- 172.21.15.129:0/971349564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee381a8220 con 0x7fee3810a730 2024-05-20T18:09:18.375 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.368+0000 7fee1ffff640 1 -- 172.21.15.129:0/971349564 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fee2c042030 con 0x7fee3810a730 2024-05-20T18:09:18.375 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.368+0000 7fee4459d740 1 Processor -- start 2024-05-20T18:09:18.375 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.368+0000 7fee4459d740 1 -- start start 2024-05-20T18:09:18.375 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee4459d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f22071b0 0x5585f25a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.375 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee4459d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21de2f0 0x5585f25a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee4459d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f21e64d0 0x5585f25a6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee4459d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5585f250b340 con 0x5585f21de2f0 2024-05-20T18:09:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee4459d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5585f24b3290 con 0x5585f22071b0 2024-05-20T18:09:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee4459d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5585f2507300 con 0x5585f21e64d0 2024-05-20T18:09:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee36ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21de2f0 0x5585f25a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee1ffff640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fee200a2cf0 0x7fee200a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee36ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21de2f0 0x5585f25a65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45706/0 (socket says 172.21.15.129:45706) 2024-05-20T18:09:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee36ffd640 1 -- 172.21.15.129:0/2861494740 learned_addr learned my addr 172.21.15.129:0/2861494740 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee367fc640 1 --2- 172.21.15.129:0/2861494740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f21e64d0 0x5585f25a6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee377fe640 1 --2- 172.21.15.129:0/2861494740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f22071b0 0x5585f25a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee377fe640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fee200a2cf0 0x7fee200a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee36ffd640 1 -- 172.21.15.129:0/2861494740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f21e64d0 msgr2=0x5585f25a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee36ffd640 1 --2- 172.21.15.129:0/2861494740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f21e64d0 0x5585f25a6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee36ffd640 1 -- 172.21.15.129:0/2861494740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f22071b0 msgr2=0x5585f25a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee36ffd640 1 --2- 172.21.15.129:0/2861494740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f22071b0 0x5585f25a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee36ffd640 1 -- 172.21.15.129:0/2861494740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f24d78e0 con 0x5585f21de2f0 2024-05-20T18:09:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee1ffff640 1 -- 172.21.15.129:0/971349564 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7fee2c0efa50 con 0x7fee3810a730 2024-05-20T18:09:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.369+0000 7fee36ffd640 1 --2- 172.21.15.129:0/2861494740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21de2f0 0x5585f25a65c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fee280103a0 tx=0x7fee28010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6eddd1bf90ac99 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.370+0000 7fee377fe640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fee200a2cf0 0x7fee200a5200 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x5585f25a31f0 tx=0x7fee30005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.370+0000 7fee1d7fa640 1 -- 172.21.15.129:0/2861494740 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee28016070 con 0x5585f21de2f0 2024-05-20T18:09:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.370+0000 7fee1d7fa640 1 -- 172.21.15.129:0/2861494740 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee28012bf0 con 0x5585f21de2f0 2024-05-20T18:09:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.370+0000 7fee1d7fa640 1 -- 172.21.15.129:0/2861494740 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2801bd10 con 0x5585f21de2f0 2024-05-20T18:09:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.370+0000 7fee4459d740 1 -- 172.21.15.129:0/2861494740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21de2f0 msgr2=0x5585f25a65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.370+0000 7fee4459d740 1 --2- 172.21.15.129:0/2861494740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21de2f0 0x5585f25a65c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fee280103a0 tx=0x7fee28010860 comp rx=0 tx=0).stop 2024-05-20T18:09:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 -- 172.21.15.129:0/2861494740 shutdown_connections 2024-05-20T18:09:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 --2- 172.21.15.129:0/2861494740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f21e64d0 0x5585f25a6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 --2- 172.21.15.129:0/2861494740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21de2f0 0x5585f25a65c0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 --2- 172.21.15.129:0/2861494740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f22071b0 0x5585f25a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 -- 172.21.15.129:0/2861494740 >> 172.21.15.129:0/2861494740 conn(0x5585f21fb300 msgr2=0x5585f25a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 -- 172.21.15.129:0/2861494740 shutdown_connections 2024-05-20T18:09:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 -- 172.21.15.129:0/2861494740 wait complete. 2024-05-20T18:09:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 -- start start 2024-05-20T18:09:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 Processor -- start 2024-05-20T18:09:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f21de2f0 0x5585f25b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f22071b0 0x5585f25b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5585f25b31b0 con 0x5585f21e64d0 2024-05-20T18:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5585f24c6790 con 0x5585f21de2f0 2024-05-20T18:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5585f25b3330 con 0x5585f22071b0 2024-05-20T18:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee4459d740 10 client.0 _ll_register_callbacks cb 0x5585f2501a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee367fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee367fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45712/0 (socket says 172.21.15.129:45712) 2024-05-20T18:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee36ffd640 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f21de2f0 0x5585f25b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee367fc640 1 -- 172.21.15.129:0/3051659331 learned_addr learned my addr 172.21.15.129:0/3051659331 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.371+0000 7fee377fe640 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f22071b0 0x5585f25b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee367fc640 1 -- 172.21.15.129:0/3051659331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f22071b0 msgr2=0x5585f25b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee367fc640 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f22071b0 0x5585f25b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee367fc640 1 -- 172.21.15.129:0/3051659331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f21de2f0 msgr2=0x5585f25b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee367fc640 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f21de2f0 0x5585f25b20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee367fc640 1 -- 172.21.15.129:0/3051659331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585f221f210 con 0x5585f21e64d0 2024-05-20T18:09:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee36ffd640 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f21de2f0 0x5585f25b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:09:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee367fc640 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b2610 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fee2c0ed1c0 tx=0x7fee2c0b7c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421d51901d3aa1e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee1aff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee1aff5640 1 -- 172.21.15.129:0/3051659331 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c049030 con 0x5585f21e64d0 2024-05-20T18:09:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee1aff5640 1 -- 172.21.15.129:0/3051659331 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7fee2c0b8660 con 0x5585f21e64d0 2024-05-20T18:09:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee4459d740 1 -- 172.21.15.129:0/3051659331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee28012bf0 con 0x5585f21e64d0 2024-05-20T18:09:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee1aff5640 1 -- 172.21.15.129:0/3051659331 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fee2c0bbb30 con 0x5585f21e64d0 2024-05-20T18:09:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee4459d740 10 client.17244 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee4459d740 1 -- 172.21.15.129:0/3051659331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5585f22eb620 con 0x5585f21e64d0 2024-05-20T18:09:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.372+0000 7fee1aff5640 1 -- 172.21.15.129:0/3051659331 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee28012bf0 con 0x5585f21e64d0 2024-05-20T18:09:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.373+0000 7fee1aff5640 1 -- 172.21.15.129:0/3051659331 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fee2c03bbb0 con 0x5585f21e64d0 2024-05-20T18:09:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.373+0000 7fee4459d740 10 client.17244 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.373+0000 7fee4459d740 10 client.17244 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.373+0000 7fee4459d740 20 client.17244 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.373+0000 7fee4459d740 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5585f21c5d80 0x5585f25c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.373+0000 7fee4459d740 4 client.17244 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.373+0000 7fee4459d740 1 -- 172.21.15.129:0/3051659331 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585f2518a70 con 0x5585f21c5d80 2024-05-20T18:09:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.373+0000 7fee36ffd640 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5585f21c5d80 0x5585f25c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee1aff5640 10 client.17244 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee36ffd640 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5585f21c5d80 0x5585f25c29c0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acd25ab9193da2f0 server_cookie=7f7a14db51f824f4 in_seq=0 out_seq=0 2024-05-20T18:09:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee1aff5640 1 -- 172.21.15.129:0/3051659331 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5585f2518a70 con 0x5585f21c5d80 2024-05-20T18:09:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee1aff5640 10 client.17244 handle_command_reply: tid=0 2024-05-20T18:09:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee4459d740 1 client.17244 shutdown 2024-05-20T18:09:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee4459d740 1 -- 172.21.15.129:0/3051659331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 msgr2=0x5585f25b2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee4459d740 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b2610 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fee2c0ed1c0 tx=0x7fee2c0b7c60 comp rx=0 tx=0).stop 2024-05-20T18:09:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee4459d740 1 -- 172.21.15.129:0/3051659331 shutdown_connections 2024-05-20T18:09:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee4459d740 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5585f21c5d80 0x5585f25c29c0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee4459d740 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f22071b0 0x5585f25b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee1aff5640 0 client.17244 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee4459d740 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b2610 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee4459d740 1 --2- 172.21.15.129:0/3051659331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f21de2f0 0x5585f25b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.374+0000 7fee4459d740 1 -- 172.21.15.129:0/3051659331 >> 172.21.15.129:0/3051659331 conn(0x5585f21fb300 msgr2=0x5585f25a5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.375+0000 7fee4459d740 1 -- 172.21.15.129:0/3051659331 shutdown_connections 2024-05-20T18:09:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.375+0000 7fee4459d740 1 -- 172.21.15.129:0/3051659331 wait complete. 2024-05-20T18:09:18.385 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.375+0000 7fee4459d740 20 client.17244 trim_cache size 0 max 16384 2024-05-20T18:09:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.389+0000 7fee4459d740 1 Processor -- start 2024-05-20T18:09:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee4459d740 1 -- start start 2024-05-20T18:09:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee4459d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f21de2f0 0x5585f25aa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee4459d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee4459d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f22071b0 0x5585f25b8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee4459d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fee381aee00 con 0x5585f21e64d0 2024-05-20T18:09:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee4459d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fee381b1660 con 0x5585f22071b0 2024-05-20T18:09:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee4459d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fee381af450 con 0x5585f21de2f0 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee367fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee367fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45720/0 (socket says 172.21.15.129:45720) 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee367fc640 1 -- 172.21.15.129:0/3654237170 learned_addr learned my addr 172.21.15.129:0/3654237170 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee36ffd640 1 --2- 172.21.15.129:0/3654237170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f21de2f0 0x5585f25aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee367fc640 1 -- 172.21.15.129:0/3654237170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f21de2f0 msgr2=0x5585f25aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee367fc640 1 --2- 172.21.15.129:0/3654237170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f21de2f0 0x5585f25aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee367fc640 1 -- 172.21.15.129:0/3654237170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f22071b0 msgr2=0x5585f25b8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee377fe640 1 --2- 172.21.15.129:0/3654237170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f22071b0 0x5585f25b8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee367fc640 1 --2- 172.21.15.129:0/3654237170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f22071b0 0x5585f25b8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee367fc640 1 -- 172.21.15.129:0/3654237170 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585f25a3420 con 0x5585f21e64d0 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee367fc640 1 --2- 172.21.15.129:0/3654237170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b85f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fee2c000c00 tx=0x7fee2c0ba500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9366d22ab947242c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.390+0000 7fee1cff9640 1 -- 172.21.15.129:0/3654237170 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c049070 con 0x5585f21e64d0 2024-05-20T18:09:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.391+0000 7fee1cff9640 1 -- 172.21.15.129:0/3654237170 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee2c0bc6f0 con 0x5585f21e64d0 2024-05-20T18:09:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.391+0000 7fee1cff9640 1 -- 172.21.15.129:0/3654237170 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c0044e0 con 0x5585f21e64d0 2024-05-20T18:09:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.391+0000 7fee4459d740 1 -- 172.21.15.129:0/3654237170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 msgr2=0x5585f25b85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.391+0000 7fee4459d740 1 --2- 172.21.15.129:0/3654237170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b85f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fee2c000c00 tx=0x7fee2c0ba500 comp rx=0 tx=0).stop 2024-05-20T18:09:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.391+0000 7fee4459d740 1 -- 172.21.15.129:0/3654237170 shutdown_connections 2024-05-20T18:09:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.391+0000 7fee4459d740 1 --2- 172.21.15.129:0/3654237170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f22071b0 0x5585f25b8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.391+0000 7fee4459d740 1 --2- 172.21.15.129:0/3654237170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b85f0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.391+0000 7fee4459d740 1 --2- 172.21.15.129:0/3654237170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f21de2f0 0x5585f25aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.391+0000 7fee4459d740 1 -- 172.21.15.129:0/3654237170 >> 172.21.15.129:0/3654237170 conn(0x5585f21fb300 msgr2=0x5585f25a9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.391+0000 7fee4459d740 1 -- 172.21.15.129:0/3654237170 shutdown_connections 2024-05-20T18:09:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.391+0000 7fee4459d740 1 -- 172.21.15.129:0/3654237170 wait complete. 2024-05-20T18:09:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee4459d740 1 -- start start 2024-05-20T18:09:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee4459d740 1 Processor -- start 2024-05-20T18:09:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee4459d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f21de2f0 0x5585f25b88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee4459d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee4459d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f22071b0 0x5585f25b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee4459d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fee381afb20 con 0x5585f21e64d0 2024-05-20T18:09:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee4459d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fee381b0f90 con 0x5585f21de2f0 2024-05-20T18:09:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee4459d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fee381b01f0 con 0x5585f22071b0 2024-05-20T18:09:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee4459d740 10 client.0 _ll_register_callbacks cb 0x7fee2c005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee367fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee367fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45730/0 (socket says 172.21.15.129:45730) 2024-05-20T18:09:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee367fc640 1 -- 172.21.15.129:0/1463047275 learned_addr learned my addr 172.21.15.129:0/1463047275 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee377fe640 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f22071b0 0x5585f25b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee367fc640 1 -- 172.21.15.129:0/1463047275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f22071b0 msgr2=0x5585f25b61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee367fc640 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f22071b0 0x5585f25b61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee367fc640 1 -- 172.21.15.129:0/1463047275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f21de2f0 msgr2=0x5585f25b88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee36ffd640 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f21de2f0 0x5585f25b88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee367fc640 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f21de2f0 0x5585f25b88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.392+0000 7fee367fc640 1 -- 172.21.15.129:0/1463047275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585f25b67e0 con 0x5585f21e64d0 2024-05-20T18:09:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee367fc640 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b5c60 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fee2c0c4730 tx=0x7fee2c0bc0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a411834be3424bce server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee1aff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee1aff5640 1 -- 172.21.15.129:0/1463047275 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c049070 con 0x5585f21e64d0 2024-05-20T18:09:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee1aff5640 1 -- 172.21.15.129:0/1463047275 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7fee2c03bbb0 con 0x5585f21e64d0 2024-05-20T18:09:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee4459d740 1 -- 172.21.15.129:0/1463047275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee2c0bc6f0 con 0x5585f21e64d0 2024-05-20T18:09:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee1aff5640 1 -- 172.21.15.129:0/1463047275 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fee2c0bb950 con 0x5585f21e64d0 2024-05-20T18:09:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee4459d740 10 client.17250 fetch_fsmap learned FSMap version 26 2024-05-20T18:09:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee4459d740 1 -- 172.21.15.129:0/1463047275 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5585f262ab20 con 0x5585f21e64d0 2024-05-20T18:09:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee1aff5640 1 -- 172.21.15.129:0/1463047275 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee2c0bc6f0 con 0x5585f21e64d0 2024-05-20T18:09:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee1aff5640 1 -- 172.21.15.129:0/1463047275 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 8322+0+0 (secure 0 0 0) 0x7fee2c0dab70 con 0x5585f21e64d0 2024-05-20T18:09:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee4459d740 10 client.17250 fetch_fsmap finished waiting for FSMap version 26 2024-05-20T18:09:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee4459d740 10 client.17250 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee4459d740 20 client.17250 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee4459d740 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5585f219b1c0 0x5585f262cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee4459d740 4 client.17250 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:09:18.400 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.393+0000 7fee4459d740 1 -- 172.21.15.129:0/1463047275 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5585f2512dc0 con 0x5585f219b1c0 2024-05-20T18:09:18.400 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.394+0000 7fee36ffd640 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5585f219b1c0 0x5585f262cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:18.400 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.394+0000 7fee1aff5640 10 client.17250 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:18.400 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.394+0000 7fee36ffd640 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5585f219b1c0 0x5585f262cf10 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57e11abf97c31a98 server_cookie=5f85548056c99739 in_seq=0 out_seq=0 2024-05-20T18:09:18.401 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee1aff5640 1 -- 172.21.15.129:0/1463047275 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5585f2512dc0 con 0x5585f219b1c0 2024-05-20T18:09:18.401 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee1aff5640 10 client.17250 handle_command_reply: tid=0 2024-05-20T18:09:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee4459d740 1 client.17250 shutdown 2024-05-20T18:09:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee4459d740 1 -- 172.21.15.129:0/1463047275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 msgr2=0x5585f25b5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee4459d740 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b5c60 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fee2c0c4730 tx=0x7fee2c0bc0c0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee4459d740 1 -- 172.21.15.129:0/1463047275 shutdown_connections 2024-05-20T18:09:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee4459d740 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5585f219b1c0 0x5585f262cf10 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee4459d740 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5585f22071b0 0x5585f25b61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee1aff5640 0 client.17250 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee4459d740 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585f21e64d0 0x5585f25b5c60 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee4459d740 1 --2- 172.21.15.129:0/1463047275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5585f21de2f0 0x5585f25b88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee4459d740 1 -- 172.21.15.129:0/1463047275 >> 172.21.15.129:0/1463047275 conn(0x5585f21fb300 msgr2=0x5585f25a5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee4459d740 1 -- 172.21.15.129:0/1463047275 shutdown_connections 2024-05-20T18:09:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.401+0000 7fee4459d740 1 -- 172.21.15.129:0/1463047275 wait complete. 2024-05-20T18:09:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee4459d740 20 client.17250 trim_cache size 0 max 16384 2024-05-20T18:09:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:09:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:09:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:09:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:09:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 -- 172.21.15.129:0/971349564 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fee200a2cf0 msgr2=0x7fee200a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fee200a2cf0 0x7fee200a5200 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x5585f25a31f0 tx=0x7fee30005f90 comp rx=0 tx=0).stop 2024-05-20T18:09:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 -- 172.21.15.129:0/971349564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee3810a730 msgr2=0x7fee38101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee3810a730 0x7fee38101ad0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fee2c006010 tx=0x7fee2c0042f0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 -- 172.21.15.129:0/971349564 shutdown_connections 2024-05-20T18:09:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fee200a2cf0 0x7fee200a5200 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee3810d090 0x7fee38102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee3810a730 0x7fee38101ad0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 --2- 172.21.15.129:0/971349564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee38103cd0 0x7fee38101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 -- 172.21.15.129:0/971349564 >> 172.21.15.129:0/971349564 conn(0x7fee380fd210 msgr2=0x7fee3810d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 -- 172.21.15.129:0/971349564 shutdown_connections 2024-05-20T18:09:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:18.402+0000 7fee3d621640 1 -- 172.21.15.129:0/971349564 wait complete. 2024-05-20T18:09:19.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:18 smithi129 ceph-mon[32029]: pgmap v335: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 525 B/s wr, 4 op/s 2024-05-20T18:09:19.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:18 smithi129 ceph-mon[32029]: daemon mds.i restarted 2024-05-20T18:09:19.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:18 smithi204 ceph-mon[37076]: pgmap v335: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 525 B/s wr, 4 op/s 2024-05-20T18:09:19.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:18 smithi204 ceph-mon[37076]: daemon mds.i restarted 2024-05-20T18:09:19.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:18 smithi154 ceph-mon[36482]: pgmap v335: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 525 B/s wr, 4 op/s 2024-05-20T18:09:19.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:18 smithi154 ceph-mon[36482]: daemon mds.i restarted 2024-05-20T18:09:20.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:19 smithi129 ceph-mon[32029]: osdmap e105: 12 total, 12 up, 12 in 2024-05-20T18:09:20.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:19 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.204:6838/1721642062,v1:172.21.15.204:6839/1721642062] up:boot 2024-05-20T18:09:20.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:19 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T18:09:20.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:19 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T18:09:20.162 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:19 smithi129 ceph-mon[32029]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T18:09:20.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:19 smithi204 ceph-mon[37076]: osdmap e105: 12 total, 12 up, 12 in 2024-05-20T18:09:20.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:19 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.204:6838/1721642062,v1:172.21.15.204:6839/1721642062] up:boot 2024-05-20T18:09:20.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:19 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T18:09:20.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:19 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T18:09:20.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:19 smithi204 ceph-mon[37076]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T18:09:20.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:19 smithi154 ceph-mon[36482]: osdmap e105: 12 total, 12 up, 12 in 2024-05-20T18:09:20.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:19 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.204:6838/1721642062,v1:172.21.15.204:6839/1721642062] up:boot 2024-05-20T18:09:20.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:19 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-20T18:09:20.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:19 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T18:09:20.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:19 smithi154 ceph-mon[36482]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-05-20T18:09:21.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:20 smithi204 ceph-mon[37076]: pgmap v337: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 661 B/s wr, 1 op/s 2024-05-20T18:09:21.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:20 smithi154 ceph-mon[36482]: pgmap v337: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 661 B/s wr, 1 op/s 2024-05-20T18:09:21.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:20 smithi129 ceph-mon[32029]: pgmap v337: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 661 B/s wr, 1 op/s 2024-05-20T18:09:23.180 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:22 smithi204 ceph-mon[37076]: pgmap v338: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 639 B/s wr, 1 op/s 2024-05-20T18:09:23.213 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:22 smithi129 ceph-mon[32029]: pgmap v338: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 639 B/s wr, 1 op/s 2024-05-20T18:09:23.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:22 smithi154 ceph-mon[36482]: pgmap v338: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 639 B/s wr, 1 op/s 2024-05-20T18:09:25.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:24 smithi204 ceph-mon[37076]: pgmap v339: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 479 B/s wr, 1 op/s 2024-05-20T18:09:25.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:24 smithi154 ceph-mon[36482]: pgmap v339: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 479 B/s wr, 1 op/s 2024-05-20T18:09:25.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:24 smithi129 ceph-mon[32029]: pgmap v339: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 479 B/s wr, 1 op/s 2024-05-20T18:09:27.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:26 smithi204 ceph-mon[37076]: pgmap v340: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 409 B/s wr, 2 op/s 2024-05-20T18:09:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:26 smithi154 ceph-mon[36482]: pgmap v340: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 409 B/s wr, 2 op/s 2024-05-20T18:09:27.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:26 smithi129 ceph-mon[32029]: pgmap v340: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 409 B/s wr, 2 op/s 2024-05-20T18:09:29.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:28 smithi204 ceph-mon[37076]: pgmap v341: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 409 B/s wr, 2 op/s 2024-05-20T18:09:29.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:28 smithi154 ceph-mon[36482]: pgmap v341: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 409 B/s wr, 2 op/s 2024-05-20T18:09:29.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:28 smithi129 ceph-mon[32029]: pgmap v341: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 409 B/s wr, 2 op/s 2024-05-20T18:09:31.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:30 smithi204 ceph-mon[37076]: pgmap v342: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 374 B/s wr, 3 op/s 2024-05-20T18:09:31.194 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:30 smithi154 ceph-mon[36482]: pgmap v342: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 374 B/s wr, 3 op/s 2024-05-20T18:09:31.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:30 smithi129 ceph-mon[32029]: pgmap v342: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 374 B/s wr, 3 op/s 2024-05-20T18:09:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.515+0000 7f44f6099640 1 -- 172.21.15.129:0/3828143695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f010d080 msgr2=0x7f44f010f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.515+0000 7f44f6099640 1 --2- 172.21.15.129:0/3828143695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f010d080 0x7f44f010f570 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009900 tx=0x7f44e802a510 comp rx=0 tx=0).stop 2024-05-20T18:09:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.515+0000 7f44f6099640 1 -- 172.21.15.129:0/3828143695 shutdown_connections 2024-05-20T18:09:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.515+0000 7f44f6099640 1 --2- 172.21.15.129:0/3828143695 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f010a6d0 0x7f44f010caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.515+0000 7f44f6099640 1 --2- 172.21.15.129:0/3828143695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f010d080 0x7f44f010f570 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.515+0000 7f44f6099640 1 --2- 172.21.15.129:0/3828143695 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44f0103c70 0x7f44f0104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.515+0000 7f44f6099640 1 -- 172.21.15.129:0/3828143695 >> 172.21.15.129:0/3828143695 conn(0x7f44f00fd210 msgr2=0x7f44f00ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.515+0000 7f44f6099640 1 -- 172.21.15.129:0/3828143695 shutdown_connections 2024-05-20T18:09:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.515+0000 7f44f6099640 1 -- 172.21.15.129:0/3828143695 wait complete. 2024-05-20T18:09:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.515+0000 7f44f6099640 1 Processor -- start 2024-05-20T18:09:31.514 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44f6099640 1 -- start start 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44f6099640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44f0103c70 0x7f44f0199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44f6099640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f010a6d0 0x7f44f0199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44f6099640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f010d080 0x7f44f019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44f6099640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44f0112e30 con 0x7f44f010d080 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44f6099640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44f0112cb0 con 0x7f44f010a6d0 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44f6099640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f44f0112fb0 con 0x7f44f0103c70 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44effff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f010d080 0x7f44f019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44effff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f010d080 0x7f44f019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53290/0 (socket says 172.21.15.129:53290) 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44effff640 1 -- 172.21.15.129:0/437360837 learned_addr learned my addr 172.21.15.129:0/437360837 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44eeffd640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f010a6d0 0x7f44f0199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44ef7fe640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44f0103c70 0x7f44f0199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44effff640 1 -- 172.21.15.129:0/437360837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44f0103c70 msgr2=0x7f44f0199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44effff640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44f0103c70 0x7f44f0199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44effff640 1 -- 172.21.15.129:0/437360837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f010a6d0 msgr2=0x7f44f0199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44effff640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f010a6d0 0x7f44f0199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44effff640 1 -- 172.21.15.129:0/437360837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f019e5f0 con 0x7f44f010d080 2024-05-20T18:09:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44eeffd640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f010a6d0 0x7f44f0199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:09:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.516+0000 7f44effff640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f010d080 0x7f44f019dfb0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f44e8005ab0 tx=0x7f44e8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.517+0000 7f44ecff9640 1 -- 172.21.15.129:0/437360837 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e803f030 con 0x7f44f010d080 2024-05-20T18:09:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.517+0000 7f44ecff9640 1 -- 172.21.15.129:0/437360837 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e8004440 con 0x7f44f010d080 2024-05-20T18:09:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.517+0000 7f44f6099640 1 -- 172.21.15.129:0/437360837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f019e880 con 0x7f44f010d080 2024-05-20T18:09:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.517+0000 7f44ecff9640 1 -- 172.21.15.129:0/437360837 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8035070 con 0x7f44f010d080 2024-05-20T18:09:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.517+0000 7f44f6099640 1 -- 172.21.15.129:0/437360837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f019ed70 con 0x7f44f010d080 2024-05-20T18:09:31.523 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.519+0000 7f44ecff9640 1 -- 172.21.15.129:0/437360837 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f44e8042030 con 0x7f44f010d080 2024-05-20T18:09:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44fd01d740 1 Processor -- start 2024-05-20T18:09:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44fd01d740 1 -- start start 2024-05-20T18:09:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e46080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e465c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a864d0 0x55d6b7e46b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44fd01d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d6b7dab340 con 0x55d6b7a864d0 2024-05-20T18:09:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44fd01d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d6b7d53290 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44fd01d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d6b7da7300 con 0x55d6b7aa71b0 2024-05-20T18:09:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a864d0 0x55d6b7e46b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44ef7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a864d0 0x55d6b7e46b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53296/0 (socket says 172.21.15.129:53296) 2024-05-20T18:09:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44ecff9640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44d80a2cf0 0x7f44d80a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44eeffd640 1 -- 172.21.15.129:0/831241055 learned_addr learned my addr 172.21.15.129:0/831241055 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44eeffd640 1 --2- 172.21.15.129:0/831241055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e46080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44effff640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44d80a2cf0 0x7f44d80a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44ef7fe640 1 -- 172.21.15.129:0/831241055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 msgr2=0x55d6b7e46080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44ef7fe640 1 --2- 172.21.15.129:0/831241055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e46080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44ef7fe640 1 -- 172.21.15.129:0/831241055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a864d0 msgr2=0x55d6b7e46b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44ef7fe640 1 --2- 172.21.15.129:0/831241055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a864d0 0x55d6b7e46b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44ef7fe640 1 -- 172.21.15.129:0/831241055 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6b7d778e0 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.520+0000 7f44ecff9640 1 -- 172.21.15.129:0/437360837 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f44e80ef7b0 con 0x7f44f010d080 2024-05-20T18:09:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.521+0000 7f44ef7fe640 1 --2- 172.21.15.129:0/831241055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e465c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009f50 tx=0x7f44e0010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8da985a0b2986f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.521+0000 7f44effff640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44d80a2cf0 0x7f44d80a5200 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f44e80ecfa0 tx=0x7f44e808ed30 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.521+0000 7f44d5ffb640 1 -- 172.21.15.129:0/831241055 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0003b70 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.521+0000 7f44d5ffb640 1 -- 172.21.15.129:0/831241055 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e0015070 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.521+0000 7f44d5ffb640 1 -- 172.21.15.129:0/831241055 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0011810 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.522+0000 7f44fd01d740 1 -- 172.21.15.129:0/831241055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a7e2f0 msgr2=0x55d6b7e465c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.522+0000 7f44fd01d740 1 --2- 172.21.15.129:0/831241055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e465c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009f50 tx=0x7f44e0010860 comp rx=0 tx=0).stop 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.522+0000 7f44fd01d740 1 -- 172.21.15.129:0/831241055 shutdown_connections 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.522+0000 7f44fd01d740 1 --2- 172.21.15.129:0/831241055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e465c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.522+0000 7f44fd01d740 1 --2- 172.21.15.129:0/831241055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a864d0 0x55d6b7e46b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.522+0000 7f44fd01d740 1 --2- 172.21.15.129:0/831241055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e46080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.522+0000 7f44fd01d740 1 -- 172.21.15.129:0/831241055 >> 172.21.15.129:0/831241055 conn(0x55d6b7a9b300 msgr2=0x55d6b7e42830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.522+0000 7f44fd01d740 1 -- 172.21.15.129:0/831241055 shutdown_connections 2024-05-20T18:09:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.522+0000 7f44fd01d740 1 -- 172.21.15.129:0/831241055 wait complete. 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.522+0000 7f44fd01d740 1 -- start start 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.522+0000 7f44fd01d740 1 Processor -- start 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e520d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e52610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e505e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44fd01d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d6b7e531b0 con 0x55d6b7aa71b0 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44fd01d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d6b7d66790 con 0x55d6b7a864d0 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44fd01d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d6b7e53330 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44fd01d740 10 client.0 _ll_register_callbacks cb 0x55d6b7da1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44ef7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e52610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e505e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53298/0 (socket says 172.21.15.129:53298) 2024-05-20T18:09:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44eeffd640 1 -- 172.21.15.129:0/3498167369 learned_addr learned my addr 172.21.15.129:0/3498167369 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44eeffd640 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44ef7fe640 1 -- 172.21.15.129:0/3498167369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7a7e2f0 msgr2=0x55d6b7e520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44ef7fe640 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44ef7fe640 1 -- 172.21.15.129:0/3498167369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7aa71b0 msgr2=0x55d6b7e505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44ef7fe640 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e505e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44ef7fe640 1 -- 172.21.15.129:0/3498167369 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6b7abf210 con 0x55d6b7a864d0 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44eeffd640 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44ef7fe640 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e52610 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x55d6b7e47430 tx=0x7f44e0004230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34ae6997fa740746 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.523+0000 7f44ceffd640 1 -- 172.21.15.129:0/3498167369 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0004420 con 0x55d6b7a864d0 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44fd01d740 1 -- 172.21.15.129:0/3498167369 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44e0015070 con 0x55d6b7a864d0 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44ceffd640 1 -- 172.21.15.129:0/3498167369 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f44e0003b80 con 0x55d6b7a864d0 2024-05-20T18:09:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44ceffd640 1 -- 172.21.15.129:0/3498167369 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44e0003d80 con 0x55d6b7a864d0 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44fd01d740 10 client.26837 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44fd01d740 1 -- 172.21.15.129:0/3498167369 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d6b7b8b620 con 0x55d6b7a864d0 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44ceffd640 1 -- 172.21.15.129:0/3498167369 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e0015070 con 0x55d6b7a864d0 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44ceffd640 1 -- 172.21.15.129:0/3498167369 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f44e001c590 con 0x55d6b7a864d0 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44fd01d740 10 client.26837 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44fd01d740 10 client.26837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44fd01d740 20 client.26837 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44fd01d740 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6b7a65d80 0x55d6b7e62870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44fd01d740 4 client.26837 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44fd01d740 1 -- 172.21.15.129:0/3498167369 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6b7db8a70 con 0x55d6b7a65d80 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.524+0000 7f44eeffd640 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6b7a65d80 0x55d6b7e62870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.525+0000 7f44ceffd640 10 client.26837 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.525+0000 7f44eeffd640 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6b7a65d80 0x55d6b7e62870 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feaedfc69b135be7 server_cookie=dafca29f9442535e in_seq=0 out_seq=0 2024-05-20T18:09:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.525+0000 7f44ceffd640 1 -- 172.21.15.129:0/3498167369 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d6b7db8a70 con 0x55d6b7a65d80 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.525+0000 7f44ceffd640 10 client.26837 handle_command_reply: tid=0 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.525+0000 7f44fd01d740 1 client.26837 shutdown 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.525+0000 7f44fd01d740 1 -- 172.21.15.129:0/3498167369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 msgr2=0x55d6b7e52610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.525+0000 7f44fd01d740 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e52610 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x55d6b7e47430 tx=0x7f44e0004230 comp rx=0 tx=0).stop 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.525+0000 7f44fd01d740 1 -- 172.21.15.129:0/3498167369 shutdown_connections 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.525+0000 7f44fd01d740 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6b7a65d80 0x55d6b7e62870 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.526+0000 7f44fd01d740 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e52610 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.526+0000 7f44fd01d740 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.526+0000 7f44fd01d740 1 --2- 172.21.15.129:0/3498167369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.526+0000 7f44ceffd640 0 client.26837 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.526+0000 7f44fd01d740 1 -- 172.21.15.129:0/3498167369 >> 172.21.15.129:0/3498167369 conn(0x55d6b7a9b300 msgr2=0x55d6b7e45e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.526+0000 7f44fd01d740 1 -- 172.21.15.129:0/3498167369 shutdown_connections 2024-05-20T18:09:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.526+0000 7f44fd01d740 1 -- 172.21.15.129:0/3498167369 wait complete. 2024-05-20T18:09:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.526+0000 7f44fd01d740 20 client.26837 trim_cache size 0 max 16384 2024-05-20T18:09:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44fd01d740 1 Processor -- start 2024-05-20T18:09:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44fd01d740 1 -- start start 2024-05-20T18:09:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e4a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e585c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e58b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44fd01d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01aca90 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44fd01d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01af310 con 0x55d6b7a864d0 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44fd01d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01ad100 con 0x55d6b7aa71b0 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e4a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e4a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53304/0 (socket says 172.21.15.129:53304) 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44eeffd640 1 -- 172.21.15.129:0/1837626536 learned_addr learned my addr 172.21.15.129:0/1837626536 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44ef7fe640 1 --2- 172.21.15.129:0/1837626536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44eeffd640 1 -- 172.21.15.129:0/1837626536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 msgr2=0x55d6b7e58b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44eeffd640 1 --2- 172.21.15.129:0/1837626536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e58b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44eeffd640 1 -- 172.21.15.129:0/1837626536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 msgr2=0x55d6b7e585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44eeffd640 1 --2- 172.21.15.129:0/1837626536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.541+0000 7f44eeffd640 1 -- 172.21.15.129:0/1837626536 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6b7e59040 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.540 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.542+0000 7f44eeffd640 1 --2- 172.21.15.129:0/1837626536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e4a3e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x55d6b7e62e10 tx=0x7f44e4009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c1e139d44c8542 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.542+0000 7f44d57fa640 1 -- 172.21.15.129:0/1837626536 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4028030 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.542+0000 7f44d57fa640 1 -- 172.21.15.129:0/1837626536 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e40044d0 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.542+0000 7f44d57fa640 1 -- 172.21.15.129:0/1837626536 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e40047a0 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.542+0000 7f44fd01d740 1 -- 172.21.15.129:0/1837626536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 msgr2=0x55d6b7e4a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.542+0000 7f44fd01d740 1 --2- 172.21.15.129:0/1837626536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e4a3e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x55d6b7e62e10 tx=0x7f44e4009ed0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.542+0000 7f44fd01d740 1 -- 172.21.15.129:0/1837626536 shutdown_connections 2024-05-20T18:09:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.542+0000 7f44fd01d740 1 --2- 172.21.15.129:0/1837626536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.542+0000 7f44fd01d740 1 --2- 172.21.15.129:0/1837626536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e58b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.542+0000 7f44fd01d740 1 --2- 172.21.15.129:0/1837626536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e4a3e0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.542+0000 7f44fd01d740 1 -- 172.21.15.129:0/1837626536 >> 172.21.15.129:0/1837626536 conn(0x55d6b7a9b300 msgr2=0x55d6b7e49cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44fd01d740 1 -- 172.21.15.129:0/1837626536 shutdown_connections 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44fd01d740 1 -- 172.21.15.129:0/1837626536 wait complete. 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44fd01d740 1 -- start start 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44fd01d740 1 Processor -- start 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e58260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e587c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44fd01d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e55720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44fd01d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01ad7d0 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44fd01d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01aec40 con 0x55d6b7a864d0 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44fd01d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01adea0 con 0x55d6b7aa71b0 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44fd01d740 10 client.0 _ll_register_callbacks cb 0x7f44e4009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e58260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e58260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53320/0 (socket says 172.21.15.129:53320) 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44eeffd640 1 -- 172.21.15.129:0/541959599 learned_addr learned my addr 172.21.15.129:0/541959599 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.543+0000 7f44ef7fe640 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44eeffd640 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e55720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44eeffd640 1 -- 172.21.15.129:0/541959599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 msgr2=0x55d6b7e55720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44eeffd640 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e55720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44eeffd640 1 -- 172.21.15.129:0/541959599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 msgr2=0x55d6b7e587c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44eeffd640 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e587c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44eeffd640 1 -- 172.21.15.129:0/541959599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6b7e55e10 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44eeffd640 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e58260 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f44e4014010 tx=0x7f44e4005550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70db99bd7f282549 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44ceffd640 1 -- 172.21.15.129:0/541959599 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4011050 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44ceffd640 1 -- 172.21.15.129:0/541959599 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f44e401a040 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44fd01d740 1 -- 172.21.15.129:0/541959599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44e40044d0 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44ceffd640 1 -- 172.21.15.129:0/541959599 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44e403f050 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44fd01d740 10 client.17268 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44fd01d740 1 -- 172.21.15.129:0/541959599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d6b7ece7c0 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44ceffd640 1 -- 172.21.15.129:0/541959599 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e40044d0 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.544+0000 7f44ceffd640 1 -- 172.21.15.129:0/541959599 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f44e4004b30 con 0x55d6b7a7e2f0 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.545+0000 7f44fd01d740 10 client.17268 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.545+0000 7f44fd01d740 10 client.17268 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.545+0000 7f44fd01d740 20 client.17268 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.545+0000 7f44fd01d740 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6b7a3b1c0 0x55d6b7ed0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.545+0000 7f44fd01d740 4 client.17268 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:09:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.545+0000 7f44fd01d740 1 -- 172.21.15.129:0/541959599 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d6b7db2dc0 con 0x55d6b7a3b1c0 2024-05-20T18:09:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.545+0000 7f44ef7fe640 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6b7a3b1c0 0x55d6b7ed0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.545+0000 7f44ceffd640 10 client.17268 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.545+0000 7f44ef7fe640 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6b7a3b1c0 0x55d6b7ed0d80 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=752a3f767cb1bc2e server_cookie=26d770bff8f982a4 in_seq=0 out_seq=0 2024-05-20T18:09:31.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44ceffd640 1 -- 172.21.15.129:0/541959599 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d6b7db2dc0 con 0x55d6b7a3b1c0 2024-05-20T18:09:31.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44ceffd640 10 client.17268 handle_command_reply: tid=0 2024-05-20T18:09:31.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 1 client.17268 shutdown 2024-05-20T18:09:31.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 1 -- 172.21.15.129:0/541959599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 msgr2=0x55d6b7e58260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:31.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e58260 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f44e4014010 tx=0x7f44e4005550 comp rx=0 tx=0).stop 2024-05-20T18:09:31.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 1 -- 172.21.15.129:0/541959599 shutdown_connections 2024-05-20T18:09:31.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55d6b7a3b1c0 0x55d6b7ed0d80 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.549 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6b7a864d0 0x55d6b7e587c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6b7aa71b0 0x55d6b7e55720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44ceffd640 0 client.17268 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 1 --2- 172.21.15.129:0/541959599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d6b7a7e2f0 0x55d6b7e58260 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 1 -- 172.21.15.129:0/541959599 >> 172.21.15.129:0/541959599 conn(0x55d6b7a9b300 msgr2=0x55d6b7e45e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 1 -- 172.21.15.129:0/541959599 shutdown_connections 2024-05-20T18:09:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 1 -- 172.21.15.129:0/541959599 wait complete. 2024-05-20T18:09:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44fd01d740 20 client.17268 trim_cache size 0 max 16384 2024-05-20T18:09:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:09:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:09:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:09:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:09:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44f6099640 1 -- 172.21.15.129:0/437360837 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44d80a2cf0 msgr2=0x7f44d80a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44f6099640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44d80a2cf0 0x7f44d80a5200 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f44e80ecfa0 tx=0x7f44e808ed30 comp rx=0 tx=0).stop 2024-05-20T18:09:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44f6099640 1 -- 172.21.15.129:0/437360837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f010d080 msgr2=0x7f44f019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.550+0000 7f44f6099640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f010d080 0x7f44f019dfb0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f44e8005ab0 tx=0x7f44e8004040 comp rx=0 tx=0).stop 2024-05-20T18:09:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.551+0000 7f44f6099640 1 -- 172.21.15.129:0/437360837 shutdown_connections 2024-05-20T18:09:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.551+0000 7f44f6099640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f44d80a2cf0 0x7f44d80a5200 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.551+0000 7f44f6099640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f010a6d0 0x7f44f0199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.551+0000 7f44f6099640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44f010d080 0x7f44f019dfb0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.551+0000 7f44f6099640 1 --2- 172.21.15.129:0/437360837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44f0103c70 0x7f44f0199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.551+0000 7f44f6099640 1 -- 172.21.15.129:0/437360837 >> 172.21.15.129:0/437360837 conn(0x7f44f00fd210 msgr2=0x7f44f010d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.551+0000 7f44f6099640 1 -- 172.21.15.129:0/437360837 shutdown_connections 2024-05-20T18:09:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:31.551+0000 7f44f6099640 1 -- 172.21.15.129:0/437360837 wait complete. 2024-05-20T18:09:33.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:32 smithi154 ceph-mon[36482]: pgmap v343: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-05-20T18:09:33.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:32 smithi129 ceph-mon[32029]: pgmap v343: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-05-20T18:09:33.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:32 smithi204 ceph-mon[37076]: pgmap v343: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1 op/s 2024-05-20T18:09:35.231 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:34 smithi204 ceph-mon[37076]: pgmap v344: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 0 B/s wr, 1 op/s 2024-05-20T18:09:35.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:34 smithi154 ceph-mon[36482]: pgmap v344: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 0 B/s wr, 1 op/s 2024-05-20T18:09:35.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:34 smithi129 ceph-mon[32029]: pgmap v344: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 0 B/s wr, 1 op/s 2024-05-20T18:09:37.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:36 smithi154 ceph-mon[36482]: pgmap v345: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:37.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:36 smithi129 ceph-mon[32029]: pgmap v345: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:37.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:36 smithi204 ceph-mon[37076]: pgmap v345: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:37.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:09:37.624 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:09:37.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.756+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3401694408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fa4108b70 msgr2=0x7f6fa4100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.756+0000 7f6fa90f7640 1 --2- 172.21.15.129:0/3401694408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fa4108b70 0x7f6fa4100ae0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009900 tx=0x7f6f9402a410 comp rx=0 tx=0).stop 2024-05-20T18:09:37.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.756+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3401694408 shutdown_connections 2024-05-20T18:09:37.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.756+0000 7f6fa90f7640 1 --2- 172.21.15.129:0/3401694408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa4101020 0x7f6fa4101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.756+0000 7f6fa90f7640 1 --2- 172.21.15.129:0/3401694408 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fa410d520 0x7f6fa410f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.756+0000 7f6fa90f7640 1 --2- 172.21.15.129:0/3401694408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fa4108b70 0x7f6fa4100ae0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.756+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3401694408 >> 172.21.15.129:0/3401694408 conn(0x7f6fa40768d0 msgr2=0x7f6fa4076d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.756+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3401694408 shutdown_connections 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.756+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3401694408 wait complete. 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.756+0000 7f6fa90f7640 1 Processor -- start 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.756+0000 7f6fa90f7640 1 -- start start 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa90f7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fa4101020 0x7f6fa419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa90f7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa4108b70 0x7f6fa419fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa90f7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fa410d520 0x7f6fa4199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa90f7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4112e30 con 0x7f6fa4101020 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa90f7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4112cb0 con 0x7f6fa4108b70 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa90f7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4112fb0 con 0x7f6fa410d520 2024-05-20T18:09:37.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fa4101020 0x7f6fa419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fa4101020 0x7f6fa419f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53338/0 (socket says 172.21.15.129:53338) 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/3981916313 learned_addr learned my addr 172.21.15.129:0/3981916313 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/3981916313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fa410d520 msgr2=0x7f6fa4199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa2d9d640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fa410d520 0x7f6fa4199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/3981916313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa4108b70 msgr2=0x7f6fa419fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa2d9d640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa4108b70 0x7f6fa419fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/3981916313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa4199d80 con 0x7f6fa4101020 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa2d9d640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fa4101020 0x7f6fa419f4d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f6f94005940 tx=0x7f6f940042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6f8bfff640 1 -- 172.21.15.129:0/3981916313 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9403f030 con 0x7f6fa4101020 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6f8bfff640 1 -- 172.21.15.129:0/3981916313 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f94003ce0 con 0x7f6fa4101020 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6f8bfff640 1 -- 172.21.15.129:0/3981916313 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94035070 con 0x7f6fa4101020 2024-05-20T18:09:37.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3981916313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa4199fb0 con 0x7f6fa4101020 2024-05-20T18:09:37.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.757+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3981916313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa41a6270 con 0x7f6fa4101020 2024-05-20T18:09:37.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.760+0000 7f6f8bfff640 1 -- 172.21.15.129:0/3981916313 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f6f94042030 con 0x7f6fa4101020 2024-05-20T18:09:37.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.760+0000 7f6faff9f740 1 Processor -- start 2024-05-20T18:09:37.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.760+0000 7f6faff9f740 1 -- start start 2024-05-20T18:09:37.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.760+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe796390 0x5555feb34d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.760+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe76d4d0 0x5555feb35270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.760+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe7756b0 0x5555feb357b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.760+0000 7f6faff9f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5555fea9ad00 con 0x5555fe7756b0 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.760+0000 7f6faff9f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5555fea424c0 con 0x5555fe796390 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.760+0000 7f6faff9f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5555fea96cc0 con 0x5555fe76d4d0 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe7756b0 0x5555feb357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe7756b0 0x5555feb357b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53342/0 (socket says 172.21.15.129:53342) 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa259c640 1 -- 172.21.15.129:0/1829993946 learned_addr learned my addr 172.21.15.129:0/1829993946 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa359e640 1 --2- 172.21.15.129:0/1829993946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe796390 0x5555feb34d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6f8bfff640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6f8c0a2d80 0x7f6f8c0a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa2d9d640 1 --2- 172.21.15.129:0/1829993946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe76d4d0 0x5555feb35270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa359e640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6f8c0a2d80 0x7f6f8c0a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa259c640 1 -- 172.21.15.129:0/1829993946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe76d4d0 msgr2=0x5555feb35270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa259c640 1 --2- 172.21.15.129:0/1829993946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe76d4d0 0x5555feb35270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6f8bfff640 1 -- 172.21.15.129:0/3981916313 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f6f940efa00 con 0x7f6fa4101020 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa259c640 1 -- 172.21.15.129:0/1829993946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe796390 msgr2=0x5555feb34d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa259c640 1 --2- 172.21.15.129:0/1829993946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe796390 0x5555feb34d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa259c640 1 -- 172.21.15.129:0/1829993946 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555fe8787d0 con 0x5555fe7756b0 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa259c640 1 --2- 172.21.15.129:0/1829993946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe7756b0 0x5555feb357b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009d70 tx=0x7f6f9800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c394d64c32b22f13 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6f897fa640 1 -- 172.21.15.129:0/1829993946 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f98016070 con 0x5555fe7756b0 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6f897fa640 1 -- 172.21.15.129:0/1829993946 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f98005020 con 0x5555fe7756b0 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6f897fa640 1 -- 172.21.15.129:0/1829993946 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f980116b0 con 0x5555fe7756b0 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.761+0000 7f6fa359e640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6f8c0a2d80 0x7f6f8c0a5290 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c005d00 tx=0x7f6f9c005c90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.762+0000 7f6faff9f740 1 -- 172.21.15.129:0/1829993946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe7756b0 msgr2=0x5555feb357b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.762+0000 7f6faff9f740 1 --2- 172.21.15.129:0/1829993946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe7756b0 0x5555feb357b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009d70 tx=0x7f6f9800e6c0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.762+0000 7f6faff9f740 1 -- 172.21.15.129:0/1829993946 shutdown_connections 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.762+0000 7f6faff9f740 1 --2- 172.21.15.129:0/1829993946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe7756b0 0x5555feb357b0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.762+0000 7f6faff9f740 1 --2- 172.21.15.129:0/1829993946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe76d4d0 0x5555feb35270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.762+0000 7f6faff9f740 1 --2- 172.21.15.129:0/1829993946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe796390 0x5555feb34d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.762+0000 7f6faff9f740 1 -- 172.21.15.129:0/1829993946 >> 172.21.15.129:0/1829993946 conn(0x5555fe78a4e0 msgr2=0x5555feb314e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.762+0000 7f6faff9f740 1 -- 172.21.15.129:0/1829993946 shutdown_connections 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.762+0000 7f6faff9f740 1 -- 172.21.15.129:0/1829993946 wait complete. 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6faff9f740 1 -- start start 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6faff9f740 1 Processor -- start 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb3f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe7756b0 0x5555feb3f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe796390 0x5555feb45560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6faff9f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5555feb42020 con 0x5555fe76d4d0 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6faff9f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5555fea55b90 con 0x5555fe796390 2024-05-20T18:09:37.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6faff9f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5555feb421a0 con 0x5555fe7756b0 2024-05-20T18:09:37.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6faff9f740 10 client.0 _ll_register_callbacks cb 0x5555fea91da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:37.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6fa2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb3f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6fa2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb3f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53352/0 (socket says 172.21.15.129:53352) 2024-05-20T18:09:37.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/3492317407 learned_addr learned my addr 172.21.15.129:0/3492317407 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:37.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6fa359e640 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe796390 0x5555feb45560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6fa259c640 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe7756b0 0x5555feb3f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/3492317407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe7756b0 msgr2=0x5555feb3f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:37.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6fa2d9d640 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe7756b0 0x5555feb3f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/3492317407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe796390 msgr2=0x5555feb45560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6fa2d9d640 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe796390 0x5555feb45560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.763+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/3492317407 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555fe7ae3f0 con 0x5555fe76d4d0 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6fa2d9d640 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb3f420 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6f940b8370 tx=0x7f6f9408bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ed03bf3f6f8301 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6f86ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6f86ff5640 1 -- 172.21.15.129:0/3492317407 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94049070 con 0x5555fe76d4d0 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6f86ff5640 1 -- 172.21.15.129:0/3492317407 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f6f940ed3c0 con 0x5555fe76d4d0 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6faff9f740 1 -- 172.21.15.129:0/3492317407 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f98005020 con 0x5555fe76d4d0 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6f86ff5640 1 -- 172.21.15.129:0/3492317407 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6f940bb870 con 0x5555fe76d4d0 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6faff9f740 10 client.17280 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6faff9f740 1 -- 172.21.15.129:0/3492317407 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5555fe76e0f0 con 0x5555fe76d4d0 2024-05-20T18:09:37.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6f86ff5640 1 -- 172.21.15.129:0/3492317407 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f98005020 con 0x5555fe76d4d0 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6f86ff5640 1 -- 172.21.15.129:0/3492317407 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f6f940b8580 con 0x5555fe76d4d0 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6faff9f740 10 client.17280 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6faff9f740 10 client.17280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6faff9f740 20 client.17280 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6faff9f740 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5555fe754f60 0x5555feb51ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6faff9f740 4 client.17280 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.764+0000 7f6faff9f740 1 -- 172.21.15.129:0/3492317407 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555feaa8430 con 0x5555fe754f60 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.765+0000 7f6fa259c640 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5555fe754f60 0x5555feb51ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.765+0000 7f6f86ff5640 10 client.17280 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.765+0000 7f6fa259c640 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5555fe754f60 0x5555feb51ae0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0d75e45538299c5 server_cookie=41eaf9e1cbcbe9e5 in_seq=0 out_seq=0 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6f86ff5640 1 -- 172.21.15.129:0/3492317407 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5555feaa8430 con 0x5555fe754f60 2024-05-20T18:09:37.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6f86ff5640 10 client.17280 handle_command_reply: tid=0 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 1 client.17280 shutdown 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 1 -- 172.21.15.129:0/3492317407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 msgr2=0x5555feb3f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb3f420 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6f940b8370 tx=0x7f6f9408bd70 comp rx=0 tx=0).stop 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 1 -- 172.21.15.129:0/3492317407 shutdown_connections 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5555fe754f60 0x5555feb51ae0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe796390 0x5555feb45560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6f86ff5640 0 client.17280 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe7756b0 0x5555feb3f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 1 --2- 172.21.15.129:0/3492317407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb3f420 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 1 -- 172.21.15.129:0/3492317407 >> 172.21.15.129:0/3492317407 conn(0x5555fe78a4e0 msgr2=0x5555feb36a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 1 -- 172.21.15.129:0/3492317407 shutdown_connections 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 1 -- 172.21.15.129:0/3492317407 wait complete. 2024-05-20T18:09:37.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.766+0000 7f6faff9f740 20 client.17280 trim_cache size 0 max 16384 2024-05-20T18:09:37.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6faff9f740 1 Processor -- start 2024-05-20T18:09:37.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6faff9f740 1 -- start start 2024-05-20T18:09:37.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe754f60 0x5555feb44e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb572a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe7756b0 0x5555feb577e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6faff9f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa41acb30 con 0x5555fe76d4d0 2024-05-20T18:09:37.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6faff9f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa41af3b0 con 0x5555fe7756b0 2024-05-20T18:09:37.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6faff9f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa41ad1a0 con 0x5555fe754f60 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6fa259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6fa259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53366/0 (socket says 172.21.15.129:53366) 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6fa259c640 1 -- 172.21.15.129:0/455050604 learned_addr learned my addr 172.21.15.129:0/455050604 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6fa359e640 1 --2- 172.21.15.129:0/455050604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe7756b0 0x5555feb577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6fa259c640 1 -- 172.21.15.129:0/455050604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe754f60 msgr2=0x5555feb44e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6fa2d9d640 1 --2- 172.21.15.129:0/455050604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe754f60 0x5555feb44e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6fa259c640 1 --2- 172.21.15.129:0/455050604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe754f60 0x5555feb44e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6fa259c640 1 -- 172.21.15.129:0/455050604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe7756b0 msgr2=0x5555feb577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6fa259c640 1 --2- 172.21.15.129:0/455050604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe7756b0 0x5555feb577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6fa259c640 1 -- 172.21.15.129:0/455050604 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555feb22990 con 0x5555fe76d4d0 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.784+0000 7f6fa359e640 1 --2- 172.21.15.129:0/455050604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe7756b0 0x5555feb577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6fa259c640 1 --2- 172.21.15.129:0/455050604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb572a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6f98004b40 tx=0x7f6f98005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7696d8ddf8960c0f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:37.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6f88ff9640 1 -- 172.21.15.129:0/455050604 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9802b070 con 0x5555fe76d4d0 2024-05-20T18:09:37.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6f88ff9640 1 -- 172.21.15.129:0/455050604 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f98027e50 con 0x5555fe76d4d0 2024-05-20T18:09:37.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6f88ff9640 1 -- 172.21.15.129:0/455050604 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f98016040 con 0x5555fe76d4d0 2024-05-20T18:09:37.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6faff9f740 1 -- 172.21.15.129:0/455050604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 msgr2=0x5555feb572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6faff9f740 1 --2- 172.21.15.129:0/455050604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb572a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6f98004b40 tx=0x7f6f98005fd0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6faff9f740 1 -- 172.21.15.129:0/455050604 shutdown_connections 2024-05-20T18:09:37.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6faff9f740 1 --2- 172.21.15.129:0/455050604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe7756b0 0x5555feb577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6faff9f740 1 --2- 172.21.15.129:0/455050604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe76d4d0 0x5555feb572a0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6faff9f740 1 --2- 172.21.15.129:0/455050604 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe754f60 0x5555feb44e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6faff9f740 1 -- 172.21.15.129:0/455050604 >> 172.21.15.129:0/455050604 conn(0x5555fe78a4e0 msgr2=0x5555feb36e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:37.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6faff9f740 1 -- 172.21.15.129:0/455050604 shutdown_connections 2024-05-20T18:09:37.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.785+0000 7f6faff9f740 1 -- 172.21.15.129:0/455050604 wait complete. 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6faff9f740 1 -- start start 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6faff9f740 1 Processor -- start 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe754f60 0x5555feb473f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe76d4d0 0x5555feb47930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6faff9f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe7756b0 0x5555feb47e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6faff9f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa41ad870 con 0x5555fe754f60 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6faff9f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa41aece0 con 0x5555fe7756b0 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6faff9f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa41adf40 con 0x5555fe76d4d0 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6faff9f740 10 client.0 _ll_register_callbacks cb 0x7f6f94005780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6fa2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe754f60 0x5555feb473f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6fa2d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe754f60 0x5555feb473f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53370/0 (socket says 172.21.15.129:53370) 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/918253897 learned_addr learned my addr 172.21.15.129:0/918253897 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:37.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6fa259c640 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe76d4d0 0x5555feb47930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/918253897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe76d4d0 msgr2=0x5555feb47930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6fa2d9d640 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe76d4d0 0x5555feb47930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/918253897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe7756b0 msgr2=0x5555feb47e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.786+0000 7f6fa359e640 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe7756b0 0x5555feb47e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6fa2d9d640 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe7756b0 0x5555feb47e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6fa2d9d640 1 -- 172.21.15.129:0/918253897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555febb8e50 con 0x5555fe754f60 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6fa259c640 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe76d4d0 0x5555feb47930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6fa2d9d640 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe754f60 0x5555feb473f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6f940b88e0 tx=0x7f6f9403b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d380450fc6701a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6f86ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6f86ff5640 1 -- 172.21.15.129:0/918253897 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94049030 con 0x5555fe754f60 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6f86ff5640 1 -- 172.21.15.129:0/918253897 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f6f940ed3c0 con 0x5555fe754f60 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6faff9f740 1 -- 172.21.15.129:0/918253897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f98027e50 con 0x5555fe754f60 2024-05-20T18:09:37.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6f86ff5640 1 -- 172.21.15.129:0/918253897 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6f940bc610 con 0x5555fe754f60 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6faff9f740 10 client.17286 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6faff9f740 1 -- 172.21.15.129:0/918253897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5555febb9ea0 con 0x5555fe754f60 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6f86ff5640 1 -- 172.21.15.129:0/918253897 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f98027e50 con 0x5555fe754f60 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6f86ff5640 1 -- 172.21.15.129:0/918253897 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f6f940bc170 con 0x5555fe754f60 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6faff9f740 10 client.17286 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6faff9f740 10 client.17286 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6faff9f740 20 client.17286 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6faff9f740 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5555fe796390 0x5555febbc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6faff9f740 4 client.17286 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.787+0000 7f6faff9f740 1 -- 172.21.15.129:0/918253897 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5555feaa2780 con 0x5555fe796390 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6fa259c640 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5555fe796390 0x5555febbc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6fa259c640 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5555fe796390 0x5555febbc520 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1c950746860a953 server_cookie=e381fe3048a9c560 in_seq=0 out_seq=0 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6f86ff5640 10 client.17286 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6f86ff5640 1 -- 172.21.15.129:0/918253897 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5555feaa2780 con 0x5555fe796390 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6f86ff5640 10 client.17286 handle_command_reply: tid=0 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6faff9f740 1 client.17286 shutdown 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6faff9f740 1 -- 172.21.15.129:0/918253897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe754f60 msgr2=0x5555feb473f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6faff9f740 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe754f60 0x5555feb473f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f6f940b88e0 tx=0x7f6f9403b690 comp rx=0 tx=0).stop 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6faff9f740 1 -- 172.21.15.129:0/918253897 shutdown_connections 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6faff9f740 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5555fe796390 0x5555febbc520 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6faff9f740 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555fe7756b0 0x5555feb47e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6faff9f740 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5555fe76d4d0 0x5555feb47930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6f86ff5640 0 client.17286 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6faff9f740 1 --2- 172.21.15.129:0/918253897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5555fe754f60 0x5555feb473f0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.788+0000 7f6faff9f740 1 -- 172.21.15.129:0/918253897 >> 172.21.15.129:0/918253897 conn(0x5555fe78a4e0 msgr2=0x5555feb40850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6faff9f740 1 -- 172.21.15.129:0/918253897 shutdown_connections 2024-05-20T18:09:37.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6faff9f740 1 -- 172.21.15.129:0/918253897 wait complete. 2024-05-20T18:09:37.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6faff9f740 20 client.17286 trim_cache size 0 max 16384 2024-05-20T18:09:37.795 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:09:37.795 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:09:37.795 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:09:37.795 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:09:37.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3981916313 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6f8c0a2d80 msgr2=0x7f6f8c0a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6f8c0a2d80 0x7f6f8c0a5290 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c005d00 tx=0x7f6f9c005c90 comp rx=0 tx=0).stop 2024-05-20T18:09:37.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3981916313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fa4101020 msgr2=0x7f6fa419f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fa4101020 0x7f6fa419f4d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f6f94005940 tx=0x7f6f940042f0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3981916313 shutdown_connections 2024-05-20T18:09:37.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6f8c0a2d80 0x7f6f8c0a5290 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa4108b70 0x7f6fa419fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fa410d520 0x7f6fa4199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 --2- 172.21.15.129:0/3981916313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6fa4101020 0x7f6fa419f4d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3981916313 >> 172.21.15.129:0/3981916313 conn(0x7f6fa40768d0 msgr2=0x7f6fa41094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:37.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3981916313 shutdown_connections 2024-05-20T18:09:37.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.789+0000 7f6fa90f7640 1 -- 172.21.15.129:0/3981916313 wait complete. 2024-05-20T18:09:37.798 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:09:37.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:09:37.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.913+0000 7f1093eca640 1 -- 172.21.15.129:0/3602788996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f108c108ac0 msgr2=0x7f108c1026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.913+0000 7f1093eca640 1 --2- 172.21.15.129:0/3602788996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f108c108ac0 0x7f108c1026b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1084009900 tx=0x7f108402a410 comp rx=0 tx=0).stop 2024-05-20T18:09:37.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.913+0000 7f1093eca640 1 -- 172.21.15.129:0/3602788996 shutdown_connections 2024-05-20T18:09:37.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.913+0000 7f1093eca640 1 --2- 172.21.15.129:0/3602788996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f108c103260 0x7f108c10f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.913+0000 7f1093eca640 1 --2- 172.21.15.129:0/3602788996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f108c108f60 0x7f108c102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.913+0000 7f1093eca640 1 --2- 172.21.15.129:0/3602788996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f108c108ac0 0x7f108c1026b0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.913+0000 7f1093eca640 1 -- 172.21.15.129:0/3602788996 >> 172.21.15.129:0/3602788996 conn(0x7f108c0fe310 msgr2=0x7f108c100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:37.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.913+0000 7f1093eca640 1 -- 172.21.15.129:0/3602788996 shutdown_connections 2024-05-20T18:09:37.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.913+0000 7f1093eca640 1 -- 172.21.15.129:0/3602788996 wait complete. 2024-05-20T18:09:37.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1093eca640 1 Processor -- start 2024-05-20T18:09:37.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1093eca640 1 -- start start 2024-05-20T18:09:37.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1093eca640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f108c103260 0x7f108c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1093eca640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f108c108ac0 0x7f108c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1093eca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f108c108f60 0x7f108c19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1093eca640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f108c112e50 con 0x7f108c108f60 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1093eca640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f108c112cd0 con 0x7f108c103260 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1093eca640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f108c112fd0 con 0x7f108c108ac0 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1092467640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f108c108f60 0x7f108c19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1092467640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f108c108f60 0x7f108c19dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53386/0 (socket says 172.21.15.129:53386) 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1092467640 1 -- 172.21.15.129:0/1694838011 learned_addr learned my addr 172.21.15.129:0/1694838011 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1091465640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f108c108ac0 0x7f108c199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1091c66640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f108c103260 0x7f108c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1092467640 1 -- 172.21.15.129:0/1694838011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f108c108ac0 msgr2=0x7f108c199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1092467640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f108c108ac0 0x7f108c199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1092467640 1 -- 172.21.15.129:0/1694838011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f108c103260 msgr2=0x7f108c1997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1092467640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f108c103260 0x7f108c1997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:37.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1092467640 1 -- 172.21.15.129:0/1694838011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108c19e610 con 0x7f108c108f60 2024-05-20T18:09:37.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.914+0000 7f1091c66640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f108c103260 0x7f108c1997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:37.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.915+0000 7f1092467640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f108c108f60 0x7f108c19dfd0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f108800e970 tx=0x7f108800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:37.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.915+0000 7f107effd640 1 -- 172.21.15.129:0/1694838011 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108800ccc0 con 0x7f108c108f60 2024-05-20T18:09:37.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.915+0000 7f107effd640 1 -- 172.21.15.129:0/1694838011 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f108800ce60 con 0x7f108c108f60 2024-05-20T18:09:37.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.915+0000 7f1093eca640 1 -- 172.21.15.129:0/1694838011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f108c19e900 con 0x7f108c108f60 2024-05-20T18:09:37.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.915+0000 7f107effd640 1 -- 172.21.15.129:0/1694838011 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1088010490 con 0x7f108c108f60 2024-05-20T18:09:37.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.915+0000 7f1093eca640 1 -- 172.21.15.129:0/1694838011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f108c19ee00 con 0x7f108c108f60 2024-05-20T18:09:37.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.916+0000 7f1093eca640 1 -- 172.21.15.129:0/1694838011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f108c069f50 con 0x7f108c108f60 2024-05-20T18:09:37.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.917+0000 7f107effd640 1 -- 172.21.15.129:0/1694838011 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f1088005000 con 0x7f108c108f60 2024-05-20T18:09:37.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.918+0000 7f107effd640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10780a2d00 0x7f10780a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:37.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.918+0000 7f1091c66640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10780a2d00 0x7f10780a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:37.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.919+0000 7f107effd640 1 -- 172.21.15.129:0/1694838011 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f10880cfb70 con 0x7f108c108f60 2024-05-20T18:09:37.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.919+0000 7f1091c66640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10780a2d00 0x7f10780a5210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f10840096c0 tx=0x7f1084005fd0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:37.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:37.922+0000 7f107effd640 1 -- 172.21.15.129:0/1694838011 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1088053850 con 0x7f108c108f60 2024-05-20T18:09:38.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.138+0000 7f1093eca640 1 -- 172.21.15.129:0/1694838011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f108c19ab00 con 0x7f108c108f60 2024-05-20T18:09:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.139+0000 7f107effd640 1 -- 172.21.15.129:0/1694838011 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11829 (secure 0 0 0) 0x7f108c19ee00 con 0x7f108c108f60 2024-05-20T18:09:38.136 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:09:38.136 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":26255,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/2724397229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":2724397229},{"type":"v1","addr":"172.21.15.154:6837","nonce":2724397229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:09:18.904334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26815":{"gid":26815,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/1721642062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":1721642062},{"type":"v1","addr":"172.21.15.204:6839","nonce":1721642062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:09:38.137 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 28 2024-05-20T18:09:38.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 -- 172.21.15.129:0/1694838011 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10780a2d00 msgr2=0x7f10780a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10780a2d00 0x7f10780a5210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f10840096c0 tx=0x7f1084005fd0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 -- 172.21.15.129:0/1694838011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f108c108f60 msgr2=0x7f108c19dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f108c108f60 0x7f108c19dfd0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f108800e970 tx=0x7f108800ed70 comp rx=0 tx=0).stop 2024-05-20T18:09:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 -- 172.21.15.129:0/1694838011 shutdown_connections 2024-05-20T18:09:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10780a2d00 0x7f10780a5210 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f108c108f60 0x7f108c19dfd0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f108c108ac0 0x7f108c199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 --2- 172.21.15.129:0/1694838011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f108c103260 0x7f108c1997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 -- 172.21.15.129:0/1694838011 >> 172.21.15.129:0/1694838011 conn(0x7f108c0fe310 msgr2=0x7f108c0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 -- 172.21.15.129:0/1694838011 shutdown_connections 2024-05-20T18:09:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.141+0000 7f1093eca640 1 -- 172.21.15.129:0/1694838011 wait complete. 2024-05-20T18:09:38.150 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:09:38.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.267+0000 7fa55f884640 1 -- 172.21.15.129:0/1293814261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5581017b0 msgr2=0x7fa55810f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:38.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.267+0000 7fa55f884640 1 --2- 172.21.15.129:0/1293814261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5581017b0 0x7fa55810f690 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa550009900 tx=0x7fa55002a410 comp rx=0 tx=0).stop 2024-05-20T18:09:38.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.267+0000 7fa55f884640 1 -- 172.21.15.129:0/1293814261 shutdown_connections 2024-05-20T18:09:38.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.267+0000 7fa55f884640 1 --2- 172.21.15.129:0/1293814261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa558109010 0x7fa558101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.267+0000 7fa55f884640 1 --2- 172.21.15.129:0/1293814261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5581017b0 0x7fa55810f690 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.267+0000 7fa55f884640 1 --2- 172.21.15.129:0/1293814261 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa558108b70 0x7fa558100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.267+0000 7fa55f884640 1 -- 172.21.15.129:0/1293814261 >> 172.21.15.129:0/1293814261 conn(0x7fa5580768d0 msgr2=0x7fa558076d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:38.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.267+0000 7fa55f884640 1 -- 172.21.15.129:0/1293814261 shutdown_connections 2024-05-20T18:09:38.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.267+0000 7fa55f884640 1 -- 172.21.15.129:0/1293814261 wait complete. 2024-05-20T18:09:38.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.267+0000 7fa55f884640 1 Processor -- start 2024-05-20T18:09:38.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.267+0000 7fa55f884640 1 -- start start 2024-05-20T18:09:38.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55f884640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5581017b0 0x7fa55819f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55f884640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa558108b70 0x7fa55819fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55f884640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa558109010 0x7fa5581996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55f884640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa558112f20 con 0x7fa558108b70 2024-05-20T18:09:38.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55f884640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa558112da0 con 0x7fa558109010 2024-05-20T18:09:38.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55f884640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581130a0 con 0x7fa5581017b0 2024-05-20T18:09:38.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa558108b70 0x7fa55819fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa558108b70 0x7fa55819fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53414/0 (socket says 172.21.15.129:53414) 2024-05-20T18:09:38.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55ce1f640 1 -- 172.21.15.129:0/1056353219 learned_addr learned my addr 172.21.15.129:0/1056353219 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:38.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55d620640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5581017b0 0x7fa55819f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55de21640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa558109010 0x7fa5581996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55ce1f640 1 -- 172.21.15.129:0/1056353219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5581017b0 msgr2=0x7fa55819f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:38.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5581017b0 0x7fa55819f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55ce1f640 1 -- 172.21.15.129:0/1056353219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa558109010 msgr2=0x7fa5581996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:38.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa558109010 0x7fa5581996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55ce1f640 1 -- 172.21.15.129:0/1056353219 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa558199df0 con 0x7fa558108b70 2024-05-20T18:09:38.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa558108b70 0x7fa55819fa80 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00e9f0 tx=0x7fa54c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:38.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa5567fc640 1 -- 172.21.15.129:0/1056353219 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c00ccc0 con 0x7fa558108b70 2024-05-20T18:09:38.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa5567fc640 1 -- 172.21.15.129:0/1056353219 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa54c0035f0 con 0x7fa558108b70 2024-05-20T18:09:38.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55f884640 1 -- 172.21.15.129:0/1056353219 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa55819a080 con 0x7fa558108b70 2024-05-20T18:09:38.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa5567fc640 1 -- 172.21.15.129:0/1056353219 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c00ce60 con 0x7fa558108b70 2024-05-20T18:09:38.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.268+0000 7fa55f884640 1 -- 172.21.15.129:0/1056353219 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5581a62e0 con 0x7fa558108b70 2024-05-20T18:09:38.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.271+0000 7fa5567fc640 1 -- 172.21.15.129:0/1056353219 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fa54c0037a0 con 0x7fa558108b70 2024-05-20T18:09:38.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.271+0000 7fa56680a740 1 Processor -- start 2024-05-20T18:09:38.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.271+0000 7fa56680a740 1 -- start start 2024-05-20T18:09:38.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.271+0000 7fa56680a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 0x5556ac131d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.271+0000 7fa56680a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac132260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.271+0000 7fa56680a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.271+0000 7fa56680a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5556ac097c40 con 0x5556abd726b0 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.271+0000 7fa56680a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5556ac03f580 con 0x5556abd6a4d0 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.271+0000 7fa56680a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5556ac093c00 con 0x5556abd93390 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55de21640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 0x5556ac131d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53422/0 (socket says 172.21.15.129:53422) 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2403012554 learned_addr learned my addr 172.21.15.129:0/2403012554 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55d620640 1 --2- 172.21.15.129:0/2403012554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac132260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa5567fc640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5400a2d80 0x7fa5400a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2403012554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 msgr2=0x5556ac131d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/2403012554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 0x5556ac131d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55de21640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5400a2d80 0x7fa5400a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2403012554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 msgr2=0x5556ac132260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:38.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/2403012554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac132260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2403012554 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556abdab3f0 con 0x5556abd726b0 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55d620640 1 --2- 172.21.15.129:0/2403012554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac132260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa5567fc640 1 -- 172.21.15.129:0/1056353219 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fa54c0cf790 con 0x7fa558108b70 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/2403012554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1327a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa54c098670 tx=0x7fa54c021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4d5628359ed3 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa53f7fe640 1 -- 172.21.15.129:0/2403012554 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c09c3a0 con 0x5556abd726b0 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa55de21640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5400a2d80 0x7fa5400a5290 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x5556ac12ee90 tx=0x7fa55002a410 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa53f7fe640 1 -- 172.21.15.129:0/2403012554 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa54c00a250 con 0x5556abd726b0 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.272+0000 7fa53f7fe640 1 -- 172.21.15.129:0/2403012554 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c016df0 con 0x5556abd726b0 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.273+0000 7fa56680a740 1 -- 172.21.15.129:0/2403012554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 msgr2=0x5556ac1327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.273+0000 7fa56680a740 1 --2- 172.21.15.129:0/2403012554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1327a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa54c098670 tx=0x7fa54c021000 comp rx=0 tx=0).stop 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.273+0000 7fa56680a740 1 -- 172.21.15.129:0/2403012554 shutdown_connections 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.273+0000 7fa56680a740 1 --2- 172.21.15.129:0/2403012554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac132260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.273+0000 7fa56680a740 1 --2- 172.21.15.129:0/2403012554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1327a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.273+0000 7fa56680a740 1 --2- 172.21.15.129:0/2403012554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 0x5556ac131d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.273+0000 7fa56680a740 1 -- 172.21.15.129:0/2403012554 >> 172.21.15.129:0/2403012554 conn(0x5556abd874e0 msgr2=0x5556ac12e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.273+0000 7fa56680a740 1 -- 172.21.15.129:0/2403012554 shutdown_connections 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.273+0000 7fa56680a740 1 -- 172.21.15.129:0/2403012554 wait complete. 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa56680a740 1 -- start start 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa56680a740 1 Processor -- start 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa56680a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac13c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa56680a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac13c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa56680a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 0x5556ac142480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa56680a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5556ac13efb0 con 0x5556abd726b0 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa56680a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5556ac052c40 con 0x5556abd6a4d0 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa56680a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5556ac13f130 con 0x5556abd93390 2024-05-20T18:09:38.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa56680a740 10 client.0 _ll_register_callbacks cb 0x5556abdaabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:38.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa55ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac13c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa55ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac13c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53428/0 (socket says 172.21.15.129:53428) 2024-05-20T18:09:38.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa55ce1f640 1 -- 172.21.15.129:0/3945064293 learned_addr learned my addr 172.21.15.129:0/3945064293 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:38.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa55d620640 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac13c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa55de21640 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 0x5556ac142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa55ce1f640 1 -- 172.21.15.129:0/3945064293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 msgr2=0x5556ac142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:38.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 0x5556ac142480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa55ce1f640 1 -- 172.21.15.129:0/3945064293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 msgr2=0x5556ac13c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.274+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac13c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa55ce1f640 1 -- 172.21.15.129:0/3945064293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556abe757d0 con 0x5556abd726b0 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac13c920 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0980b0 tx=0x7fa54c00c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12039a930d5ef52 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa53cff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa53cff9640 1 -- 172.21.15.129:0/3945064293 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c09bca0 con 0x5556abd726b0 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa53cff9640 1 -- 172.21.15.129:0/3945064293 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fa54c016df0 con 0x5556abd726b0 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa56680a740 1 -- 172.21.15.129:0/3945064293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa54c00a250 con 0x5556abd726b0 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa53cff9640 1 -- 172.21.15.129:0/3945064293 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa54c09c3a0 con 0x5556abd726b0 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa56680a740 10 client.17304 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa56680a740 1 -- 172.21.15.129:0/3945064293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5556ac14b790 con 0x5556abd726b0 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa53cff9640 1 -- 172.21.15.129:0/3945064293 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa54c00a250 con 0x5556abd726b0 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa53cff9640 1 -- 172.21.15.129:0/3945064293 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fa54c098310 con 0x5556abd726b0 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa56680a740 10 client.17304 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa56680a740 10 client.17304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa56680a740 20 client.17304 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa56680a740 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556abd51f60 0x5556ac14eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa56680a740 4 client.17304 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.275+0000 7fa56680a740 1 -- 172.21.15.129:0/3945064293 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556ac0a5370 con 0x5556abd51f60 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.276+0000 7fa55d620640 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556abd51f60 0x5556ac14eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.276+0000 7fa53cff9640 10 client.17304 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.276+0000 7fa55d620640 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556abd51f60 0x5556ac14eaf0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef3dc941c5dcfc5 server_cookie=6422e311cc494684 in_seq=0 out_seq=0 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa53cff9640 1 -- 172.21.15.129:0/3945064293 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5556ac0a5370 con 0x5556abd51f60 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa53cff9640 10 client.17304 handle_command_reply: tid=0 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 1 client.17304 shutdown 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 1 -- 172.21.15.129:0/3945064293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 msgr2=0x5556ac13c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:38.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac13c920 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0980b0 tx=0x7fa54c00c7d0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 1 -- 172.21.15.129:0/3945064293 shutdown_connections 2024-05-20T18:09:38.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556abd51f60 0x5556ac14eaf0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 0x5556ac142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac13c920 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa53cff9640 0 client.17304 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:38.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 1 --2- 172.21.15.129:0/3945064293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac13c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 1 -- 172.21.15.129:0/3945064293 >> 172.21.15.129:0/3945064293 conn(0x5556abd874e0 msgr2=0x5556ac133a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:38.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 1 -- 172.21.15.129:0/3945064293 shutdown_connections 2024-05-20T18:09:38.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 1 -- 172.21.15.129:0/3945064293 wait complete. 2024-05-20T18:09:38.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.277+0000 7fa56680a740 20 client.17304 trim_cache size 0 max 16384 2024-05-20T18:09:38.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa56680a740 1 Processor -- start 2024-05-20T18:09:38.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa56680a740 1 -- start start 2024-05-20T18:09:38.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa56680a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac13bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa56680a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1359d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa56680a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 0x5556ac135f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa56680a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581acc20 con 0x5556abd726b0 2024-05-20T18:09:38.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa56680a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581af4a0 con 0x5556abd6a4d0 2024-05-20T18:09:38.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa56680a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581ad290 con 0x5556abd93390 2024-05-20T18:09:38.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa55ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa55ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1359d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53432/0 (socket says 172.21.15.129:53432) 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2288426987 learned_addr learned my addr 172.21.15.129:0/2288426987 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa55d620640 1 --2- 172.21.15.129:0/2288426987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac13bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2288426987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 msgr2=0x5556ac135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/2288426987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 0x5556ac135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2288426987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 msgr2=0x5556ac13bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/2288426987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac13bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2288426987 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556ac13a720 con 0x5556abd726b0 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.291+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/2288426987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1359d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa54c098670 tx=0x7fa54c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72e9f934f25643a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa53effd640 1 -- 172.21.15.129:0/2288426987 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c09bca0 con 0x5556abd726b0 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa53effd640 1 -- 172.21.15.129:0/2288426987 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa54c099fa0 con 0x5556abd726b0 2024-05-20T18:09:38.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa53effd640 1 -- 172.21.15.129:0/2288426987 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c092240 con 0x5556abd726b0 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa56680a740 1 -- 172.21.15.129:0/2288426987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 msgr2=0x5556ac1359d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa56680a740 1 --2- 172.21.15.129:0/2288426987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1359d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa54c098670 tx=0x7fa54c00c840 comp rx=0 tx=0).stop 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa56680a740 1 -- 172.21.15.129:0/2288426987 shutdown_connections 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa56680a740 1 --2- 172.21.15.129:0/2288426987 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd93390 0x5556ac135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa56680a740 1 --2- 172.21.15.129:0/2288426987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1359d0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa56680a740 1 --2- 172.21.15.129:0/2288426987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd6a4d0 0x5556ac13bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa56680a740 1 -- 172.21.15.129:0/2288426987 >> 172.21.15.129:0/2288426987 conn(0x5556abd874e0 msgr2=0x5556ac1324e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa56680a740 1 -- 172.21.15.129:0/2288426987 shutdown_connections 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.292+0000 7fa56680a740 1 -- 172.21.15.129:0/2288426987 wait complete. 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa56680a740 1 -- start start 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa56680a740 1 Processor -- start 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa56680a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd6a4d0 0x5556ac1b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa56680a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa56680a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd93390 0x5556ac1ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa56680a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581ad960 con 0x5556abd726b0 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa56680a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581aedd0 con 0x5556abd93390 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa56680a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581ae030 con 0x5556abd6a4d0 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa56680a740 10 client.0 _ll_register_callbacks cb 0x7fa548009660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1b3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53446/0 (socket says 172.21.15.129:53446) 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2166758431 learned_addr learned my addr 172.21.15.129:0/2166758431 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55d620640 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd6a4d0 0x5556ac1b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55de21640 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd93390 0x5556ac1ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2166758431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd6a4d0 msgr2=0x5556ac1b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd6a4d0 0x5556ac1b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2166758431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd93390 msgr2=0x5556ac1ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd93390 0x5556ac1ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55ce1f640 1 -- 172.21.15.129:0/2166758431 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556ac1ad7d0 con 0x5556abd726b0 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55d620640 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd6a4d0 0x5556ac1b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.293+0000 7fa55de21640 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd93390 0x5556ac1ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa55ce1f640 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1b3310 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0cd4d0 tx=0x7fa54c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84a70d0bd420795 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa53cff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa53cff9640 1 -- 172.21.15.129:0/2166758431 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c0b8070 con 0x5556abd726b0 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa53cff9640 1 -- 172.21.15.129:0/2166758431 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fa54c09bf70 con 0x5556abd726b0 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa56680a740 1 -- 172.21.15.129:0/2166758431 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa54c099fa0 con 0x5556abd726b0 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa53cff9640 1 -- 172.21.15.129:0/2166758431 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa54c081050 con 0x5556abd726b0 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa56680a740 10 client.17310 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa56680a740 1 -- 172.21.15.129:0/2166758431 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5556ac1ad7d0 con 0x5556abd726b0 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa53cff9640 1 -- 172.21.15.129:0/2166758431 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa54c099fa0 con 0x5556abd726b0 2024-05-20T18:09:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa53cff9640 1 -- 172.21.15.129:0/2166758431 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fa54c09ab60 con 0x5556abd726b0 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa56680a740 10 client.17310 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa56680a740 10 client.17310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa56680a740 20 client.17310 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa56680a740 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556abd273a0 0x5556ac1bc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa56680a740 4 client.17310 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.294+0000 7fa56680a740 1 -- 172.21.15.129:0/2166758431 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5556ac09f6c0 con 0x5556abd273a0 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa55d620640 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556abd273a0 0x5556ac1bc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa53cff9640 10 client.17310 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa55d620640 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556abd273a0 0x5556ac1bc720 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e5d9a24ccf0308e server_cookie=3119c75b6278289d in_seq=0 out_seq=0 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa53cff9640 1 -- 172.21.15.129:0/2166758431 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5556ac09f6c0 con 0x5556abd273a0 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa53cff9640 10 client.17310 handle_command_reply: tid=0 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 1 client.17310 shutdown 2024-05-20T18:09:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 1 -- 172.21.15.129:0/2166758431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 msgr2=0x5556ac1b3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:38.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1b3310 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0cd4d0 tx=0x7fa54c017000 comp rx=0 tx=0).stop 2024-05-20T18:09:38.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 1 -- 172.21.15.129:0/2166758431 shutdown_connections 2024-05-20T18:09:38.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5556abd273a0 0x5556ac1bc720 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5556abd93390 0x5556ac1ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556abd726b0 0x5556ac1b3310 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa53cff9640 0 client.17310 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:38.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 1 --2- 172.21.15.129:0/2166758431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5556abd6a4d0 0x5556ac1b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 1 -- 172.21.15.129:0/2166758431 >> 172.21.15.129:0/2166758431 conn(0x5556abd874e0 msgr2=0x5556ac132240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:38.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 1 -- 172.21.15.129:0/2166758431 shutdown_connections 2024-05-20T18:09:38.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 1 -- 172.21.15.129:0/2166758431 wait complete. 2024-05-20T18:09:38.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.295+0000 7fa56680a740 20 client.17310 trim_cache size 0 max 16384 2024-05-20T18:09:38.301 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:09:38.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 -- 172.21.15.129:0/1056353219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5400a2d80 msgr2=0x7fa5400a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:38.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5400a2d80 0x7fa5400a5290 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x5556ac12ee90 tx=0x7fa55002a410 comp rx=0 tx=0).stop 2024-05-20T18:09:38.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 -- 172.21.15.129:0/1056353219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa558108b70 msgr2=0x7fa55819fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:38.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa558108b70 0x7fa55819fa80 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00e9f0 tx=0x7fa54c00eeb0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 -- 172.21.15.129:0/1056353219 shutdown_connections 2024-05-20T18:09:38.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fa5400a2d80 0x7fa5400a5290 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa558109010 0x7fa5581996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa558108b70 0x7fa55819fa80 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 --2- 172.21.15.129:0/1056353219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa5581017b0 0x7fa55819f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:38.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 -- 172.21.15.129:0/1056353219 >> 172.21.15.129:0/1056353219 conn(0x7fa5580768d0 msgr2=0x7fa5580ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:38.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 -- 172.21.15.129:0/1056353219 shutdown_connections 2024-05-20T18:09:38.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:38.296+0000 7fa55f884640 1 -- 172.21.15.129:0/1056353219 wait complete. 2024-05-20T18:09:39.305 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:09:39.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:09:39.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:09:39.333 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:38 smithi129 ceph-mon[32029]: pgmap v346: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:39.333 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:38 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1694838011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:09:39.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:38 smithi154 ceph-mon[36482]: pgmap v346: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:39.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:38 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1694838011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:09:39.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:38 smithi204 ceph-mon[37076]: pgmap v346: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:39.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:38 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1694838011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:09:39.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.437+0000 7f5eab5b1640 1 -- 172.21.15.129:0/3490778873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ea410c360 msgr2=0x7f5ea410e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.437+0000 7f5eab5b1640 1 --2- 172.21.15.129:0/3490778873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ea410c360 0x7f5ea410e780 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009ca0 tx=0x7f5e9c02a8c0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.437+0000 7f5eab5b1640 1 -- 172.21.15.129:0/3490778873 shutdown_connections 2024-05-20T18:09:39.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.437+0000 7f5eab5b1640 1 --2- 172.21.15.129:0/3490778873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ea410c360 0x7f5ea410e780 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.437+0000 7f5eab5b1640 1 --2- 172.21.15.129:0/3490778873 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea4101b20 0x7f5ea4101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.437+0000 7f5eab5b1640 1 --2- 172.21.15.129:0/3490778873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ea4107a00 0x7f5ea41015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.437+0000 7f5eab5b1640 1 -- 172.21.15.129:0/3490778873 >> 172.21.15.129:0/3490778873 conn(0x7f5ea40770c0 msgr2=0x7f5ea4077500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.437+0000 7f5eab5b1640 1 -- 172.21.15.129:0/3490778873 shutdown_connections 2024-05-20T18:09:39.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.437+0000 7f5eab5b1640 1 -- 172.21.15.129:0/3490778873 wait complete. 2024-05-20T18:09:39.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.438+0000 7f5eab5b1640 1 Processor -- start 2024-05-20T18:09:39.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.438+0000 7f5eab5b1640 1 -- start start 2024-05-20T18:09:39.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.438+0000 7f5eab5b1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ea4101b20 0x7f5ea419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.438+0000 7f5eab5b1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ea4107a00 0x7f5ea419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.438+0000 7f5eab5b1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea410c360 0x7f5ea419b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.438+0000 7f5eab5b1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea4111d30 con 0x7f5ea4101b20 2024-05-20T18:09:39.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.438+0000 7f5eab5b1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea4111bb0 con 0x7f5ea4107a00 2024-05-20T18:09:39.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.438+0000 7f5eab5b1640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea4111eb0 con 0x7f5ea410c360 2024-05-20T18:09:39.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea8b4c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ea4107a00 0x7f5ea419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea9b4e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea410c360 0x7f5ea419b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea8b4c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ea4107a00 0x7f5ea419ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:33608/0 (socket says 172.21.15.129:33608) 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/1837397423 learned_addr learned my addr 172.21.15.129:0/1837397423 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea934d640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ea4101b20 0x7f5ea419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/1837397423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea410c360 msgr2=0x7f5ea419b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea410c360 0x7f5ea419b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/1837397423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ea4101b20 msgr2=0x7f5ea419a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ea4101b20 0x7f5ea419a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/1837397423 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ea41a58e0 con 0x7f5ea4107a00 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea934d640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ea4101b20 0x7f5ea419a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ea4107a00 0x7f5ea419ab10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5e94005d00 tx=0x7f5e9400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5e9a7fc640 1 -- 172.21.15.129:0/1837397423 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94017070 con 0x7f5ea4107a00 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5eab5b1640 1 -- 172.21.15.129:0/1837397423 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ea41a5bd0 con 0x7f5ea4107a00 2024-05-20T18:09:39.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5e9a7fc640 1 -- 172.21.15.129:0/1837397423 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e94009c30 con 0x7f5ea4107a00 2024-05-20T18:09:39.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5eab5b1640 1 -- 172.21.15.129:0/1837397423 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ea41a6120 con 0x7f5ea4107a00 2024-05-20T18:09:39.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.439+0000 7f5e9a7fc640 1 -- 172.21.15.129:0/1837397423 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9400f620 con 0x7f5ea4107a00 2024-05-20T18:09:39.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5eb2458740 1 Processor -- start 2024-05-20T18:09:39.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5eb2458740 1 -- start start 2024-05-20T18:09:39.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677695e410 0x556776cfd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5e9a7fc640 1 -- 172.21.15.129:0/1837397423 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5e9401a070 con 0x7f5ea4107a00 2024-05-20T18:09:39.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776cfd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776cfdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5eb2458740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556776c62990 con 0x55677693d730 2024-05-20T18:09:39.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5eb2458740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556776c0a5b0 con 0x556776935550 2024-05-20T18:09:39.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5eb2458740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x556776c5e950 con 0x55677695e410 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5ea8b4c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776cfdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5ea8b4c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776cfdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53474/0 (socket says 172.21.15.129:53474) 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/1045963476 learned_addr learned my addr 172.21.15.129:0/1045963476 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5ea9b4e640 1 --2- 172.21.15.129:0/1045963476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677695e410 0x556776cfd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.442+0000 7f5ea934d640 1 --2- 172.21.15.129:0/1045963476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776cfd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/1045963476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677695e410 msgr2=0x556776cfd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/1045963476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677695e410 0x556776cfd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/1045963476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 msgr2=0x556776cfd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/1045963476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776cfd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/1045963476 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556776a3e7e0 con 0x55677693d730 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/1045963476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776cfdf30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f5e940cc480 tx=0x7f5e940d74b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac14b6545773689 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5e9a7fc640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5e900a2d00 0x7f5e900a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5e8f7fe640 1 -- 172.21.15.129:0/1045963476 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e940e2070 con 0x55677693d730 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5ea9b4e640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5e900a2d00 0x7f5e900a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5e8f7fe640 1 -- 172.21.15.129:0/1045963476 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e94095cb0 con 0x55677693d730 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5e8f7fe640 1 -- 172.21.15.129:0/1045963476 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94066500 con 0x55677693d730 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.443+0000 7f5e9a7fc640 1 -- 172.21.15.129:0/1837397423 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f5e940cee60 con 0x7f5ea4107a00 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5eb2458740 1 -- 172.21.15.129:0/1045963476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 msgr2=0x556776cfdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5eb2458740 1 --2- 172.21.15.129:0/1045963476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776cfdf30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f5e940cc480 tx=0x7f5e940d74b0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5ea9b4e640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5e900a2d00 0x7f5e900a5210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4074e60 tx=0x7f5e9c004a80 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5eb2458740 1 -- 172.21.15.129:0/1045963476 shutdown_connections 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5eb2458740 1 --2- 172.21.15.129:0/1045963476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776cfd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5eb2458740 1 --2- 172.21.15.129:0/1045963476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776cfdf30 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5eb2458740 1 --2- 172.21.15.129:0/1045963476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677695e410 0x556776cfd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5eb2458740 1 -- 172.21.15.129:0/1045963476 >> 172.21.15.129:0/1045963476 conn(0x556776952560 msgr2=0x556776cf9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5eb2458740 1 -- 172.21.15.129:0/1045963476 shutdown_connections 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5eb2458740 1 -- 172.21.15.129:0/1045963476 wait complete. 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5eb2458740 1 -- start start 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.444+0000 7f5eb2458740 1 Processor -- start 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556776935550 0x556776d0da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776d0dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55677695e410 0x556776d0bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5eb2458740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556776d0ed80 con 0x55677693d730 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5eb2458740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556776c1db40 con 0x55677695e410 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5eb2458740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x556776d0ef00 con 0x556776935550 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5eb2458740 10 client.0 _ll_register_callbacks cb 0x556776c120f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea8b4c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776d0dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea8b4c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776d0dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53486/0 (socket says 172.21.15.129:53486) 2024-05-20T18:09:39.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/3717645702 learned_addr learned my addr 172.21.15.129:0/3717645702 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea9b4e640 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55677695e410 0x556776d0bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea934d640 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556776935550 0x556776d0da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/3717645702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556776935550 msgr2=0x556776d0da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556776935550 0x556776d0da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/3717645702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55677695e410 msgr2=0x556776d0bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55677695e410 0x556776d0bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea8b4c640 1 -- 172.21.15.129:0/3717645702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556776a9ef70 con 0x55677693d730 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea9b4e640 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55677695e410 0x556776d0bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776d0dfd0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009610 tx=0x7f5e94010c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945aa874466ab2e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5e8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5e8cff9640 1 -- 172.21.15.129:0/3717645702 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94075b30 con 0x55677693d730 2024-05-20T18:09:39.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5e8cff9640 1 -- 172.21.15.129:0/3717645702 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f5e940d2a80 con 0x55677693d730 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5eb2458740 1 -- 172.21.15.129:0/3717645702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e94095cb0 con 0x55677693d730 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.445+0000 7f5e8cff9640 1 -- 172.21.15.129:0/3717645702 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5e94075d00 con 0x55677693d730 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.446+0000 7f5eb2458740 10 client.17322 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.446+0000 7f5eb2458740 1 -- 172.21.15.129:0/3717645702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556776936170 con 0x55677693d730 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.446+0000 7f5e8cff9640 1 -- 172.21.15.129:0/3717645702 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e94095cb0 con 0x55677693d730 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.446+0000 7f5e8cff9640 1 -- 172.21.15.129:0/3717645702 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f5e94097a30 con 0x55677693d730 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.446+0000 7f5eb2458740 10 client.17322 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.446+0000 7f5eb2458740 10 client.17322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.446+0000 7f5eb2458740 20 client.17322 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.446+0000 7f5eb2458740 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55677691d1f0 0x556776d1e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.446+0000 7f5eb2458740 4 client.17322 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.446+0000 7f5eb2458740 1 -- 172.21.15.129:0/3717645702 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556776c700c0 con 0x55677691d1f0 2024-05-20T18:09:39.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.446+0000 7f5ea934d640 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55677691d1f0 0x556776d1e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.447+0000 7f5ea934d640 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55677691d1f0 0x556776d1e3d0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f419dd75cc7ecf server_cookie=9056e1662e7a0e9 in_seq=0 out_seq=0 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.447+0000 7f5e8cff9640 10 client.17322 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5e8cff9640 1 -- 172.21.15.129:0/3717645702 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556776c700c0 con 0x55677691d1f0 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5e8cff9640 10 client.17322 handle_command_reply: tid=0 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 1 client.17322 shutdown 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 1 -- 172.21.15.129:0/3717645702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 msgr2=0x556776d0dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776d0dfd0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009610 tx=0x7f5e94010c70 comp rx=0 tx=0).stop 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 1 -- 172.21.15.129:0/3717645702 shutdown_connections 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55677691d1f0 0x556776d1e3d0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55677695e410 0x556776d0bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677693d730 0x556776d0dfd0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5e8cff9640 0 client.17322 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:39.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 1 --2- 172.21.15.129:0/3717645702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556776935550 0x556776d0da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 1 -- 172.21.15.129:0/3717645702 >> 172.21.15.129:0/3717645702 conn(0x556776952560 msgr2=0x556776cff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 1 -- 172.21.15.129:0/3717645702 shutdown_connections 2024-05-20T18:09:39.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 1 -- 172.21.15.129:0/3717645702 wait complete. 2024-05-20T18:09:39.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.448+0000 7f5eb2458740 20 client.17322 trim_cache size 0 max 16384 2024-05-20T18:09:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.466+0000 7f5eb2458740 1 Processor -- start 2024-05-20T18:09:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.466+0000 7f5eb2458740 1 -- start start 2024-05-20T18:09:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.466+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776cffa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.466+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677693d730 0x556776d05860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.466+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d05da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.466+0000 7f5eb2458740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea41aa9b0 con 0x55677695e410 2024-05-20T18:09:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.466+0000 7f5eb2458740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea41ad230 con 0x556776935550 2024-05-20T18:09:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.466+0000 7f5eb2458740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea41ab020 con 0x55677693d730 2024-05-20T18:09:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.466+0000 7f5ea9b4e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d05da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.466+0000 7f5ea9b4e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d05da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53502/0 (socket says 172.21.15.129:53502) 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.466+0000 7f5ea9b4e640 1 -- 172.21.15.129:0/3101544491 learned_addr learned my addr 172.21.15.129:0/3101544491 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5ea934d640 1 --2- 172.21.15.129:0/3101544491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776cffa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5ea9b4e640 1 -- 172.21.15.129:0/3101544491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677693d730 msgr2=0x556776d05860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/3101544491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677693d730 0x556776d05860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5ea9b4e640 1 --2- 172.21.15.129:0/3101544491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677693d730 0x556776d05860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5ea9b4e640 1 -- 172.21.15.129:0/3101544491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 msgr2=0x556776cffa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5ea9b4e640 1 --2- 172.21.15.129:0/3101544491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776cffa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5ea9b4e640 1 -- 172.21.15.129:0/3101544491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556776cff0f0 con 0x55677695e410 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5ea934d640 1 --2- 172.21.15.129:0/3101544491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776cffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5ea9b4e640 1 --2- 172.21.15.129:0/3101544491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d05da0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c02ade0 tx=0x7f5e9c041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930bbfed049e23e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5e8effd640 1 -- 172.21.15.129:0/3101544491 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c003e00 con 0x55677695e410 2024-05-20T18:09:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5e8effd640 1 -- 172.21.15.129:0/3101544491 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e9c036040 con 0x55677695e410 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.467+0000 7f5e8effd640 1 -- 172.21.15.129:0/3101544491 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c02faa0 con 0x55677695e410 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.468+0000 7f5eb2458740 1 -- 172.21.15.129:0/3101544491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 msgr2=0x556776d05da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.468+0000 7f5eb2458740 1 --2- 172.21.15.129:0/3101544491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d05da0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c02ade0 tx=0x7f5e9c041000 comp rx=0 tx=0).stop 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.468+0000 7f5eb2458740 1 -- 172.21.15.129:0/3101544491 shutdown_connections 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.468+0000 7f5eb2458740 1 --2- 172.21.15.129:0/3101544491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d05da0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.468+0000 7f5eb2458740 1 --2- 172.21.15.129:0/3101544491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677693d730 0x556776d05860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.468+0000 7f5eb2458740 1 --2- 172.21.15.129:0/3101544491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776cffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.468+0000 7f5eb2458740 1 -- 172.21.15.129:0/3101544491 >> 172.21.15.129:0/3101544491 conn(0x556776952560 msgr2=0x556776cff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.468+0000 7f5eb2458740 1 -- 172.21.15.129:0/3101544491 shutdown_connections 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.468+0000 7f5eb2458740 1 -- 172.21.15.129:0/3101544491 wait complete. 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.468+0000 7f5eb2458740 1 -- start start 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.468+0000 7f5eb2458740 1 Processor -- start 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776d21910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677693d730 0x556776d1ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5eb2458740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d1f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5eb2458740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea41ab6f0 con 0x55677695e410 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5eb2458740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea41acb60 con 0x556776935550 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5eb2458740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea41abdc0 con 0x55677693d730 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5eb2458740 10 client.0 _ll_register_callbacks cb 0x7f5e940e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea9b4e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d1f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea9b4e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d1f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53514/0 (socket says 172.21.15.129:53514) 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea9b4e640 1 -- 172.21.15.129:0/2341198499 learned_addr learned my addr 172.21.15.129:0/2341198499 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677693d730 0x556776d1ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea934d640 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776d21910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea9b4e640 1 -- 172.21.15.129:0/2341198499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677693d730 msgr2=0x556776d1ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea9b4e640 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677693d730 0x556776d1ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea9b4e640 1 -- 172.21.15.129:0/2341198499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 msgr2=0x556776d21910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea9b4e640 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776d21910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea9b4e640 1 -- 172.21.15.129:0/2341198499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556776d1f880 con 0x55677695e410 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea8b4c640 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677693d730 0x556776d1ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5ea9b4e640 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d1f1c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009a60 tx=0x7f5e9c031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348bd5c63ed11814 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5e8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5e8cff9640 1 -- 172.21.15.129:0/2341198499 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c02fd70 con 0x55677695e410 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5e8cff9640 1 -- 172.21.15.129:0/2341198499 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f5e9c0305c0 con 0x55677695e410 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.469+0000 7f5eb2458740 1 -- 172.21.15.129:0/2341198499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e9c036040 con 0x55677695e410 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5e8cff9640 1 -- 172.21.15.129:0/2341198499 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5e9c0307c0 con 0x55677695e410 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5eb2458740 10 client.17328 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5eb2458740 1 -- 172.21.15.129:0/2341198499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556776d863b0 con 0x55677695e410 2024-05-20T18:09:39.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5e8cff9640 1 -- 172.21.15.129:0/2341198499 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e9c036040 con 0x55677695e410 2024-05-20T18:09:39.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5e8cff9640 1 -- 172.21.15.129:0/2341198499 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f5e9c004560 con 0x55677695e410 2024-05-20T18:09:39.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5eb2458740 10 client.17328 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:39.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5eb2458740 10 client.17328 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:39.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5eb2458740 20 client.17328 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:39.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5eb2458740 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5567768f2630 0x556776d88ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5eb2458740 4 client.17328 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:09:39.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5eb2458740 1 -- 172.21.15.129:0/2341198499 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556776c6a410 con 0x5567768f2630 2024-05-20T18:09:39.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.470+0000 7f5ea934d640 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5567768f2630 0x556776d88ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.471+0000 7f5e8cff9640 10 client.17328 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:39.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.471+0000 7f5ea934d640 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5567768f2630 0x556776d88ae0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b52e2f37018e4a0 server_cookie=e4d3c9fb05979069 in_seq=0 out_seq=0 2024-05-20T18:09:39.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5e8cff9640 1 -- 172.21.15.129:0/2341198499 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556776c6a410 con 0x5567768f2630 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5e8cff9640 10 client.17328 handle_command_reply: tid=0 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 1 client.17328 shutdown 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 1 -- 172.21.15.129:0/2341198499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 msgr2=0x556776d1f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d1f1c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c009a60 tx=0x7f5e9c031ea0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 1 -- 172.21.15.129:0/2341198499 shutdown_connections 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5567768f2630 0x556776d88ae0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55677695e410 0x556776d1f1c0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55677693d730 0x556776d1ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5e8cff9640 0 client.17328 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 1 --2- 172.21.15.129:0/2341198499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556776935550 0x556776d21910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 1 -- 172.21.15.129:0/2341198499 >> 172.21.15.129:0/2341198499 conn(0x556776952560 msgr2=0x556776d04850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 1 -- 172.21.15.129:0/2341198499 shutdown_connections 2024-05-20T18:09:39.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 1 -- 172.21.15.129:0/2341198499 wait complete. 2024-05-20T18:09:39.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.472+0000 7f5eb2458740 20 client.17328 trim_cache size 0 max 16384 2024-05-20T18:09:39.477 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:09:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:09:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "80d1d9c2-80c2-4a71-b7d0-b3ce14f5bd35", 2024-05-20T18:09:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:09:39.477 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:09:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 -- 172.21.15.129:0/1837397423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5e900a2d00 msgr2=0x7f5e900a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5e900a2d00 0x7f5e900a5210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4074e60 tx=0x7f5e9c004a80 comp rx=0 tx=0).stop 2024-05-20T18:09:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 -- 172.21.15.129:0/1837397423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ea4107a00 msgr2=0x7f5ea419ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ea4107a00 0x7f5ea419ab10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5e94005d00 tx=0x7f5e9400dde0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 -- 172.21.15.129:0/1837397423 shutdown_connections 2024-05-20T18:09:39.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5e900a2d00 0x7f5e900a5210 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ea4107a00 0x7f5ea419ab10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5ea410c360 0x7f5ea419b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 --2- 172.21.15.129:0/1837397423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ea4101b20 0x7f5ea419a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 -- 172.21.15.129:0/1837397423 >> 172.21.15.129:0/1837397423 conn(0x7f5ea40770c0 msgr2=0x7f5ea4103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 -- 172.21.15.129:0/1837397423 shutdown_connections 2024-05-20T18:09:39.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.473+0000 7f5eab5b1640 1 -- 172.21.15.129:0/1837397423 wait complete. 2024-05-20T18:09:39.482 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:09:39.482 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:09:39.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.639+0000 7fb82ccbc640 1 -- 172.21.15.129:0/305667009 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb828108aa0 msgr2=0x7fb828100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.639+0000 7fb82ccbc640 1 --2- 172.21.15.129:0/305667009 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb828108aa0 0x7fb828100b20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009ca0 tx=0x7fb81c02a8c0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.639+0000 7fb82ccbc640 1 -- 172.21.15.129:0/305667009 shutdown_connections 2024-05-20T18:09:39.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.639+0000 7fb82ccbc640 1 --2- 172.21.15.129:0/305667009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8281016d0 0x7fb82810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.639+0000 7fb82ccbc640 1 --2- 172.21.15.129:0/305667009 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb828108f40 0x7fb828101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.639+0000 7fb82ccbc640 1 --2- 172.21.15.129:0/305667009 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb828108aa0 0x7fb828100b20 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.639+0000 7fb82ccbc640 1 -- 172.21.15.129:0/305667009 >> 172.21.15.129:0/305667009 conn(0x7fb828076830 msgr2=0x7fb828076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.639+0000 7fb82ccbc640 1 -- 172.21.15.129:0/305667009 shutdown_connections 2024-05-20T18:09:39.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.639+0000 7fb82ccbc640 1 -- 172.21.15.129:0/305667009 wait complete. 2024-05-20T18:09:39.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb82ccbc640 1 Processor -- start 2024-05-20T18:09:39.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb82ccbc640 1 -- start start 2024-05-20T18:09:39.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb82ccbc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8281016d0 0x7fb82819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb82ccbc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb828108aa0 0x7fb82819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb82ccbc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb828108f40 0x7fb8281a7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb82ccbc640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb828112e00 con 0x7fb828108f40 2024-05-20T18:09:39.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb82ccbc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb828112c80 con 0x7fb8281016d0 2024-05-20T18:09:39.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb82ccbc640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb828112f80 con 0x7fb828108aa0 2024-05-20T18:09:39.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb826d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb828108f40 0x7fb8281a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb826d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb828108f40 0x7fb8281a7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33138/0 (socket says 172.21.15.129:33138) 2024-05-20T18:09:39.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb826d9d640 1 -- 172.21.15.129:0/3049863644 learned_addr learned my addr 172.21.15.129:0/3049863644 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:39.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb82659c640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8281016d0 0x7fb82819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.640+0000 7fb825d9b640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb828108aa0 0x7fb82819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.641+0000 7fb826d9d640 1 -- 172.21.15.129:0/3049863644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb828108aa0 msgr2=0x7fb82819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.641+0000 7fb826d9d640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb828108aa0 0x7fb82819cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.641+0000 7fb826d9d640 1 -- 172.21.15.129:0/3049863644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8281016d0 msgr2=0x7fb82819c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.641+0000 7fb826d9d640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8281016d0 0x7fb82819c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.641+0000 7fb826d9d640 1 -- 172.21.15.129:0/3049863644 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8281a7ca0 con 0x7fb828108f40 2024-05-20T18:09:39.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.641+0000 7fb826d9d640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb828108f40 0x7fb8281a7760 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb82000c970 tx=0x7fb82000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.641+0000 7fb8177fe640 1 -- 172.21.15.129:0/3049863644 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb820007bb0 con 0x7fb828108f40 2024-05-20T18:09:39.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.641+0000 7fb8177fe640 1 -- 172.21.15.129:0/3049863644 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb820007d50 con 0x7fb828108f40 2024-05-20T18:09:39.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.641+0000 7fb82ccbc640 1 -- 172.21.15.129:0/3049863644 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8281a7e70 con 0x7fb828108f40 2024-05-20T18:09:39.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.641+0000 7fb8177fe640 1 -- 172.21.15.129:0/3049863644 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb820018800 con 0x7fb828108f40 2024-05-20T18:09:39.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.641+0000 7fb82ccbc640 1 -- 172.21.15.129:0/3049863644 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8281a82d0 con 0x7fb828108f40 2024-05-20T18:09:39.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.643+0000 7fb8177fe640 1 -- 172.21.15.129:0/3049863644 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fb82000f450 con 0x7fb828108f40 2024-05-20T18:09:39.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb8337fd740 1 Processor -- start 2024-05-20T18:09:39.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb8337fd740 1 -- start start 2024-05-20T18:09:39.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facde8390 0x564fad186d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad187270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdc76b0 0x564fad1877b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb8337fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564fad0ecd00 con 0x564facdbf4d0 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb8337fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564fad0944c0 con 0x564facdc76b0 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb8337fd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564fad0e8cc0 con 0x564facde8390 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb82659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad187270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb82659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad187270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33152/0 (socket says 172.21.15.129:33152) 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb82659c640 1 -- 172.21.15.129:0/3543630290 learned_addr learned my addr 172.21.15.129:0/3543630290 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb8177fe640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb8100a2d80 0x7fb8100a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb825d9b640 1 --2- 172.21.15.129:0/3543630290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facde8390 0x564fad186d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb82659c640 1 -- 172.21.15.129:0/3543630290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facde8390 msgr2=0x564fad186d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb82659c640 1 --2- 172.21.15.129:0/3543630290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facde8390 0x564fad186d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb82659c640 1 -- 172.21.15.129:0/3543630290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdc76b0 msgr2=0x564fad1877b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb825d9b640 1 --2- 172.21.15.129:0/3543630290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdc76b0 0x564fad1877b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb82659c640 1 --2- 172.21.15.129:0/3543630290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdc76b0 0x564fad1877b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb82659c640 1 -- 172.21.15.129:0/3543630290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564faceca7d0 con 0x564facdbf4d0 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb826d9d640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb8100a2d80 0x7fb8100a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.644+0000 7fb825d9b640 1 --2- 172.21.15.129:0/3543630290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdc76b0 0x564fad1877b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.645+0000 7fb8177fe640 1 -- 172.21.15.129:0/3049863644 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fb8200cf550 con 0x7fb828108f40 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.645+0000 7fb82659c640 1 --2- 172.21.15.129:0/3543630290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad187270 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009920 tx=0x7fb81c009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265ca116fd36004c server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.645+0000 7fb814ff9640 1 -- 172.21.15.129:0/3543630290 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c003970 con 0x564facdbf4d0 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.645+0000 7fb814ff9640 1 -- 172.21.15.129:0/3543630290 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb81c036050 con 0x564facdbf4d0 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.645+0000 7fb826d9d640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb8100a2d80 0x7fb8100a5290 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb82006eaa0 tx=0x7fb820086040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.645+0000 7fb814ff9640 1 -- 172.21.15.129:0/3543630290 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c0043c0 con 0x564facdbf4d0 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 -- 172.21.15.129:0/3543630290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 msgr2=0x564fad187270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3543630290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad187270 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb81c009920 tx=0x7fb81c009950 comp rx=0 tx=0).stop 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 -- 172.21.15.129:0/3543630290 shutdown_connections 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3543630290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdc76b0 0x564fad1877b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3543630290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad187270 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3543630290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facde8390 0x564fad186d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 -- 172.21.15.129:0/3543630290 >> 172.21.15.129:0/3543630290 conn(0x564facddc4e0 msgr2=0x564fad1834e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 -- 172.21.15.129:0/3543630290 shutdown_connections 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 -- 172.21.15.129:0/3543630290 wait complete. 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 -- start start 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 Processor -- start 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdbf4d0 0x564fad192f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdc76b0 0x564fad193440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad191330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564fad193ea0 con 0x564facde8390 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564fad0a7b90 con 0x564facdc76b0 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.646+0000 7fb8337fd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564fad194020 con 0x564facdbf4d0 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb8337fd740 10 client.0 _ll_register_callbacks cb 0x564fad0e3da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb825d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad191330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb825d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad191330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33162/0 (socket says 172.21.15.129:33162) 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb825d9b640 1 -- 172.21.15.129:0/986716801 learned_addr learned my addr 172.21.15.129:0/986716801 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb82659c640 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdc76b0 0x564fad193440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb825d9b640 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdbf4d0 0x564fad192f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb825d9b640 1 -- 172.21.15.129:0/986716801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdbf4d0 msgr2=0x564fad192f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb825d9b640 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdbf4d0 0x564fad192f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb825d9b640 1 -- 172.21.15.129:0/986716801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdc76b0 msgr2=0x564fad193440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb825d9b640 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdc76b0 0x564fad193440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb825d9b640 1 -- 172.21.15.129:0/986716801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564face003f0 con 0x564facde8390 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb825d9b640 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad191330 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb818005fd0 tx=0x7fb81800d470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e25702f1f6c7b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb80dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb80dffb640 1 -- 172.21.15.129:0/986716801 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818013e60 con 0x564facde8390 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb80dffb640 1 -- 172.21.15.129:0/986716801 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fb8180110d0 con 0x564facde8390 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb8337fd740 1 -- 172.21.15.129:0/986716801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb81c036050 con 0x564facde8390 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb80dffb640 1 -- 172.21.15.129:0/986716801 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb818025050 con 0x564facde8390 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb8337fd740 10 client.17340 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.647+0000 7fb8337fd740 1 -- 172.21.15.129:0/986716801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564facdc00f0 con 0x564facde8390 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.648+0000 7fb80dffb640 1 -- 172.21.15.129:0/986716801 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb81c036050 con 0x564facde8390 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.648+0000 7fb80dffb640 1 -- 172.21.15.129:0/986716801 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fb81800e3b0 con 0x564facde8390 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.648+0000 7fb8337fd740 10 client.17340 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.648+0000 7fb8337fd740 10 client.17340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.648+0000 7fb8337fd740 20 client.17340 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.648+0000 7fb8337fd740 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564facda6f60 0x564fad1a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.648+0000 7fb8337fd740 4 client.17340 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.648+0000 7fb8337fd740 1 -- 172.21.15.129:0/986716801 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564fad0fa430 con 0x564facda6f60 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.648+0000 7fb82659c640 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564facda6f60 0x564fad1a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.649+0000 7fb80dffb640 10 client.17340 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.649+0000 7fb82659c640 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564facda6f60 0x564fad1a39c0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb9c19ef1c67db5d server_cookie=c68143b3070586d9 in_seq=0 out_seq=0 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.649+0000 7fb80dffb640 1 -- 172.21.15.129:0/986716801 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564fad0fa430 con 0x564facda6f60 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.649+0000 7fb80dffb640 10 client.17340 handle_command_reply: tid=0 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.649+0000 7fb8337fd740 1 client.17340 shutdown 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.649+0000 7fb8337fd740 1 -- 172.21.15.129:0/986716801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 msgr2=0x564fad191330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.649+0000 7fb8337fd740 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad191330 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb818005fd0 tx=0x7fb81800d470 comp rx=0 tx=0).stop 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.649+0000 7fb8337fd740 1 -- 172.21.15.129:0/986716801 shutdown_connections 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.649+0000 7fb8337fd740 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564facda6f60 0x564fad1a39c0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.649+0000 7fb8337fd740 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdc76b0 0x564fad193440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.649+0000 7fb8337fd740 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad191330 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.650+0000 7fb80dffb640 0 client.17340 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:39.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.650+0000 7fb8337fd740 1 --2- 172.21.15.129:0/986716801 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdbf4d0 0x564fad192f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.650+0000 7fb8337fd740 1 -- 172.21.15.129:0/986716801 >> 172.21.15.129:0/986716801 conn(0x564facddc4e0 msgr2=0x564fad188a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.650+0000 7fb8337fd740 1 -- 172.21.15.129:0/986716801 shutdown_connections 2024-05-20T18:09:39.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.650+0000 7fb8337fd740 1 -- 172.21.15.129:0/986716801 wait complete. 2024-05-20T18:09:39.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.650+0000 7fb8337fd740 20 client.17340 trim_cache size 0 max 16384 2024-05-20T18:09:39.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.667+0000 7fb8337fd740 1 Processor -- start 2024-05-20T18:09:39.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.667+0000 7fb8337fd740 1 -- start start 2024-05-20T18:09:39.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad18c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdc76b0 0x564fad1874f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facde8390 0x564fad1a9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb8337fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb8281aeb80 con 0x564facdbf4d0 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb8337fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb8281b1400 con 0x564facde8390 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb8337fd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb8281af1f0 con 0x564facdc76b0 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb825d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad18c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb825d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad18c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33166/0 (socket says 172.21.15.129:33166) 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb825d9b640 1 -- 172.21.15.129:0/3771803724 learned_addr learned my addr 172.21.15.129:0/3771803724 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb825d9b640 1 -- 172.21.15.129:0/3771803724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdc76b0 msgr2=0x564fad1874f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb825d9b640 1 --2- 172.21.15.129:0/3771803724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdc76b0 0x564fad1874f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb825d9b640 1 -- 172.21.15.129:0/3771803724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facde8390 msgr2=0x564fad1a9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb825d9b640 1 --2- 172.21.15.129:0/3771803724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facde8390 0x564fad1a9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb825d9b640 1 -- 172.21.15.129:0/3771803724 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fad17d8c0 con 0x564facdbf4d0 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb825d9b640 1 --2- 172.21.15.129:0/3771803724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad18c6d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb818011c70 tx=0x7fb818011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce0160e64132e55 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb80ffff640 1 -- 172.21.15.129:0/3771803724 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81802e070 con 0x564facdbf4d0 2024-05-20T18:09:39.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb80ffff640 1 -- 172.21.15.129:0/3771803724 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb81800d4e0 con 0x564facdbf4d0 2024-05-20T18:09:39.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.668+0000 7fb80ffff640 1 -- 172.21.15.129:0/3771803724 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818013e60 con 0x564facdbf4d0 2024-05-20T18:09:39.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.669+0000 7fb8337fd740 1 -- 172.21.15.129:0/3771803724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 msgr2=0x564fad18c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.669+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3771803724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad18c6d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fb818011c70 tx=0x7fb818011ca0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.669+0000 7fb8337fd740 1 -- 172.21.15.129:0/3771803724 shutdown_connections 2024-05-20T18:09:39.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.669+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3771803724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facde8390 0x564fad1a9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.669+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3771803724 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdc76b0 0x564fad1874f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.669+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3771803724 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facdbf4d0 0x564fad18c6d0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.669+0000 7fb8337fd740 1 -- 172.21.15.129:0/3771803724 >> 172.21.15.129:0/3771803724 conn(0x564facddc4e0 msgr2=0x564fad189050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.669+0000 7fb8337fd740 1 -- 172.21.15.129:0/3771803724 shutdown_connections 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.669+0000 7fb8337fd740 1 -- 172.21.15.129:0/3771803724 wait complete. 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.669+0000 7fb8337fd740 1 -- start start 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb8337fd740 1 Processor -- start 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdbf4d0 0x564fad1990a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdc76b0 0x564fad1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb8337fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad196580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb8337fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb8281af8c0 con 0x564facde8390 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb8337fd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb8281b0d30 con 0x564facdbf4d0 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb8337fd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb8281aff90 con 0x564facdc76b0 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb8337fd740 10 client.0 _ll_register_callbacks cb 0x7fb81800d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb825d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb825d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33178/0 (socket says 172.21.15.129:33178) 2024-05-20T18:09:39.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb825d9b640 1 -- 172.21.15.129:0/3903669230 learned_addr learned my addr 172.21.15.129:0/3903669230 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb825d9b640 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdbf4d0 0x564fad1990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb82659c640 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdc76b0 0x564fad1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb825d9b640 1 -- 172.21.15.129:0/3903669230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdc76b0 msgr2=0x564fad1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb825d9b640 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdc76b0 0x564fad1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb825d9b640 1 -- 172.21.15.129:0/3903669230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdbf4d0 msgr2=0x564fad1990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb825d9b640 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdbf4d0 0x564fad1990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb825d9b640 1 -- 172.21.15.129:0/3903669230 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fad196bc0 con 0x564facde8390 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb82659c640 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdc76b0 0x564fad1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb825d9b640 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad196580 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb818011d90 tx=0x7fb818011210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e7a2c655420be7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.670+0000 7fb80dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb80dffb640 1 -- 172.21.15.129:0/3903669230 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81802e040 con 0x564facde8390 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb80dffb640 1 -- 172.21.15.129:0/3903669230 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fb818005250 con 0x564facde8390 2024-05-20T18:09:39.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb8337fd740 1 -- 172.21.15.129:0/3903669230 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb81800d4e0 con 0x564facde8390 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb80dffb640 1 -- 172.21.15.129:0/3903669230 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb81800e590 con 0x564facde8390 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb8337fd740 10 client.17346 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb8337fd740 1 -- 172.21.15.129:0/3903669230 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564fad20b660 con 0x564facde8390 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb80dffb640 1 -- 172.21.15.129:0/3903669230 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb81800d4e0 con 0x564facde8390 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb80dffb640 1 -- 172.21.15.129:0/3903669230 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fb818034030 con 0x564facde8390 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb8337fd740 10 client.17346 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb8337fd740 10 client.17346 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb8337fd740 20 client.17346 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564facd7c3a0 0x564fad20dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb8337fd740 4 client.17346 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb8337fd740 1 -- 172.21.15.129:0/3903669230 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564fad0f4780 con 0x564facd7c3a0 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.671+0000 7fb82659c640 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564facd7c3a0 0x564fad20dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:39.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb80dffb640 10 client.17346 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb82659c640 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564facd7c3a0 0x564fad20dc60 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5618aa3b8277b2f server_cookie=9b97d29b2aaade98 in_seq=0 out_seq=0 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb80dffb640 1 -- 172.21.15.129:0/3903669230 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564fad0f4780 con 0x564facd7c3a0 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb80dffb640 10 client.17346 handle_command_reply: tid=0 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 1 client.17346 shutdown 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 1 -- 172.21.15.129:0/3903669230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 msgr2=0x564fad196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad196580 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb818011d90 tx=0x7fb818011210 comp rx=0 tx=0).stop 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 1 -- 172.21.15.129:0/3903669230 shutdown_connections 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x564facd7c3a0 0x564fad20dc60 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564facde8390 0x564fad196580 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564facdc76b0 0x564fad1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb80dffb640 0 client.17346 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 1 --2- 172.21.15.129:0/3903669230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564facdbf4d0 0x564fad1990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 1 -- 172.21.15.129:0/3903669230 >> 172.21.15.129:0/3903669230 conn(0x564facddc4e0 msgr2=0x564fad189050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 1 -- 172.21.15.129:0/3903669230 shutdown_connections 2024-05-20T18:09:39.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 1 -- 172.21.15.129:0/3903669230 wait complete. 2024-05-20T18:09:39.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.672+0000 7fb8337fd740 20 client.17346 trim_cache size 0 max 16384 2024-05-20T18:09:39.677 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:09:39.677 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:09:39.677 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:09:39.677 INFO:teuthology.orchestra.run.smithi129.stdout: "80d1d9c2-80c2-4a71-b7d0-b3ce14f5bd35": { 2024-05-20T18:09:39.677 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:09:39.677 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "80d1d9c2-80c2-4a71-b7d0-b3ce14f5bd35", 2024-05-20T18:09:39.677 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:09:39.678 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:09:39.678 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:09:39.678 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:09:39.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 -- 172.21.15.129:0/3049863644 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb8100a2d80 msgr2=0x7fb8100a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb8100a2d80 0x7fb8100a5290 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb82006eaa0 tx=0x7fb820086040 comp rx=0 tx=0).stop 2024-05-20T18:09:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 -- 172.21.15.129:0/3049863644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb828108f40 msgr2=0x7fb8281a7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb828108f40 0x7fb8281a7760 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb82000c970 tx=0x7fb82000ce30 comp rx=0 tx=0).stop 2024-05-20T18:09:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 -- 172.21.15.129:0/3049863644 shutdown_connections 2024-05-20T18:09:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb8100a2d80 0x7fb8100a5290 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb828108f40 0x7fb8281a7760 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb828108aa0 0x7fb82819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 --2- 172.21.15.129:0/3049863644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8281016d0 0x7fb82819c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 -- 172.21.15.129:0/3049863644 >> 172.21.15.129:0/3049863644 conn(0x7fb828076830 msgr2=0x7fb8280ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 -- 172.21.15.129:0/3049863644 shutdown_connections 2024-05-20T18:09:39.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:09:39.673+0000 7fb82ccbc640 1 -- 172.21.15.129:0/3049863644 wait complete. 2024-05-20T18:09:39.682 INFO:tasks.cephfs.filesystem:scrub status for tag:80d1d9c2-80c2-4a71-b7d0-b3ce14f5bd35 - {'path': '/', 'tag': '80d1d9c2-80c2-4a71-b7d0-b3ce14f5bd35', 'options': 'recursive,force'} 2024-05-20T18:09:40.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:39 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:09:40.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:39 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:40.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:39 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:09:40.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:39 smithi154 ceph-mon[36482]: pgmap v347: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:40.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:39 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:09:40.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:39 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:40.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:39 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:09:40.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:39 smithi129 ceph-mon[32029]: pgmap v347: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:40.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:39 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:09:40.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:39 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:40.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:39 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:09:40.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:39 smithi204 ceph-mon[37076]: pgmap v347: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:41.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:40 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:41.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:40 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:09:41.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:40 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:41.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:40 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:09:41.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:40 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:09:41.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:40 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:09:42.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:41 smithi154 ceph-mon[36482]: pgmap v348: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:42.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:41 smithi129 ceph-mon[32029]: pgmap v348: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:42.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:41 smithi204 ceph-mon[37076]: pgmap v348: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:44.661 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.661+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/3404889016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d68107eb0 msgr2=0x7f9d68101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:44.661 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.661+0000 7f9d6cdaa640 1 --2- 172.21.15.129:0/3404889016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d68107eb0 0x7f9d68101b00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009900 tx=0x7f9d5002a410 comp rx=0 tx=0).stop 2024-05-20T18:09:44.661 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.661+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/3404889016 shutdown_connections 2024-05-20T18:09:44.661 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.661+0000 7f9d6cdaa640 1 --2- 172.21.15.129:0/3404889016 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d68102170 0x7f9d6810e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.661 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.661+0000 7f9d6cdaa640 1 --2- 172.21.15.129:0/3404889016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d68107eb0 0x7f9d68101b00 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.661 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.661+0000 7f9d6cdaa640 1 --2- 172.21.15.129:0/3404889016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d68107a10 0x7f9d681015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.661+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/3404889016 >> 172.21.15.129:0/3404889016 conn(0x7f9d68077160 msgr2=0x7f9d680775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.661+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/3404889016 shutdown_connections 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.661+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/3404889016 wait complete. 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d6cdaa640 1 Processor -- start 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d6cdaa640 1 -- start start 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d6cdaa640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d68102170 0x7f9d6819a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d6cdaa640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d68107a10 0x7f9d6819ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d6cdaa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d68107eb0 0x7f9d6819b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d6cdaa640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d68111d80 con 0x7f9d68102170 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d6cdaa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d68111c00 con 0x7f9d68107eb0 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d6cdaa640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d68111f00 con 0x7f9d68107a10 2024-05-20T18:09:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d6659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d68102170 0x7f9d6819a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d6659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d68102170 0x7f9d6819a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33208/0 (socket says 172.21.15.129:33208) 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d6659c640 1 -- 172.21.15.129:0/2994741404 learned_addr learned my addr 172.21.15.129:0/2994741404 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d68107eb0 0x7f9d6819b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.662+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d68107a10 0x7f9d6819ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d6659c640 1 -- 172.21.15.129:0/2994741404 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d68107a10 msgr2=0x7f9d6819ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d6659c640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d68107a10 0x7f9d6819ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d6659c640 1 -- 172.21.15.129:0/2994741404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d68107eb0 msgr2=0x7f9d6819b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d6659c640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d68107eb0 0x7f9d6819b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d6659c640 1 -- 172.21.15.129:0/2994741404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d681a5a00 con 0x7f9d68102170 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d68107a10 0x7f9d6819ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d6659c640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d68102170 0x7f9d6819a6c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800d950 tx=0x7f9d5800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d5ffff640 1 -- 172.21.15.129:0/2994741404 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58017070 con 0x7f9d68102170 2024-05-20T18:09:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d5ffff640 1 -- 172.21.15.129:0/2994741404 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d58009c30 con 0x7f9d68102170 2024-05-20T18:09:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/2994741404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d681a5cf0 con 0x7f9d68102170 2024-05-20T18:09:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d5ffff640 1 -- 172.21.15.129:0/2994741404 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d580047a0 con 0x7f9d68102170 2024-05-20T18:09:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.663+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/2994741404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d681a61e0 con 0x7f9d68102170 2024-05-20T18:09:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.666+0000 7f9d73c54740 1 Processor -- start 2024-05-20T18:09:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.666+0000 7f9d73c54740 1 -- start start 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.666+0000 7f9d5ffff640 1 -- 172.21.15.129:0/2994741404 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9d58004020 con 0x7f9d68102170 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac758f1b0 0x558ac792e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac75662f0 0x558ac792e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac756e4d0 0x558ac792eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d73c54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558ac7893340 con 0x558ac758f1b0 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d73c54740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558ac783b290 con 0x558ac756e4d0 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d73c54740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558ac788f300 con 0x558ac75662f0 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d66d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac758f1b0 0x558ac792e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d66d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac758f1b0 0x558ac792e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33214/0 (socket says 172.21.15.129:33214) 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d66d9d640 1 -- 172.21.15.129:0/781702000 learned_addr learned my addr 172.21.15.129:0/781702000 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d6659c640 1 --2- 172.21.15.129:0/781702000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac75662f0 0x558ac792e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/781702000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac756e4d0 0x558ac792eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d66d9d640 1 -- 172.21.15.129:0/781702000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac75662f0 msgr2=0x558ac792e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/781702000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac75662f0 0x558ac792e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d66d9d640 1 -- 172.21.15.129:0/781702000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac756e4d0 msgr2=0x558ac792eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/781702000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac756e4d0 0x558ac792eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d66d9d640 1 -- 172.21.15.129:0/781702000 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ac785f8e0 con 0x558ac758f1b0 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d6659c640 1 --2- 172.21.15.129:0/781702000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac75662f0 0x558ac792e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/781702000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac756e4d0 0x558ac792eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d5ffff640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d540a2cf0 0x7f9d540a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/781702000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac758f1b0 0x558ac792e080 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9d60009fa0 tx=0x7f9d60010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29de3dc6d3db1d1f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d6659c640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d540a2cf0 0x7f9d540a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d5cd99640 1 -- 172.21.15.129:0/781702000 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60016070 con 0x558ac758f1b0 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.667+0000 7f9d5ffff640 1 -- 172.21.15.129:0/2994741404 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f9d580cf040 con 0x7f9d68102170 2024-05-20T18:09:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.668+0000 7f9d5cd99640 1 -- 172.21.15.129:0/781702000 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d60012930 con 0x558ac758f1b0 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.668+0000 7f9d5cd99640 1 -- 172.21.15.129:0/781702000 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60011b50 con 0x558ac758f1b0 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.668+0000 7f9d73c54740 1 -- 172.21.15.129:0/781702000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac758f1b0 msgr2=0x558ac792e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.668+0000 7f9d6659c640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d540a2cf0 0x7f9d540a5200 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9d580cc830 tx=0x7f9d58005230 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.668+0000 7f9d73c54740 1 --2- 172.21.15.129:0/781702000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac758f1b0 0x558ac792e080 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9d60009fa0 tx=0x7f9d60010800 comp rx=0 tx=0).stop 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.668+0000 7f9d73c54740 1 -- 172.21.15.129:0/781702000 shutdown_connections 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.668+0000 7f9d73c54740 1 --2- 172.21.15.129:0/781702000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac756e4d0 0x558ac792eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.668+0000 7f9d73c54740 1 --2- 172.21.15.129:0/781702000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac75662f0 0x558ac792e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.668+0000 7f9d73c54740 1 --2- 172.21.15.129:0/781702000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac758f1b0 0x558ac792e080 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.668+0000 7f9d73c54740 1 -- 172.21.15.129:0/781702000 >> 172.21.15.129:0/781702000 conn(0x558ac7583300 msgr2=0x558ac792a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d73c54740 1 -- 172.21.15.129:0/781702000 shutdown_connections 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d73c54740 1 -- 172.21.15.129:0/781702000 wait complete. 2024-05-20T18:09:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d73c54740 1 -- start start 2024-05-20T18:09:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d73c54740 1 Processor -- start 2024-05-20T18:09:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac75662f0 0x558ac793a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac756e4d0 0x558ac793a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac7938580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d73c54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558ac793b1b0 con 0x558ac75662f0 2024-05-20T18:09:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d73c54740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558ac784e790 con 0x558ac756e4d0 2024-05-20T18:09:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d73c54740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558ac793b330 con 0x558ac758f1b0 2024-05-20T18:09:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d73c54740 10 client.0 _ll_register_callbacks cb 0x558ac7889a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d66d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac75662f0 0x558ac793a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d66d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac75662f0 0x558ac793a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33224/0 (socket says 172.21.15.129:33224) 2024-05-20T18:09:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.669+0000 7f9d66d9d640 1 -- 172.21.15.129:0/3897460654 learned_addr learned my addr 172.21.15.129:0/3897460654 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac756e4d0 0x558ac793a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d66d9d640 1 -- 172.21.15.129:0/3897460654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 msgr2=0x558ac7938580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac7938580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d66d9d640 1 -- 172.21.15.129:0/3897460654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac756e4d0 msgr2=0x558ac793a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac756e4d0 0x558ac793a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d66d9d640 1 -- 172.21.15.129:0/3897460654 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ac75a7210 con 0x558ac75662f0 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac75662f0 0x558ac793a090 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f9d600116d0 tx=0x7f9d600115b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414025a6afd01d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d4dffb640 1 -- 172.21.15.129:0/3897460654 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60016070 con 0x558ac75662f0 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d4dffb640 1 -- 172.21.15.129:0/3897460654 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f9d60004a40 con 0x558ac75662f0 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d73c54740 1 -- 172.21.15.129:0/3897460654 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d60012930 con 0x558ac75662f0 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d4dffb640 1 -- 172.21.15.129:0/3897460654 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9d60004c40 con 0x558ac75662f0 2024-05-20T18:09:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d73c54740 10 client.17358 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d73c54740 1 -- 172.21.15.129:0/3897460654 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558ac7673620 con 0x558ac75662f0 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.670+0000 7f9d4dffb640 1 -- 172.21.15.129:0/3897460654 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d60012930 con 0x558ac75662f0 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.671+0000 7f9d4dffb640 1 -- 172.21.15.129:0/3897460654 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f9d6000f040 con 0x558ac75662f0 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.671+0000 7f9d73c54740 10 client.17358 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.671+0000 7f9d73c54740 10 client.17358 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.671+0000 7f9d73c54740 20 client.17358 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.671+0000 7f9d73c54740 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558ac754dd80 0x558ac794a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.671+0000 7f9d73c54740 4 client.17358 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.671+0000 7f9d73c54740 1 -- 172.21.15.129:0/3897460654 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ac78a0a70 con 0x558ac754dd80 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.671+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558ac754dd80 0x558ac794a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d4dffb640 10 client.17358 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558ac754dd80 0x558ac794a860 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c495fbf1935bf0f server_cookie=a9fcb4cd73942775 in_seq=0 out_seq=0 2024-05-20T18:09:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d4dffb640 1 -- 172.21.15.129:0/3897460654 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558ac78a0a70 con 0x558ac754dd80 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d4dffb640 10 client.17358 handle_command_reply: tid=0 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d73c54740 1 client.17358 shutdown 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d73c54740 1 -- 172.21.15.129:0/3897460654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac75662f0 msgr2=0x558ac793a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d73c54740 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac75662f0 0x558ac793a090 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f9d600116d0 tx=0x7f9d600115b0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d73c54740 1 -- 172.21.15.129:0/3897460654 shutdown_connections 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d73c54740 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558ac754dd80 0x558ac794a860 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d73c54740 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac756e4d0 0x558ac793a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d73c54740 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac7938580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d4dffb640 0 client.17358 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d73c54740 1 --2- 172.21.15.129:0/3897460654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac75662f0 0x558ac793a090 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.672+0000 7f9d73c54740 1 -- 172.21.15.129:0/3897460654 >> 172.21.15.129:0/3897460654 conn(0x558ac7583300 msgr2=0x558ac792de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.673+0000 7f9d73c54740 1 -- 172.21.15.129:0/3897460654 shutdown_connections 2024-05-20T18:09:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.673+0000 7f9d73c54740 1 -- 172.21.15.129:0/3897460654 wait complete. 2024-05-20T18:09:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.673+0000 7f9d73c54740 20 client.17358 trim_cache size 0 max 16384 2024-05-20T18:09:44.690 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d73c54740 1 Processor -- start 2024-05-20T18:09:44.690 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d73c54740 1 -- start start 2024-05-20T18:09:44.690 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac75662f0 0x558ac793a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.690 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac794d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac794ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d73c54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d681acbc0 con 0x558ac756e4d0 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d73c54740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d681af440 con 0x558ac75662f0 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d73c54740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d681ad230 con 0x558ac758f1b0 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d5dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac794d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d5dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac794d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33230/0 (socket says 172.21.15.129:33230) 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d5dd9b640 1 -- 172.21.15.129:0/486477942 learned_addr learned my addr 172.21.15.129:0/486477942 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/486477942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac75662f0 0x558ac793a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/486477942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac794ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d5dd9b640 1 -- 172.21.15.129:0/486477942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 msgr2=0x558ac794ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/486477942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac794ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d5dd9b640 1 -- 172.21.15.129:0/486477942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac75662f0 msgr2=0x558ac793a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/486477942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac75662f0 0x558ac793a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d5dd9b640 1 -- 172.21.15.129:0/486477942 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ac794e310 con 0x558ac756e4d0 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.688+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/486477942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac794ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/486477942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac794d890 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9d50006010 tx=0x7f9d50032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0174b88f266047b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d4ffff640 1 -- 172.21.15.129:0/486477942 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d50035040 con 0x558ac756e4d0 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d4ffff640 1 -- 172.21.15.129:0/486477942 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d50005040 con 0x558ac756e4d0 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d4ffff640 1 -- 172.21.15.129:0/486477942 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d50004ae0 con 0x558ac756e4d0 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d73c54740 1 -- 172.21.15.129:0/486477942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 msgr2=0x558ac794d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d73c54740 1 --2- 172.21.15.129:0/486477942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac794d890 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9d50006010 tx=0x7f9d50032f60 comp rx=0 tx=0).stop 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d73c54740 1 -- 172.21.15.129:0/486477942 shutdown_connections 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d73c54740 1 --2- 172.21.15.129:0/486477942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac794ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d73c54740 1 --2- 172.21.15.129:0/486477942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac794d890 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d73c54740 1 --2- 172.21.15.129:0/486477942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac75662f0 0x558ac793a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d73c54740 1 -- 172.21.15.129:0/486477942 >> 172.21.15.129:0/486477942 conn(0x558ac7583300 msgr2=0x558ac7939ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.689+0000 7f9d73c54740 1 -- 172.21.15.129:0/486477942 shutdown_connections 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d73c54740 1 -- 172.21.15.129:0/486477942 wait complete. 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d73c54740 1 -- start start 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d73c54740 1 Processor -- start 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac75662f0 0x558ac79ab240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac79ab780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d73c54740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac79a5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d73c54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d681ad900 con 0x558ac756e4d0 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d73c54740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d681aed70 con 0x558ac75662f0 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d73c54740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9d681adfd0 con 0x558ac758f1b0 2024-05-20T18:09:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d73c54740 10 client.0 _ll_register_callbacks cb 0x7f9d60009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d5dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac79ab780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d5dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac79ab780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33234/0 (socket says 172.21.15.129:33234) 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.690+0000 7f9d5dd9b640 1 -- 172.21.15.129:0/1276627582 learned_addr learned my addr 172.21.15.129:0/1276627582 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac75662f0 0x558ac79ab240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac79a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d5dd9b640 1 -- 172.21.15.129:0/1276627582 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 msgr2=0x558ac79a5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac79a5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d5dd9b640 1 -- 172.21.15.129:0/1276627582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac75662f0 msgr2=0x558ac79ab240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac75662f0 0x558ac79ab240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d5dd9b640 1 -- 172.21.15.129:0/1276627582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ac79a5c40 con 0x558ac756e4d0 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac79a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d5dd9b640 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac79ab780 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9d50005f70 tx=0x7f9d500045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49bfcf52788a7798 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d4dffb640 1 -- 172.21.15.129:0/1276627582 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5004f070 con 0x558ac756e4d0 2024-05-20T18:09:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d4dffb640 1 -- 172.21.15.129:0/1276627582 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f9d500051f0 con 0x558ac756e4d0 2024-05-20T18:09:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d73c54740 1 -- 172.21.15.129:0/1276627582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d50005040 con 0x558ac756e4d0 2024-05-20T18:09:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d4dffb640 1 -- 172.21.15.129:0/1276627582 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9d5005b050 con 0x558ac756e4d0 2024-05-20T18:09:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d73c54740 10 client.17364 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d73c54740 1 -- 172.21.15.129:0/1276627582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558ac79b2860 con 0x558ac756e4d0 2024-05-20T18:09:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d4dffb640 1 -- 172.21.15.129:0/1276627582 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d50005040 con 0x558ac756e4d0 2024-05-20T18:09:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.691+0000 7f9d4dffb640 1 -- 172.21.15.129:0/1276627582 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f9d50042ce0 con 0x558ac756e4d0 2024-05-20T18:09:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.692+0000 7f9d73c54740 10 client.17364 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.692+0000 7f9d73c54740 10 client.17364 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.692+0000 7f9d73c54740 20 client.17364 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.692+0000 7f9d73c54740 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558ac75231c0 0x558ac79b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.692+0000 7f9d73c54740 4 client.17364 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:09:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.692+0000 7f9d73c54740 1 -- 172.21.15.129:0/1276627582 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558ac789adc0 con 0x558ac75231c0 2024-05-20T18:09:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.692+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558ac75231c0 0x558ac79b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.692+0000 7f9d4dffb640 10 client.17364 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:44.696 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.692+0000 7f9d66d9d640 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558ac75231c0 0x558ac79b4fd0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f569a35dff29768 server_cookie=b2e5fc046673eb7f in_seq=0 out_seq=0 2024-05-20T18:09:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d4dffb640 1 -- 172.21.15.129:0/1276627582 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558ac789adc0 con 0x558ac75231c0 2024-05-20T18:09:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d4dffb640 10 client.17364 handle_command_reply: tid=0 2024-05-20T18:09:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 1 client.17364 shutdown 2024-05-20T18:09:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 1 -- 172.21.15.129:0/1276627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 msgr2=0x558ac79ab780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac79ab780 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9d50005f70 tx=0x7f9d500045c0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 1 -- 172.21.15.129:0/1276627582 shutdown_connections 2024-05-20T18:09:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558ac75231c0 0x558ac79b4fd0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558ac758f1b0 0x558ac79a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558ac756e4d0 0x558ac79ab780 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d4dffb640 0 client.17364 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 1 --2- 172.21.15.129:0/1276627582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558ac75662f0 0x558ac79ab240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 1 -- 172.21.15.129:0/1276627582 >> 172.21.15.129:0/1276627582 conn(0x558ac7583300 msgr2=0x558ac7939c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 1 -- 172.21.15.129:0/1276627582 shutdown_connections 2024-05-20T18:09:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 1 -- 172.21.15.129:0/1276627582 wait complete. 2024-05-20T18:09:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.697+0000 7f9d73c54740 20 client.17364 trim_cache size 0 max 16384 2024-05-20T18:09:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:09:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:09:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:09:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:09:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/2994741404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d540a2cf0 msgr2=0x7f9d540a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d540a2cf0 0x7f9d540a5200 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9d580cc830 tx=0x7f9d58005230 comp rx=0 tx=0).stop 2024-05-20T18:09:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/2994741404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d68102170 msgr2=0x7f9d6819a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d68102170 0x7f9d6819a6c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800d950 tx=0x7f9d5800de10 comp rx=0 tx=0).stop 2024-05-20T18:09:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/2994741404 shutdown_connections 2024-05-20T18:09:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9d540a2cf0 0x7f9d540a5200 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d68107eb0 0x7f9d6819b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9d68107a10 0x7f9d6819ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 --2- 172.21.15.129:0/2994741404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d68102170 0x7f9d6819a6c0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/2994741404 >> 172.21.15.129:0/2994741404 conn(0x7f9d68077160 msgr2=0x7f9d681030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/2994741404 shutdown_connections 2024-05-20T18:09:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:44.698+0000 7f9d6cdaa640 1 -- 172.21.15.129:0/2994741404 wait complete. 2024-05-20T18:09:45.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:44 smithi129 ceph-mon[32029]: pgmap v349: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:45.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:44 smithi204 ceph-mon[37076]: pgmap v349: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:45.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:44 smithi154 ceph-mon[36482]: pgmap v349: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:47.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:46 smithi129 ceph-mon[32029]: pgmap v350: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-20T18:09:47.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:46 smithi204 ceph-mon[37076]: pgmap v350: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-20T18:09:47.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:46 smithi154 ceph-mon[36482]: pgmap v350: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-20T18:09:49.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:48 smithi129 ceph-mon[32029]: pgmap v351: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:49.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:48 smithi204 ceph-mon[37076]: pgmap v351: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:49.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:48 smithi154 ceph-mon[36482]: pgmap v351: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-20T18:09:51.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:50 smithi204 ceph-mon[37076]: pgmap v352: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-20T18:09:51.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:50 smithi154 ceph-mon[36482]: pgmap v352: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-20T18:09:51.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:50 smithi129 ceph-mon[32029]: pgmap v352: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-20T18:09:53.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:52 smithi204 ceph-mon[37076]: pgmap v353: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:53.213 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:52 smithi129 ceph-mon[32029]: pgmap v353: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:53.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:52 smithi154 ceph-mon[36482]: pgmap v353: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:55.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:54 smithi204 ceph-mon[37076]: pgmap v354: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:55.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:54 smithi154 ceph-mon[36482]: pgmap v354: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:55.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:54 smithi129 ceph-mon[32029]: pgmap v354: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:57.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:56 smithi204 ceph-mon[37076]: pgmap v355: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:57.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:56 smithi154 ceph-mon[36482]: pgmap v355: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:57.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:56 smithi129 ceph-mon[32029]: pgmap v355: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:09:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.810+0000 7fefec121640 1 -- 172.21.15.129:0/3940199517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefe4107a10 msgr2=0x7fefe41015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.810+0000 7fefec121640 1 --2- 172.21.15.129:0/3940199517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefe4107a10 0x7fefe41015c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009900 tx=0x7fefe002a410 comp rx=0 tx=0).stop 2024-05-20T18:09:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.810+0000 7fefec121640 1 -- 172.21.15.129:0/3940199517 shutdown_connections 2024-05-20T18:09:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.810+0000 7fefec121640 1 --2- 172.21.15.129:0/3940199517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe4102170 0x7fefe410e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.810+0000 7fefec121640 1 --2- 172.21.15.129:0/3940199517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefe4107eb0 0x7fefe4101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.810+0000 7fefec121640 1 --2- 172.21.15.129:0/3940199517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefe4107a10 0x7fefe41015c0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.810+0000 7fefec121640 1 -- 172.21.15.129:0/3940199517 >> 172.21.15.129:0/3940199517 conn(0x7fefe4077160 msgr2=0x7fefe40775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.810+0000 7fefec121640 1 -- 172.21.15.129:0/3940199517 shutdown_connections 2024-05-20T18:09:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.810+0000 7fefec121640 1 -- 172.21.15.129:0/3940199517 wait complete. 2024-05-20T18:09:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.810+0000 7fefec121640 1 Processor -- start 2024-05-20T18:09:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.810+0000 7fefec121640 1 -- start start 2024-05-20T18:09:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefec121640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefe4102170 0x7fefe41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefec121640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe4107a10 0x7fefe4198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefec121640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefe4107eb0 0x7fefe419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefec121640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fefe4111d60 con 0x7fefe4102170 2024-05-20T18:09:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefec121640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefe4111be0 con 0x7fefe4107a10 2024-05-20T18:09:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefec121640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fefe4111ee0 con 0x7fefe4107eb0 2024-05-20T18:09:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefe9ebd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefe4102170 0x7fefe41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.811 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefe9ebd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefe4102170 0x7fefe41986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45738/0 (socket says 172.21.15.129:45738) 2024-05-20T18:09:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefe9ebd640 1 -- 172.21.15.129:0/564177446 learned_addr learned my addr 172.21.15.129:0/564177446 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefea6be640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefe4107eb0 0x7fefe419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefe96bc640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe4107a10 0x7fefe4198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefe9ebd640 1 -- 172.21.15.129:0/564177446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefe4107eb0 msgr2=0x7fefe419cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefe9ebd640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefe4107eb0 0x7fefe419cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefe9ebd640 1 -- 172.21.15.129:0/564177446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe4107a10 msgr2=0x7fefe4198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefe9ebd640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe4107a10 0x7fefe4198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefe9ebd640 1 -- 172.21.15.129:0/564177446 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe419d540 con 0x7fefe4102170 2024-05-20T18:09:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefea6be640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefe4107eb0 0x7fefe419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefe96bc640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe4107a10 0x7fefe4198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:09:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.811+0000 7fefe9ebd640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefe4102170 0x7fefe41986e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fefe0005940 tx=0x7fefe00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.812+0000 7fefdaffd640 1 -- 172.21.15.129:0/564177446 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe003f030 con 0x7fefe4102170 2024-05-20T18:09:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.812+0000 7fefdaffd640 1 -- 172.21.15.129:0/564177446 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefe0003ce0 con 0x7fefe4102170 2024-05-20T18:09:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.812+0000 7fefec121640 1 -- 172.21.15.129:0/564177446 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe419d7d0 con 0x7fefe4102170 2024-05-20T18:09:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.812+0000 7fefdaffd640 1 -- 172.21.15.129:0/564177446 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe0035070 con 0x7fefe4102170 2024-05-20T18:09:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.812+0000 7fefec121640 1 -- 172.21.15.129:0/564177446 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe419dcc0 con 0x7fefe4102170 2024-05-20T18:09:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.814+0000 7fefdaffd640 1 -- 172.21.15.129:0/564177446 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fefe0042030 con 0x7fefe4102170 2024-05-20T18:09:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.814+0000 7feff2fc7740 1 Processor -- start 2024-05-20T18:09:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.814+0000 7feff2fc7740 1 -- start start 2024-05-20T18:09:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813771b0 0x55e681715ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e68134e2f0 0x55e681716410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e6813564d0 0x55e681716950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7feff2fc7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e68167b340 con 0x55e68134e2f0 2024-05-20T18:09:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7feff2fc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e681623290 con 0x55e6813771b0 2024-05-20T18:09:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7feff2fc7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e681677300 con 0x55e6813564d0 2024-05-20T18:09:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefea6be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813771b0 0x55e681715ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefdaffd640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fefd00a2cf0 0x7fefd00a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefe9ebd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e68134e2f0 0x55e681716410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefea6be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813771b0 0x55e681715ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:52104/0 (socket says 172.21.15.129:52104) 2024-05-20T18:09:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefea6be640 1 -- 172.21.15.129:0/497752652 learned_addr learned my addr 172.21.15.129:0/497752652 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefe96bc640 1 --2- 172.21.15.129:0/497752652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e6813564d0 0x55e681716950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefea6be640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fefd00a2cf0 0x7fefd00a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefdaffd640 1 -- 172.21.15.129:0/564177446 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fefe00efa80 con 0x7fefe4102170 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefe96bc640 1 -- 172.21.15.129:0/497752652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813771b0 msgr2=0x55e681715ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefe96bc640 1 --2- 172.21.15.129:0/497752652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813771b0 0x55e681715ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefe96bc640 1 -- 172.21.15.129:0/497752652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e68134e2f0 msgr2=0x55e681716410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefe96bc640 1 --2- 172.21.15.129:0/497752652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e68134e2f0 0x55e681716410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefe96bc640 1 -- 172.21.15.129:0/497752652 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6816478e0 con 0x55e6813564d0 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefe9ebd640 1 --2- 172.21.15.129:0/497752652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e68134e2f0 0x55e681716410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefe96bc640 1 --2- 172.21.15.129:0/497752652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e6813564d0 0x55e681716950 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fefd4005d00 tx=0x7fefd400e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1bf18822aca5f17 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.815+0000 7fefea6be640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fefd00a2cf0 0x7fefd00a5200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55e681713a90 tx=0x7fefdc004ab0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.816+0000 7fefcffff640 1 -- 172.21.15.129:0/497752652 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4016070 con 0x55e6813564d0 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.816+0000 7fefcffff640 1 -- 172.21.15.129:0/497752652 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd4011950 con 0x55e6813564d0 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.816+0000 7fefcffff640 1 -- 172.21.15.129:0/497752652 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4010830 con 0x55e6813564d0 2024-05-20T18:09:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.816+0000 7feff2fc7740 1 -- 172.21.15.129:0/497752652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e6813564d0 msgr2=0x55e681716950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.816+0000 7feff2fc7740 1 --2- 172.21.15.129:0/497752652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e6813564d0 0x55e681716950 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fefd4005d00 tx=0x7fefd400e6c0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 -- 172.21.15.129:0/497752652 shutdown_connections 2024-05-20T18:09:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 --2- 172.21.15.129:0/497752652 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e6813564d0 0x55e681716950 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 --2- 172.21.15.129:0/497752652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e68134e2f0 0x55e681716410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 --2- 172.21.15.129:0/497752652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813771b0 0x55e681715ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 -- 172.21.15.129:0/497752652 >> 172.21.15.129:0/497752652 conn(0x55e68136b300 msgr2=0x55e681712830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 -- 172.21.15.129:0/497752652 shutdown_connections 2024-05-20T18:09:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 -- 172.21.15.129:0/497752652 wait complete. 2024-05-20T18:09:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 -- start start 2024-05-20T18:09:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 Processor -- start 2024-05-20T18:09:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e681722950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 0x55e681722e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813771b0 0x55e6817288d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e681725420 con 0x55e6813564d0 2024-05-20T18:09:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e681636790 con 0x55e6813771b0 2024-05-20T18:09:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e6817255a0 con 0x55e68134e2f0 2024-05-20T18:09:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.817+0000 7feff2fc7740 10 client.0 _ll_register_callbacks cb 0x55e681671a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefe9ebd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e681722950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefe96bc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 0x55e681722e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefe9ebd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e681722950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:45152/0 (socket says 172.21.15.129:45152) 2024-05-20T18:09:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefe9ebd640 1 -- 172.21.15.129:0/402533682 learned_addr learned my addr 172.21.15.129:0/402533682 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefea6be640 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813771b0 0x55e6817288d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefe96bc640 1 -- 172.21.15.129:0/402533682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 msgr2=0x55e681722950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefe96bc640 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e681722950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefe96bc640 1 -- 172.21.15.129:0/402533682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813771b0 msgr2=0x55e6817288d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefe96bc640 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813771b0 0x55e6817288d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefe96bc640 1 -- 172.21.15.129:0/402533682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e68138f210 con 0x55e6813564d0 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefe9ebd640 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e681722950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefe96bc640 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 0x55e681722e90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fefd4000c00 tx=0x7fefd4009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3c96d056b108ce server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefcd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefcd7fa640 1 -- 172.21.15.129:0/402533682 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4016050 con 0x55e6813564d0 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefcd7fa640 1 -- 172.21.15.129:0/402533682 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fefd4005040 con 0x55e6813564d0 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7feff2fc7740 1 -- 172.21.15.129:0/402533682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefd4011950 con 0x55e6813564d0 2024-05-20T18:09:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7fefcd7fa640 1 -- 172.21.15.129:0/402533682 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fefd4030050 con 0x55e6813564d0 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7feff2fc7740 10 client.17376 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.818+0000 7feff2fc7740 1 -- 172.21.15.129:0/402533682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e68145b620 con 0x55e6813564d0 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.819+0000 7fefcd7fa640 1 -- 172.21.15.129:0/402533682 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd4011950 con 0x55e6813564d0 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.819+0000 7fefcd7fa640 1 -- 172.21.15.129:0/402533682 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fefd402b030 con 0x55e6813564d0 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.819+0000 7feff2fc7740 10 client.17376 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.819+0000 7feff2fc7740 10 client.17376 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.819+0000 7feff2fc7740 20 client.17376 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.819+0000 7feff2fc7740 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e681335d80 0x55e6817349f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.819+0000 7feff2fc7740 4 client.17376 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.819+0000 7feff2fc7740 1 -- 172.21.15.129:0/402533682 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e681688a70 con 0x55e681335d80 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.819+0000 7fefe9ebd640 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e681335d80 0x55e6817349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.820+0000 7fefcd7fa640 10 client.17376 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.820+0000 7fefe9ebd640 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e681335d80 0x55e6817349f0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a19b7e47489d472 server_cookie=4b890c9bf712cd2c in_seq=0 out_seq=0 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.820+0000 7fefcd7fa640 1 -- 172.21.15.129:0/402533682 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e681688a70 con 0x55e681335d80 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.820+0000 7fefcd7fa640 10 client.17376 handle_command_reply: tid=0 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.820+0000 7feff2fc7740 1 client.17376 shutdown 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.820+0000 7feff2fc7740 1 -- 172.21.15.129:0/402533682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 msgr2=0x55e681722e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.820+0000 7feff2fc7740 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 0x55e681722e90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fefd4000c00 tx=0x7fefd4009eb0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.821+0000 7feff2fc7740 1 -- 172.21.15.129:0/402533682 shutdown_connections 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.821+0000 7feff2fc7740 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e681335d80 0x55e6817349f0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.821+0000 7feff2fc7740 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813771b0 0x55e6817288d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.821+0000 7feff2fc7740 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 0x55e681722e90 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.821+0000 7fefcd7fa640 0 client.17376 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.821+0000 7feff2fc7740 1 --2- 172.21.15.129:0/402533682 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e681722950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.828 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.821+0000 7feff2fc7740 1 -- 172.21.15.129:0/402533682 >> 172.21.15.129:0/402533682 conn(0x55e68136b300 msgr2=0x55e681717d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.821+0000 7feff2fc7740 1 -- 172.21.15.129:0/402533682 shutdown_connections 2024-05-20T18:09:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.821+0000 7feff2fc7740 1 -- 172.21.15.129:0/402533682 wait complete. 2024-05-20T18:09:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.821+0000 7feff2fc7740 20 client.17376 trim_cache size 0 max 16384 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.836+0000 7feff2fc7740 1 Processor -- start 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.836+0000 7feff2fc7740 1 -- start start 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e681335d80 0x55e681724c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e681716840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 0x55e68171c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7feff2fc7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41aa940 con 0x55e6813564d0 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7feff2fc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41ad1c0 con 0x55e681335d80 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7feff2fc7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41aafb0 con 0x55e68134e2f0 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefea6be640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 0x55e68171c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefea6be640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 0x55e68171c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45756/0 (socket says 172.21.15.129:45756) 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefea6be640 1 -- 172.21.15.129:0/3659175636 learned_addr learned my addr 172.21.15.129:0/3659175636 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefe9ebd640 1 --2- 172.21.15.129:0/3659175636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e681335d80 0x55e681724c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefe96bc640 1 --2- 172.21.15.129:0/3659175636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e681716840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefea6be640 1 -- 172.21.15.129:0/3659175636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 msgr2=0x55e681716840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefea6be640 1 --2- 172.21.15.129:0/3659175636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e681716840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefea6be640 1 -- 172.21.15.129:0/3659175636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e681335d80 msgr2=0x55e681724c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefea6be640 1 --2- 172.21.15.129:0/3659175636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e681335d80 0x55e681724c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefea6be640 1 -- 172.21.15.129:0/3659175636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e681720cb0 con 0x55e6813564d0 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefea6be640 1 --2- 172.21.15.129:0/3659175636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 0x55e68171c0c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fefdc013f00 tx=0x7fefdc009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ebdeb25f23bc25a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefcf7fe640 1 -- 172.21.15.129:0/3659175636 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc004d20 con 0x55e6813564d0 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefcf7fe640 1 -- 172.21.15.129:0/3659175636 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefdc010db0 con 0x55e6813564d0 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.837+0000 7fefcf7fe640 1 -- 172.21.15.129:0/3659175636 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc0058c0 con 0x55e6813564d0 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.838+0000 7feff2fc7740 1 -- 172.21.15.129:0/3659175636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 msgr2=0x55e68171c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.838+0000 7feff2fc7740 1 --2- 172.21.15.129:0/3659175636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 0x55e68171c0c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fefdc013f00 tx=0x7fefdc009890 comp rx=0 tx=0).stop 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.838+0000 7feff2fc7740 1 -- 172.21.15.129:0/3659175636 shutdown_connections 2024-05-20T18:09:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.838+0000 7feff2fc7740 1 --2- 172.21.15.129:0/3659175636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e6813564d0 0x55e68171c0c0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.838+0000 7feff2fc7740 1 --2- 172.21.15.129:0/3659175636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e681716840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.838+0000 7feff2fc7740 1 --2- 172.21.15.129:0/3659175636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e681335d80 0x55e681724c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.838+0000 7feff2fc7740 1 -- 172.21.15.129:0/3659175636 >> 172.21.15.129:0/3659175636 conn(0x55e68136b300 msgr2=0x55e681718200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.838+0000 7feff2fc7740 1 -- 172.21.15.129:0/3659175636 shutdown_connections 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.838+0000 7feff2fc7740 1 -- 172.21.15.129:0/3659175636 wait complete. 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7feff2fc7740 1 -- start start 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7feff2fc7740 1 Processor -- start 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e681335d80 0x55e68172bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e68172c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7feff2fc7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813564d0 0x55e68179ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7feff2fc7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41ab680 con 0x55e681335d80 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7feff2fc7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41acaf0 con 0x55e6813564d0 2024-05-20T18:09:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7feff2fc7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41abd50 con 0x55e68134e2f0 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7feff2fc7740 10 client.0 _ll_register_callbacks cb 0x7fefe00f4040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7fefe9ebd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e681335d80 0x55e68172bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7fefe9ebd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e681335d80 0x55e68172bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45762/0 (socket says 172.21.15.129:45762) 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7fefe9ebd640 1 -- 172.21.15.129:0/1431444641 learned_addr learned my addr 172.21.15.129:0/1431444641 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7fefe96bc640 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e68172c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7fefea6be640 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813564d0 0x55e68179ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7fefe9ebd640 1 -- 172.21.15.129:0/1431444641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 msgr2=0x55e68172c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7fefe9ebd640 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e68172c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7fefe9ebd640 1 -- 172.21.15.129:0/1431444641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813564d0 msgr2=0x55e68179ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7fefe9ebd640 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813564d0 0x55e68179ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.839+0000 7fefe9ebd640 1 -- 172.21.15.129:0/1431444641 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e68172cb70 con 0x55e681335d80 2024-05-20T18:09:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7fefe9ebd640 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e681335d80 0x55e68172bdc0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fefe00bbf20 tx=0x7fefe00dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cce6d6235a6ce3d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7fefcd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7fefcd7fa640 1 -- 172.21.15.129:0/1431444641 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe0049040 con 0x55e681335d80 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7fefcd7fa640 1 -- 172.21.15.129:0/1431444641 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fefe00ed440 con 0x55e681335d80 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7feff2fc7740 1 -- 172.21.15.129:0/1431444641 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefdc010db0 con 0x55e681335d80 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7fefcd7fa640 1 -- 172.21.15.129:0/1431444641 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fefe00bc690 con 0x55e681335d80 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7feff2fc7740 10 client.17382 fetch_fsmap learned FSMap version 28 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7feff2fc7740 1 -- 172.21.15.129:0/1431444641 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e68179c020 con 0x55e681335d80 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7fefcd7fa640 1 -- 172.21.15.129:0/1431444641 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefdc010db0 con 0x55e681335d80 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7fefcd7fa640 1 -- 172.21.15.129:0/1431444641 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fefe003bc50 con 0x55e681335d80 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7feff2fc7740 10 client.17382 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7feff2fc7740 10 client.17382 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7feff2fc7740 20 client.17382 populate_metadata read hostname 'smithi129' 2024-05-20T18:09:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7feff2fc7740 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e6813771b0 0x55e68179e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:09:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7feff2fc7740 4 client.17382 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:09:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.840+0000 7feff2fc7740 1 -- 172.21.15.129:0/1431444641 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e681682dc0 con 0x55e6813771b0 2024-05-20T18:09:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.841+0000 7fefe96bc640 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e6813771b0 0x55e68179e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:09:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.841+0000 7fefe96bc640 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e6813771b0 0x55e68179e740 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f35c69f022f03b23 server_cookie=99ab07507030b7f0 in_seq=0 out_seq=0 2024-05-20T18:09:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.841+0000 7fefcd7fa640 10 client.17382 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7fefcd7fa640 1 -- 172.21.15.129:0/1431444641 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e681682dc0 con 0x55e6813771b0 2024-05-20T18:09:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7fefcd7fa640 10 client.17382 handle_command_reply: tid=0 2024-05-20T18:09:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 1 client.17382 shutdown 2024-05-20T18:09:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 1 -- 172.21.15.129:0/1431444641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e681335d80 msgr2=0x55e68172bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e681335d80 0x55e68172bdc0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fefe00bbf20 tx=0x7fefe00dc040 comp rx=0 tx=0).stop 2024-05-20T18:09:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 1 -- 172.21.15.129:0/1431444641 shutdown_connections 2024-05-20T18:09:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55e6813771b0 0x55e68179e740 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6813564d0 0x55e68179ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e68134e2f0 0x55e68172c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7fefcd7fa640 0 client.17382 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:09:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 1 --2- 172.21.15.129:0/1431444641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e681335d80 0x55e68172bdc0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 1 -- 172.21.15.129:0/1431444641 >> 172.21.15.129:0/1431444641 conn(0x55e68136b300 msgr2=0x55e681718200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 1 -- 172.21.15.129:0/1431444641 shutdown_connections 2024-05-20T18:09:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 1 -- 172.21.15.129:0/1431444641 wait complete. 2024-05-20T18:09:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7feff2fc7740 20 client.17382 trim_cache size 0 max 16384 2024-05-20T18:09:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:09:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:09:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:09:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:09:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7fefec121640 1 -- 172.21.15.129:0/564177446 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fefd00a2cf0 msgr2=0x7fefd00a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7fefec121640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fefd00a2cf0 0x7fefd00a5200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55e681713a90 tx=0x7fefdc004ab0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.846+0000 7fefec121640 1 -- 172.21.15.129:0/564177446 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefe4102170 msgr2=0x7fefe41986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:09:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.847+0000 7fefec121640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefe4102170 0x7fefe41986e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fefe0005940 tx=0x7fefe00042f0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.847+0000 7fefec121640 1 -- 172.21.15.129:0/564177446 shutdown_connections 2024-05-20T18:09:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.847+0000 7fefec121640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fefd00a2cf0 0x7fefd00a5200 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.847+0000 7fefec121640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe4107a10 0x7fefe4198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.847+0000 7fefec121640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fefe4107eb0 0x7fefe419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.847+0000 7fefec121640 1 --2- 172.21.15.129:0/564177446 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefe4102170 0x7fefe41986e0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:09:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.847+0000 7fefec121640 1 -- 172.21.15.129:0/564177446 >> 172.21.15.129:0/564177446 conn(0x7fefe4077160 msgr2=0x7fefe41030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:09:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.847+0000 7fefec121640 1 -- 172.21.15.129:0/564177446 shutdown_connections 2024-05-20T18:09:57.848 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:09:57.847+0000 7fefec121640 1 -- 172.21.15.129:0/564177446 wait complete. 2024-05-20T18:09:59.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:09:58 smithi204 ceph-mon[37076]: pgmap v356: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 0 B/s wr, 1 op/s 2024-05-20T18:09:59.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:09:58 smithi154 ceph-mon[36482]: pgmap v356: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 0 B/s wr, 1 op/s 2024-05-20T18:09:59.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:09:58 smithi129 ceph-mon[32029]: pgmap v356: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 0 B/s wr, 1 op/s 2024-05-20T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:00 smithi154 ceph-mon[36482]: pgmap v357: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-20T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:00 smithi154 ceph-mon[36482]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-05-20T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:00 smithi154 ceph-mon[36482]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-20T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:00 smithi154 ceph-mon[36482]: daemon mds.i on smithi204 is in unknown state 2024-05-20T18:10:01.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:00 smithi129 ceph-mon[32029]: pgmap v357: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-20T18:10:01.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:00 smithi129 ceph-mon[32029]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-05-20T18:10:01.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:00 smithi129 ceph-mon[32029]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-20T18:10:01.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:00 smithi129 ceph-mon[32029]: daemon mds.i on smithi204 is in unknown state 2024-05-20T18:10:01.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:00 smithi204 ceph-mon[37076]: pgmap v357: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-05-20T18:10:01.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:00 smithi204 ceph-mon[37076]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-05-20T18:10:01.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:00 smithi204 ceph-mon[37076]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-20T18:10:01.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:00 smithi204 ceph-mon[37076]: daemon mds.i on smithi204 is in unknown state 2024-05-20T18:10:03.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:02 smithi154 ceph-mon[36482]: pgmap v358: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:10:03.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:02 smithi129 ceph-mon[32029]: pgmap v358: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:10:03.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:02 smithi204 ceph-mon[37076]: pgmap v358: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-05-20T18:10:05.230 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:04 smithi204 ceph-mon[37076]: pgmap v359: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:10:05.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:04 smithi154 ceph-mon[36482]: pgmap v359: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:10:05.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:04 smithi129 ceph-mon[32029]: pgmap v359: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-05-20T18:10:07.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:06 smithi154 ceph-mon[36482]: pgmap v360: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:10:07.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:06 smithi129 ceph-mon[32029]: pgmap v360: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:10:07.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:06 smithi204 ceph-mon[37076]: pgmap v360: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:10:09.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:08 smithi154 ceph-mon[36482]: pgmap v361: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:10:09.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:08 smithi129 ceph-mon[32029]: pgmap v361: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:10:09.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:08 smithi204 ceph-mon[37076]: pgmap v361: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:10:09.683 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:10:09.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:10:09.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.815+0000 7f07a952f640 1 -- 172.21.15.129:0/4040514986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a41079d0 msgr2=0x7f07a4101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.815+0000 7f07a952f640 1 --2- 172.21.15.129:0/4040514986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a41079d0 0x7f07a4101560 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f0794009900 tx=0x7f079402a410 comp rx=0 tx=0).stop 2024-05-20T18:10:09.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.815+0000 7f07a952f640 1 -- 172.21.15.129:0/4040514986 shutdown_connections 2024-05-20T18:10:09.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.815+0000 7f07a952f640 1 --2- 172.21.15.129:0/4040514986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a4101aa0 0x7f07a4101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.815+0000 7f07a952f640 1 --2- 172.21.15.129:0/4040514986 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a410c330 0x7f07a410e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.815+0000 7f07a952f640 1 --2- 172.21.15.129:0/4040514986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a41079d0 0x7f07a4101560 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.815+0000 7f07a952f640 1 -- 172.21.15.129:0/4040514986 >> 172.21.15.129:0/4040514986 conn(0x7f07a40770b0 msgr2=0x7f07a40774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.815+0000 7f07a952f640 1 -- 172.21.15.129:0/4040514986 shutdown_connections 2024-05-20T18:10:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.815+0000 7f07a952f640 1 -- 172.21.15.129:0/4040514986 wait complete. 2024-05-20T18:10:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.816+0000 7f07a952f640 1 Processor -- start 2024-05-20T18:10:09.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.816+0000 7f07a952f640 1 -- start start 2024-05-20T18:10:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.816+0000 7f07a952f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a4101aa0 0x7f07a4199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.816+0000 7f07a952f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a41079d0 0x7f07a4199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.816+0000 7f07a952f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410c330 0x7f07a419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.816+0000 7f07a952f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07a4111ce0 con 0x7f07a4101aa0 2024-05-20T18:10:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.816+0000 7f07a952f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07a4111b60 con 0x7f07a410c330 2024-05-20T18:10:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.816+0000 7f07a952f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f07a4111e60 con 0x7f07a41079d0 2024-05-20T18:10:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.816+0000 7f07a2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a4101aa0 0x7f07a4199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a4101aa0 0x7f07a4199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43354/0 (socket says 172.21.15.129:43354) 2024-05-20T18:10:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f079bfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a41079d0 0x7f07a4199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a37fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410c330 0x7f07a419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a2ffd640 1 -- 172.21.15.129:0/3738135567 learned_addr learned my addr 172.21.15.129:0/3738135567 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a2ffd640 1 -- 172.21.15.129:0/3738135567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a41079d0 msgr2=0x7f07a4199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a2ffd640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a41079d0 0x7f07a4199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a2ffd640 1 -- 172.21.15.129:0/3738135567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410c330 msgr2=0x7f07a419df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a2ffd640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410c330 0x7f07a419df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a2ffd640 1 -- 172.21.15.129:0/3738135567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a419e5d0 con 0x7f07a4101aa0 2024-05-20T18:10:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a2ffd640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a4101aa0 0x7f07a4199770 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0794005940 tx=0x7f0794004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a0ff9640 1 -- 172.21.15.129:0/3738135567 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079403f030 con 0x7f07a4101aa0 2024-05-20T18:10:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a0ff9640 1 -- 172.21.15.129:0/3738135567 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0794003ad0 con 0x7f07a4101aa0 2024-05-20T18:10:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a952f640 1 -- 172.21.15.129:0/3738135567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a419e860 con 0x7f07a4101aa0 2024-05-20T18:10:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a0ff9640 1 -- 172.21.15.129:0/3738135567 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794035070 con 0x7f07a4101aa0 2024-05-20T18:10:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.817+0000 7f07a952f640 1 -- 172.21.15.129:0/3738135567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a419eda0 con 0x7f07a4101aa0 2024-05-20T18:10:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07a0ff9640 1 -- 172.21.15.129:0/3738135567 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f0794042030 con 0x7f07a4101aa0 2024-05-20T18:10:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07b03d5740 1 Processor -- start 2024-05-20T18:10:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07b03d5740 1 -- start start 2024-05-20T18:10:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560181158390 0x5601814f6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x5601814f7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601811376b0 0x5601814f77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07b03d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56018145cd00 con 0x56018112f4d0 2024-05-20T18:10:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07b03d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5601814044c0 con 0x5601811376b0 2024-05-20T18:10:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07b03d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560181458cc0 con 0x560181158390 2024-05-20T18:10:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07a0ff9640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07900a2d80 0x7f07900a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07a2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x5601814f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07a2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x5601814f7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43358/0 (socket says 172.21.15.129:43358) 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f07a2ffd640 1 -- 172.21.15.129:0/2012408276 learned_addr learned my addr 172.21.15.129:0/2012408276 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.820+0000 7f079bfff640 1 --2- 172.21.15.129:0/2012408276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601811376b0 0x5601814f77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f07a37fe640 1 --2- 172.21.15.129:0/2012408276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560181158390 0x5601814f6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f07a37fe640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07900a2d80 0x7f07900a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f07a0ff9640 1 -- 172.21.15.129:0/3738135567 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f07940efa80 con 0x7f07a4101aa0 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f079bfff640 1 -- 172.21.15.129:0/2012408276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560181158390 msgr2=0x5601814f6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f079bfff640 1 --2- 172.21.15.129:0/2012408276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560181158390 0x5601814f6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f079bfff640 1 -- 172.21.15.129:0/2012408276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 msgr2=0x5601814f7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f079bfff640 1 --2- 172.21.15.129:0/2012408276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x5601814f7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f079bfff640 1 -- 172.21.15.129:0/2012408276 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56018123a7d0 con 0x5601811376b0 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f07a37fe640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07900a2d80 0x7f07900a5290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x5601814f3ea0 tx=0x7f079c009500 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:09.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f079bfff640 1 --2- 172.21.15.129:0/2012408276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601811376b0 0x5601814f77b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f07a4074f80 tx=0x7f078c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88ce4a641bb504c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f07997fa640 1 -- 172.21.15.129:0/2012408276 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c004030 con 0x5601811376b0 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f07997fa640 1 -- 172.21.15.129:0/2012408276 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f078c005500 con 0x5601811376b0 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.821+0000 7f07997fa640 1 -- 172.21.15.129:0/2012408276 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c004c20 con 0x5601811376b0 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.822+0000 7f07b03d5740 1 -- 172.21.15.129:0/2012408276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601811376b0 msgr2=0x5601814f77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.822+0000 7f07b03d5740 1 --2- 172.21.15.129:0/2012408276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601811376b0 0x5601814f77b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f07a4074f80 tx=0x7f078c0107a0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.822+0000 7f07b03d5740 1 -- 172.21.15.129:0/2012408276 shutdown_connections 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.822+0000 7f07b03d5740 1 --2- 172.21.15.129:0/2012408276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601811376b0 0x5601814f77b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.822+0000 7f07b03d5740 1 --2- 172.21.15.129:0/2012408276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x5601814f7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.822+0000 7f07b03d5740 1 --2- 172.21.15.129:0/2012408276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560181158390 0x5601814f6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.822+0000 7f07b03d5740 1 -- 172.21.15.129:0/2012408276 >> 172.21.15.129:0/2012408276 conn(0x56018114c4e0 msgr2=0x5601814f34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.822+0000 7f07b03d5740 1 -- 172.21.15.129:0/2012408276 shutdown_connections 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.822+0000 7f07b03d5740 1 -- 172.21.15.129:0/2012408276 wait complete. 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07b03d5740 1 -- start start 2024-05-20T18:10:09.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07b03d5740 1 Processor -- start 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181502e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601811376b0 0x560181503360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560181158390 0x5601815012c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07b03d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560181503fc0 con 0x56018112f4d0 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07b03d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560181417b90 con 0x5601811376b0 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07b03d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560181504140 con 0x560181158390 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07b03d5740 10 client.0 _ll_register_callbacks cb 0x560181453da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07a2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181502e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07a2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181502e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43364/0 (socket says 172.21.15.129:43364) 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07a2ffd640 1 -- 172.21.15.129:0/518186562 learned_addr learned my addr 172.21.15.129:0/518186562 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f079bfff640 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601811376b0 0x560181503360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.823+0000 7f07a37fe640 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560181158390 0x5601815012c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f07a2ffd640 1 -- 172.21.15.129:0/518186562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560181158390 msgr2=0x5601815012c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f07a2ffd640 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560181158390 0x5601815012c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f07a2ffd640 1 -- 172.21.15.129:0/518186562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601811376b0 msgr2=0x560181503360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f07a2ffd640 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601811376b0 0x560181503360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f07a2ffd640 1 -- 172.21.15.129:0/518186562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601811703f0 con 0x56018112f4d0 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f07a2ffd640 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181502e20 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f07940bbd40 tx=0x7f07940b7c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21aa1e63c8bc24b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f078a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f078a7fc640 1 -- 172.21.15.129:0/518186562 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794049040 con 0x56018112f4d0 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f078a7fc640 1 -- 172.21.15.129:0/518186562 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f07940ed440 con 0x56018112f4d0 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f07b03d5740 1 -- 172.21.15.129:0/518186562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f078c005500 con 0x56018112f4d0 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f078a7fc640 1 -- 172.21.15.129:0/518186562 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f07940bb9a0 con 0x56018112f4d0 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f07b03d5740 10 client.17394 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f07b03d5740 1 -- 172.21.15.129:0/518186562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5601811300f0 con 0x56018112f4d0 2024-05-20T18:10:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f078a7fc640 1 -- 172.21.15.129:0/518186562 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f078c005500 con 0x56018112f4d0 2024-05-20T18:10:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f078a7fc640 1 -- 172.21.15.129:0/518186562 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f07940bc2a0 con 0x56018112f4d0 2024-05-20T18:10:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f07b03d5740 10 client.17394 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.824+0000 7f07b03d5740 10 client.17394 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.825+0000 7f07b03d5740 20 client.17394 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.825+0000 7f07b03d5740 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560181116f60 0x560181513b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.825+0000 7f07b03d5740 4 client.17394 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:10:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.825+0000 7f07b03d5740 1 -- 172.21.15.129:0/518186562 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56018146a430 con 0x560181116f60 2024-05-20T18:10:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.825+0000 7f079bfff640 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560181116f60 0x560181513b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.825+0000 7f078a7fc640 10 client.17394 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.825+0000 7f079bfff640 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560181116f60 0x560181513b50 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dc05bed28a58bbc server_cookie=9c0e6db5fe52443a in_seq=0 out_seq=0 2024-05-20T18:10:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f078a7fc640 1 -- 172.21.15.129:0/518186562 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56018146a430 con 0x560181116f60 2024-05-20T18:10:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f078a7fc640 10 client.17394 handle_command_reply: tid=0 2024-05-20T18:10:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 1 client.17394 shutdown 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 1 -- 172.21.15.129:0/518186562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 msgr2=0x560181502e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181502e20 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f07940bbd40 tx=0x7f07940b7c80 comp rx=0 tx=0).stop 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 1 -- 172.21.15.129:0/518186562 shutdown_connections 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x560181116f60 0x560181513b50 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601811376b0 0x560181503360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560181158390 0x5601815012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f078a7fc640 0 client.17394 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 1 --2- 172.21.15.129:0/518186562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181502e20 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 1 -- 172.21.15.129:0/518186562 >> 172.21.15.129:0/518186562 conn(0x56018114c4e0 msgr2=0x5601814f8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 1 -- 172.21.15.129:0/518186562 shutdown_connections 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 1 -- 172.21.15.129:0/518186562 wait complete. 2024-05-20T18:10:09.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.826+0000 7f07b03d5740 20 client.17394 trim_cache size 0 max 16384 2024-05-20T18:10:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.844+0000 7f07b03d5740 1 Processor -- start 2024-05-20T18:10:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.844+0000 7f07b03d5740 1 -- start start 2024-05-20T18:10:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.844+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018112f4d0 0x5601814fcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.844+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5601811376b0 0x560181501260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.844+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560181158390 0x560181519310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.844+0000 7f07b03d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07a41aca00 con 0x560181158390 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.844+0000 7f07b03d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07a41af280 con 0x56018112f4d0 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.844+0000 7f07b03d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f07a41ad070 con 0x5601811376b0 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f07a37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560181158390 0x560181519310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f07a37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560181158390 0x560181519310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43366/0 (socket says 172.21.15.129:43366) 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f07a37fe640 1 -- 172.21.15.129:0/2918687879 learned_addr learned my addr 172.21.15.129:0/2918687879 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f079bfff640 1 --2- 172.21.15.129:0/2918687879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5601811376b0 0x560181501260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f07a2ffd640 1 --2- 172.21.15.129:0/2918687879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018112f4d0 0x5601814fcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f07a37fe640 1 -- 172.21.15.129:0/2918687879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5601811376b0 msgr2=0x560181501260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f07a37fe640 1 --2- 172.21.15.129:0/2918687879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5601811376b0 0x560181501260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f07a37fe640 1 -- 172.21.15.129:0/2918687879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018112f4d0 msgr2=0x5601814fcaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f07a37fe640 1 --2- 172.21.15.129:0/2918687879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018112f4d0 0x5601814fcaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f07a37fe640 1 -- 172.21.15.129:0/2918687879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601814ff810 con 0x560181158390 2024-05-20T18:10:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f07a2ffd640 1 --2- 172.21.15.129:0/2918687879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018112f4d0 0x5601814fcaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f07a37fe640 1 --2- 172.21.15.129:0/2918687879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560181158390 0x560181519310 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f079c00fc20 tx=0x7f079c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f360d08864b5ecbd server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f0798ff9640 1 -- 172.21.15.129:0/2918687879 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079c016070 con 0x560181158390 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f0798ff9640 1 -- 172.21.15.129:0/2918687879 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f079c02b410 con 0x560181158390 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.845+0000 7f0798ff9640 1 -- 172.21.15.129:0/2918687879 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079c019040 con 0x560181158390 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.846+0000 7f07b03d5740 1 -- 172.21.15.129:0/2918687879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560181158390 msgr2=0x560181519310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.846+0000 7f07b03d5740 1 --2- 172.21.15.129:0/2918687879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560181158390 0x560181519310 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f079c00fc20 tx=0x7f079c015000 comp rx=0 tx=0).stop 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.846+0000 7f07b03d5740 1 -- 172.21.15.129:0/2918687879 shutdown_connections 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.846+0000 7f07b03d5740 1 --2- 172.21.15.129:0/2918687879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560181158390 0x560181519310 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.846+0000 7f07b03d5740 1 --2- 172.21.15.129:0/2918687879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5601811376b0 0x560181501260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.846+0000 7f07b03d5740 1 --2- 172.21.15.129:0/2918687879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018112f4d0 0x5601814fcaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.846+0000 7f07b03d5740 1 -- 172.21.15.129:0/2918687879 >> 172.21.15.129:0/2918687879 conn(0x56018114c4e0 msgr2=0x560181500b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.846+0000 7f07b03d5740 1 -- 172.21.15.129:0/2918687879 shutdown_connections 2024-05-20T18:10:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.846+0000 7f07b03d5740 1 -- 172.21.15.129:0/2918687879 wait complete. 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.846+0000 7f07b03d5740 1 -- start start 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.846+0000 7f07b03d5740 1 Processor -- start 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181506a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5601811376b0 0x560181506f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07b03d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560181158390 0x560181508f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07b03d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07a41ad740 con 0x56018112f4d0 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07b03d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07a41aebb0 con 0x560181158390 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07b03d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f07a41ade10 con 0x5601811376b0 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07b03d5740 10 client.0 _ll_register_callbacks cb 0x7f0794005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07a2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181506a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07a2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181506a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43382/0 (socket says 172.21.15.129:43382) 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07a2ffd640 1 -- 172.21.15.129:0/1904548727 learned_addr learned my addr 172.21.15.129:0/1904548727 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f079bfff640 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5601811376b0 0x560181506f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07a37fe640 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560181158390 0x560181508f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07a2ffd640 1 -- 172.21.15.129:0/1904548727 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5601811376b0 msgr2=0x560181506f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07a2ffd640 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5601811376b0 0x560181506f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07a2ffd640 1 -- 172.21.15.129:0/1904548727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560181158390 msgr2=0x560181508f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07a2ffd640 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560181158390 0x560181508f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07a2ffd640 1 -- 172.21.15.129:0/1904548727 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601815094c0 con 0x56018112f4d0 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07a37fe640 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560181158390 0x560181508f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07a2ffd640 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181506a00 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f07940ed270 tx=0x7f07940b8730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5489b553e799c398 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f078a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f078a7fc640 1 -- 172.21.15.129:0/1904548727 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794049070 con 0x56018112f4d0 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f078a7fc640 1 -- 172.21.15.129:0/1904548727 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f07940d4030 con 0x56018112f4d0 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.847+0000 7f07b03d5740 1 -- 172.21.15.129:0/1904548727 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f079c02b410 con 0x56018112f4d0 2024-05-20T18:10:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f078a7fc640 1 -- 172.21.15.129:0/1904548727 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f079409ae60 con 0x56018112f4d0 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f07b03d5740 10 client.17400 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f07b03d5740 1 -- 172.21.15.129:0/1904548727 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56018157b850 con 0x56018112f4d0 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f078a7fc640 1 -- 172.21.15.129:0/1904548727 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f079c02b410 con 0x56018112f4d0 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f078a7fc640 1 -- 172.21.15.129:0/1904548727 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f07940ed440 con 0x56018112f4d0 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f07b03d5740 10 client.17400 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f07b03d5740 10 client.17400 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f07b03d5740 20 client.17400 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f07b03d5740 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5601810ec3a0 0x56018157de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f07b03d5740 4 client.17400 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f07b03d5740 1 -- 172.21.15.129:0/1904548727 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560181464780 con 0x5601810ec3a0 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.848+0000 7f079bfff640 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5601810ec3a0 0x56018157de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f078a7fc640 10 client.17400 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f079bfff640 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5601810ec3a0 0x56018157de50 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8629053f458d165e server_cookie=6f78e5e316975734 in_seq=0 out_seq=0 2024-05-20T18:10:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f078a7fc640 1 -- 172.21.15.129:0/1904548727 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560181464780 con 0x5601810ec3a0 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f078a7fc640 10 client.17400 handle_command_reply: tid=0 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 1 client.17400 shutdown 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 1 -- 172.21.15.129:0/1904548727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 msgr2=0x560181506a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181506a00 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f07940ed270 tx=0x7f07940b8730 comp rx=0 tx=0).stop 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 1 -- 172.21.15.129:0/1904548727 shutdown_connections 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5601810ec3a0 0x56018157de50 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560181158390 0x560181508f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5601811376b0 0x560181506f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 1 --2- 172.21.15.129:0/1904548727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56018112f4d0 0x560181506a00 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f078a7fc640 0 client.17400 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 1 -- 172.21.15.129:0/1904548727 >> 172.21.15.129:0/1904548727 conn(0x56018114c4e0 msgr2=0x5601814ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 1 -- 172.21.15.129:0/1904548727 shutdown_connections 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 1 -- 172.21.15.129:0/1904548727 wait complete. 2024-05-20T18:10:09.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.849+0000 7f07b03d5740 20 client.17400 trim_cache size 0 max 16384 2024-05-20T18:10:09.854 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:10:09.854 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:10:09.854 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:10:09.854 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:10:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 -- 172.21.15.129:0/3738135567 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07900a2d80 msgr2=0x7f07900a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07900a2d80 0x7f07900a5290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x5601814f3ea0 tx=0x7f079c009500 comp rx=0 tx=0).stop 2024-05-20T18:10:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 -- 172.21.15.129:0/3738135567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a4101aa0 msgr2=0x7f07a4199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a4101aa0 0x7f07a4199770 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0794005940 tx=0x7f0794004270 comp rx=0 tx=0).stop 2024-05-20T18:10:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 -- 172.21.15.129:0/3738135567 shutdown_connections 2024-05-20T18:10:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f07900a2d80 0x7f07900a5290 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a410c330 0x7f07a419df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f07a41079d0 0x7f07a4199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 --2- 172.21.15.129:0/3738135567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07a4101aa0 0x7f07a4199770 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 -- 172.21.15.129:0/3738135567 >> 172.21.15.129:0/3738135567 conn(0x7f07a40770b0 msgr2=0x7f07a4068e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 -- 172.21.15.129:0/3738135567 shutdown_connections 2024-05-20T18:10:09.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.850+0000 7f07a952f640 1 -- 172.21.15.129:0/3738135567 wait complete. 2024-05-20T18:10:09.858 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:10:09.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:10:09.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.977+0000 7f7f61fb7640 1 -- 172.21.15.129:0/231980734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5c101010 msgr2=0x7f7f5c101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.977+0000 7f7f61fb7640 1 --2- 172.21.15.129:0/231980734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5c101010 0x7f7f5c101420 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009900 tx=0x7f7f5002a410 comp rx=0 tx=0).stop 2024-05-20T18:10:09.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.977+0000 7f7f61fb7640 1 -- 172.21.15.129:0/231980734 shutdown_connections 2024-05-20T18:10:09.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.977+0000 7f7f61fb7640 1 --2- 172.21.15.129:0/231980734 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f5c10d510 0x7f7f5c10f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.977+0000 7f7f61fb7640 1 --2- 172.21.15.129:0/231980734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5c101010 0x7f7f5c101420 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.977+0000 7f7f61fb7640 1 --2- 172.21.15.129:0/231980734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f5c108b60 0x7f7f5c100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.977+0000 7f7f61fb7640 1 -- 172.21.15.129:0/231980734 >> 172.21.15.129:0/231980734 conn(0x7f7f5c0fc800 msgr2=0x7f7f5c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.977+0000 7f7f61fb7640 1 -- 172.21.15.129:0/231980734 shutdown_connections 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.977+0000 7f7f61fb7640 1 -- 172.21.15.129:0/231980734 wait complete. 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f61fb7640 1 Processor -- start 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f61fb7640 1 -- start start 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f61fb7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f5c101010 0x7f7f5c1997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f61fb7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5c108b60 0x7f7f5c199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f61fb7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f5c10d510 0x7f7f5c19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f61fb7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c112e20 con 0x7f7f5c108b60 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f61fb7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c112ca0 con 0x7f7f5c101010 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f61fb7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5c112fa0 con 0x7f7f5c10d510 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f5affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5c108b60 0x7f7f5c199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f5affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5c108b60 0x7f7f5c199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43396/0 (socket says 172.21.15.129:43396) 2024-05-20T18:10:09.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f5affd640 1 -- 172.21.15.129:0/980675040 learned_addr learned my addr 172.21.15.129:0/980675040 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f5bfff640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f5c10d510 0x7f7f5c19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.978+0000 7f7f5b7fe640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f5c101010 0x7f7f5c1997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f5affd640 1 -- 172.21.15.129:0/980675040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f5c10d510 msgr2=0x7f7f5c19dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f5affd640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f5c10d510 0x7f7f5c19dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f5affd640 1 -- 172.21.15.129:0/980675040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f5c101010 msgr2=0x7f7f5c1997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f5affd640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f5c101010 0x7f7f5c1997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f5affd640 1 -- 172.21.15.129:0/980675040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5c19e600 con 0x7f7f5c108b60 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f5b7fe640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f5c101010 0x7f7f5c1997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f5affd640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5c108b60 0x7f7f5c199d00 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f7f50005940 tx=0x7f7f500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f58ff9640 1 -- 172.21.15.129:0/980675040 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5003f030 con 0x7f7f5c108b60 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f58ff9640 1 -- 172.21.15.129:0/980675040 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f50003ce0 con 0x7f7f5c108b60 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f61fb7640 1 -- 172.21.15.129:0/980675040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5c19e890 con 0x7f7f5c108b60 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f58ff9640 1 -- 172.21.15.129:0/980675040 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f50035070 con 0x7f7f5c108b60 2024-05-20T18:10:09.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.979+0000 7f7f61fb7640 1 -- 172.21.15.129:0/980675040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f5c19ed80 con 0x7f7f5c108b60 2024-05-20T18:10:09.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.980+0000 7f7f61fb7640 1 -- 172.21.15.129:0/980675040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f5c069140 con 0x7f7f5c108b60 2024-05-20T18:10:09.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.981+0000 7f7f58ff9640 1 -- 172.21.15.129:0/980675040 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7f50042030 con 0x7f7f5c108b60 2024-05-20T18:10:09.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.982+0000 7f7f58ff9640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7f440a2d00 0x7f7f440a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:09.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.982+0000 7f7f5b7fe640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7f440a2d00 0x7f7f440a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:09.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.982+0000 7f7f58ff9640 1 -- 172.21.15.129:0/980675040 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7f7f500efcb0 con 0x7f7f5c108b60 2024-05-20T18:10:09.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.983+0000 7f7f5b7fe640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7f440a2d00 0x7f7f440a5210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c005d00 tx=0x7f7f4c005ef0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:09.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:09.986+0000 7f7f58ff9640 1 -- 172.21.15.129:0/980675040 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7f500bc8c0 con 0x7f7f5c108b60 2024-05-20T18:10:10.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.203+0000 7f7f61fb7640 1 -- 172.21.15.129:0/980675040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f5c1a6330 con 0x7f7f5c108b60 2024-05-20T18:10:10.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.204+0000 7f7f58ff9640 1 -- 172.21.15.129:0/980675040 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11829 (secure 0 0 0) 0x7f7f500bc1c0 con 0x7f7f5c108b60 2024-05-20T18:10:10.202 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:10:10.202 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":26255,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/2724397229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":2724397229},{"type":"v1","addr":"172.21.15.154:6837","nonce":2724397229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:09:18.904334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26815":{"gid":26815,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/1721642062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":1721642062},{"type":"v1","addr":"172.21.15.204:6839","nonce":1721642062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:10:10.202 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 28 2024-05-20T18:10:10.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 -- 172.21.15.129:0/980675040 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7f440a2d00 msgr2=0x7f7f440a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7f440a2d00 0x7f7f440a5210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c005d00 tx=0x7f7f4c005ef0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 -- 172.21.15.129:0/980675040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5c108b60 msgr2=0x7f7f5c199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5c108b60 0x7f7f5c199d00 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f7f50005940 tx=0x7f7f500042f0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 -- 172.21.15.129:0/980675040 shutdown_connections 2024-05-20T18:10:10.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7f440a2d00 0x7f7f440a5210 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7f5c10d510 0x7f7f5c19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5c108b60 0x7f7f5c199d00 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 --2- 172.21.15.129:0/980675040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f5c101010 0x7f7f5c1997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 -- 172.21.15.129:0/980675040 >> 172.21.15.129:0/980675040 conn(0x7f7f5c0fc800 msgr2=0x7f7f5c0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:10.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 -- 172.21.15.129:0/980675040 shutdown_connections 2024-05-20T18:10:10.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.206+0000 7f7f61fb7640 1 -- 172.21.15.129:0/980675040 wait complete. 2024-05-20T18:10:10.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:10:10.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.331+0000 7fe92646e640 1 -- 172.21.15.129:0/4222717544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9201079d0 msgr2=0x7fe920101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.331+0000 7fe92646e640 1 --2- 172.21.15.129:0/4222717544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9201079d0 0x7fe920101560 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fe908009900 tx=0x7fe90802a410 comp rx=0 tx=0).stop 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.331+0000 7fe92646e640 1 -- 172.21.15.129:0/4222717544 shutdown_connections 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.331+0000 7fe92646e640 1 --2- 172.21.15.129:0/4222717544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe920101aa0 0x7fe920101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.331+0000 7fe92646e640 1 --2- 172.21.15.129:0/4222717544 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe92010c330 0x7fe92010e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.331+0000 7fe92646e640 1 --2- 172.21.15.129:0/4222717544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9201079d0 0x7fe920101560 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.331+0000 7fe92646e640 1 -- 172.21.15.129:0/4222717544 >> 172.21.15.129:0/4222717544 conn(0x7fe9200770b0 msgr2=0x7fe9200774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.331+0000 7fe92646e640 1 -- 172.21.15.129:0/4222717544 shutdown_connections 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.331+0000 7fe92646e640 1 -- 172.21.15.129:0/4222717544 wait complete. 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.331+0000 7fe92646e640 1 Processor -- start 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe92646e640 1 -- start start 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe92646e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe920101aa0 0x7fe920198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe92646e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9201079d0 0x7fe920198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe92646e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe92010c330 0x7fe92019ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe92646e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe920111ce0 con 0x7fe9201079d0 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe92646e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe920111b60 con 0x7fe92010c330 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe92646e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe920111e60 con 0x7fe920101aa0 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe91f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9201079d0 0x7fe920198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe91f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9201079d0 0x7fe920198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43412/0 (socket says 172.21.15.129:43412) 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1449813534 learned_addr learned my addr 172.21.15.129:0/1449813534 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe91ffff640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe920101aa0 0x7fe920198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe924a0b640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe92010c330 0x7fe92019ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1449813534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe920101aa0 msgr2=0x7fe920198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe920101aa0 0x7fe920198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1449813534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe92010c330 msgr2=0x7fe92019ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe92010c330 0x7fe92019ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1449813534 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92019d4c0 con 0x7fe9201079d0 2024-05-20T18:10:10.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9201079d0 0x7fe920198bc0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fe91000e9f0 tx=0x7fe91000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:10.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.332+0000 7fe91d7fa640 1 -- 172.21.15.129:0/1449813534 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91000ccc0 con 0x7fe9201079d0 2024-05-20T18:10:10.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.333+0000 7fe91d7fa640 1 -- 172.21.15.129:0/1449813534 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9100035f0 con 0x7fe9201079d0 2024-05-20T18:10:10.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.333+0000 7fe91d7fa640 1 -- 172.21.15.129:0/1449813534 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91000ce60 con 0x7fe9201079d0 2024-05-20T18:10:10.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.333+0000 7fe92646e640 1 -- 172.21.15.129:0/1449813534 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe92019d7b0 con 0x7fe9201079d0 2024-05-20T18:10:10.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.333+0000 7fe92646e640 1 -- 172.21.15.129:0/1449813534 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe92019dcb0 con 0x7fe9201079d0 2024-05-20T18:10:10.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.335+0000 7fe91d7fa640 1 -- 172.21.15.129:0/1449813534 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fe9100037a0 con 0x7fe9201079d0 2024-05-20T18:10:10.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.335+0000 7fe92d3ea740 1 Processor -- start 2024-05-20T18:10:10.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.335+0000 7fe92d3ea740 1 -- start start 2024-05-20T18:10:10.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c48390 0x5627f1fe6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c1f4d0 0x5627f1fe70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c276b0 0x5627f1fe75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5627f1f4cc40 con 0x5627f1c48390 2024-05-20T18:10:10.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5627f1ef4580 con 0x5627f1c1f4d0 2024-05-20T18:10:10.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5627f1f48c00 con 0x5627f1c276b0 2024-05-20T18:10:10.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe91d7fa640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe90c0a2d00 0x7fe90c0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe924a0b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c48390 0x5627f1fe6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe924a0b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c48390 0x5627f1fe6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43422/0 (socket says 172.21.15.129:43422) 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe924a0b640 1 -- 172.21.15.129:0/2544264493 learned_addr learned my addr 172.21.15.129:0/2544264493 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe924a0b640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe90c0a2d00 0x7fe90c0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe91ffff640 1 --2- 172.21.15.129:0/2544264493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c1f4d0 0x5627f1fe70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe91d7fa640 1 -- 172.21.15.129:0/1449813534 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fe9100cf810 con 0x7fe9201079d0 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe924a0b640 1 -- 172.21.15.129:0/2544264493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c276b0 msgr2=0x5627f1fe75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/2544264493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c276b0 0x5627f1fe75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe924a0b640 1 --2- 172.21.15.129:0/2544264493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c276b0 0x5627f1fe75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe924a0b640 1 -- 172.21.15.129:0/2544264493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c1f4d0 msgr2=0x5627f1fe70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe924a0b640 1 --2- 172.21.15.129:0/2544264493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c1f4d0 0x5627f1fe70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe924a0b640 1 -- 172.21.15.129:0/2544264493 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627f1c603f0 con 0x5627f1c48390 2024-05-20T18:10:10.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/2544264493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c276b0 0x5627f1fe75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.336+0000 7fe924a0b640 1 --2- 172.21.15.129:0/2544264493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c48390 0x5627f1fe6b70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fe918012da0 tx=0x7fe9180107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12f6a3cfb14af92 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.337+0000 7fe924a0b640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe90c0a2d00 0x7fe90c0a5210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe918005c40 tx=0x7fe91801d000 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.337+0000 7fe9167fc640 1 -- 172.21.15.129:0/2544264493 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91802a070 con 0x5627f1c48390 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.337+0000 7fe9167fc640 1 -- 172.21.15.129:0/2544264493 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe918003e50 con 0x5627f1c48390 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.337+0000 7fe9167fc640 1 -- 172.21.15.129:0/2544264493 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe918004050 con 0x5627f1c48390 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.337+0000 7fe92d3ea740 1 -- 172.21.15.129:0/2544264493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c48390 msgr2=0x5627f1fe6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.337+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/2544264493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c48390 0x5627f1fe6b70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fe918012da0 tx=0x7fe9180107a0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 -- 172.21.15.129:0/2544264493 shutdown_connections 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/2544264493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c1f4d0 0x5627f1fe70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/2544264493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c276b0 0x5627f1fe75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/2544264493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c48390 0x5627f1fe6b70 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 -- 172.21.15.129:0/2544264493 >> 172.21.15.129:0/2544264493 conn(0x5627f1c3c4e0 msgr2=0x5627f1fe34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:10.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 -- 172.21.15.129:0/2544264493 shutdown_connections 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 -- 172.21.15.129:0/2544264493 wait complete. 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 -- start start 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 Processor -- start 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c1f4d0 0x5627f1ff35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c276b0 0x5627f1ff3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c48390 0x5627f1ff9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5627f1ff6250 con 0x5627f1c276b0 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5627f1f07c40 con 0x5627f1c48390 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5627f1ff63d0 con 0x5627f1c1f4d0 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe92d3ea740 10 client.0 _ll_register_callbacks cb 0x5627f1c5fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe91f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c276b0 0x5627f1ff3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe91f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c276b0 0x5627f1ff3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43432/0 (socket says 172.21.15.129:43432) 2024-05-20T18:10:10.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.338+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1392789236 learned_addr learned my addr 172.21.15.129:0/1392789236 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe91ffff640 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c1f4d0 0x5627f1ff35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe924a0b640 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c48390 0x5627f1ff9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1392789236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c1f4d0 msgr2=0x5627f1ff35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c1f4d0 0x5627f1ff35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1392789236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c48390 msgr2=0x5627f1ff9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c48390 0x5627f1ff9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1392789236 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627f1d2a7d0 con 0x5627f1c276b0 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe924a0b640 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c48390 0x5627f1ff9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c276b0 0x5627f1ff3b10 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe91009ba20 tx=0x7fe9100a1320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f01d0cec4977ab server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe8ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/1392789236 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9100a00d0 con 0x5627f1c276b0 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/1392789236 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fe91009bf80 con 0x5627f1c276b0 2024-05-20T18:10:10.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1392789236 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe918003e50 con 0x5627f1c276b0 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/1392789236 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe9100a15e0 con 0x5627f1c276b0 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe92d3ea740 10 client.17418 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1392789236 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5627f20029a0 con 0x5627f1c276b0 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.339+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/1392789236 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe918003e50 con 0x5627f1c276b0 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.340+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/1392789236 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fe91007ab40 con 0x5627f1c276b0 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.340+0000 7fe92d3ea740 10 client.17418 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.340+0000 7fe92d3ea740 10 client.17418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.340+0000 7fe92d3ea740 20 client.17418 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.340+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627f1c06f60 0x5627f2005dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.340+0000 7fe92d3ea740 4 client.17418 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.340+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1392789236 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627f1f5a370 con 0x5627f1c06f60 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.340+0000 7fe91ffff640 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627f1c06f60 0x5627f2005dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.340+0000 7fe8ff7fe640 10 client.17418 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.340+0000 7fe91ffff640 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627f1c06f60 0x5627f2005dc0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7e1174135fabbc6 server_cookie=f33a334f17180d76 in_seq=0 out_seq=0 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/1392789236 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5627f1f5a370 con 0x5627f1c06f60 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe8ff7fe640 10 client.17418 handle_command_reply: tid=0 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe92d3ea740 1 client.17418 shutdown 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1392789236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c276b0 msgr2=0x5627f1ff3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c276b0 0x5627f1ff3b10 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe91009ba20 tx=0x7fe9100a1320 comp rx=0 tx=0).stop 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1392789236 shutdown_connections 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627f1c06f60 0x5627f2005dc0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c48390 0x5627f1ff9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe8ff7fe640 0 client.17418 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c276b0 0x5627f1ff3b10 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/1392789236 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c1f4d0 0x5627f1ff35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1392789236 >> 172.21.15.129:0/1392789236 conn(0x5627f1c3c4e0 msgr2=0x5627f1fe8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:10.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.341+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1392789236 shutdown_connections 2024-05-20T18:10:10.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.342+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1392789236 wait complete. 2024-05-20T18:10:10.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.342+0000 7fe92d3ea740 20 client.17418 trim_cache size 0 max 16384 2024-05-20T18:10:10.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe92d3ea740 1 Processor -- start 2024-05-20T18:10:10.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe92d3ea740 1 -- start start 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c06f60 0x5627f1fe92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c1f4d0 0x5627f1ff9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c276b0 0x5627f1fffb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe9201aa800 con 0x5627f1c1f4d0 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe9201ad080 con 0x5627f1c276b0 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe9201aae70 con 0x5627f1c06f60 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe91f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c1f4d0 0x5627f1ff9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe91f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c1f4d0 0x5627f1ff9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43442/0 (socket says 172.21.15.129:43442) 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1946357913 learned_addr learned my addr 172.21.15.129:0/1946357913 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.355+0000 7fe91ffff640 1 --2- 172.21.15.129:0/1946357913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c06f60 0x5627f1fe92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.356+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1946357913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c06f60 msgr2=0x5627f1fe92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.356+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/1946357913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c06f60 0x5627f1fe92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.356+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1946357913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c276b0 msgr2=0x5627f1fffb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:10.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.356+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/1946357913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c276b0 0x5627f1fffb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.356+0000 7fe91f7fe640 1 -- 172.21.15.129:0/1946357913 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627f1ff42e0 con 0x5627f1c1f4d0 2024-05-20T18:10:10.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.356+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/1946357913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c1f4d0 0x5627f1ff9640 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fe910098180 tx=0x7fe9100cfb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6963f7a01e9e2899 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:10.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.356+0000 7fe915ffb640 1 -- 172.21.15.129:0/1946357913 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91009b680 con 0x5627f1c1f4d0 2024-05-20T18:10:10.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.356+0000 7fe915ffb640 1 -- 172.21.15.129:0/1946357913 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9100a00d0 con 0x5627f1c1f4d0 2024-05-20T18:10:10.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.356+0000 7fe915ffb640 1 -- 172.21.15.129:0/1946357913 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910097fc0 con 0x5627f1c1f4d0 2024-05-20T18:10:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1946357913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c1f4d0 msgr2=0x5627f1ff9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/1946357913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c1f4d0 0x5627f1ff9640 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fe910098180 tx=0x7fe9100cfb80 comp rx=0 tx=0).stop 2024-05-20T18:10:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1946357913 shutdown_connections 2024-05-20T18:10:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/1946357913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c276b0 0x5627f1fffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/1946357913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c1f4d0 0x5627f1ff9640 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/1946357913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c06f60 0x5627f1fe92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1946357913 >> 172.21.15.129:0/1946357913 conn(0x5627f1c3c4e0 msgr2=0x5627f1fe6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1946357913 shutdown_connections 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 -- 172.21.15.129:0/1946357913 wait complete. 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 -- start start 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 Processor -- start 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c06f60 0x5627f1ff1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c1f4d0 0x5627f200af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c276b0 0x5627f200b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe9201ab540 con 0x5627f1c06f60 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe9201ac9b0 con 0x5627f1c1f4d0 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe9201abc10 con 0x5627f1c276b0 2024-05-20T18:10:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe92d3ea740 10 client.0 _ll_register_callbacks cb 0x7fe9100046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe91ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c06f60 0x5627f1ff1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe91ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c06f60 0x5627f1ff1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43450/0 (socket says 172.21.15.129:43450) 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe91ffff640 1 -- 172.21.15.129:0/865788594 learned_addr learned my addr 172.21.15.129:0/865788594 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.357+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c1f4d0 0x5627f200af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe924a0b640 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c276b0 0x5627f200b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe91ffff640 1 -- 172.21.15.129:0/865788594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c276b0 msgr2=0x5627f200b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe91ffff640 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c276b0 0x5627f200b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe91ffff640 1 -- 172.21.15.129:0/865788594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c1f4d0 msgr2=0x5627f200af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe91ffff640 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c1f4d0 0x5627f200af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe91ffff640 1 -- 172.21.15.129:0/865788594 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627f200bd20 con 0x5627f1c06f60 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c1f4d0 0x5627f200af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:10:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe91ffff640 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c06f60 0x5627f1ff1090 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe908009900 tx=0x7fe908031aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a18e00c25d83a12 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe8ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/865788594 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908035040 con 0x5627f1c06f60 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/865788594 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8543+0+0 (secure 0 0 0) 0x7fe908004450 con 0x5627f1c06f60 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe92d3ea740 1 -- 172.21.15.129:0/865788594 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9100a00d0 con 0x5627f1c06f60 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/865788594 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe908057050 con 0x5627f1c06f60 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe92d3ea740 10 client.17424 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe92d3ea740 1 -- 172.21.15.129:0/865788594 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5627f2070170 con 0x5627f1c06f60 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/865788594 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9100a00d0 con 0x5627f1c06f60 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.358+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/865788594 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fe908003aa0 con 0x5627f1c06f60 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe92d3ea740 10 client.17424 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe92d3ea740 10 client.17424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe92d3ea740 20 client.17424 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627f1c48390 0x5627f2072960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe92d3ea740 4 client.17424 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe92d3ea740 1 -- 172.21.15.129:0/865788594 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5627f1f546c0 con 0x5627f1c48390 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627f1c48390 0x5627f2072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe91f7fe640 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627f1c48390 0x5627f2072960 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44e9cbb38ed309ac server_cookie=25015ca9e4d02ec0 in_seq=0 out_seq=0 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe8ff7fe640 10 client.17424 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe8ff7fe640 1 -- 172.21.15.129:0/865788594 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5627f1f546c0 con 0x5627f1c48390 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe8ff7fe640 10 client.17424 handle_command_reply: tid=0 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe92d3ea740 1 client.17424 shutdown 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe92d3ea740 1 -- 172.21.15.129:0/865788594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c06f60 msgr2=0x5627f1ff1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.359+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c06f60 0x5627f1ff1090 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe908009900 tx=0x7fe908031aa0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92d3ea740 1 -- 172.21.15.129:0/865788594 shutdown_connections 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5627f1c48390 0x5627f2072960 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5627f1c1f4d0 0x5627f200af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe8ff7fe640 0 client.17424 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:10.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5627f1c276b0 0x5627f200b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92d3ea740 1 --2- 172.21.15.129:0/865788594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5627f1c06f60 0x5627f1ff1090 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92d3ea740 1 -- 172.21.15.129:0/865788594 >> 172.21.15.129:0/865788594 conn(0x5627f1c3c4e0 msgr2=0x5627f1fe74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:10.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92d3ea740 1 -- 172.21.15.129:0/865788594 shutdown_connections 2024-05-20T18:10:10.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92d3ea740 1 -- 172.21.15.129:0/865788594 wait complete. 2024-05-20T18:10:10.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92d3ea740 20 client.17424 trim_cache size 0 max 16384 2024-05-20T18:10:10.364 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:10:10.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 -- 172.21.15.129:0/1449813534 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe90c0a2d00 msgr2=0x7fe90c0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe90c0a2d00 0x7fe90c0a5210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe918005c40 tx=0x7fe91801d000 comp rx=0 tx=0).stop 2024-05-20T18:10:10.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 -- 172.21.15.129:0/1449813534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9201079d0 msgr2=0x7fe920198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9201079d0 0x7fe920198bc0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fe91000e9f0 tx=0x7fe91000eeb0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 -- 172.21.15.129:0/1449813534 shutdown_connections 2024-05-20T18:10:10.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fe90c0a2d00 0x7fe90c0a5210 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe92010c330 0x7fe92019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9201079d0 0x7fe920198bc0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 --2- 172.21.15.129:0/1449813534 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe920101aa0 0x7fe920198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 -- 172.21.15.129:0/1449813534 >> 172.21.15.129:0/1449813534 conn(0x7fe9200770b0 msgr2=0x7fe920103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 -- 172.21.15.129:0/1449813534 shutdown_connections 2024-05-20T18:10:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:10.360+0000 7fe92646e640 1 -- 172.21.15.129:0/1449813534 wait complete. 2024-05-20T18:10:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.952+0000 7f9ef5196640 1 -- 172.21.15.129:0/1938006239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef0108b20 msgr2=0x7f9ef01026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.952+0000 7f9ef5196640 1 --2- 172.21.15.129:0/1938006239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef0108b20 0x7f9ef01026c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0009900 tx=0x7f9ee002a410 comp rx=0 tx=0).stop 2024-05-20T18:10:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.952+0000 7f9ef5196640 1 -- 172.21.15.129:0/1938006239 shutdown_connections 2024-05-20T18:10:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.952+0000 7f9ef5196640 1 --2- 172.21.15.129:0/1938006239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef0103270 0x7f9ef010f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.952+0000 7f9ef5196640 1 --2- 172.21.15.129:0/1938006239 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ef0108fc0 0x7f9ef0102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.952+0000 7f9ef5196640 1 --2- 172.21.15.129:0/1938006239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef0108b20 0x7f9ef01026c0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.952+0000 7f9ef5196640 1 -- 172.21.15.129:0/1938006239 >> 172.21.15.129:0/1938006239 conn(0x7f9ef00fe1c0 msgr2=0x7f9ef0100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.952+0000 7f9ef5196640 1 -- 172.21.15.129:0/1938006239 shutdown_connections 2024-05-20T18:10:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.952+0000 7f9ef5196640 1 -- 172.21.15.129:0/1938006239 wait complete. 2024-05-20T18:10:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.953+0000 7f9ef5196640 1 Processor -- start 2024-05-20T18:10:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.953+0000 7f9ef5196640 1 -- start start 2024-05-20T18:10:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9ef5196640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef0103270 0x7f9ef019f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9ef5196640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef0108b20 0x7f9ef019fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9ef5196640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ef0108fc0 0x7f9ef0199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9ef5196640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef0112eb0 con 0x7f9ef0103270 2024-05-20T18:10:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9ef5196640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef0112d30 con 0x7f9ef0108b20 2024-05-20T18:10:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9ef5196640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef0113030 con 0x7f9ef0108fc0 2024-05-20T18:10:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9eeed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef0103270 0x7f9ef019f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9eeed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef0103270 0x7f9ef019f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43474/0 (socket says 172.21.15.129:43474) 2024-05-20T18:10:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9eeed9d640 1 -- 172.21.15.129:0/947633967 learned_addr learned my addr 172.21.15.129:0/947633967 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9eee59c640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef0108b20 0x7f9ef019fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9eef59e640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ef0108fc0 0x7f9ef0199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9eeed9d640 1 -- 172.21.15.129:0/947633967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ef0108fc0 msgr2=0x7f9ef0199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9eeed9d640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ef0108fc0 0x7f9ef0199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9eeed9d640 1 -- 172.21.15.129:0/947633967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef0108b20 msgr2=0x7f9ef019fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9eeed9d640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef0108b20 0x7f9ef019fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9eeed9d640 1 -- 172.21.15.129:0/947633967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef0199da0 con 0x7f9ef0103270 2024-05-20T18:10:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.954+0000 7f9eeed9d640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef0103270 0x7f9ef019f4f0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f9ee002a930 tx=0x7f9ee0003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.962+0000 7f9ed7fff640 1 -- 172.21.15.129:0/947633967 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee003f070 con 0x7f9ef0103270 2024-05-20T18:10:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.962+0000 7f9ed7fff640 1 -- 172.21.15.129:0/947633967 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee0031c90 con 0x7f9ef0103270 2024-05-20T18:10:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.962+0000 7f9ed7fff640 1 -- 172.21.15.129:0/947633967 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0035030 con 0x7f9ef0103270 2024-05-20T18:10:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.963+0000 7f9ef5196640 1 -- 172.21.15.129:0/947633967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef0199fd0 con 0x7f9ef0103270 2024-05-20T18:10:10.962 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.963+0000 7f9ef5196640 1 -- 172.21.15.129:0/947633967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef019a440 con 0x7f9ef0103270 2024-05-20T18:10:10.966 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.969+0000 7f9efc044740 1 Processor -- start 2024-05-20T18:10:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.969+0000 7f9efc044740 1 -- start start 2024-05-20T18:10:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.970+0000 7f9efc044740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49971b0 0x5595d4d36080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.970+0000 7f9efc044740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d496e2f0 0x5595d4d365c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.970+0000 7f9efc044740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49764d0 0x5595d4d36b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.970+0000 7f9eef59e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49971b0 0x5595d4d36080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.970+0000 7f9eef59e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49971b0 0x5595d4d36080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:42332/0 (socket says 172.21.15.129:42332) 2024-05-20T18:10:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.970+0000 7f9eef59e640 1 -- 172.21.15.129:0/1938112820 learned_addr learned my addr 172.21.15.129:0/1938112820 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.970+0000 7f9efc044740 1 -- 172.21.15.129:0/1938112820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5595d4c9b340 con 0x5595d496e2f0 2024-05-20T18:10:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.970+0000 7f9efc044740 1 -- 172.21.15.129:0/1938112820 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5595d4c43290 con 0x5595d49764d0 2024-05-20T18:10:10.968 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.971+0000 7f9efc044740 1 -- 172.21.15.129:0/1938112820 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5595d4c97300 con 0x5595d49971b0 2024-05-20T18:10:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.972+0000 7f9eef59e640 1 -- 172.21.15.129:0/1938112820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49764d0 msgr2=0x5595d4d36b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:10:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.972+0000 7f9eef59e640 1 --2- 172.21.15.129:0/1938112820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49764d0 0x5595d4d36b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.972+0000 7f9eef59e640 1 -- 172.21.15.129:0/1938112820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d496e2f0 msgr2=0x5595d4d365c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:10:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.972+0000 7f9eef59e640 1 --2- 172.21.15.129:0/1938112820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d496e2f0 0x5595d4d365c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.972+0000 7f9eef59e640 1 -- 172.21.15.129:0/1938112820 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595d4c678e0 con 0x5595d49971b0 2024-05-20T18:10:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.972+0000 7f9ed7fff640 1 -- 172.21.15.129:0/947633967 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f9ee0042070 con 0x7f9ef0103270 2024-05-20T18:10:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.973+0000 7f9ed7fff640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9ed80a2d70 0x7f9ed80a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.974+0000 7f9ed7fff640 1 -- 172.21.15.129:0/947633967 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8631+0+0 (secure 0 0 0) 0x7f9ee00ba190 con 0x7f9ef0103270 2024-05-20T18:10:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.974+0000 7f9eeed9d640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9ed80a2d70 0x7f9ed80a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.975+0000 7f9eeed9d640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9ed80a2d70 0x7f9ed80a5280 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ee00bbdf0 tx=0x7f9ee003b720 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.983+0000 7f9eef59e640 1 --2- 172.21.15.129:0/1938112820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49971b0 0x5595d4d36080 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009fc0 tx=0x7f9ee8010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12f57c5a1998551d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.987+0000 7f9ed57fa640 1 -- 172.21.15.129:0/1938112820 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8003b70 con 0x5595d49971b0 2024-05-20T18:10:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.987+0000 7f9ed57fa640 1 -- 172.21.15.129:0/1938112820 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee8015df0 con 0x5595d49971b0 2024-05-20T18:10:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.987+0000 7f9ed57fa640 1 -- 172.21.15.129:0/1938112820 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee801f420 con 0x5595d49971b0 2024-05-20T18:10:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.990+0000 7f9efc044740 1 -- 172.21.15.129:0/1938112820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49971b0 msgr2=0x5595d4d36080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:10.988 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.990+0000 7f9efc044740 1 --2- 172.21.15.129:0/1938112820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49971b0 0x5595d4d36080 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009fc0 tx=0x7f9ee8010860 comp rx=0 tx=0).stop 2024-05-20T18:10:10.988 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.990+0000 7f9efc044740 1 -- 172.21.15.129:0/1938112820 shutdown_connections 2024-05-20T18:10:10.988 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.990+0000 7f9efc044740 1 --2- 172.21.15.129:0/1938112820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49764d0 0x5595d4d36b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.988 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.990+0000 7f9efc044740 1 --2- 172.21.15.129:0/1938112820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d496e2f0 0x5595d4d365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.988 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.990+0000 7f9efc044740 1 --2- 172.21.15.129:0/1938112820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49971b0 0x5595d4d36080 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.988 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.990+0000 7f9efc044740 1 -- 172.21.15.129:0/1938112820 >> 172.21.15.129:0/1938112820 conn(0x5595d498b300 msgr2=0x5595d4d32830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:10.988 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.990+0000 7f9efc044740 1 -- 172.21.15.129:0/1938112820 shutdown_connections 2024-05-20T18:10:10.988 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.990+0000 7f9efc044740 1 -- 172.21.15.129:0/1938112820 wait complete. 2024-05-20T18:10:10.989 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.991+0000 7f9efc044740 1 -- start start 2024-05-20T18:10:10.989 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.991+0000 7f9efc044740 1 Processor -- start 2024-05-20T18:10:10.989 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.992+0000 7f9efc044740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d496e2f0 0x5595d4d42170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.989 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.992+0000 7f9efc044740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 0x5595d4d426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.990 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.992+0000 7f9eef59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d496e2f0 0x5595d4d42170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.990 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.992+0000 7f9eef59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d496e2f0 0x5595d4d42170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59696/0 (socket says 172.21.15.129:59696) 2024-05-20T18:10:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.992+0000 7f9eef59e640 1 -- 172.21.15.129:0/1158800370 learned_addr learned my addr 172.21.15.129:0/1158800370 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.992+0000 7f9eee59c640 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 0x5595d4d426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.992+0000 7f9efc044740 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d49971b0 0x5595d4d40610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.992+0000 7f9efc044740 1 -- 172.21.15.129:0/1158800370 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5595d4d431b0 con 0x5595d49971b0 2024-05-20T18:10:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.992+0000 7f9efc044740 1 -- 172.21.15.129:0/1158800370 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5595d4c56790 con 0x5595d496e2f0 2024-05-20T18:10:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.993+0000 7f9efc044740 1 -- 172.21.15.129:0/1158800370 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5595d4d43330 con 0x5595d49764d0 2024-05-20T18:10:10.991 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.993+0000 7f9efc044740 10 client.0 _ll_register_callbacks cb 0x5595d4c91a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.993+0000 7f9eee59c640 1 -- 172.21.15.129:0/1158800370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d496e2f0 msgr2=0x5595d4d42170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.993+0000 7f9eee59c640 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d496e2f0 0x5595d4d42170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.993+0000 7f9eee59c640 1 -- 172.21.15.129:0/1158800370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d49971b0 msgr2=0x5595d4d40610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.993+0000 7f9eee59c640 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d49971b0 0x5595d4d40610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.993+0000 7f9eee59c640 1 -- 172.21.15.129:0/1158800370 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595d49af210 con 0x5595d49764d0 2024-05-20T18:10:10.992 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.994+0000 7f9eee59c640 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 0x5595d4d426b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9ef019aea0 tx=0x7f9ee4011b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15d1191b4e4aac7b server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:10.993 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.994+0000 7f9ed2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:10:10.993 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.995+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1158800370 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee400fd00 con 0x5595d49764d0 2024-05-20T18:10:10.993 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.995+0000 7f9efc044740 1 -- 172.21.15.129:0/1158800370 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ee8015df0 con 0x5595d49764d0 2024-05-20T18:10:10.993 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.995+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1158800370 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8631+0+0 (secure 0 0 0) 0x7f9ee4004020 con 0x5595d49764d0 2024-05-20T18:10:10.993 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.995+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1158800370 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9ee400fea0 con 0x5595d49764d0 2024-05-20T18:10:10.994 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.995+0000 7f9efc044740 10 client.26959 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:10.994 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:10.996+0000 7f9efc044740 1 -- 172.21.15.129:0/1158800370 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595d4a7b620 con 0x5595d49764d0 2024-05-20T18:10:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.001+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1158800370 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee8015df0 con 0x5595d49764d0 2024-05-20T18:10:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.001+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1158800370 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f9ee4019430 con 0x5595d49764d0 2024-05-20T18:10:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.001+0000 7f9efc044740 10 client.26959 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:10.998 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.001+0000 7f9efc044740 10 client.26959 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.001+0000 7f9efc044740 20 client.26959 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.001+0000 7f9efc044740 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5595d4955d80 0x5595d4d52cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.001+0000 7f9efc044740 4 client.26959 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:10:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.001+0000 7f9efc044740 1 -- 172.21.15.129:0/1158800370 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595d4ca8a70 con 0x5595d4955d80 2024-05-20T18:10:10.999 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.002+0000 7f9eef59e640 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5595d4955d80 0x5595d4d52cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.007+0000 7f9eef59e640 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5595d4955d80 0x5595d4d52cd0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64a8088914f24218 server_cookie=1241b9f0e926670b in_seq=0 out_seq=0 2024-05-20T18:10:11.004 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.007+0000 7f9ed2ff5640 10 client.26959 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.012+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1158800370 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5595d4ca8a70 con 0x5595d4955d80 2024-05-20T18:10:11.011 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.012+0000 7f9ed2ff5640 10 client.26959 handle_command_reply: tid=0 2024-05-20T18:10:11.012 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.013+0000 7f9efc044740 1 client.26959 shutdown 2024-05-20T18:10:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.017+0000 7f9efc044740 1 -- 172.21.15.129:0/1158800370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 msgr2=0x5595d4d426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.015 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.017+0000 7f9efc044740 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 0x5595d4d426b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9ef019aea0 tx=0x7f9ee4011b50 comp rx=0 tx=0).stop 2024-05-20T18:10:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.018+0000 7f9efc044740 1 -- 172.21.15.129:0/1158800370 shutdown_connections 2024-05-20T18:10:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.018+0000 7f9efc044740 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5595d4955d80 0x5595d4d52cd0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.018+0000 7f9efc044740 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d49971b0 0x5595d4d40610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.018+0000 7f9efc044740 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 0x5595d4d426b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.018+0000 7f9efc044740 1 --2- 172.21.15.129:0/1158800370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d496e2f0 0x5595d4d42170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.018+0000 7f9efc044740 1 -- 172.21.15.129:0/1158800370 >> 172.21.15.129:0/1158800370 conn(0x5595d498b300 msgr2=0x5595d4d37da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.016 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.019+0000 7f9ed2ff5640 0 client.26959 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:11.019 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.021+0000 7f9efc044740 1 -- 172.21.15.129:0/1158800370 shutdown_connections 2024-05-20T18:10:11.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.024+0000 7f9efc044740 1 -- 172.21.15.129:0/1158800370 wait complete. 2024-05-20T18:10:11.022 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.024+0000 7f9efc044740 20 client.26959 trim_cache size 0 max 16384 2024-05-20T18:10:11.050 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.052+0000 7f9efc044740 1 Processor -- start 2024-05-20T18:10:11.050 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.052+0000 7f9efc044740 1 -- start start 2024-05-20T18:10:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.053+0000 7f9efc044740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d496e2f0 0x5595d4d584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.053+0000 7f9efc044740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 0x5595d4d58a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.053+0000 7f9eef59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d496e2f0 0x5595d4d584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.053+0000 7f9eef59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d496e2f0 0x5595d4d584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43486/0 (socket says 172.21.15.129:43486) 2024-05-20T18:10:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.053+0000 7f9eef59e640 1 -- 172.21.15.129:0/907017796 learned_addr learned my addr 172.21.15.129:0/907017796 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.053+0000 7f9efc044740 1 --2- 172.21.15.129:0/907017796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49971b0 0x5595d4d48600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.053+0000 7f9efc044740 1 -- 172.21.15.129:0/907017796 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef01aabe0 con 0x5595d496e2f0 2024-05-20T18:10:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.053+0000 7f9efc044740 1 -- 172.21.15.129:0/907017796 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef01ad440 con 0x5595d49971b0 2024-05-20T18:10:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.053+0000 7f9efc044740 1 -- 172.21.15.129:0/907017796 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef01ab230 con 0x5595d49764d0 2024-05-20T18:10:11.051 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.053+0000 7f9eee59c640 1 --2- 172.21.15.129:0/907017796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 0x5595d4d58a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.052 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.055+0000 7f9eee59c640 1 -- 172.21.15.129:0/907017796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49971b0 msgr2=0x5595d4d48600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-20T18:10:11.053 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.055+0000 7f9eee59c640 1 --2- 172.21.15.129:0/907017796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49971b0 0x5595d4d48600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.053 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.055+0000 7f9eee59c640 1 -- 172.21.15.129:0/907017796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d496e2f0 msgr2=0x5595d4d584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.053 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.055+0000 7f9eee59c640 1 --2- 172.21.15.129:0/907017796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d496e2f0 0x5595d4d584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.053 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.055+0000 7f9eee59c640 1 -- 172.21.15.129:0/907017796 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595d4d2ca80 con 0x5595d49764d0 2024-05-20T18:10:11.054 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.056+0000 7f9eee59c640 1 --2- 172.21.15.129:0/907017796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 0x5595d4d58a10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40198d0 tx=0x7f9ee400fe30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=462c5e72fd2974d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.054 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.056+0000 7f9ed4ff9640 1 -- 172.21.15.129:0/907017796 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4019630 con 0x5595d49764d0 2024-05-20T18:10:11.054 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.056+0000 7f9ed4ff9640 1 -- 172.21.15.129:0/907017796 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee4005520 con 0x5595d49764d0 2024-05-20T18:10:11.054 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.056+0000 7f9ed4ff9640 1 -- 172.21.15.129:0/907017796 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee40039e0 con 0x5595d49764d0 2024-05-20T18:10:11.055 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.058+0000 7f9efc044740 1 -- 172.21.15.129:0/907017796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 msgr2=0x5595d4d58a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.055 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.058+0000 7f9efc044740 1 --2- 172.21.15.129:0/907017796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 0x5595d4d58a10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40198d0 tx=0x7f9ee400fe30 comp rx=0 tx=0).stop 2024-05-20T18:10:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.061+0000 7f9efc044740 1 -- 172.21.15.129:0/907017796 shutdown_connections 2024-05-20T18:10:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.061+0000 7f9efc044740 1 --2- 172.21.15.129:0/907017796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49971b0 0x5595d4d48600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.061+0000 7f9efc044740 1 --2- 172.21.15.129:0/907017796 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d49764d0 0x5595d4d58a10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.062+0000 7f9efc044740 1 --2- 172.21.15.129:0/907017796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d496e2f0 0x5595d4d584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.059 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.062+0000 7f9efc044740 1 -- 172.21.15.129:0/907017796 >> 172.21.15.129:0/907017796 conn(0x5595d498b300 msgr2=0x5595d4d3eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.063+0000 7f9efc044740 1 -- 172.21.15.129:0/907017796 shutdown_connections 2024-05-20T18:10:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.064+0000 7f9efc044740 1 -- 172.21.15.129:0/907017796 wait complete. 2024-05-20T18:10:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.065+0000 7f9efc044740 1 -- start start 2024-05-20T18:10:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.065+0000 7f9efc044740 1 Processor -- start 2024-05-20T18:10:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.065+0000 7f9efc044740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d496e2f0 0x5595d4d484e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.066+0000 7f9efc044740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d49764d0 0x5595d4d48a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.066+0000 7f9efc044740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49971b0 0x5595d4d459f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.066+0000 7f9efc044740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef01ab900 con 0x5595d49764d0 2024-05-20T18:10:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.066+0000 7f9efc044740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef01acd70 con 0x5595d49971b0 2024-05-20T18:10:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.066+0000 7f9efc044740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef01abfd0 con 0x5595d496e2f0 2024-05-20T18:10:11.063 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.066+0000 7f9efc044740 10 client.0 _ll_register_callbacks cb 0x7f9ee4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.066+0000 7f9eef59e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d496e2f0 0x5595d4d484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.066+0000 7f9eef59e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d496e2f0 0x5595d4d484e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:42344/0 (socket says 172.21.15.129:42344) 2024-05-20T18:10:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.066+0000 7f9eef59e640 1 -- 172.21.15.129:0/1490292802 learned_addr learned my addr 172.21.15.129:0/1490292802 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:11.064 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.066+0000 7f9eee59c640 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d49764d0 0x5595d4d48a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.067+0000 7f9eef59e640 1 -- 172.21.15.129:0/1490292802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49971b0 msgr2=0x5595d4d459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.067+0000 7f9eef59e640 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49971b0 0x5595d4d459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.067+0000 7f9eef59e640 1 -- 172.21.15.129:0/1490292802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d49764d0 msgr2=0x5595d4d48a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.067+0000 7f9eef59e640 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d49764d0 0x5595d4d48a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.067+0000 7f9eef59e640 1 -- 172.21.15.129:0/1490292802 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595d4d460e0 con 0x5595d496e2f0 2024-05-20T18:10:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.067+0000 7f9eef59e640 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d496e2f0 0x5595d4d484e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8015bb0 tx=0x7f9ee8015be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d69ff28de35e33b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.067+0000 7f9ed2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:10:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.068+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1490292802 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8015400 con 0x5595d496e2f0 2024-05-20T18:10:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.068+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1490292802 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8631+0+0 (secure 0 0 0) 0x7f9ee80155a0 con 0x5595d496e2f0 2024-05-20T18:10:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.068+0000 7f9efc044740 1 -- 172.21.15.129:0/1490292802 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ee4005520 con 0x5595d496e2f0 2024-05-20T18:10:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.069+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1490292802 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9ee8036050 con 0x5595d496e2f0 2024-05-20T18:10:11.067 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.069+0000 7f9efc044740 10 client.26965 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:11.067 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.069+0000 7f9efc044740 1 -- 172.21.15.129:0/1490292802 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595d4dc31d0 con 0x5595d496e2f0 2024-05-20T18:10:11.068 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.071+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1490292802 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee4005520 con 0x5595d496e2f0 2024-05-20T18:10:11.068 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.071+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1490292802 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f9ee802e760 con 0x5595d496e2f0 2024-05-20T18:10:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.071+0000 7f9efc044740 10 client.26965 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.071+0000 7f9efc044740 10 client.26965 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.071+0000 7f9efc044740 20 client.26965 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.071+0000 7f9efc044740 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5595d492b1c0 0x5595d4dc56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.071+0000 7f9efc044740 4 client.26965 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-20T18:10:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.071+0000 7f9efc044740 1 -- 172.21.15.129:0/1490292802 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5595d4ca2dc0 con 0x5595d492b1c0 2024-05-20T18:10:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.072+0000 7f9eee59c640 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5595d492b1c0 0x5595d4dc56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.070 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.073+0000 7f9eee59c640 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5595d492b1c0 0x5595d4dc56e0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed34e4102201e213 server_cookie=ac12ddc464f265ff in_seq=0 out_seq=0 2024-05-20T18:10:11.070 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.073+0000 7f9ed2ff5640 10 client.26965 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:11.119 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.122+0000 7f9ed2ff5640 1 -- 172.21.15.129:0/1490292802 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5595d4ca2dc0 con 0x5595d492b1c0 2024-05-20T18:10:11.119 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.122+0000 7f9ed2ff5640 10 client.26965 handle_command_reply: tid=0 2024-05-20T18:10:11.120 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.122+0000 7f9efc044740 1 client.26965 shutdown 2024-05-20T18:10:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.122+0000 7f9efc044740 1 -- 172.21.15.129:0/1490292802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d496e2f0 msgr2=0x5595d4d484e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.122+0000 7f9efc044740 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d496e2f0 0x5595d4d484e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8015bb0 tx=0x7f9ee8015be0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.123+0000 7f9efc044740 1 -- 172.21.15.129:0/1490292802 shutdown_connections 2024-05-20T18:10:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.123+0000 7f9efc044740 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x5595d492b1c0 0x5595d4dc56e0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.123+0000 7f9ed2ff5640 0 client.26965 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:11.121 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.123+0000 7f9efc044740 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595d49971b0 0x5595d4d459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.123+0000 7f9efc044740 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5595d49764d0 0x5595d4d48a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.123+0000 7f9efc044740 1 --2- 172.21.15.129:0/1490292802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5595d496e2f0 0x5595d4d484e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.123+0000 7f9efc044740 1 -- 172.21.15.129:0/1490292802 >> 172.21.15.129:0/1490292802 conn(0x5595d498b300 msgr2=0x5595d4d36000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.123+0000 7f9efc044740 1 -- 172.21.15.129:0/1490292802 shutdown_connections 2024-05-20T18:10:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.123+0000 7f9efc044740 1 -- 172.21.15.129:0/1490292802 wait complete. 2024-05-20T18:10:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.124+0000 7f9efc044740 20 client.26965 trim_cache size 0 max 16384 2024-05-20T18:10:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:{ 2024-05-20T18:10:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "message": "", 2024-05-20T18:10:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout: "return_code": 0 2024-05-20T18:10:11.122 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stdout:} 2024-05-20T18:10:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.124+0000 7f9ef5196640 1 -- 172.21.15.129:0/947633967 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9ed80a2d70 msgr2=0x7f9ed80a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.124+0000 7f9ef5196640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9ed80a2d70 0x7f9ed80a5280 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ee00bbdf0 tx=0x7f9ee003b720 comp rx=0 tx=0).stop 2024-05-20T18:10:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.125+0000 7f9ef5196640 1 -- 172.21.15.129:0/947633967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef0103270 msgr2=0x7f9ef019f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.125+0000 7f9ef5196640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef0103270 0x7f9ef019f4f0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f9ee002a930 tx=0x7f9ee0003b20 comp rx=0 tx=0).stop 2024-05-20T18:10:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.125+0000 7f9ef5196640 1 -- 172.21.15.129:0/947633967 shutdown_connections 2024-05-20T18:10:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.125+0000 7f9ef5196640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f9ed80a2d70 0x7f9ed80a5280 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.125+0000 7f9ef5196640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ef0108b20 0x7f9ef019fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.125+0000 7f9ef5196640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ef0108fc0 0x7f9ef0199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.125+0000 7f9ef5196640 1 --2- 172.21.15.129:0/947633967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ef0103270 0x7f9ef019f4f0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.125+0000 7f9ef5196640 1 -- 172.21.15.129:0/947633967 >> 172.21.15.129:0/947633967 conn(0x7f9ef00fe1c0 msgr2=0x7f9ef00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.125+0000 7f9ef5196640 1 -- 172.21.15.129:0/947633967 shutdown_connections 2024-05-20T18:10:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi129.front.sepia.ceph.com.smithi129.stderr:2024-05-20T18:10:11.125+0000 7f9ef5196640 1 -- 172.21.15.129:0/947633967 wait complete. 2024-05-20T18:10:11.369 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-20T18:10:11.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:10:11.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-20T18:10:11.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:10 smithi154 ceph-mon[36482]: pgmap v362: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2024-05-20T18:10:11.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:10 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/980675040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:10:11.404 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:10 smithi129 ceph-mon[32029]: pgmap v362: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2024-05-20T18:10:11.404 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:10 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/980675040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:10:11.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:10 smithi204 ceph-mon[37076]: pgmap v362: 97 pgs: 97 active+clean; 32 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2024-05-20T18:10:11.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:10 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/980675040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:10:11.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.570+0000 7f4bb0a79640 1 -- 172.21.15.129:0/184069938 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bac06a200 msgr2=0x7f4bac06a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.570+0000 7f4bb0a79640 1 --2- 172.21.15.129:0/184069938 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bac06a200 0x7f4bac06a610 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0013f10 tx=0x7f4ba00329c0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.571+0000 7f4bb0a79640 1 -- 172.21.15.129:0/184069938 shutdown_connections 2024-05-20T18:10:11.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.571+0000 7f4bb0a79640 1 --2- 172.21.15.129:0/184069938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bac101ac0 0x7f4bac101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.571+0000 7f4bb0a79640 1 --2- 172.21.15.129:0/184069938 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bac06a200 0x7f4bac06a610 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.571+0000 7f4bb0a79640 1 --2- 172.21.15.129:0/184069938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bac1079f0 0x7f4bac101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.571+0000 7f4bb0a79640 1 -- 172.21.15.129:0/184069938 >> 172.21.15.129:0/184069938 conn(0x7f4bac077130 msgr2=0x7f4bac077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.571+0000 7f4bb0a79640 1 -- 172.21.15.129:0/184069938 shutdown_connections 2024-05-20T18:10:11.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.573+0000 7f4bb0a79640 1 -- 172.21.15.129:0/184069938 wait complete. 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.574+0000 7f4bb0a79640 1 Processor -- start 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.574+0000 7f4bb0a79640 1 -- start start 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.575+0000 7f4bb0a79640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bac06a200 0x7f4bac0733a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.575+0000 7f4bb0a79640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bac101ac0 0x7f4bac06d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.575+0000 7f4bb0a79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bac1079f0 0x7f4bac06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.575+0000 7f4bb0a79640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac06bb60 con 0x7f4bac101ac0 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.575+0000 7f4bb0a79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac06b9e0 con 0x7f4bac1079f0 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.575+0000 7f4bb0a79640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac06bce0 con 0x7f4bac06a200 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.575+0000 7f4baad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bac1079f0 0x7f4bac06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.575+0000 7f4baad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bac1079f0 0x7f4bac06d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59716/0 (socket says 172.21.15.129:59716) 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.575+0000 7f4baad9d640 1 -- 172.21.15.129:0/494855999 learned_addr learned my addr 172.21.15.129:0/494855999 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:11.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.575+0000 7f4baa59c640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bac06a200 0x7f4bac0733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.576+0000 7f4baad9d640 1 -- 172.21.15.129:0/494855999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bac06a200 msgr2=0x7f4bac0733a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.576+0000 7f4baad9d640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bac06a200 0x7f4bac0733a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.576+0000 7f4baad9d640 1 -- 172.21.15.129:0/494855999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bac101ac0 msgr2=0x7f4bac06d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.576+0000 7f4baad9d640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bac101ac0 0x7f4bac06d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.576+0000 7f4baad9d640 1 -- 172.21.15.129:0/494855999 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac06df30 con 0x7f4bac1079f0 2024-05-20T18:10:11.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.578+0000 7f4baad9d640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bac1079f0 0x7f4bac06d9f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0006010 tx=0x7f4ba0005540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.579+0000 7f4b9b7fe640 1 -- 172.21.15.129:0/494855999 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0045070 con 0x7f4bac1079f0 2024-05-20T18:10:11.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.579+0000 7f4bb0a79640 1 -- 172.21.15.129:0/494855999 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bac06e1c0 con 0x7f4bac1079f0 2024-05-20T18:10:11.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.580+0000 7f4bb0a79640 1 -- 172.21.15.129:0/494855999 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bac1bd6e0 con 0x7f4bac1079f0 2024-05-20T18:10:11.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.580+0000 7f4b9b7fe640 1 -- 172.21.15.129:0/494855999 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba0012740 con 0x7f4bac1079f0 2024-05-20T18:10:11.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.581+0000 7f4b9b7fe640 1 -- 172.21.15.129:0/494855999 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0003a20 con 0x7f4bac1079f0 2024-05-20T18:10:11.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.584+0000 7f4b9b7fe640 1 -- 172.21.15.129:0/494855999 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f4ba0048070 con 0x7f4bac1079f0 2024-05-20T18:10:11.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.585+0000 7f4b9b7fe640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b940a2de0 0x7f4b940a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.585+0000 7f4b9b7fe640 1 -- 172.21.15.129:0/494855999 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8631+0+0 (secure 0 0 0) 0x7f4ba00ef590 con 0x7f4bac1079f0 2024-05-20T18:10:11.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.585+0000 7f4baa59c640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b940a2de0 0x7f4b940a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.586+0000 7f4baa59c640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b940a2de0 0x7f4b940a52f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4bac1029f0 tx=0x7f4ba400d040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.588+0000 7f4bb79f5740 1 Processor -- start 2024-05-20T18:10:11.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.588+0000 7f4bb79f5740 1 -- start start 2024-05-20T18:10:11.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.588+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcfc410 0x55b04e09b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.588+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcd3550 0x55b04e09bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.588+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcdb730 0x55b04e09c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.588+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b04e000990 con 0x55b04dcfc410 2024-05-20T18:10:11.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.588+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b04dfa85b0 con 0x55b04dcdb730 2024-05-20T18:10:11.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.588+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b04dffc950 con 0x55b04dcd3550 2024-05-20T18:10:11.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.589+0000 7f4baad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcdb730 0x55b04e09c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.589+0000 7f4baad9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcdb730 0x55b04e09c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59726/0 (socket says 172.21.15.129:59726) 2024-05-20T18:10:11.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.589+0000 7f4baad9d640 1 -- 172.21.15.129:0/3307085228 learned_addr learned my addr 172.21.15.129:0/3307085228 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:11.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.590+0000 7f4baa59c640 1 --2- 172.21.15.129:0/3307085228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcfc410 0x55b04e09b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.590+0000 7f4baad9d640 1 -- 172.21.15.129:0/3307085228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcd3550 msgr2=0x55b04e09bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.590+0000 7f4baad9d640 1 --2- 172.21.15.129:0/3307085228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcd3550 0x55b04e09bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.590+0000 7f4baad9d640 1 -- 172.21.15.129:0/3307085228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcfc410 msgr2=0x55b04e09b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.590+0000 7f4baad9d640 1 --2- 172.21.15.129:0/3307085228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcfc410 0x55b04e09b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.590+0000 7f4baad9d640 1 -- 172.21.15.129:0/3307085228 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b04dddc7e0 con 0x55b04dcdb730 2024-05-20T18:10:11.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.591+0000 7f4baad9d640 1 --2- 172.21.15.129:0/3307085228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcdb730 0x55b04e09c0a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00b83b0 tx=0x7f4ba00bb7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec368cdbb1dcce61 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.594+0000 7f4b98ff9640 1 -- 172.21.15.129:0/3307085228 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba00d4070 con 0x55b04dcdb730 2024-05-20T18:10:11.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.594+0000 7f4b98ff9640 1 -- 172.21.15.129:0/3307085228 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba0004e10 con 0x55b04dcdb730 2024-05-20T18:10:11.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.595+0000 7f4b98ff9640 1 -- 172.21.15.129:0/3307085228 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0004b10 con 0x55b04dcdb730 2024-05-20T18:10:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.595+0000 7f4bb79f5740 1 -- 172.21.15.129:0/3307085228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcdb730 msgr2=0x55b04e09c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.595+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/3307085228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcdb730 0x55b04e09c0a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00b83b0 tx=0x7f4ba00bb7b0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.595+0000 7f4bb79f5740 1 -- 172.21.15.129:0/3307085228 shutdown_connections 2024-05-20T18:10:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.595+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/3307085228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcdb730 0x55b04e09c0a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.595+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/3307085228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcd3550 0x55b04e09bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.595+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/3307085228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcfc410 0x55b04e09b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.595+0000 7f4bb79f5740 1 -- 172.21.15.129:0/3307085228 >> 172.21.15.129:0/3307085228 conn(0x55b04dcf0560 msgr2=0x55b04e097dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.595+0000 7f4bb79f5740 1 -- 172.21.15.129:0/3307085228 shutdown_connections 2024-05-20T18:10:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.595+0000 7f4bb79f5740 1 -- 172.21.15.129:0/3307085228 wait complete. 2024-05-20T18:10:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.596+0000 7f4bb79f5740 1 -- start start 2024-05-20T18:10:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.596+0000 7f4bb79f5740 1 Processor -- start 2024-05-20T18:10:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.596+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcd3550 0x55b04e0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.596+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcdb730 0x55b04e0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.596+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 0x55b04e0abe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.596+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b04e0a87f0 con 0x55b04dcdb730 2024-05-20T18:10:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.596+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b04dfbbb40 con 0x55b04dcfc410 2024-05-20T18:10:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.596+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b04e0a8970 con 0x55b04dcd3550 2024-05-20T18:10:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.596+0000 7f4bb79f5740 10 client.0 _ll_register_callbacks cb 0x55b04dfb00f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.597+0000 7f4ba9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcdb730 0x55b04e0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.597+0000 7f4ba9d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcdb730 0x55b04e0a6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43520/0 (socket says 172.21.15.129:43520) 2024-05-20T18:10:11.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.597+0000 7f4ba9d9b640 1 -- 172.21.15.129:0/1397798001 learned_addr learned my addr 172.21.15.129:0/1397798001 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:11.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.597+0000 7f4baad9d640 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 0x55b04e0abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.597+0000 7f4baa59c640 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcd3550 0x55b04e0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.598+0000 7f4ba9d9b640 1 -- 172.21.15.129:0/1397798001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcd3550 msgr2=0x55b04e0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.598+0000 7f4ba9d9b640 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcd3550 0x55b04e0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.598+0000 7f4ba9d9b640 1 -- 172.21.15.129:0/1397798001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 msgr2=0x55b04e0abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.598+0000 7f4ba9d9b640 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 0x55b04e0abe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.598+0000 7f4ba9d9b640 1 -- 172.21.15.129:0/1397798001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b04de3cf70 con 0x55b04dcdb730 2024-05-20T18:10:11.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.598+0000 7f4ba9d9b640 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcdb730 0x55b04e0a6250 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f4bac06ec70 tx=0x7f4b9c00b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e07ff0ac5fc041 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.598+0000 7f4b91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:10:11.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.598+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1397798001 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c015020 con 0x55b04dcdb730 2024-05-20T18:10:11.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.598+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1397798001 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8631+0+0 (secure 0 0 0) 0x7f4b9c011420 con 0x55b04dcdb730 2024-05-20T18:10:11.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.598+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1397798001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b04e00e0c0 con 0x55b04dcdb730 2024-05-20T18:10:11.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.599+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1397798001 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b9c020050 con 0x55b04dcdb730 2024-05-20T18:10:11.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.599+0000 7f4bb79f5740 10 client.17442 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:11.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.599+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1397798001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b04dcd4170 con 0x55b04dcdb730 2024-05-20T18:10:11.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.599+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1397798001 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b04e00e0c0 con 0x55b04dcdb730 2024-05-20T18:10:11.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.599+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1397798001 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f4b9c0198c0 con 0x55b04dcdb730 2024-05-20T18:10:11.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.599+0000 7f4bb79f5740 10 client.17442 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:11.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.599+0000 7f4bb79f5740 10 client.17442 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:11.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.599+0000 7f4bb79f5740 20 client.17442 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:11.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.599+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b04dcbb1f0 0x55b04e0b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.600+0000 7f4bb79f5740 4 client.17442 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:10:11.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.600+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1397798001 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b04e008410 con 0x55b04dcbb1f0 2024-05-20T18:10:11.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.600+0000 7f4baa59c640 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b04dcbb1f0 0x55b04e0b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.600+0000 7f4b91ffb640 10 client.17442 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:11.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.601+0000 7f4baa59c640 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b04dcbb1f0 0x55b04e0b81f0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1693d8702eb9760 server_cookie=55946a1baf35652 in_seq=0 out_seq=0 2024-05-20T18:10:11.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1397798001 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f4ba40165d0 con 0x55b04dcbb1f0 2024-05-20T18:10:11.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4b91ffb640 10 client.17442 handle_command_reply: tid=0 2024-05-20T18:10:11.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4bb79f5740 1 client.17442 shutdown 2024-05-20T18:10:11.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1397798001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcdb730 msgr2=0x55b04e0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcdb730 0x55b04e0a6250 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f4bac06ec70 tx=0x7f4b9c00b4b0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1397798001 shutdown_connections 2024-05-20T18:10:11.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b04dcbb1f0 0x55b04e0b81f0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 0x55b04e0abe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcdb730 0x55b04e0a6250 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1397798001 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcd3550 0x55b04e0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1397798001 >> 172.21.15.129:0/1397798001 conn(0x55b04dcf0560 msgr2=0x55b04e09d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.602+0000 7f4b91ffb640 0 client.17442 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:11.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.603+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1397798001 shutdown_connections 2024-05-20T18:10:11.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.603+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1397798001 wait complete. 2024-05-20T18:10:11.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.604+0000 7f4bb79f5740 20 client.17442 trim_cache size 0 max 16384 2024-05-20T18:10:11.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.641+0000 7f4bb79f5740 1 Processor -- start 2024-05-20T18:10:11.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.642+0000 7f4bb79f5740 1 -- start start 2024-05-20T18:10:11.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.642+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0a12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.642+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcdb730 0x55b04e09d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.642+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 0x55b04e09dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.642+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c00c930 con 0x55b04dcd3550 2024-05-20T18:10:11.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.642+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c00f190 con 0x55b04dcfc410 2024-05-20T18:10:11.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.642+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c00cf80 con 0x55b04dcdb730 2024-05-20T18:10:11.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.644+0000 7f4baa59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0a12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.644+0000 7f4baa59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0a12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43528/0 (socket says 172.21.15.129:43528) 2024-05-20T18:10:11.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.644+0000 7f4baa59c640 1 -- 172.21.15.129:0/3382498369 learned_addr learned my addr 172.21.15.129:0/3382498369 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:11.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.644+0000 7f4baa59c640 1 -- 172.21.15.129:0/3382498369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcdb730 msgr2=0x55b04e09d860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:10:11.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.644+0000 7f4baa59c640 1 --2- 172.21.15.129:0/3382498369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcdb730 0x55b04e09d860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.644+0000 7f4baa59c640 1 -- 172.21.15.129:0/3382498369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 msgr2=0x55b04e09dda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-20T18:10:11.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.644+0000 7f4baa59c640 1 --2- 172.21.15.129:0/3382498369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 0x55b04e09dda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.644+0000 7f4baa59c640 1 -- 172.21.15.129:0/3382498369 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b04e0a4ab0 con 0x55b04dcd3550 2024-05-20T18:10:11.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.644+0000 7f4baa59c640 1 --2- 172.21.15.129:0/3382498369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0a12e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400ac60 tx=0x7f4ba4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3947d26fcf72e26e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.645+0000 7f4b93fff640 1 -- 172.21.15.129:0/3382498369 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4022970 con 0x55b04dcd3550 2024-05-20T18:10:11.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.645+0000 7f4b93fff640 1 -- 172.21.15.129:0/3382498369 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba402b030 con 0x55b04dcd3550 2024-05-20T18:10:11.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.647+0000 7f4bb79f5740 1 -- 172.21.15.129:0/3382498369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 msgr2=0x55b04e0a12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.647+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/3382498369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0a12e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400ac60 tx=0x7f4ba4011000 comp rx=0 tx=0).stop 2024-05-20T18:10:11.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.648+0000 7f4bb79f5740 1 -- 172.21.15.129:0/3382498369 shutdown_connections 2024-05-20T18:10:11.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.648+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/3382498369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 0x55b04e09dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.648+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/3382498369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcdb730 0x55b04e09d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.648+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/3382498369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0a12e0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.648+0000 7f4bb79f5740 1 -- 172.21.15.129:0/3382498369 >> 172.21.15.129:0/3382498369 conn(0x55b04dcf0560 msgr2=0x55b04e09d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.649+0000 7f4bb79f5740 1 -- 172.21.15.129:0/3382498369 shutdown_connections 2024-05-20T18:10:11.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.649+0000 7f4bb79f5740 1 -- 172.21.15.129:0/3382498369 wait complete. 2024-05-20T18:10:11.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.649+0000 7f4bb79f5740 1 -- start start 2024-05-20T18:10:11.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.650+0000 7f4bb79f5740 1 Processor -- start 2024-05-20T18:10:11.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.650+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0abd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.650+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcdb730 0x55b04e0ac2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.650+0000 7f4bb79f5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 0x55b04e0b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.650+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c00d650 con 0x55b04dcd3550 2024-05-20T18:10:11.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.650+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c00eac0 con 0x55b04dcfc410 2024-05-20T18:10:11.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.650+0000 7f4bb79f5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c00dd20 con 0x55b04dcdb730 2024-05-20T18:10:11.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.651+0000 7f4bb79f5740 10 client.0 _ll_register_callbacks cb 0x7f4b9c006e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:11.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.651+0000 7f4baa59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0abd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.651+0000 7f4baa59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0abd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43542/0 (socket says 172.21.15.129:43542) 2024-05-20T18:10:11.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.651+0000 7f4baa59c640 1 -- 172.21.15.129:0/1429372476 learned_addr learned my addr 172.21.15.129:0/1429372476 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:11.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.652+0000 7f4ba9d9b640 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcdb730 0x55b04e0ac2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.652+0000 7f4baad9d640 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 0x55b04e0b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.652+0000 7f4baa59c640 1 -- 172.21.15.129:0/1429372476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcdb730 msgr2=0x55b04e0ac2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.652+0000 7f4baa59c640 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcdb730 0x55b04e0ac2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.652+0000 7f4baa59c640 1 -- 172.21.15.129:0/1429372476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 msgr2=0x55b04e0b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.652+0000 7f4baa59c640 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 0x55b04e0b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.652+0000 7f4baa59c640 1 -- 172.21.15.129:0/1429372476 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b04e0980a0 con 0x55b04dcd3550 2024-05-20T18:10:11.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.653+0000 7f4baa59c640 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0abd60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4022700 tx=0x7f4ba40278b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a08571b8b3b918 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.653+0000 7f4b91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:10:11.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.653+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1429372476 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4022b70 con 0x55b04dcd3550 2024-05-20T18:10:11.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.653+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1429372476 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ba402b030 con 0x55b04dcd3550 2024-05-20T18:10:11.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.653+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1429372476 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8631+0+0 (secure 0 0 0) 0x7f4ba4022d10 con 0x55b04dcd3550 2024-05-20T18:10:11.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.654+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1429372476 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4ba403f050 con 0x55b04dcd3550 2024-05-20T18:10:11.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.654+0000 7f4bb79f5740 10 client.17448 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:11.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.654+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1429372476 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b04e120a80 con 0x55b04dcd3550 2024-05-20T18:10:11.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.655+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1429372476 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba402b030 con 0x55b04dcd3550 2024-05-20T18:10:11.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.656+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1429372476 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f4ba4030530 con 0x55b04dcd3550 2024-05-20T18:10:11.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.656+0000 7f4bb79f5740 10 client.17448 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:11.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.656+0000 7f4bb79f5740 10 client.17448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:11.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.656+0000 7f4bb79f5740 20 client.17448 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:11.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.657+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b04dc90630 0x55b04e123150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.657+0000 7f4bb79f5740 4 client.17448 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-20T18:10:11.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.657+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1429372476 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b04df85da0 con 0x55b04dc90630 2024-05-20T18:10:11.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.658+0000 7f4ba9d9b640 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b04dc90630 0x55b04e123150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.663+0000 7f4ba9d9b640 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b04dc90630 0x55b04e123150 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f5ce5c4ba6fb2c6 server_cookie=c5fa592155fd021a in_seq=0 out_seq=0 2024-05-20T18:10:11.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.663+0000 7f4b91ffb640 10 client.17448 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:11.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.664+0000 7f4b91ffb640 1 -- 172.21.15.129:0/1429372476 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b04df85da0 con 0x55b04dc90630 2024-05-20T18:10:11.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.665+0000 7f4b91ffb640 10 client.17448 handle_command_reply: tid=0 2024-05-20T18:10:11.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.666+0000 7f4bb79f5740 1 client.17448 shutdown 2024-05-20T18:10:11.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.668+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1429372476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 msgr2=0x55b04e0abd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.668+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0abd60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4022700 tx=0x7f4ba40278b0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.668+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1429372476 shutdown_connections 2024-05-20T18:10:11.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.668+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55b04dc90630 0x55b04e123150 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.668+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b04dcfc410 0x55b04e0b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.668+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b04dcdb730 0x55b04e0ac2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.668+0000 7f4bb79f5740 1 --2- 172.21.15.129:0/1429372476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b04dcd3550 0x55b04e0abd60 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.668+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1429372476 >> 172.21.15.129:0/1429372476 conn(0x55b04dcf0560 msgr2=0x55b04e09d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.668 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:10:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-05-20T18:10:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "1bb21371-3dd1-4008-a7d0-6c6d8414d188", 2024-05-20T18:10:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-05-20T18:10:11.669 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:10:11.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.668+0000 7f4b91ffb640 0 client.17448 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:11.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.670+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1429372476 shutdown_connections 2024-05-20T18:10:11.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.670+0000 7f4bb79f5740 1 -- 172.21.15.129:0/1429372476 wait complete. 2024-05-20T18:10:11.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.671+0000 7f4bb79f5740 20 client.17448 trim_cache size 0 max 16384 2024-05-20T18:10:11.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.673+0000 7f4bb0a79640 1 -- 172.21.15.129:0/494855999 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b940a2de0 msgr2=0x7f4b940a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.673+0000 7f4bb0a79640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b940a2de0 0x7f4b940a52f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4bac1029f0 tx=0x7f4ba400d040 comp rx=0 tx=0).stop 2024-05-20T18:10:11.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.673+0000 7f4bb0a79640 1 -- 172.21.15.129:0/494855999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bac1079f0 msgr2=0x7f4bac06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.673+0000 7f4bb0a79640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bac1079f0 0x7f4bac06d9f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0006010 tx=0x7f4ba0005540 comp rx=0 tx=0).stop 2024-05-20T18:10:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.673+0000 7f4bb0a79640 1 -- 172.21.15.129:0/494855999 shutdown_connections 2024-05-20T18:10:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.673+0000 7f4bb0a79640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f4b940a2de0 0x7f4b940a52f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.673+0000 7f4bb0a79640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bac1079f0 0x7f4bac06d9f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.673+0000 7f4bb0a79640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bac101ac0 0x7f4bac06d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.673+0000 7f4bb0a79640 1 --2- 172.21.15.129:0/494855999 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4bac06a200 0x7f4bac0733a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.673+0000 7f4bb0a79640 1 -- 172.21.15.129:0/494855999 >> 172.21.15.129:0/494855999 conn(0x7f4bac077130 msgr2=0x7f4bac103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.674+0000 7f4bb0a79640 1 -- 172.21.15.129:0/494855999 shutdown_connections 2024-05-20T18:10:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.674+0000 7f4bb0a79640 1 -- 172.21.15.129:0/494855999 wait complete. 2024-05-20T18:10:11.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:10:11.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:10:11.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.931+0000 7fbd68a14640 1 -- 172.21.15.129:0/3523266070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64101560 msgr2=0x7fbd641058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.931+0000 7fbd68a14640 1 --2- 172.21.15.129:0/3523266070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64101560 0x7fbd641058b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009900 tx=0x7fbd5802a440 comp rx=0 tx=0).stop 2024-05-20T18:10:11.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.932+0000 7fbd68a14640 1 -- 172.21.15.129:0/3523266070 shutdown_connections 2024-05-20T18:10:11.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.932+0000 7fbd68a14640 1 --2- 172.21.15.129:0/3523266070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd64105df0 0x7fbd64106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.932+0000 7fbd68a14640 1 --2- 172.21.15.129:0/3523266070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64101560 0x7fbd641058b0 secure :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009900 tx=0x7fbd5802a440 comp rx=0 tx=0).stop 2024-05-20T18:10:11.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.932+0000 7fbd68a14640 1 --2- 172.21.15.129:0/3523266070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd640689d0 0x7fbd64100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.932+0000 7fbd68a14640 1 -- 172.21.15.129:0/3523266070 >> 172.21.15.129:0/3523266070 conn(0x7fbd64076830 msgr2=0x7fbd64076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.934+0000 7fbd68a14640 1 -- 172.21.15.129:0/3523266070 shutdown_connections 2024-05-20T18:10:11.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.935+0000 7fbd68a14640 1 -- 172.21.15.129:0/3523266070 wait complete. 2024-05-20T18:10:11.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.936+0000 7fbd68a14640 1 Processor -- start 2024-05-20T18:10:11.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.936+0000 7fbd68a14640 1 -- start start 2024-05-20T18:10:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.937+0000 7fbd68a14640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd640689d0 0x7fbd641a8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.937+0000 7fbd68a14640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64105df0 0x7fbd641a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.937+0000 7fbd68a14640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd641ace80 0x7fbd641ad290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.937+0000 7fbd68a14640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6410ec00 con 0x7fbd64105df0 2024-05-20T18:10:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.937+0000 7fbd68a14640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6410ea80 con 0x7fbd640689d0 2024-05-20T18:10:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.937+0000 7fbd68a14640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64110e80 con 0x7fbd641ace80 2024-05-20T18:10:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.938+0000 7fbd6259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd640689d0 0x7fbd641a8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.938+0000 7fbd6259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd640689d0 0x7fbd641a8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59780/0 (socket says 172.21.15.129:59780) 2024-05-20T18:10:11.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.938+0000 7fbd6259c640 1 -- 172.21.15.129:0/2260354879 learned_addr learned my addr 172.21.15.129:0/2260354879 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:11.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.939+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64105df0 0x7fbd641a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.939+0000 7fbd62d9d640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd641ace80 0x7fbd641ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.940+0000 7fbd62d9d640 1 -- 172.21.15.129:0/2260354879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd640689d0 msgr2=0x7fbd641a8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.940+0000 7fbd62d9d640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd640689d0 0x7fbd641a8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.940+0000 7fbd62d9d640 1 -- 172.21.15.129:0/2260354879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64105df0 msgr2=0x7fbd641a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.940+0000 7fbd62d9d640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64105df0 0x7fbd641a8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.940+0000 7fbd62d9d640 1 -- 172.21.15.129:0/2260354879 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd641ad990 con 0x7fbd641ace80 2024-05-20T18:10:11.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.942+0000 7fbd62d9d640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd641ace80 0x7fbd641ad290 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009fd0 tx=0x7fbd5c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.942+0000 7fbd537fe640 1 -- 172.21.15.129:0/2260354879 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c00ce20 con 0x7fbd641ace80 2024-05-20T18:10:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.942+0000 7fbd537fe640 1 -- 172.21.15.129:0/2260354879 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd5c005020 con 0x7fbd641ace80 2024-05-20T18:10:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.943+0000 7fbd537fe640 1 -- 172.21.15.129:0/2260354879 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c010720 con 0x7fbd641ace80 2024-05-20T18:10:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.943+0000 7fbd68a14640 1 -- 172.21.15.129:0/2260354879 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd641adc80 con 0x7fbd641ace80 2024-05-20T18:10:11.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.943+0000 7fbd68a14640 1 -- 172.21.15.129:0/2260354879 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd641b5430 con 0x7fbd641ace80 2024-05-20T18:10:11.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.948+0000 7fbd537fe640 1 -- 172.21.15.129:0/2260354879 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fbd5c005540 con 0x7fbd641ace80 2024-05-20T18:10:11.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.952+0000 7fbd6f996740 1 Processor -- start 2024-05-20T18:10:11.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.952+0000 7fbd6f996740 1 -- start start 2024-05-20T18:10:11.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.952+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d692d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.952+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d693270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.952+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6937b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.952+0000 7fbd6f996740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d5f8d00 con 0x55aa4d2d36b0 2024-05-20T18:10:11.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.952+0000 7fbd6f996740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d5a04c0 con 0x55aa4d2cb4d0 2024-05-20T18:10:11.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.952+0000 7fbd6f996740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d5f4cc0 con 0x55aa4d2f4390 2024-05-20T18:10:11.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.953+0000 7fbd537fe640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd4c0a2f00 0x7fbd4c0a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.953+0000 7fbd62d9d640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd4c0a2f00 0x7fbd4c0a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.954+0000 7fbd61d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d692d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.954+0000 7fbd61d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d692d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:42430/0 (socket says 172.21.15.129:42430) 2024-05-20T18:10:11.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.954+0000 7fbd61d9b640 1 -- 172.21.15.129:0/833117025 learned_addr learned my addr 172.21.15.129:0/833117025 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:11.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.954+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/833117025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.954+0000 7fbd537fe640 1 -- 172.21.15.129:0/2260354879 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8631+0+0 (secure 0 0 0) 0x7fbd5c0cf840 con 0x7fbd641ace80 2024-05-20T18:10:11.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.954+0000 7fbd6259c640 1 --2- 172.21.15.129:0/833117025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d693270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.956+0000 7fbd62d9d640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd4c0a2f00 0x7fbd4c0a5410 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0a43d0 tx=0x7fbd5c097810 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.959+0000 7fbd61d9b640 1 -- 172.21.15.129:0/833117025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2cb4d0 msgr2=0x55aa4d693270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.959+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/833117025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d693270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.959+0000 7fbd61d9b640 1 -- 172.21.15.129:0/833117025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2d36b0 msgr2=0x55aa4d6937b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.959+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/833117025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6937b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.959+0000 7fbd61d9b640 1 -- 172.21.15.129:0/833117025 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa4d3d67d0 con 0x55aa4d2f4390 2024-05-20T18:10:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.959+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/833117025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d692d30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009230 tx=0x7fbd58005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6df7c2409206e130 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.960+0000 7fbd50ff9640 1 -- 172.21.15.129:0/833117025 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58035070 con 0x55aa4d2f4390 2024-05-20T18:10:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.960+0000 7fbd50ff9640 1 -- 172.21.15.129:0/833117025 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd58004750 con 0x55aa4d2f4390 2024-05-20T18:10:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.961+0000 7fbd6f996740 1 -- 172.21.15.129:0/833117025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 msgr2=0x55aa4d692d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.961+0000 7fbd6f996740 1 --2- 172.21.15.129:0/833117025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d692d30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009230 tx=0x7fbd58005fd0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.961+0000 7fbd6f996740 1 -- 172.21.15.129:0/833117025 shutdown_connections 2024-05-20T18:10:11.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.961+0000 7fbd6f996740 1 --2- 172.21.15.129:0/833117025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d693270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.961+0000 7fbd6f996740 1 --2- 172.21.15.129:0/833117025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6937b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.961+0000 7fbd6f996740 1 --2- 172.21.15.129:0/833117025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d692d30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.961+0000 7fbd6f996740 1 -- 172.21.15.129:0/833117025 >> 172.21.15.129:0/833117025 conn(0x55aa4d2e84e0 msgr2=0x55aa4d68f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.963+0000 7fbd6f996740 1 -- 172.21.15.129:0/833117025 shutdown_connections 2024-05-20T18:10:11.961 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:11 smithi129 ceph-mon[32029]: osdmap e106: 12 total, 12 up, 12 in 2024-05-20T18:10:11.961 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:11 smithi129 ceph-mon[32029]: pgmap v364: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 31 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-05-20T18:10:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.965+0000 7fbd6f996740 1 -- 172.21.15.129:0/833117025 wait complete. 2024-05-20T18:10:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.966+0000 7fbd6f996740 1 -- start start 2024-05-20T18:10:11.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.966+0000 7fbd6f996740 1 Processor -- start 2024-05-20T18:10:11.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.966+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d69ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.966+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2d36b0 0x55aa4d69f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.966+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2f4390 0x55aa4d69d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.967+0000 7fbd6f996740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d6b1070 con 0x55aa4d2f4390 2024-05-20T18:10:11.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.967+0000 7fbd6f996740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d5b3b90 con 0x55aa4d2d36b0 2024-05-20T18:10:11.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.967+0000 7fbd6f996740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55aa4d6b11f0 con 0x55aa4d2cb4d0 2024-05-20T18:10:11.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.967+0000 7fbd61d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d69ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.967+0000 7fbd6f996740 10 client.0 _ll_register_callbacks cb 0x55aa4d5efda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:11.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.967+0000 7fbd61d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d69ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:42434/0 (socket says 172.21.15.129:42434) 2024-05-20T18:10:11.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.967+0000 7fbd61d9b640 1 -- 172.21.15.129:0/1307367032 learned_addr learned my addr 172.21.15.129:0/1307367032 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:11.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.968+0000 7fbd6259c640 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2d36b0 0x55aa4d69f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.968+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2f4390 0x55aa4d69d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.968+0000 7fbd61d9b640 1 -- 172.21.15.129:0/1307367032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2d36b0 msgr2=0x55aa4d69f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.968+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2d36b0 0x55aa4d69f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.968+0000 7fbd61d9b640 1 -- 172.21.15.129:0/1307367032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2f4390 msgr2=0x55aa4d69d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:11.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.968+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2f4390 0x55aa4d69d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.968+0000 7fbd61d9b640 1 -- 172.21.15.129:0/1307367032 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa4d30c3f0 con 0x55aa4d2cb4d0 2024-05-20T18:10:11.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.968+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d69ed70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbd58005dc0 tx=0x7fbd5802a440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dea2e862b8054a7a server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:11.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.968+0000 7fbd49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-05-20T18:10:11.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.969+0000 7fbd49ffb640 1 -- 172.21.15.129:0/1307367032 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58035070 con 0x55aa4d2cb4d0 2024-05-20T18:10:11.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.969+0000 7fbd6f996740 1 -- 172.21.15.129:0/1307367032 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55aa4d606430 con 0x55aa4d2cb4d0 2024-05-20T18:10:11.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.969+0000 7fbd49ffb640 1 -- 172.21.15.129:0/1307367032 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8631+0+0 (secure 0 0 0) 0x7fbd58005250 con 0x55aa4d2cb4d0 2024-05-20T18:10:11.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.969+0000 7fbd49ffb640 1 -- 172.21.15.129:0/1307367032 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbd58005450 con 0x55aa4d2cb4d0 2024-05-20T18:10:11.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.970+0000 7fbd6f996740 10 client.26989 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:11.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.970+0000 7fbd6f996740 1 -- 172.21.15.129:0/1307367032 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aa4d2cc0f0 con 0x55aa4d2cb4d0 2024-05-20T18:10:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.970+0000 7fbd49ffb640 1 -- 172.21.15.129:0/1307367032 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55aa4d606430 con 0x55aa4d2cb4d0 2024-05-20T18:10:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.970+0000 7fbd49ffb640 1 -- 172.21.15.129:0/1307367032 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fbd58031660 con 0x55aa4d2cb4d0 2024-05-20T18:10:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.970+0000 7fbd6f996740 10 client.26989 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.971+0000 7fbd6f996740 10 client.26989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.971+0000 7fbd6f996740 20 client.26989 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.971+0000 7fbd6f996740 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aa4d2b2f60 0x55aa4d6c0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.971+0000 7fbd6f996740 4 client.26989 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:10:11.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.971+0000 7fbd6f996740 1 -- 172.21.15.129:0/1307367032 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa4d600780 con 0x55aa4d2b2f60 2024-05-20T18:10:11.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.973+0000 7fbd6259c640 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aa4d2b2f60 0x55aa4d6c0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:11.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.978+0000 7fbd6259c640 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aa4d2b2f60 0x55aa4d6c0de0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac203a743f3cff47 server_cookie=54dcf3549863f192 in_seq=0 out_seq=0 2024-05-20T18:10:11.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.978+0000 7fbd49ffb640 10 client.26989 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.980+0000 7fbd49ffb640 1 -- 172.21.15.129:0/1307367032 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55aa4d600780 con 0x55aa4d2b2f60 2024-05-20T18:10:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.980+0000 7fbd49ffb640 10 client.26989 handle_command_reply: tid=0 2024-05-20T18:10:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.980+0000 7fbd6f996740 1 client.26989 shutdown 2024-05-20T18:10:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.981+0000 7fbd6f996740 1 -- 172.21.15.129:0/1307367032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2cb4d0 msgr2=0x55aa4d69ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:11.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.981+0000 7fbd6f996740 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d69ed70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbd58005dc0 tx=0x7fbd5802a440 comp rx=0 tx=0).stop 2024-05-20T18:10:11.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.984+0000 7fbd6f996740 1 -- 172.21.15.129:0/1307367032 shutdown_connections 2024-05-20T18:10:11.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.984+0000 7fbd6f996740 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aa4d2b2f60 0x55aa4d6c0de0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.984+0000 7fbd6f996740 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2d36b0 0x55aa4d69f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.984+0000 7fbd6f996740 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2f4390 0x55aa4d69d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.984+0000 7fbd6f996740 1 --2- 172.21.15.129:0/1307367032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d69ed70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:11.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.984+0000 7fbd6f996740 1 -- 172.21.15.129:0/1307367032 >> 172.21.15.129:0/1307367032 conn(0x55aa4d2e84e0 msgr2=0x55aa4d692c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:11.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.984+0000 7fbd49ffb640 0 client.26989 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:11.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.986+0000 7fbd6f996740 1 -- 172.21.15.129:0/1307367032 shutdown_connections 2024-05-20T18:10:11.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.986+0000 7fbd6f996740 1 -- 172.21.15.129:0/1307367032 wait complete. 2024-05-20T18:10:11.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:11.989+0000 7fbd6f996740 20 client.26989 trim_cache size 0 max 16384 2024-05-20T18:10:12.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.025+0000 7fbd6f996740 1 Processor -- start 2024-05-20T18:10:12.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.025+0000 7fbd6f996740 1 -- start start 2024-05-20T18:10:12.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.025+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d696ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:12.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.025+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6b6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:12.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.025+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d6b6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:12.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.025+0000 7fbd6f996740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64068290 con 0x55aa4d2cb4d0 2024-05-20T18:10:12.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.026+0000 7fbd6f996740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd641b60a0 con 0x55aa4d2d36b0 2024-05-20T18:10:12.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.026+0000 7fbd6f996740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64072510 con 0x55aa4d2f4390 2024-05-20T18:10:12.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.026+0000 7fbd61d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d696ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:12.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.026+0000 7fbd61d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d6b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:12.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.026+0000 7fbd61d9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d6b6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.129:42438/0 (socket says 172.21.15.129:42438) 2024-05-20T18:10:12.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.026+0000 7fbd61d9b640 1 -- 172.21.15.129:0/4049018351 learned_addr learned my addr 172.21.15.129:0/4049018351 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:12.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.029+0000 7fbd6259c640 1 --2- 172.21.15.129:0/4049018351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.030+0000 7fbd61d9b640 1 -- 172.21.15.129:0/4049018351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2d36b0 msgr2=0x55aa4d6b6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.030+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/4049018351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6b6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.030+0000 7fbd61d9b640 1 -- 172.21.15.129:0/4049018351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2cb4d0 msgr2=0x55aa4d696ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.030+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/4049018351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d696ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.030+0000 7fbd61d9b640 1 -- 172.21.15.129:0/4049018351 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa4d2b3190 con 0x55aa4d2f4390 2024-05-20T18:10:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.031+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/4049018351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d6b6980 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009900 tx=0x7fbd58005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f425ad7ddab1bbe server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.032+0000 7fbd4bfff640 1 -- 172.21.15.129:0/4049018351 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58035040 con 0x55aa4d2f4390 2024-05-20T18:10:12.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.032+0000 7fbd4bfff640 1 -- 172.21.15.129:0/4049018351 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd5802ac40 con 0x55aa4d2f4390 2024-05-20T18:10:12.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.032+0000 7fbd4bfff640 1 -- 172.21.15.129:0/4049018351 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5804ea30 con 0x55aa4d2f4390 2024-05-20T18:10:12.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.033+0000 7fbd6f996740 1 -- 172.21.15.129:0/4049018351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 msgr2=0x55aa4d6b6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:12.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.033+0000 7fbd6f996740 1 --2- 172.21.15.129:0/4049018351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d6b6980 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009900 tx=0x7fbd58005fd0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.035+0000 7fbd6f996740 1 -- 172.21.15.129:0/4049018351 shutdown_connections 2024-05-20T18:10:12.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.035+0000 7fbd6f996740 1 --2- 172.21.15.129:0/4049018351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6b6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.035+0000 7fbd6f996740 1 --2- 172.21.15.129:0/4049018351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d6b6980 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.035+0000 7fbd6f996740 1 --2- 172.21.15.129:0/4049018351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d696ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.035+0000 7fbd6f996740 1 -- 172.21.15.129:0/4049018351 >> 172.21.15.129:0/4049018351 conn(0x55aa4d2e84e0 msgr2=0x55aa4d6967e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:12.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.040+0000 7fbd6f996740 1 -- 172.21.15.129:0/4049018351 shutdown_connections 2024-05-20T18:10:12.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.040+0000 7fbd6f996740 1 -- 172.21.15.129:0/4049018351 wait complete. 2024-05-20T18:10:12.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.041+0000 7fbd6f996740 1 -- start start 2024-05-20T18:10:12.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.041+0000 7fbd6f996740 1 Processor -- start 2024-05-20T18:10:12.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.042+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d6b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:12.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.042+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6b6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:12.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.043+0000 7fbd6f996740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d6b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:12.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.043+0000 7fbd6f996740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64072bc0 con 0x55aa4d2d36b0 2024-05-20T18:10:12.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.043+0000 7fbd6f996740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd641b59d0 con 0x55aa4d2cb4d0 2024-05-20T18:10:12.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.043+0000 7fbd61d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d6b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:12.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.043+0000 7fbd6f996740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64073290 con 0x55aa4d2f4390 2024-05-20T18:10:12.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.043+0000 7fbd6f996740 10 client.0 _ll_register_callbacks cb 0x7fbd58004100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:12.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.043+0000 7fbd61d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d6b63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.129:59812/0 (socket says 172.21.15.129:59812) 2024-05-20T18:10:12.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.043+0000 7fbd61d9b640 1 -- 172.21.15.129:0/2647863048 learned_addr learned my addr 172.21.15.129:0/2647863048 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:12.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.043+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d6b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:12.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.043+0000 7fbd6259c640 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:12.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.044+0000 7fbd6259c640 1 -- 172.21.15.129:0/2647863048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 msgr2=0x55aa4d6b27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:12.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.044+0000 7fbd6259c640 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d6b27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.044+0000 7fbd6259c640 1 -- 172.21.15.129:0/2647863048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2cb4d0 msgr2=0x55aa4d6b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:12.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.044+0000 7fbd6259c640 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d6b63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.044+0000 7fbd6259c640 1 -- 172.21.15.129:0/2647863048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa4d6b2f20 con 0x55aa4d2d36b0 2024-05-20T18:10:12.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.044+0000 7fbd6259c640 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6b6900 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x55aa4d6942f0 tx=0x7fbd540039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962cb05a472baf3f server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:12.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.045+0000 7fbd49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:10:12.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.046+0000 7fbd49ffb640 1 -- 172.21.15.129:0/2647863048 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd54024070 con 0x55aa4d2d36b0 2024-05-20T18:10:12.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.046+0000 7fbd49ffb640 1 -- 172.21.15.129:0/2647863048 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8631+0+0 (secure 0 0 0) 0x7fbd540108a0 con 0x55aa4d2d36b0 2024-05-20T18:10:12.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.046+0000 7fbd6f996740 1 -- 172.21.15.129:0/2647863048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd5802ac40 con 0x55aa4d2d36b0 2024-05-20T18:10:12.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.047+0000 7fbd49ffb640 1 -- 172.21.15.129:0/2647863048 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbd5402e050 con 0x55aa4d2d36b0 2024-05-20T18:10:12.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.050+0000 7fbd6f996740 10 client.17466 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:12.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.050+0000 7fbd6f996740 1 -- 172.21.15.129:0/2647863048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aa4d727bc0 con 0x55aa4d2d36b0 2024-05-20T18:10:12.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.055+0000 7fbd49ffb640 1 -- 172.21.15.129:0/2647863048 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd5802ac40 con 0x55aa4d2d36b0 2024-05-20T18:10:12.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.055+0000 7fbd49ffb640 1 -- 172.21.15.129:0/2647863048 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fbd54020870 con 0x55aa4d2d36b0 2024-05-20T18:10:12.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.055+0000 7fbd6f996740 10 client.17466 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:12.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.055+0000 7fbd6f996740 10 client.17466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:12.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.055+0000 7fbd6f996740 20 client.17466 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:12.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.055+0000 7fbd6f996740 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aa4d2883a0 0x55aa4d72a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:12.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.055+0000 7fbd6f996740 4 client.17466 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:10:12.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.055+0000 7fbd6f996740 1 -- 172.21.15.129:0/2647863048 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55aa4d57d6f0 con 0x55aa4d2883a0 2024-05-20T18:10:12.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.056+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aa4d2883a0 0x55aa4d72a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:12.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.057+0000 7fbd61d9b640 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aa4d2883a0 0x55aa4d72a380 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf9dab2e922f5a54 server_cookie=45bd5f4e1083c4ae in_seq=0 out_seq=0 2024-05-20T18:10:12.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.057+0000 7fbd49ffb640 10 client.17466 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:12.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.057+0000 7fbd49ffb640 1 -- 172.21.15.129:0/2647863048 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fbd5802adf0 con 0x55aa4d2883a0 2024-05-20T18:10:12.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.057+0000 7fbd49ffb640 10 client.17466 handle_command_reply: tid=0 2024-05-20T18:10:12.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.060+0000 7fbd6f996740 1 client.17466 shutdown 2024-05-20T18:10:12.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.061+0000 7fbd6f996740 1 -- 172.21.15.129:0/2647863048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2d36b0 msgr2=0x55aa4d6b6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:12.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.061+0000 7fbd6f996740 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6b6900 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x55aa4d6942f0 tx=0x7fbd540039c0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.063+0000 7fbd6f996740 1 -- 172.21.15.129:0/2647863048 shutdown_connections 2024-05-20T18:10:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.063+0000 7fbd6f996740 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x55aa4d2883a0 0x55aa4d72a380 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.063+0000 7fbd6f996740 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55aa4d2f4390 0x55aa4d6b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.063+0000 7fbd6f996740 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aa4d2d36b0 0x55aa4d6b6900 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.063+0000 7fbd6f996740 1 --2- 172.21.15.129:0/2647863048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aa4d2cb4d0 0x55aa4d6b63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.063+0000 7fbd6f996740 1 -- 172.21.15.129:0/2647863048 >> 172.21.15.129:0/2647863048 conn(0x55aa4d2e84e0 msgr2=0x55aa4d693250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:12.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.064+0000 7fbd49ffb640 0 client.17466 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:12.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.065+0000 7fbd6f996740 1 -- 172.21.15.129:0/2647863048 shutdown_connections 2024-05-20T18:10:12.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.065+0000 7fbd6f996740 1 -- 172.21.15.129:0/2647863048 wait complete. 2024-05-20T18:10:12.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.066+0000 7fbd6f996740 20 client.17466 trim_cache size 0 max 16384 2024-05-20T18:10:12.071 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:10:12.071 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-20T18:10:12.071 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": { 2024-05-20T18:10:12.071 INFO:teuthology.orchestra.run.smithi129.stdout: "1bb21371-3dd1-4008-a7d0-6c6d8414d188": { 2024-05-20T18:10:12.071 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/", 2024-05-20T18:10:12.072 INFO:teuthology.orchestra.run.smithi129.stdout: "tag": "1bb21371-3dd1-4008-a7d0-6c6d8414d188", 2024-05-20T18:10:12.072 INFO:teuthology.orchestra.run.smithi129.stdout: "options": "recursive,force" 2024-05-20T18:10:12.072 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:12.072 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:12.072 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:10:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.077+0000 7fbd68a14640 1 -- 172.21.15.129:0/2260354879 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd4c0a2f00 msgr2=0x7fbd4c0a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.077+0000 7fbd68a14640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd4c0a2f00 0x7fbd4c0a5410 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0a43d0 tx=0x7fbd5c097810 comp rx=0 tx=0).stop 2024-05-20T18:10:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.077+0000 7fbd68a14640 1 -- 172.21.15.129:0/2260354879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd641ace80 msgr2=0x7fbd641ad290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.077+0000 7fbd68a14640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd641ace80 0x7fbd641ad290 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009fd0 tx=0x7fbd5c00ef10 comp rx=0 tx=0).stop 2024-05-20T18:10:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.077+0000 7fbd68a14640 1 -- 172.21.15.129:0/2260354879 shutdown_connections 2024-05-20T18:10:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.077+0000 7fbd68a14640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fbd4c0a2f00 0x7fbd4c0a5410 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.077+0000 7fbd68a14640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd641ace80 0x7fbd641ad290 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.077+0000 7fbd68a14640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64105df0 0x7fbd641a8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.077+0000 7fbd68a14640 1 --2- 172.21.15.129:0/2260354879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd640689d0 0x7fbd641a8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.077+0000 7fbd68a14640 1 -- 172.21.15.129:0/2260354879 >> 172.21.15.129:0/2260354879 conn(0x7fbd64076830 msgr2=0x7fbd6410f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:12.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.080+0000 7fbd68a14640 1 -- 172.21.15.129:0/2260354879 shutdown_connections 2024-05-20T18:10:12.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:12.080+0000 7fbd68a14640 1 -- 172.21.15.129:0/2260354879 wait complete. 2024-05-20T18:10:12.099 INFO:tasks.cephfs.filesystem:scrub status for tag:1bb21371-3dd1-4008-a7d0-6c6d8414d188 - {'path': '/', 'tag': '1bb21371-3dd1-4008-a7d0-6c6d8414d188', 'options': 'recursive,force'} 2024-05-20T18:10:12.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:11 smithi154 ceph-mon[36482]: osdmap e106: 12 total, 12 up, 12 in 2024-05-20T18:10:12.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:11 smithi154 ceph-mon[36482]: pgmap v364: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 31 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-05-20T18:10:12.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:11 smithi204 ceph-mon[37076]: osdmap e106: 12 total, 12 up, 12 in 2024-05-20T18:10:12.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:11 smithi204 ceph-mon[37076]: pgmap v364: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 31 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-05-20T18:10:13.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:12 smithi154 ceph-mon[36482]: scrub queued for path: / 2024-05-20T18:10:13.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:12 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:10:13.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:12 smithi154 ceph-mon[36482]: scrub summary: active paths [/] 2024-05-20T18:10:13.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:12 smithi154 ceph-mon[36482]: scrub summary: idle+waiting paths [/] 2024-05-20T18:10:13.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:12 smithi129 ceph-mon[32029]: scrub queued for path: / 2024-05-20T18:10:13.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:12 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:10:13.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:12 smithi129 ceph-mon[32029]: scrub summary: active paths [/] 2024-05-20T18:10:13.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:12 smithi129 ceph-mon[32029]: scrub summary: idle+waiting paths [/] 2024-05-20T18:10:13.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:12 smithi204 ceph-mon[37076]: scrub queued for path: / 2024-05-20T18:10:13.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:12 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:10:13.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:12 smithi204 ceph-mon[37076]: scrub summary: active paths [/] 2024-05-20T18:10:13.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:12 smithi204 ceph-mon[37076]: scrub summary: idle+waiting paths [/] 2024-05-20T18:10:14.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:13 smithi154 ceph-mon[36482]: pgmap v365: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 29 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 28 KiB/s wr, 6 op/s 2024-05-20T18:10:14.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:13 smithi129 ceph-mon[32029]: pgmap v365: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 29 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 28 KiB/s wr, 6 op/s 2024-05-20T18:10:14.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:13 smithi204 ceph-mon[37076]: pgmap v365: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 29 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 28 KiB/s wr, 6 op/s 2024-05-20T18:10:15.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:10:15.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:10:15.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:14 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:10:15.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:10:15.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:10:15.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:14 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:10:15.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:10:15.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:10:15.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:14 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:10:16.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:15 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:10:16.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:15 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:10:16.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:15 smithi154 ceph-mon[36482]: pgmap v366: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 22 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-20T18:10:16.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:15 smithi154 ceph-mon[36482]: pgmap v367: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 22 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-05-20T18:10:16.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:15 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:10:16.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:15 smithi154 ceph-mon[36482]: scrub summary: idle 2024-05-20T18:10:16.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:15 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:10:16.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:15 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:10:16.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:15 smithi129 ceph-mon[32029]: pgmap v366: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 22 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-20T18:10:16.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:15 smithi129 ceph-mon[32029]: pgmap v367: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 22 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-05-20T18:10:16.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:15 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:10:16.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:15 smithi129 ceph-mon[32029]: scrub summary: idle 2024-05-20T18:10:16.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:15 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:10:16.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:15 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:10:16.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:15 smithi204 ceph-mon[37076]: pgmap v366: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 22 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-20T18:10:16.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:15 smithi204 ceph-mon[37076]: pgmap v367: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 22 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-05-20T18:10:16.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:15 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:10:16.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:15 smithi204 ceph-mon[37076]: scrub summary: idle 2024-05-20T18:10:17.958 INFO:tasks.workunit.client.0.smithi129.stdout:stopping iogen 2024-05-20T18:10:17.958 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'stopping iogen' 2024-05-20T18:10:17.958 INFO:tasks.workunit.client.0.smithi129.stderr:+ ./iogen -k 2024-05-20T18:10:17.959 INFO:tasks.workunit.client.0.smithi129.stdout:OK 2024-05-20T18:10:17.960 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo OK 2024-05-20T18:10:18.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:18 smithi154 ceph-mon[36482]: pgmap v368: 97 pgs: 97 active+clean; 22 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-20T18:10:18.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:18 smithi129 ceph-mon[32029]: pgmap v368: 97 pgs: 97 active+clean; 22 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-20T18:10:18.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:18 smithi204 ceph-mon[37076]: pgmap v368: 97 pgs: 97 active+clean; 22 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-20T18:10:19.977 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-20T18:10:19.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-20T18:10:20.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:20 smithi154 ceph-mon[36482]: pgmap v369: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-20T18:10:20.644 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:20 smithi129 ceph-mon[32029]: pgmap v369: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-20T18:10:20.645 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-05-20T18:10:20.645 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-20T18:10:20.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:20 smithi204 ceph-mon[37076]: pgmap v369: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-20T18:10:20.950 DEBUG:teuthology.parallel:result is None 2024-05-20T18:10:20.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-20T18:10:20.984 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-20T18:10:20.984 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-05-20T18:10:20.996 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi129.front.sepia.ceph.com 2024-05-20T18:10:21.033 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-20T18:10:21.043 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-20T18:10:21.043 INFO:teuthology.task.exec:Executing custom commands... 2024-05-20T18:10:21.043 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi129.front.sepia.ceph.com 2024-05-20T18:10:21.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-20T18:10:21.105 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-20T18:10:21.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-20T18:10:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.345+0000 7fd9c122b640 1 -- 172.21.15.129:0/2579038696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9bc102170 msgr2=0x7fd9bc10e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.345+0000 7fd9c122b640 1 --2- 172.21.15.129:0/2579038696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9bc102170 0x7fd9bc10e4a0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009ca0 tx=0x7fd9b402a8c0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.345+0000 7fd9c122b640 1 -- 172.21.15.129:0/2579038696 shutdown_connections 2024-05-20T18:10:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.345+0000 7fd9c122b640 1 --2- 172.21.15.129:0/2579038696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9bc102170 0x7fd9bc10e4a0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.345+0000 7fd9c122b640 1 --2- 172.21.15.129:0/2579038696 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9bc107eb0 0x7fd9bc101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.345+0000 7fd9c122b640 1 --2- 172.21.15.129:0/2579038696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9bc107a10 0x7fd9bc1015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.345+0000 7fd9c122b640 1 -- 172.21.15.129:0/2579038696 >> 172.21.15.129:0/2579038696 conn(0x7fd9bc077160 msgr2=0x7fd9bc0775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:21.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.345+0000 7fd9c122b640 1 -- 172.21.15.129:0/2579038696 shutdown_connections 2024-05-20T18:10:21.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.345+0000 7fd9c122b640 1 -- 172.21.15.129:0/2579038696 wait complete. 2024-05-20T18:10:21.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9c122b640 1 Processor -- start 2024-05-20T18:10:21.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9c122b640 1 -- start start 2024-05-20T18:10:21.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9c122b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9bc102170 0x7fd9bc19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9c122b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9bc107a10 0x7fd9bc19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9c122b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9bc107eb0 0x7fd9bc19b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9c122b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc111d80 con 0x7fd9bc107eb0 2024-05-20T18:10:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9c122b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc111c00 con 0x7fd9bc102170 2024-05-20T18:10:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9c122b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc111f00 con 0x7fd9bc107a10 2024-05-20T18:10:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9bb59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9bc107eb0 0x7fd9bc19b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9bb59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9bc107eb0 0x7fd9bc19b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45308/0 (socket says 172.21.15.129:45308) 2024-05-20T18:10:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9bb59e640 1 -- 172.21.15.129:0/3291834327 learned_addr learned my addr 172.21.15.129:0/3291834327 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9bad9d640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9bc102170 0x7fd9bc19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.346+0000 7fd9ba59c640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9bc107a10 0x7fd9bc19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.347+0000 7fd9bb59e640 1 -- 172.21.15.129:0/3291834327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9bc107a10 msgr2=0x7fd9bc19ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.347+0000 7fd9bb59e640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9bc107a10 0x7fd9bc19ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.347+0000 7fd9bb59e640 1 -- 172.21.15.129:0/3291834327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9bc102170 msgr2=0x7fd9bc19a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.347+0000 7fd9bb59e640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9bc102170 0x7fd9bc19a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.347+0000 7fd9bb59e640 1 -- 172.21.15.129:0/3291834327 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9bc1a5980 con 0x7fd9bc107eb0 2024-05-20T18:10:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.347+0000 7fd9bb59e640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9bc107eb0 0x7fd9bc19b0e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4031a20 tx=0x7fd9b4031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.347+0000 7fd9a3fff640 1 -- 172.21.15.129:0/3291834327 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4003c30 con 0x7fd9bc107eb0 2024-05-20T18:10:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.347+0000 7fd9c122b640 1 -- 172.21.15.129:0/3291834327 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9bc1a5c10 con 0x7fd9bc107eb0 2024-05-20T18:10:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.347+0000 7fd9a3fff640 1 -- 172.21.15.129:0/3291834327 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9b4003dd0 con 0x7fd9bc107eb0 2024-05-20T18:10:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.347+0000 7fd9a3fff640 1 -- 172.21.15.129:0/3291834327 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b402fcf0 con 0x7fd9bc107eb0 2024-05-20T18:10:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.347+0000 7fd9c122b640 1 -- 172.21.15.129:0/3291834327 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9bc1a61b0 con 0x7fd9bc107eb0 2024-05-20T18:10:21.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.348+0000 7fd9c122b640 1 -- 172.21.15.129:0/3291834327 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9bc0690c0 con 0x7fd9bc107eb0 2024-05-20T18:10:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.349+0000 7fd9a3fff640 1 -- 172.21.15.129:0/3291834327 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fd9b4004050 con 0x7fd9bc107eb0 2024-05-20T18:10:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.350+0000 7fd9a3fff640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd9a40a2cf0 0x7fd9a40a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.350+0000 7fd9bad9d640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd9a40a2cf0 0x7fd9a40a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.350+0000 7fd9a3fff640 1 -- 172.21.15.129:0/3291834327 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7fd9b40f0680 con 0x7fd9bc107eb0 2024-05-20T18:10:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.351+0000 7fd9bad9d640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd9a40a2cf0 0x7fd9a40a5200 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac005d00 tx=0x7fd9ac00b410 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.355+0000 7fd9a3fff640 1 -- 172.21.15.129:0/3291834327 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd9b40bd400 con 0x7fd9bc107eb0 2024-05-20T18:10:21.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.555+0000 7fd9c122b640 1 -- 172.21.15.129:0/3291834327 --> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fd9bc109b20 con 0x7fd9a40a2cf0 2024-05-20T18:10:21.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.563+0000 7fd9a3fff640 1 -- 172.21.15.129:0/3291834327 <== mgr.15228 v2:172.21.15.129:6800/3515662734 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fd9ac00bc40 con 0x7fd9a40a2cf0 2024-05-20T18:10:21.561 INFO:teuthology.orchestra.run.smithi129.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-20T00:00:00", "created": "2024-05-20T17:58:49", "first": "2024-05-20T17:59:00", "last": "2024-05-20T18:10:00", "last_pruned": "2024-05-20T18:10:00", "created_count": 12, "pruned_count": 5, "active": true} 2024-05-20T18:10:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 -- 172.21.15.129:0/3291834327 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd9a40a2cf0 msgr2=0x7fd9a40a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd9a40a2cf0 0x7fd9a40a5200 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac005d00 tx=0x7fd9ac00b410 comp rx=0 tx=0).stop 2024-05-20T18:10:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 -- 172.21.15.129:0/3291834327 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9bc107eb0 msgr2=0x7fd9bc19b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9bc107eb0 0x7fd9bc19b0e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4031a20 tx=0x7fd9b4031a50 comp rx=0 tx=0).stop 2024-05-20T18:10:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 -- 172.21.15.129:0/3291834327 shutdown_connections 2024-05-20T18:10:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd9a40a2cf0 0x7fd9a40a5200 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9bc107eb0 0x7fd9bc19b0e0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9bc107a10 0x7fd9bc19ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 --2- 172.21.15.129:0/3291834327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9bc102170 0x7fd9bc19a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 -- 172.21.15.129:0/3291834327 >> 172.21.15.129:0/3291834327 conn(0x7fd9bc077160 msgr2=0x7fd9bc0ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 -- 172.21.15.129:0/3291834327 shutdown_connections 2024-05-20T18:10:21.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.566+0000 7fd9c122b640 1 -- 172.21.15.129:0/3291834327 wait complete. 2024-05-20T18:10:21.573 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:21 smithi129 ceph-mon[32029]: osdmap e107: 12 total, 12 up, 12 in 2024-05-20T18:10:21.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-20T18:10:21.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:21 smithi154 ceph-mon[36482]: osdmap e107: 12 total, 12 up, 12 in 2024-05-20T18:10:21.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:21 smithi204 ceph-mon[37076]: osdmap e107: 12 total, 12 up, 12 in 2024-05-20T18:10:21.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.691+0000 7f162e109640 1 -- 172.21.15.129:0/2657268710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1628107ed0 msgr2=0x7f1628102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:21.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.691+0000 7f162e109640 1 --2- 172.21.15.129:0/2657268710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1628107ed0 0x7f1628102170 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f161c009900 tx=0x7f161c02a410 comp rx=0 tx=0).stop 2024-05-20T18:10:21.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.691+0000 7f162e109640 1 -- 172.21.15.129:0/2657268710 shutdown_connections 2024-05-20T18:10:21.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.691+0000 7f162e109640 1 --2- 172.21.15.129:0/2657268710 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f16281027e0 0x7f162810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.691+0000 7f162e109640 1 --2- 172.21.15.129:0/2657268710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1628107ed0 0x7f1628102170 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.691+0000 7f162e109640 1 --2- 172.21.15.129:0/2657268710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1628107a30 0x7f1628101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.691+0000 7f162e109640 1 -- 172.21.15.129:0/2657268710 >> 172.21.15.129:0/2657268710 conn(0x7f16280fc850 msgr2=0x7f16280feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:21.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.691+0000 7f162e109640 1 -- 172.21.15.129:0/2657268710 shutdown_connections 2024-05-20T18:10:21.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.691+0000 7f162e109640 1 -- 172.21.15.129:0/2657268710 wait complete. 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.692+0000 7f162e109640 1 Processor -- start 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.692+0000 7f162e109640 1 -- start start 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f162e109640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16281027e0 0x7f162819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f162e109640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1628107a30 0x7f162819acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f162e109640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1628107ed0 0x7f162819b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f162e109640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1628111dd0 con 0x7f1628107a30 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f162e109640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1628111c50 con 0x7f16281027e0 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f162e109640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1628111f50 con 0x7f1628107ed0 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f1626ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1628107a30 0x7f162819acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f1626ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1628107a30 0x7f162819acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45328/0 (socket says 172.21.15.129:45328) 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f1626ffd640 1 -- 172.21.15.129:0/2721118780 learned_addr learned my addr 172.21.15.129:0/2721118780 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:21.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f1627fff640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1628107ed0 0x7f162819b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f16277fe640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16281027e0 0x7f162819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f1626ffd640 1 -- 172.21.15.129:0/2721118780 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1628107ed0 msgr2=0x7f162819b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f1626ffd640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1628107ed0 0x7f162819b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f1626ffd640 1 -- 172.21.15.129:0/2721118780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16281027e0 msgr2=0x7f162819a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f1626ffd640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16281027e0 0x7f162819a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f1626ffd640 1 -- 172.21.15.129:0/2721118780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16281a5ac0 con 0x7f1628107a30 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f16277fe640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16281027e0 0x7f162819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.693+0000 7f1626ffd640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1628107a30 0x7f162819acf0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f161c005940 tx=0x7f161c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.694+0000 7f1624ff9640 1 -- 172.21.15.129:0/2721118780 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c03f030 con 0x7f1628107a30 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.694+0000 7f1624ff9640 1 -- 172.21.15.129:0/2721118780 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f161c003ce0 con 0x7f1628107a30 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.694+0000 7f1624ff9640 1 -- 172.21.15.129:0/2721118780 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c035070 con 0x7f1628107a30 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.694+0000 7f162e109640 1 -- 172.21.15.129:0/2721118780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16281a5d50 con 0x7f1628107a30 2024-05-20T18:10:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.694+0000 7f162e109640 1 -- 172.21.15.129:0/2721118780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16281a61e0 con 0x7f1628107a30 2024-05-20T18:10:21.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.695+0000 7f162e109640 1 -- 172.21.15.129:0/2721118780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f162804f720 con 0x7f1628107a30 2024-05-20T18:10:21.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.696+0000 7f1624ff9640 1 -- 172.21.15.129:0/2721118780 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f161c042030 con 0x7f1628107a30 2024-05-20T18:10:21.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.697+0000 7f1624ff9640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16100a2cf0 0x7f16100a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:21.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.697+0000 7f16277fe640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16100a2cf0 0x7f16100a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:21.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.697+0000 7f1624ff9640 1 -- 172.21.15.129:0/2721118780 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7f161c0efc70 con 0x7f1628107a30 2024-05-20T18:10:21.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.697+0000 7f16277fe640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16100a2cf0 0x7f16100a5200 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f162806a1e0 tx=0x7f1618008040 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:21.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.701+0000 7f1624ff9640 1 -- 172.21.15.129:0/2721118780 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f161c0bc850 con 0x7f1628107a30 2024-05-20T18:10:21.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.901+0000 7f162e109640 1 -- 172.21.15.129:0/2721118780 --> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f162810dd00 con 0x7f16100a2cf0 2024-05-20T18:10:21.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.909+0000 7f1624ff9640 1 -- 172.21.15.129:0/2721118780 <== mgr.15228 v2:172.21.15.129:6800/3515662734 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f162810dd00 con 0x7f16100a2cf0 2024-05-20T18:10:21.907 INFO:teuthology.orchestra.run.smithi129.stdout:/ 1m 6m3h 2024-05-20T18:10:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.911+0000 7f162e109640 1 -- 172.21.15.129:0/2721118780 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16100a2cf0 msgr2=0x7f16100a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.911+0000 7f162e109640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16100a2cf0 0x7f16100a5200 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f162806a1e0 tx=0x7f1618008040 comp rx=0 tx=0).stop 2024-05-20T18:10:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.911+0000 7f162e109640 1 -- 172.21.15.129:0/2721118780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1628107a30 msgr2=0x7f162819acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.911+0000 7f162e109640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1628107a30 0x7f162819acf0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f161c005940 tx=0x7f161c0042f0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.911+0000 7f162e109640 1 -- 172.21.15.129:0/2721118780 shutdown_connections 2024-05-20T18:10:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.911+0000 7f162e109640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f16100a2cf0 0x7f16100a5200 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.911+0000 7f162e109640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1628107ed0 0x7f162819b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.911+0000 7f162e109640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1628107a30 0x7f162819acf0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.911+0000 7f162e109640 1 --2- 172.21.15.129:0/2721118780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16281027e0 0x7f162819a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:21.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.911+0000 7f162e109640 1 -- 172.21.15.129:0/2721118780 >> 172.21.15.129:0/2721118780 conn(0x7f16280fc850 msgr2=0x7f162810d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:21.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.912+0000 7f162e109640 1 -- 172.21.15.129:0/2721118780 shutdown_connections 2024-05-20T18:10:21.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:21.912+0000 7f162e109640 1 -- 172.21.15.129:0/2721118780 wait complete. 2024-05-20T18:10:21.920 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-20T18:10:21.931 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-20T18:10:22.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:22 smithi154 ceph-mon[36482]: pgmap v371: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-20T18:10:22.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:22 smithi154 ceph-mon[36482]: from='client.17472 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T18:10:22.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:22 smithi154 ceph-mon[36482]: from='client.17478 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T18:10:22.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:22 smithi129 ceph-mon[32029]: pgmap v371: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-20T18:10:22.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:22 smithi129 ceph-mon[32029]: from='client.17472 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T18:10:22.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:22 smithi129 ceph-mon[32029]: from='client.17478 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T18:10:22.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:22 smithi204 ceph-mon[37076]: pgmap v371: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-20T18:10:22.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:22 smithi204 ceph-mon[37076]: from='client.17472 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T18:10:22.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:22 smithi204 ceph-mon[37076]: from='client.17478 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-20T18:10:24.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:24 smithi154 ceph-mon[36482]: pgmap v372: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-20T18:10:24.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:24 smithi129 ceph-mon[32029]: pgmap v372: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-20T18:10:24.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:24 smithi204 ceph-mon[37076]: pgmap v372: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-20T18:10:26.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:26 smithi154 ceph-mon[36482]: pgmap v373: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 6 op/s 2024-05-20T18:10:26.662 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:26 smithi129 ceph-mon[32029]: pgmap v373: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 6 op/s 2024-05-20T18:10:26.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:26 smithi204 ceph-mon[37076]: pgmap v373: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 6 op/s 2024-05-20T18:10:28.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:28 smithi154 ceph-mon[36482]: pgmap v374: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-05-20T18:10:28.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:28 smithi129 ceph-mon[32029]: pgmap v374: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-05-20T18:10:28.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:28 smithi204 ceph-mon[37076]: pgmap v374: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2024-05-20T18:10:30.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:30 smithi154 ceph-mon[36482]: pgmap v375: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 39 op/s 2024-05-20T18:10:30.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:30 smithi129 ceph-mon[32029]: pgmap v375: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 39 op/s 2024-05-20T18:10:30.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:30 smithi204 ceph-mon[37076]: pgmap v375: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 39 op/s 2024-05-20T18:10:32.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:32 smithi154 ceph-mon[36482]: pgmap v376: 97 pgs: 97 active+clean; 22 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 205 op/s 2024-05-20T18:10:32.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:32 smithi129 ceph-mon[32029]: pgmap v376: 97 pgs: 97 active+clean; 22 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 205 op/s 2024-05-20T18:10:32.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:32 smithi204 ceph-mon[37076]: pgmap v376: 97 pgs: 97 active+clean; 22 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 205 op/s 2024-05-20T18:10:34.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:34 smithi154 ceph-mon[36482]: pgmap v377: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 218 op/s 2024-05-20T18:10:34.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:34 smithi129 ceph-mon[32029]: pgmap v377: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 218 op/s 2024-05-20T18:10:34.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:34 smithi204 ceph-mon[37076]: pgmap v377: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 218 op/s 2024-05-20T18:10:36.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:36 smithi154 ceph-mon[36482]: pgmap v378: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 222 op/s 2024-05-20T18:10:36.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:36 smithi129 ceph-mon[32029]: pgmap v378: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 222 op/s 2024-05-20T18:10:36.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:36 smithi204 ceph-mon[37076]: pgmap v378: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 222 op/s 2024-05-20T18:10:38.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:38 smithi154 ceph-mon[36482]: pgmap v379: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.2 KiB/s wr, 218 op/s 2024-05-20T18:10:38.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:38 smithi129 ceph-mon[32029]: pgmap v379: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.2 KiB/s wr, 218 op/s 2024-05-20T18:10:38.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:38 smithi204 ceph-mon[37076]: pgmap v379: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.2 KiB/s wr, 218 op/s 2024-05-20T18:10:40.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:40 smithi154 ceph-mon[36482]: pgmap v380: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.2 KiB/s wr, 217 op/s 2024-05-20T18:10:40.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:40 smithi129 ceph-mon[32029]: pgmap v380: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.2 KiB/s wr, 217 op/s 2024-05-20T18:10:40.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:40 smithi204 ceph-mon[37076]: pgmap v380: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.2 KiB/s wr, 217 op/s 2024-05-20T18:10:42.100 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-20T18:10:42.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-20T18:10:42.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.233+0000 7fc852307640 1 -- 172.21.15.129:0/83762509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc84c103bb0 msgr2=0x7fc84c103fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.233+0000 7fc852307640 1 --2- 172.21.15.129:0/83762509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc84c103bb0 0x7fc84c103fc0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009900 tx=0x7fc83c02a410 comp rx=0 tx=0).stop 2024-05-20T18:10:42.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.233+0000 7fc852307640 1 -- 172.21.15.129:0/83762509 shutdown_connections 2024-05-20T18:10:42.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.233+0000 7fc852307640 1 --2- 172.21.15.129:0/83762509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c10d000 0x7fc84c10f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.233+0000 7fc852307640 1 --2- 172.21.15.129:0/83762509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc84c10a610 0x7fc84c10ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.233+0000 7fc852307640 1 --2- 172.21.15.129:0/83762509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc84c103bb0 0x7fc84c103fc0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.233+0000 7fc852307640 1 -- 172.21.15.129:0/83762509 >> 172.21.15.129:0/83762509 conn(0x7fc84c0fd210 msgr2=0x7fc84c0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.233+0000 7fc852307640 1 -- 172.21.15.129:0/83762509 shutdown_connections 2024-05-20T18:10:42.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.233+0000 7fc852307640 1 -- 172.21.15.129:0/83762509 wait complete. 2024-05-20T18:10:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.234+0000 7fc852307640 1 Processor -- start 2024-05-20T18:10:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.234+0000 7fc852307640 1 -- start start 2024-05-20T18:10:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.234+0000 7fc852307640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc84c103bb0 0x7fc84c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.234+0000 7fc852307640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc84c10a610 0x7fc84c19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.234+0000 7fc852307640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c10d000 0x7fc84c1996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.234+0000 7fc852307640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c112e40 con 0x7fc84c10a610 2024-05-20T18:10:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.234+0000 7fc852307640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c112cc0 con 0x7fc84c10d000 2024-05-20T18:10:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.234+0000 7fc852307640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c112fc0 con 0x7fc84c103bb0 2024-05-20T18:10:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.234+0000 7fc843fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc84c10a610 0x7fc84c19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc843fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc84c10a610 0x7fc84c19fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60986/0 (socket says 172.21.15.129:60986) 2024-05-20T18:10:42.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc843fff640 1 -- 172.21.15.129:0/2339895448 learned_addr learned my addr 172.21.15.129:0/2339895448 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc84bfff640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc84c103bb0 0x7fc84c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc8508a4640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c10d000 0x7fc84c1996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc843fff640 1 -- 172.21.15.129:0/2339895448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc84c103bb0 msgr2=0x7fc84c19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc843fff640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc84c103bb0 0x7fc84c19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc843fff640 1 -- 172.21.15.129:0/2339895448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c10d000 msgr2=0x7fc84c1996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc843fff640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c10d000 0x7fc84c1996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc843fff640 1 -- 172.21.15.129:0/2339895448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c199dd0 con 0x7fc84c10a610 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc843fff640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc84c10a610 0x7fc84c19fa60 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fc83400d750 tx=0x7fc83400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc849ffb640 1 -- 172.21.15.129:0/2339895448 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834017070 con 0x7fc84c10a610 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc849ffb640 1 -- 172.21.15.129:0/2339895448 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc834005500 con 0x7fc84c10a610 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc852307640 1 -- 172.21.15.129:0/2339895448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84c19a060 con 0x7fc84c10a610 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc849ffb640 1 -- 172.21.15.129:0/2339895448 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834004e00 con 0x7fc84c10a610 2024-05-20T18:10:42.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.235+0000 7fc852307640 1 -- 172.21.15.129:0/2339895448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84c1a62c0 con 0x7fc84c10a610 2024-05-20T18:10:42.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.238+0000 7fc849ffb640 1 -- 172.21.15.129:0/2339895448 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc834004fa0 con 0x7fc84c10a610 2024-05-20T18:10:42.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.238+0000 7fc8591a8740 1 Processor -- start 2024-05-20T18:10:42.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.238+0000 7fc8591a8740 1 -- start start 2024-05-20T18:10:42.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.238+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe038d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.238+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc714d0 0x563afe039270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.238+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc796b0 0x563afe0397b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.238+0000 7fc8591a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563afdf9ed00 con 0x563afdc9a390 2024-05-20T18:10:42.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.238+0000 7fc8591a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563afdf464c0 con 0x563afdc796b0 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.238+0000 7fc8591a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x563afdf9acc0 con 0x563afdc714d0 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8508a4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8508a4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe038d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60994/0 (socket says 172.21.15.129:60994) 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8508a4640 1 -- 172.21.15.129:0/348381284 learned_addr learned my addr 172.21.15.129:0/348381284 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc849ffb640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8380a2d80 0x7fc8380a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc843fff640 1 --2- 172.21.15.129:0/348381284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc796b0 0x563afe0397b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc84bfff640 1 --2- 172.21.15.129:0/348381284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc714d0 0x563afe039270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8508a4640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8380a2d80 0x7fc8380a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8508a4640 1 -- 172.21.15.129:0/348381284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc714d0 msgr2=0x563afe039270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8508a4640 1 --2- 172.21.15.129:0/348381284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc714d0 0x563afe039270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc849ffb640 1 -- 172.21.15.129:0/2339895448 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7fc8340ced20 con 0x7fc84c10a610 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8508a4640 1 -- 172.21.15.129:0/348381284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc796b0 msgr2=0x563afe0397b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8508a4640 1 --2- 172.21.15.129:0/348381284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc796b0 0x563afe0397b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8508a4640 1 -- 172.21.15.129:0/348381284 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563afdd7c7d0 con 0x563afdc9a390 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8508a4640 1 --2- 172.21.15.129:0/348381284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe038d30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc844005c40 tx=0x7fc844013f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea62910430fbb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8427fc640 1 -- 172.21.15.129:0/348381284 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844022020 con 0x563afdc9a390 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8427fc640 1 -- 172.21.15.129:0/348381284 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8440134e0 con 0x563afdc9a390 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8427fc640 1 -- 172.21.15.129:0/348381284 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8440109c0 con 0x563afdc9a390 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.239+0000 7fc8508a4640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8380a2d80 0x7fc8380a5290 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc844023000 tx=0x7fc844021000 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.240+0000 7fc8591a8740 1 -- 172.21.15.129:0/348381284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 msgr2=0x563afe038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.240+0000 7fc8591a8740 1 --2- 172.21.15.129:0/348381284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe038d30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc844005c40 tx=0x7fc844013f50 comp rx=0 tx=0).stop 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.240+0000 7fc8591a8740 1 -- 172.21.15.129:0/348381284 shutdown_connections 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.240+0000 7fc8591a8740 1 --2- 172.21.15.129:0/348381284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc796b0 0x563afe0397b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.240+0000 7fc8591a8740 1 --2- 172.21.15.129:0/348381284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc714d0 0x563afe039270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.240+0000 7fc8591a8740 1 --2- 172.21.15.129:0/348381284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe038d30 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.240+0000 7fc8591a8740 1 -- 172.21.15.129:0/348381284 >> 172.21.15.129:0/348381284 conn(0x563afdc8e4e0 msgr2=0x563afe0354e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.240+0000 7fc8591a8740 1 -- 172.21.15.129:0/348381284 shutdown_connections 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.240+0000 7fc8591a8740 1 -- 172.21.15.129:0/348381284 wait complete. 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8591a8740 1 -- start start 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8591a8740 1 Processor -- start 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc714d0 0x563afe044e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc796b0 0x563afe0453d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe043310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8591a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563afe045fc0 con 0x563afdc9a390 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8591a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563afdf59b90 con 0x563afdc714d0 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8591a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x563afe046140 con 0x563afdc796b0 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8591a8740 10 client.0 _ll_register_callbacks cb 0x563afdf95da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:42.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8508a4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe043310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8508a4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe043310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60996/0 (socket says 172.21.15.129:60996) 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.241+0000 7fc8508a4640 1 -- 172.21.15.129:0/1214429417 learned_addr learned my addr 172.21.15.129:0/1214429417 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc84bfff640 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc714d0 0x563afe044e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc843fff640 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc796b0 0x563afe0453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc8508a4640 1 -- 172.21.15.129:0/1214429417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc796b0 msgr2=0x563afe0453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc8508a4640 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc796b0 0x563afe0453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc8508a4640 1 -- 172.21.15.129:0/1214429417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc714d0 msgr2=0x563afe044e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc8508a4640 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc714d0 0x563afe044e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc8508a4640 1 -- 172.21.15.129:0/1214429417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563afdcb23f0 con 0x563afdc9a390 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc8508a4640 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe043310 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc844010f80 tx=0x7fc844010490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a939d818044f79b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc82b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:10:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc82b7fe640 1 -- 172.21.15.129:0/1214429417 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844034060 con 0x563afdc9a390 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc82b7fe640 1 -- 172.21.15.129:0/1214429417 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7fc844010890 con 0x563afdc9a390 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc8591a8740 1 -- 172.21.15.129:0/1214429417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563afdfac430 con 0x563afdc9a390 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc82b7fe640 1 -- 172.21.15.129:0/1214429417 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc844022070 con 0x563afdc9a390 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc8591a8740 10 client.17490 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc8591a8740 1 -- 172.21.15.129:0/1214429417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563afdc720f0 con 0x563afdc9a390 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.242+0000 7fc82b7fe640 1 -- 172.21.15.129:0/1214429417 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563afdfac430 con 0x563afdc9a390 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.243+0000 7fc82b7fe640 1 -- 172.21.15.129:0/1214429417 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fc84402e030 con 0x563afdc9a390 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.243+0000 7fc8591a8740 10 client.17490 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.243+0000 7fc8591a8740 10 client.17490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.243+0000 7fc8591a8740 20 client.17490 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.243+0000 7fc8591a8740 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x563afdc58f60 0x563afe055a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.243+0000 7fc8591a8740 4 client.17490 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.243+0000 7fc8591a8740 1 -- 172.21.15.129:0/1214429417 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563afdfa6780 con 0x563afdc58f60 2024-05-20T18:10:42.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.243+0000 7fc84bfff640 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x563afdc58f60 0x563afe055a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.243+0000 7fc82b7fe640 10 client.17490 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.243+0000 7fc84bfff640 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x563afdc58f60 0x563afe055a90 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6893db5063f94a49 server_cookie=3071a53628651899 in_seq=0 out_seq=0 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc82b7fe640 1 -- 172.21.15.129:0/1214429417 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563afdfa6780 con 0x563afdc58f60 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc82b7fe640 10 client.17490 handle_command_reply: tid=0 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc8591a8740 1 client.17490 shutdown 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc8591a8740 1 -- 172.21.15.129:0/1214429417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 msgr2=0x563afe043310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc8591a8740 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe043310 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc844010f80 tx=0x7fc844010490 comp rx=0 tx=0).stop 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc8591a8740 1 -- 172.21.15.129:0/1214429417 shutdown_connections 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc8591a8740 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x563afdc58f60 0x563afe055a90 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc8591a8740 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc9a390 0x563afe043310 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc8591a8740 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc796b0 0x563afe0453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc82b7fe640 0 client.17490 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:42.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc8591a8740 1 --2- 172.21.15.129:0/1214429417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc714d0 0x563afe044e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.244+0000 7fc8591a8740 1 -- 172.21.15.129:0/1214429417 >> 172.21.15.129:0/1214429417 conn(0x563afdc8e4e0 msgr2=0x563afe03aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.245+0000 7fc8591a8740 1 -- 172.21.15.129:0/1214429417 shutdown_connections 2024-05-20T18:10:42.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.245+0000 7fc8591a8740 1 -- 172.21.15.129:0/1214429417 wait complete. 2024-05-20T18:10:42.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.245+0000 7fc8591a8740 20 client.17490 trim_cache size 0 max 16384 2024-05-20T18:10:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.262+0000 7fc8591a8740 1 Processor -- start 2024-05-20T18:10:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8591a8740 1 -- start start 2024-05-20T18:10:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc8e4e0 0x563afe03e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc58f60 0x563afe0424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe04f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8591a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1acb60 con 0x563afdc714d0 2024-05-20T18:10:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8591a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1af3c0 con 0x563afdc8e4e0 2024-05-20T18:10:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8591a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1ad1b0 con 0x563afdc58f60 2024-05-20T18:10:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8508a4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe04f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8508a4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe04f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32768/0 (socket says 172.21.15.129:32768) 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8508a4640 1 -- 172.21.15.129:0/819865087 learned_addr learned my addr 172.21.15.129:0/819865087 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc84bfff640 1 --2- 172.21.15.129:0/819865087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc8e4e0 0x563afe03e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc843fff640 1 --2- 172.21.15.129:0/819865087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc58f60 0x563afe0424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8508a4640 1 -- 172.21.15.129:0/819865087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc58f60 msgr2=0x563afe0424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8508a4640 1 --2- 172.21.15.129:0/819865087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc58f60 0x563afe0424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8508a4640 1 -- 172.21.15.129:0/819865087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc8e4e0 msgr2=0x563afe03e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8508a4640 1 --2- 172.21.15.129:0/819865087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc8e4e0 0x563afe03e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8508a4640 1 -- 172.21.15.129:0/819865087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563afe045f60 con 0x563afdc714d0 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc843fff640 1 --2- 172.21.15.129:0/819865087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc58f60 0x563afe0424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc8508a4640 1 --2- 172.21.15.129:0/819865087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe04f970 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc844010fd0 tx=0x7fc844012350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcee65139aa6495d server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc841ffb640 1 -- 172.21.15.129:0/819865087 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844034070 con 0x563afdc714d0 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc841ffb640 1 -- 172.21.15.129:0/819865087 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc844005bd0 con 0x563afdc714d0 2024-05-20T18:10:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.263+0000 7fc841ffb640 1 -- 172.21.15.129:0/819865087 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8440049d0 con 0x563afdc714d0 2024-05-20T18:10:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.264+0000 7fc8591a8740 1 -- 172.21.15.129:0/819865087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 msgr2=0x563afe04f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.264+0000 7fc8591a8740 1 --2- 172.21.15.129:0/819865087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe04f970 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc844010fd0 tx=0x7fc844012350 comp rx=0 tx=0).stop 2024-05-20T18:10:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.264+0000 7fc8591a8740 1 -- 172.21.15.129:0/819865087 shutdown_connections 2024-05-20T18:10:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.264+0000 7fc8591a8740 1 --2- 172.21.15.129:0/819865087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe04f970 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.264+0000 7fc8591a8740 1 --2- 172.21.15.129:0/819865087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc58f60 0x563afe0424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.264+0000 7fc8591a8740 1 --2- 172.21.15.129:0/819865087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc8e4e0 0x563afe03e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.264+0000 7fc8591a8740 1 -- 172.21.15.129:0/819865087 >> 172.21.15.129:0/819865087 conn(0x563afe03ec90 msgr2=0x563afe043210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.264+0000 7fc8591a8740 1 -- 172.21.15.129:0/819865087 shutdown_connections 2024-05-20T18:10:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.264+0000 7fc8591a8740 1 -- 172.21.15.129:0/819865087 wait complete. 2024-05-20T18:10:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc8591a8740 1 -- start start 2024-05-20T18:10:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc8591a8740 1 Processor -- start 2024-05-20T18:10:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc58f60 0x563afe0343a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe034900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc8591a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc8e4e0 0x563afe0b9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc8591a8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1ad880 con 0x563afdc714d0 2024-05-20T18:10:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc8591a8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1aecf0 con 0x563afdc58f60 2024-05-20T18:10:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc8591a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c1adf50 con 0x563afdc8e4e0 2024-05-20T18:10:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc8591a8740 10 client.0 _ll_register_callbacks cb 0x7fc83409b080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc843fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe034900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc843fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe034900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32772/0 (socket says 172.21.15.129:32772) 2024-05-20T18:10:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc843fff640 1 -- 172.21.15.129:0/3619502129 learned_addr learned my addr 172.21.15.129:0/3619502129 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc84bfff640 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc58f60 0x563afe0343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc8508a4640 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc8e4e0 0x563afe0b9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc843fff640 1 -- 172.21.15.129:0/3619502129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc8e4e0 msgr2=0x563afe0b9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc843fff640 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc8e4e0 0x563afe0b9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc843fff640 1 -- 172.21.15.129:0/3619502129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc58f60 msgr2=0x563afe0343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc843fff640 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc58f60 0x563afe0343a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc843fff640 1 -- 172.21.15.129:0/3619502129 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563afe0ba300 con 0x563afdc714d0 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc84bfff640 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc58f60 0x563afe0343a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.265+0000 7fc8508a4640 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc8e4e0 0x563afe0b9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc843fff640 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe034900 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fc83409afb0 tx=0x7fc8340048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea355869ffad8f04 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc82b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc82b7fe640 1 -- 172.21.15.129:0/3619502129 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834020040 con 0x563afdc714d0 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc82b7fe640 1 -- 172.21.15.129:0/3619502129 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7fc8340995b0 con 0x563afdc714d0 2024-05-20T18:10:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc8591a8740 1 -- 172.21.15.129:0/3619502129 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc844005bd0 con 0x563afdc714d0 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc82b7fe640 1 -- 172.21.15.129:0/3619502129 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc83409f5b0 con 0x563afdc714d0 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc8591a8740 10 client.17496 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc8591a8740 1 -- 172.21.15.129:0/3619502129 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563afe0c16b0 con 0x563afdc714d0 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc82b7fe640 1 -- 172.21.15.129:0/3619502129 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc844005bd0 con 0x563afdc714d0 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc82b7fe640 1 -- 172.21.15.129:0/3619502129 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7fc834097920 con 0x563afdc714d0 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc8591a8740 10 client.17496 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc8591a8740 10 client.17496 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc8591a8740 20 client.17496 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc8591a8740 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x563afdc796b0 0x563afe0c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc8591a8740 4 client.17496 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.266+0000 7fc8591a8740 1 -- 172.21.15.129:0/3619502129 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563afdf236f0 con 0x563afdc796b0 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc84bfff640 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x563afdc796b0 0x563afe0c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc82b7fe640 10 client.17496 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc84bfff640 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x563afdc796b0 0x563afe0c3e00 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b1ff87ed3673f99 server_cookie=ca41bf671bf668ba in_seq=0 out_seq=0 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc82b7fe640 1 -- 172.21.15.129:0/3619502129 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563afdf236f0 con 0x563afdc796b0 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc82b7fe640 10 client.17496 handle_command_reply: tid=0 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc8591a8740 1 client.17496 shutdown 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc8591a8740 1 -- 172.21.15.129:0/3619502129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 msgr2=0x563afe034900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc8591a8740 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe034900 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fc83409afb0 tx=0x7fc8340048a0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc8591a8740 1 -- 172.21.15.129:0/3619502129 shutdown_connections 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc8591a8740 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x563afdc796b0 0x563afe0c3e00 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc8591a8740 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x563afdc8e4e0 0x563afe0b9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc8591a8740 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563afdc714d0 0x563afe034900 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc82b7fe640 0 client.17496 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc8591a8740 1 --2- 172.21.15.129:0/3619502129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563afdc58f60 0x563afe0343a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc8591a8740 1 -- 172.21.15.129:0/3619502129 >> 172.21.15.129:0/3619502129 conn(0x563afe03ec90 msgr2=0x563afe03a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc8591a8740 1 -- 172.21.15.129:0/3619502129 shutdown_connections 2024-05-20T18:10:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.267+0000 7fc8591a8740 1 -- 172.21.15.129:0/3619502129 wait complete. 2024-05-20T18:10:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc8591a8740 20 client.17496 trim_cache size 0 max 16384 2024-05-20T18:10:42.272 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:10:42.272 INFO:teuthology.orchestra.run.smithi129.stdout: "status": "no active scrubs running", 2024-05-20T18:10:42.272 INFO:teuthology.orchestra.run.smithi129.stdout: "scrubs": {} 2024-05-20T18:10:42.272 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:10:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 -- 172.21.15.129:0/2339895448 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8380a2d80 msgr2=0x7fc8380a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8380a2d80 0x7fc8380a5290 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc844023000 tx=0x7fc844021000 comp rx=0 tx=0).stop 2024-05-20T18:10:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 -- 172.21.15.129:0/2339895448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc84c10a610 msgr2=0x7fc84c19fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc84c10a610 0x7fc84c19fa60 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fc83400d750 tx=0x7fc83400dc10 comp rx=0 tx=0).stop 2024-05-20T18:10:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 -- 172.21.15.129:0/2339895448 shutdown_connections 2024-05-20T18:10:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc8380a2d80 0x7fc8380a5290 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc84c10d000 0x7fc84c1996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc84c10a610 0x7fc84c19fa60 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 --2- 172.21.15.129:0/2339895448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc84c103bb0 0x7fc84c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 -- 172.21.15.129:0/2339895448 >> 172.21.15.129:0/2339895448 conn(0x7fc84c0fd210 msgr2=0x7fc84c10d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 -- 172.21.15.129:0/2339895448 shutdown_connections 2024-05-20T18:10:42.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.268+0000 7fc852307640 1 -- 172.21.15.129:0/2339895448 wait complete. 2024-05-20T18:10:42.277 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-20T18:10:42.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:10:42.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.391+0000 7fc4b519d640 1 -- 172.21.15.129:0/4167065648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4b010c370 msgr2=0x7fc4b010e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.391+0000 7fc4b519d640 1 --2- 172.21.15.129:0/4167065648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4b010c370 0x7fc4b010e790 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009900 tx=0x7fc4a802a410 comp rx=0 tx=0).stop 2024-05-20T18:10:42.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.391+0000 7fc4b519d640 1 -- 172.21.15.129:0/4167065648 shutdown_connections 2024-05-20T18:10:42.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.391+0000 7fc4b519d640 1 --2- 172.21.15.129:0/4167065648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4b010c370 0x7fc4b010e790 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.391+0000 7fc4b519d640 1 --2- 172.21.15.129:0/4167065648 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4b0101b30 0x7fc4b0101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.391+0000 7fc4b519d640 1 --2- 172.21.15.129:0/4167065648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4b0107a10 0x7fc4b01015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.391+0000 7fc4b519d640 1 -- 172.21.15.129:0/4167065648 >> 172.21.15.129:0/4167065648 conn(0x7fc4b0077130 msgr2=0x7fc4b0077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.391+0000 7fc4b519d640 1 -- 172.21.15.129:0/4167065648 shutdown_connections 2024-05-20T18:10:42.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.391+0000 7fc4b519d640 1 -- 172.21.15.129:0/4167065648 wait complete. 2024-05-20T18:10:42.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.392+0000 7fc4b519d640 1 Processor -- start 2024-05-20T18:10:42.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.392+0000 7fc4b519d640 1 -- start start 2024-05-20T18:10:42.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.392+0000 7fc4b519d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4b0101b30 0x7fc4b019a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.392+0000 7fc4b519d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4b0107a10 0x7fc4b019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.392+0000 7fc4b519d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4b010c370 0x7fc4b019afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.392+0000 7fc4b519d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0111d40 con 0x7fc4b0107a10 2024-05-20T18:10:42.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.392+0000 7fc4b519d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0111bc0 con 0x7fc4b0101b30 2024-05-20T18:10:42.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.392+0000 7fc4b519d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0111ec0 con 0x7fc4b010c370 2024-05-20T18:10:42.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4ae59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4b0107a10 0x7fc4b019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4ae59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4b0107a10 0x7fc4b019aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32796/0 (socket says 172.21.15.129:32796) 2024-05-20T18:10:42.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4ae59c640 1 -- 172.21.15.129:0/3264287558 learned_addr learned my addr 172.21.15.129:0/3264287558 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:42.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4aed9d640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4b0101b30 0x7fc4b019a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4af59e640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4b010c370 0x7fc4b019afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4ae59c640 1 -- 172.21.15.129:0/3264287558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4b010c370 msgr2=0x7fc4b019afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4ae59c640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4b010c370 0x7fc4b019afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4ae59c640 1 -- 172.21.15.129:0/3264287558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4b0101b30 msgr2=0x7fc4b019a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4ae59c640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4b0101b30 0x7fc4b019a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4ae59c640 1 -- 172.21.15.129:0/3264287558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b01a5870 con 0x7fc4b0107a10 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4af59e640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4b010c370 0x7fc4b019afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4ae59c640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4b0107a10 0x7fc4b019aaa0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0005d00 tx=0x7fc4a000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4a7fff640 1 -- 172.21.15.129:0/3264287558 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0017070 con 0x7fc4b0107a10 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4b519d640 1 -- 172.21.15.129:0/3264287558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b01a5b60 con 0x7fc4b0107a10 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4a7fff640 1 -- 172.21.15.129:0/3264287558 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a0009c30 con 0x7fc4b0107a10 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4a7fff640 1 -- 172.21.15.129:0/3264287558 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a000f620 con 0x7fc4b0107a10 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.393+0000 7fc4b519d640 1 -- 172.21.15.129:0/3264287558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b01a60b0 con 0x7fc4b0107a10 2024-05-20T18:10:42.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.394+0000 7fc4b519d640 1 -- 172.21.15.129:0/3264287558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4b0069140 con 0x7fc4b0107a10 2024-05-20T18:10:42.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.395+0000 7fc4a7fff640 1 -- 172.21.15.129:0/3264287558 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc4a001a070 con 0x7fc4b0107a10 2024-05-20T18:10:42.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.396+0000 7fc4a7fff640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc49c0a2d00 0x7fc49c0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.396+0000 7fc4aed9d640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc49c0a2d00 0x7fc49c0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.396+0000 7fc4a7fff640 1 -- 172.21.15.129:0/3264287558 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7fc4a00cee60 con 0x7fc4b0107a10 2024-05-20T18:10:42.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.397+0000 7fc4aed9d640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc49c0a2d00 0x7fc49c0a5210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0102a10 tx=0x7fc498009680 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.400+0000 7fc4a7fff640 1 -- 172.21.15.129:0/3264287558 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc4a0009de0 con 0x7fc4b0107a10 2024-05-20T18:10:42.410 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:42 smithi129 ceph-mon[32029]: pgmap v381: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.2 KiB/s wr, 192 op/s 2024-05-20T18:10:42.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.611+0000 7fc4b519d640 1 -- 172.21.15.129:0/3264287558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4b0074370 con 0x7fc4b0107a10 2024-05-20T18:10:42.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.612+0000 7fc4a7fff640 1 -- 172.21.15.129:0/3264287558 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11829 (secure 0 0 0) 0x7fc4a009ba40 con 0x7fc4b0107a10 2024-05-20T18:10:42.609 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:10:42.609 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":26255,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/2724397229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":2724397229},{"type":"v1","addr":"172.21.15.154:6837","nonce":2724397229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:09:18.904334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26815":{"gid":26815,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/1721642062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":1721642062},{"type":"v1","addr":"172.21.15.204:6839","nonce":1721642062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:10:42.609 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 28 2024-05-20T18:10:42.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 -- 172.21.15.129:0/3264287558 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc49c0a2d00 msgr2=0x7fc49c0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc49c0a2d00 0x7fc49c0a5210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0102a10 tx=0x7fc498009680 comp rx=0 tx=0).stop 2024-05-20T18:10:42.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 -- 172.21.15.129:0/3264287558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4b0107a10 msgr2=0x7fc4b019aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4b0107a10 0x7fc4b019aaa0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0005d00 tx=0x7fc4a000dde0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 -- 172.21.15.129:0/3264287558 shutdown_connections 2024-05-20T18:10:42.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc49c0a2d00 0x7fc49c0a5210 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4b010c370 0x7fc4b019afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4b0107a10 0x7fc4b019aaa0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 --2- 172.21.15.129:0/3264287558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4b0101b30 0x7fc4b019a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 -- 172.21.15.129:0/3264287558 >> 172.21.15.129:0/3264287558 conn(0x7fc4b0077130 msgr2=0x7fc4b01030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 -- 172.21.15.129:0/3264287558 shutdown_connections 2024-05-20T18:10:42.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.614+0000 7fc4b519d640 1 -- 172.21.15.129:0/3264287558 wait complete. 2024-05-20T18:10:42.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-20T18:10:42.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:42 smithi154 ceph-mon[36482]: pgmap v381: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.2 KiB/s wr, 192 op/s 2024-05-20T18:10:42.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:42 smithi204 ceph-mon[37076]: pgmap v381: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.2 KiB/s wr, 192 op/s 2024-05-20T18:10:42.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.739+0000 7f5df4b16640 1 -- 172.21.15.129:0/375000917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df010c330 msgr2=0x7f5df010e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.739+0000 7f5df4b16640 1 --2- 172.21.15.129:0/375000917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df010c330 0x7f5df010e750 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009ca0 tx=0x7f5de802a8c0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.739+0000 7f5df4b16640 1 -- 172.21.15.129:0/375000917 shutdown_connections 2024-05-20T18:10:42.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.739+0000 7f5df4b16640 1 --2- 172.21.15.129:0/375000917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df0101aa0 0x7f5df0101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.739+0000 7f5df4b16640 1 --2- 172.21.15.129:0/375000917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df010c330 0x7f5df010e750 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.739+0000 7f5df4b16640 1 --2- 172.21.15.129:0/375000917 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5df01079d0 0x7f5df0101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.739+0000 7f5df4b16640 1 -- 172.21.15.129:0/375000917 >> 172.21.15.129:0/375000917 conn(0x7f5df00770b0 msgr2=0x7f5df00774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.739+0000 7f5df4b16640 1 -- 172.21.15.129:0/375000917 shutdown_connections 2024-05-20T18:10:42.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.739+0000 7f5df4b16640 1 -- 172.21.15.129:0/375000917 wait complete. 2024-05-20T18:10:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5df4b16640 1 Processor -- start 2024-05-20T18:10:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5df4b16640 1 -- start start 2024-05-20T18:10:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5df4b16640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df0101aa0 0x7f5df01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5df4b16640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df01079d0 0x7f5df0198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5df4b16640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5df010c330 0x7f5df019cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5df4b16640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5df0111d00 con 0x7f5df01079d0 2024-05-20T18:10:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5df4b16640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5df0111b80 con 0x7f5df0101aa0 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5df4b16640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5df0111e80 con 0x7f5df010c330 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5de5d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df01079d0 0x7f5df0198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5de5d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df01079d0 0x7f5df0198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32816/0 (socket says 172.21.15.129:32816) 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5de5d9b640 1 -- 172.21.15.129:0/4108746062 learned_addr learned my addr 172.21.15.129:0/4108746062 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.740+0000 7f5deed9d640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5df010c330 0x7f5df019cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5dee59c640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df0101aa0 0x7f5df01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5de5d9b640 1 -- 172.21.15.129:0/4108746062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5df010c330 msgr2=0x7f5df019cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5df010c330 0x7f5df019cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5de5d9b640 1 -- 172.21.15.129:0/4108746062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df0101aa0 msgr2=0x7f5df01986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df0101aa0 0x7f5df01986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5de5d9b640 1 -- 172.21.15.129:0/4108746062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df019d540 con 0x7f5df01079d0 2024-05-20T18:10:42.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5dee59c640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df0101aa0 0x7f5df01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:10:42.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df01079d0 0x7f5df0198c40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800d740 tx=0x7f5dd800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5de7fff640 1 -- 172.21.15.129:0/4108746062 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8017070 con 0x7f5df01079d0 2024-05-20T18:10:42.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5de7fff640 1 -- 172.21.15.129:0/4108746062 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd800de50 con 0x7f5df01079d0 2024-05-20T18:10:42.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5df4b16640 1 -- 172.21.15.129:0/4108746062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df019d830 con 0x7f5df01079d0 2024-05-20T18:10:42.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5de7fff640 1 -- 172.21.15.129:0/4108746062 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8003e00 con 0x7f5df01079d0 2024-05-20T18:10:42.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.741+0000 7f5df4b16640 1 -- 172.21.15.129:0/4108746062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df01a51b0 con 0x7f5df01079d0 2024-05-20T18:10:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.744+0000 7f5dfb9b7740 1 Processor -- start 2024-05-20T18:10:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.744+0000 7f5dfb9b7740 1 -- start start 2024-05-20T18:10:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5de7fff640 1 -- 172.21.15.129:0/4108746062 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5dd8010460 con 0x7f5df01079d0 2024-05-20T18:10:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188b6390 0x558c18c54d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c1888d4d0 0x558c18c55260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c188956b0 0x558c18c557a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558c18bbac40 con 0x558c188b6390 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558c18b62580 con 0x558c188956b0 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558c18bb6c00 con 0x558c1888d4d0 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5deed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188b6390 0x558c18c54d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5deed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188b6390 0x558c18c54d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32828/0 (socket says 172.21.15.129:32828) 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5deed9d640 1 -- 172.21.15.129:0/2068400251 learned_addr learned my addr 172.21.15.129:0/2068400251 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5dee59c640 1 --2- 172.21.15.129:0/2068400251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c1888d4d0 0x558c18c55260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/2068400251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c188956b0 0x558c18c557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5deed9d640 1 -- 172.21.15.129:0/2068400251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c1888d4d0 msgr2=0x558c18c55260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5deed9d640 1 --2- 172.21.15.129:0/2068400251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c1888d4d0 0x558c18c55260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5deed9d640 1 -- 172.21.15.129:0/2068400251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c188956b0 msgr2=0x558c18c557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5deed9d640 1 --2- 172.21.15.129:0/2068400251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c188956b0 0x558c18c557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5deed9d640 1 -- 172.21.15.129:0/2068400251 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c188ce3f0 con 0x558c188b6390 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/2068400251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c188956b0 0x558c18c557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5de7fff640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ddc0a2d00 0x7f5ddc0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.745+0000 7f5deed9d640 1 --2- 172.21.15.129:0/2068400251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188b6390 0x558c18c54d20 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009ca0 tx=0x7f5de802f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744bd327eafa4119 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.746+0000 7f5dee59c640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ddc0a2d00 0x7f5ddc0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.746+0000 7f5de7fff640 1 -- 172.21.15.129:0/4108746062 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7f5dd80ceeb0 con 0x7f5df01079d0 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.746+0000 7f5de4d99640 1 -- 172.21.15.129:0/2068400251 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de8031bb0 con 0x558c188b6390 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.746+0000 7f5de4d99640 1 -- 172.21.15.129:0/2068400251 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5de8036040 con 0x558c188b6390 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.746+0000 7f5de4d99640 1 -- 172.21.15.129:0/2068400251 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de80042a0 con 0x558c188b6390 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/2068400251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188b6390 msgr2=0x558c18c54d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dee59c640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ddc0a2d00 0x7f5ddc0a5210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x558c18c562e0 tx=0x7f5de0005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/2068400251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188b6390 0x558c18c54d20 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009ca0 tx=0x7f5de802f3c0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/2068400251 shutdown_connections 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/2068400251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c188956b0 0x558c18c557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/2068400251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c1888d4d0 0x558c18c55260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/2068400251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188b6390 0x558c18c54d20 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/2068400251 >> 172.21.15.129:0/2068400251 conn(0x558c188aa4e0 msgr2=0x558c18c514d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/2068400251 shutdown_connections 2024-05-20T18:10:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/2068400251 wait complete. 2024-05-20T18:10:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 -- start start 2024-05-20T18:10:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 Processor -- start 2024-05-20T18:10:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.747+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c1888d4d0 0x558c18c67850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188956b0 0x558c18c67d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c188b6390 0x558c18c62340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558c18c61fb0 con 0x558c188956b0 2024-05-20T18:10:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558c18b75c40 con 0x558c188b6390 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558c18c62130 con 0x558c1888d4d0 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dfb9b7740 10 client.0 _ll_register_callbacks cb 0x558c188cdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5de5d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188956b0 0x558c18c67d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5de5d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188956b0 0x558c18c67d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32842/0 (socket says 172.21.15.129:32842) 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5de5d9b640 1 -- 172.21.15.129:0/1136210789 learned_addr learned my addr 172.21.15.129:0/1136210789 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dee59c640 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c1888d4d0 0x558c18c67850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5deed9d640 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c188b6390 0x558c18c62340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5de5d9b640 1 -- 172.21.15.129:0/1136210789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c1888d4d0 msgr2=0x558c18c67850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c1888d4d0 0x558c18c67850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5de5d9b640 1 -- 172.21.15.129:0/1136210789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c188b6390 msgr2=0x558c18c62340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c188b6390 0x558c18c62340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5de5d9b640 1 -- 172.21.15.129:0/1136210789 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c189987d0 con 0x558c188956b0 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188956b0 0x558c18c67d90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80977f0 tx=0x7f5dd809af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc550026efa8cea5 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1136210789 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8020050 con 0x558c188956b0 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1136210789 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7f5dd8097a00 con 0x558c188956b0 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1136210789 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5de8036040 con 0x558c188956b0 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1136210789 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5dd809b390 con 0x558c188956b0 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dfb9b7740 10 client.17514 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.748+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1136210789 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c18c70900 con 0x558c188956b0 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.749+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1136210789 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5de8036040 con 0x558c188956b0 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.749+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1136210789 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f5dd809a7e0 con 0x558c188956b0 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.749+0000 7f5dfb9b7740 10 client.17514 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.749+0000 7f5dfb9b7740 10 client.17514 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.749+0000 7f5dfb9b7740 20 client.17514 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.749+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558c18874f60 0x558c18c73df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.749+0000 7f5dfb9b7740 4 client.17514 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-20T18:10:42.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.749+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1136210789 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c18bc8370 con 0x558c18874f60 2024-05-20T18:10:42.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.749+0000 7f5dee59c640 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558c18874f60 0x558c18c73df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dd5ffb640 10 client.17514 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:42.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dee59c640 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558c18874f60 0x558c18c73df0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7c59f913fca5e41 server_cookie=cd5f5d36aa473867 in_seq=0 out_seq=0 2024-05-20T18:10:42.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1136210789 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558c18bc8370 con 0x558c18874f60 2024-05-20T18:10:42.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dd5ffb640 10 client.17514 handle_command_reply: tid=0 2024-05-20T18:10:42.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dfb9b7740 1 client.17514 shutdown 2024-05-20T18:10:42.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1136210789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188956b0 msgr2=0x558c18c67d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188956b0 0x558c18c67d90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80977f0 tx=0x7f5dd809af90 comp rx=0 tx=0).stop 2024-05-20T18:10:42.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1136210789 shutdown_connections 2024-05-20T18:10:42.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558c18874f60 0x558c18c73df0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c188b6390 0x558c18c62340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c188956b0 0x558c18c67d90 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dd5ffb640 0 client.17514 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:42.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1136210789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c1888d4d0 0x558c18c67850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.750+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1136210789 >> 172.21.15.129:0/1136210789 conn(0x558c188aa4e0 msgr2=0x558c18c56a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.751+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1136210789 shutdown_connections 2024-05-20T18:10:42.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.751+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1136210789 wait complete. 2024-05-20T18:10:42.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.751+0000 7f5dfb9b7740 20 client.17514 trim_cache size 0 max 16384 2024-05-20T18:10:42.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.764+0000 7f5dfb9b7740 1 Processor -- start 2024-05-20T18:10:42.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.764+0000 7f5dfb9b7740 1 -- start start 2024-05-20T18:10:42.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.764+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c18874f60 0x558c18c58aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.764+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c1888d4d0 0x558c18c58fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.764+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c188956b0 0x558c18c795b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.764+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01aa7d0 con 0x558c18874f60 2024-05-20T18:10:42.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.764+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01ad030 con 0x558c1888d4d0 2024-05-20T18:10:42.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.764+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01aae20 con 0x558c188956b0 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.764+0000 7f5dee59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c18874f60 0x558c18c58aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.764+0000 7f5dee59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c18874f60 0x558c18c58aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32854/0 (socket says 172.21.15.129:32854) 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.764+0000 7f5dee59c640 1 -- 172.21.15.129:0/1653961816 learned_addr learned my addr 172.21.15.129:0/1653961816 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/1653961816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c1888d4d0 0x558c18c58fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5deed9d640 1 --2- 172.21.15.129:0/1653961816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c188956b0 0x558c18c795b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5dee59c640 1 -- 172.21.15.129:0/1653961816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c188956b0 msgr2=0x558c18c795b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5dee59c640 1 --2- 172.21.15.129:0/1653961816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c188956b0 0x558c18c795b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5dee59c640 1 -- 172.21.15.129:0/1653961816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c1888d4d0 msgr2=0x558c18c58fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5dee59c640 1 --2- 172.21.15.129:0/1653961816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c1888d4d0 0x558c18c58fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5dee59c640 1 -- 172.21.15.129:0/1653961816 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c18c56540 con 0x558c18874f60 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/1653961816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c1888d4d0 0x558c18c58fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5dee59c640 1 --2- 172.21.15.129:0/1653961816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c18874f60 0x558c18c58aa0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x558c18c52810 tx=0x7f5de0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2db4f2a6d071f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5dd7fff640 1 -- 172.21.15.129:0/1653961816 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0005220 con 0x558c18874f60 2024-05-20T18:10:42.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5dd7fff640 1 -- 172.21.15.129:0/1653961816 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5de0004c80 con 0x558c18874f60 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.765+0000 7f5dd7fff640 1 -- 172.21.15.129:0/1653961816 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de002a890 con 0x558c18874f60 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1653961816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c18874f60 msgr2=0x558c18c58aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1653961816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c18874f60 0x558c18c58aa0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x558c18c52810 tx=0x7f5de0016000 comp rx=0 tx=0).stop 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1653961816 shutdown_connections 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1653961816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c1888d4d0 0x558c18c58fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1653961816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c188956b0 0x558c18c795b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1653961816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c18874f60 0x558c18c58aa0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1653961816 >> 172.21.15.129:0/1653961816 conn(0x558c188aa4e0 msgr2=0x558c18c5db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1653961816 shutdown_connections 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1653961816 wait complete. 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 -- start start 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 Processor -- start 2024-05-20T18:10:42.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c18874f60 0x558c18c6e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c1888d4d0 0x558c18c69460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c188956b0 0x558c18c699c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01ab4f0 con 0x558c1888d4d0 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01ac960 con 0x558c18874f60 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01abbc0 con 0x558c188956b0 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5dfb9b7740 10 client.0 _ll_register_callbacks cb 0x7f5dd8005ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5de5d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c1888d4d0 0x558c18c69460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5de5d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c1888d4d0 0x558c18c69460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32870/0 (socket says 172.21.15.129:32870) 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.766+0000 7f5de5d9b640 1 -- 172.21.15.129:0/1165831142 learned_addr learned my addr 172.21.15.129:0/1165831142 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dee59c640 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c18874f60 0x558c18c6e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5deed9d640 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c188956b0 0x558c18c699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5de5d9b640 1 -- 172.21.15.129:0/1165831142 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c188956b0 msgr2=0x558c18c699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c188956b0 0x558c18c699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5de5d9b640 1 -- 172.21.15.129:0/1165831142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c18874f60 msgr2=0x558c18c6e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c18874f60 0x558c18c6e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5de5d9b640 1 -- 172.21.15.129:0/1165831142 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c18c6a150 con 0x558c1888d4d0 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5de5d9b640 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c1888d4d0 0x558c18c69460 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f5dd809ac80 tx=0x7f5dd809b230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a8c88a396a945e server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1165831142 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8020030 con 0x558c1888d4d0 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1165831142 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7f5dd809a7e0 con 0x558c1888d4d0 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1165831142 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5de0004c80 con 0x558c1888d4d0 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1165831142 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5dd8009e50 con 0x558c1888d4d0 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dfb9b7740 10 client.17520 fetch_fsmap learned FSMap version 28 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1165831142 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c18cdc030 con 0x558c1888d4d0 2024-05-20T18:10:42.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1165831142 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5de0004c80 con 0x558c1888d4d0 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1165831142 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 28) ==== 8322+0+0 (secure 0 0 0) 0x7f5dd80bb0b0 con 0x558c1888d4d0 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dfb9b7740 10 client.17520 fetch_fsmap finished waiting for FSMap version 28 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dfb9b7740 10 client.17520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.767+0000 7f5dfb9b7740 20 client.17520 populate_metadata read hostname 'smithi129' 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.768+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558c188b6390 0x558c18cde630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.768+0000 7f5dfb9b7740 4 client.17520 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.768+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1165831142 --> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558c18bc26c0 con 0x558c188b6390 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.768+0000 7f5dee59c640 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558c188b6390 0x558c18cde630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.768+0000 7f5dd5ffb640 10 client.17520 ms_handle_connect on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.768+0000 7f5dee59c640 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558c188b6390 0x558c18cde630 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb683de183cf151b server_cookie=c9e0cf2965017e10 in_seq=0 out_seq=0 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dd5ffb640 1 -- 172.21.15.129:0/1165831142 <== mds.0 v2:172.21.15.154:6838/793468142 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558c18bc26c0 con 0x558c188b6390 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dd5ffb640 10 client.17520 handle_command_reply: tid=0 2024-05-20T18:10:42.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 1 client.17520 shutdown 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1165831142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c1888d4d0 msgr2=0x558c18c69460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c1888d4d0 0x558c18c69460 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f5dd809ac80 tx=0x7f5dd809b230 comp rx=0 tx=0).stop 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1165831142 shutdown_connections 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.154:6838/793468142,v1:172.21.15.154:6839/793468142] conn(0x558c188b6390 0x558c18cde630 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c188956b0 0x558c18c699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dd5ffb640 0 client.17520 ms_handle_reset on v2:172.21.15.154:6838/793468142 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558c1888d4d0 0x558c18c69460 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 1 --2- 172.21.15.129:0/1165831142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558c18874f60 0x558c18c6e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1165831142 >> 172.21.15.129:0/1165831142 conn(0x558c188aa4e0 msgr2=0x558c18c567c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1165831142 shutdown_connections 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 1 -- 172.21.15.129:0/1165831142 wait complete. 2024-05-20T18:10:42.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5dfb9b7740 20 client.17520 trim_cache size 0 max 16384 2024-05-20T18:10:42.774 INFO:teuthology.orchestra.run.smithi129.stdout:[] 2024-05-20T18:10:42.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5df4b16640 1 -- 172.21.15.129:0/4108746062 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ddc0a2d00 msgr2=0x7f5ddc0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5df4b16640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ddc0a2d00 0x7f5ddc0a5210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x558c18c562e0 tx=0x7f5de0005f90 comp rx=0 tx=0).stop 2024-05-20T18:10:42.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5df4b16640 1 -- 172.21.15.129:0/4108746062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df01079d0 msgr2=0x7f5df0198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:42.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5df4b16640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df01079d0 0x7f5df0198c40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800d740 tx=0x7f5dd800dc00 comp rx=0 tx=0).stop 2024-05-20T18:10:42.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5df4b16640 1 -- 172.21.15.129:0/4108746062 shutdown_connections 2024-05-20T18:10:42.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5df4b16640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5ddc0a2d00 0x7f5ddc0a5210 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5df4b16640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5df010c330 0x7f5df019cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5df4b16640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df01079d0 0x7f5df0198c40 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5df4b16640 1 --2- 172.21.15.129:0/4108746062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df0101aa0 0x7f5df01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:42.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.769+0000 7f5df4b16640 1 -- 172.21.15.129:0/4108746062 >> 172.21.15.129:0/4108746062 conn(0x7f5df00770b0 msgr2=0x7f5df0103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:42.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.770+0000 7f5df4b16640 1 -- 172.21.15.129:0/4108746062 shutdown_connections 2024-05-20T18:10:42.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:42.770+0000 7f5df4b16640 1 -- 172.21.15.129:0/4108746062 wait complete. 2024-05-20T18:10:43.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:43 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3264287558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:10:43.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:43 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3264287558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:10:43.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:43 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3264287558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:10:43.782 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-05-20T18:10:43.782 INFO:tasks.fwd_scrub:done joining 2024-05-20T18:10:43.782 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-20T18:10:43.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T18:10:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.921+0000 7f0ec6884640 1 -- 172.21.15.129:0/3647471250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec010c330 msgr2=0x7f0ec010e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.921+0000 7f0ec6884640 1 --2- 172.21.15.129:0/3647471250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec010c330 0x7f0ec010e750 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009ca0 tx=0x7f0eb802a8c0 comp rx=0 tx=0).stop 2024-05-20T18:10:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.921+0000 7f0ec6884640 1 -- 172.21.15.129:0/3647471250 shutdown_connections 2024-05-20T18:10:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.921+0000 7f0ec6884640 1 --2- 172.21.15.129:0/3647471250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec0101aa0 0x7f0ec0101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.921+0000 7f0ec6884640 1 --2- 172.21.15.129:0/3647471250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec010c330 0x7f0ec010e750 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:43.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.921+0000 7f0ec6884640 1 --2- 172.21.15.129:0/3647471250 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ec01079d0 0x7f0ec0101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:43.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.921+0000 7f0ec6884640 1 -- 172.21.15.129:0/3647471250 >> 172.21.15.129:0/3647471250 conn(0x7f0ec00770c0 msgr2=0x7f0ec0077500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:43.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.921+0000 7f0ec6884640 1 -- 172.21.15.129:0/3647471250 shutdown_connections 2024-05-20T18:10:43.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.921+0000 7f0ec6884640 1 -- 172.21.15.129:0/3647471250 wait complete. 2024-05-20T18:10:43.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.922+0000 7f0ec6884640 1 Processor -- start 2024-05-20T18:10:43.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.922+0000 7f0ec6884640 1 -- start start 2024-05-20T18:10:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.922+0000 7f0ec6884640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec0101aa0 0x7f0ec01985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.922+0000 7f0ec6884640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec01079d0 0x7f0ec0198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.922+0000 7f0ec6884640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ec010c330 0x7f0ec019cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.922+0000 7f0ec6884640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec0111ce0 con 0x7f0ec0101aa0 2024-05-20T18:10:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.922+0000 7f0ec6884640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec0111b60 con 0x7f0ec01079d0 2024-05-20T18:10:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.922+0000 7f0ec6884640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec0111e60 con 0x7f0ec010c330 2024-05-20T18:10:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec0101aa0 0x7f0ec01985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec0101aa0 0x7f0ec01985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32902/0 (socket says 172.21.15.129:32902) 2024-05-20T18:10:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebffff640 1 -- 172.21.15.129:0/883266633 learned_addr learned my addr 172.21.15.129:0/883266633 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ec4e21640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ec010c330 0x7f0ec019cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebf7fe640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec01079d0 0x7f0ec0198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebffff640 1 -- 172.21.15.129:0/883266633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ec010c330 msgr2=0x7f0ec019cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebffff640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ec010c330 0x7f0ec019cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebffff640 1 -- 172.21.15.129:0/883266633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec01079d0 msgr2=0x7f0ec0198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebffff640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec01079d0 0x7f0ec0198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebffff640 1 -- 172.21.15.129:0/883266633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec019d3a0 con 0x7f0ec0101aa0 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebffff640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec0101aa0 0x7f0ec01985c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000c9a0 tx=0x7f0eb000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebd7fa640 1 -- 172.21.15.129:0/883266633 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0007bb0 con 0x7f0ec0101aa0 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebd7fa640 1 -- 172.21.15.129:0/883266633 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0eb0007d50 con 0x7f0ec0101aa0 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ec6884640 1 -- 172.21.15.129:0/883266633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec019d690 con 0x7f0ec0101aa0 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ebd7fa640 1 -- 172.21.15.129:0/883266633 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb000f6b0 con 0x7f0ec0101aa0 2024-05-20T18:10:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.923+0000 7f0ec6884640 1 -- 172.21.15.129:0/883266633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec019dbe0 con 0x7f0ec0101aa0 2024-05-20T18:10:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.924+0000 7f0ec6884640 1 -- 172.21.15.129:0/883266633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ec0102c80 con 0x7f0ec0101aa0 2024-05-20T18:10:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.926+0000 7f0ebd7fa640 1 -- 172.21.15.129:0/883266633 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f0eb00046c0 con 0x7f0ec0101aa0 2024-05-20T18:10:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.926+0000 7f0ebd7fa640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0eac0a2d00 0x7f0eac0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.927+0000 7f0ebf7fe640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0eac0a2d00 0x7f0eac0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.927+0000 7f0ebd7fa640 1 -- 172.21.15.129:0/883266633 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7f0eb00cf4e0 con 0x7f0ec0101aa0 2024-05-20T18:10:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.927+0000 7f0ebf7fe640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0eac0a2d00 0x7f0eac0a5210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ea80058c0 tx=0x7f0ea8005850 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:43.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:43.930+0000 7f0ebd7fa640 1 -- 172.21.15.129:0/883266633 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0eb0014030 con 0x7f0ec0101aa0 2024-05-20T18:10:44.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.110+0000 7f0ec6884640 1 -- 172.21.15.129:0/883266633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ec0105900 con 0x7f0ec0101aa0 2024-05-20T18:10:44.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.111+0000 7f0ebd7fa640 1 -- 172.21.15.129:0/883266633 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+23222 (secure 0 0 0) 0x7f0eb009c0c0 con 0x7f0ec0101aa0 2024-05-20T18:10:44.109 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:10:44.109 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":107,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T18:10:20.313450+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":18,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4},{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2249940021":"2024-05-21T17:58:43.562501+0000","172.21.15.129:6800/2870089245":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/1943655536":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/2870089245":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1369259428":"2024-05-21T17:58:43.562501+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/549964701":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":18,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T18:10:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 -- 172.21.15.129:0/883266633 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0eac0a2d00 msgr2=0x7f0eac0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0eac0a2d00 0x7f0eac0a5210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ea80058c0 tx=0x7f0ea8005850 comp rx=0 tx=0).stop 2024-05-20T18:10:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 -- 172.21.15.129:0/883266633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec0101aa0 msgr2=0x7f0ec01985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec0101aa0 0x7f0ec01985c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000c9a0 tx=0x7f0eb000ce60 comp rx=0 tx=0).stop 2024-05-20T18:10:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 -- 172.21.15.129:0/883266633 shutdown_connections 2024-05-20T18:10:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0eac0a2d00 0x7f0eac0a5210 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ec01079d0 0x7f0ec0198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ec010c330 0x7f0ec019cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 --2- 172.21.15.129:0/883266633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec0101aa0 0x7f0ec01985c0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 -- 172.21.15.129:0/883266633 >> 172.21.15.129:0/883266633 conn(0x7f0ec00770c0 msgr2=0x7f0ec0103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 -- 172.21.15.129:0/883266633 shutdown_connections 2024-05-20T18:10:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.114+0000 7f0ec6884640 1 -- 172.21.15.129:0/883266633 wait complete. 2024-05-20T18:10:44.123 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-20T17:56:42.965073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-20T17:56:44.277499+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 18, 'snap_epoch': 106, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T18:10:44.124 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-20T18:10:44.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.241+0000 7f8b00f65640 1 -- 172.21.15.129:0/3955076215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8afc10c350 msgr2=0x7f8afc10e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.241+0000 7f8b00f65640 1 --2- 172.21.15.129:0/3955076215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8afc10c350 0x7f8afc10e770 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f8af4009900 tx=0x7f8af402a410 comp rx=0 tx=0).stop 2024-05-20T18:10:44.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.241+0000 7f8b00f65640 1 -- 172.21.15.129:0/3955076215 shutdown_connections 2024-05-20T18:10:44.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.241+0000 7f8b00f65640 1 --2- 172.21.15.129:0/3955076215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8afc10c350 0x7f8afc10e770 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.241+0000 7f8b00f65640 1 --2- 172.21.15.129:0/3955076215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8afc101b10 0x7f8afc101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.241+0000 7f8b00f65640 1 --2- 172.21.15.129:0/3955076215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8afc1079f0 0x7f8afc1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.241+0000 7f8b00f65640 1 -- 172.21.15.129:0/3955076215 >> 172.21.15.129:0/3955076215 conn(0x7f8afc0fd210 msgr2=0x7f8afc0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:44.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.241+0000 7f8b00f65640 1 -- 172.21.15.129:0/3955076215 shutdown_connections 2024-05-20T18:10:44.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.241+0000 7f8b00f65640 1 -- 172.21.15.129:0/3955076215 wait complete. 2024-05-20T18:10:44.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8b00f65640 1 Processor -- start 2024-05-20T18:10:44.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8b00f65640 1 -- start start 2024-05-20T18:10:44.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8b00f65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8afc101b10 0x7f8afc199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8b00f65640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8afc1079f0 0x7f8afc199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8b00f65640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8afc10c350 0x7f8afc19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8b00f65640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc112e20 con 0x7f8afc10c350 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8b00f65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc112ca0 con 0x7f8afc101b10 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8b00f65640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc112fa0 con 0x7f8afc1079f0 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8afad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8afc10c350 0x7f8afc19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8afad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8afc10c350 0x7f8afc19df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32918/0 (socket says 172.21.15.129:32918) 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8afad9d640 1 -- 172.21.15.129:0/1018275642 learned_addr learned my addr 172.21.15.129:0/1018275642 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8af9d9b640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8afc1079f0 0x7f8afc199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.242+0000 7f8afa59c640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8afc101b10 0x7f8afc199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8afad9d640 1 -- 172.21.15.129:0/1018275642 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8afc1079f0 msgr2=0x7f8afc199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8afad9d640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8afc1079f0 0x7f8afc199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8afad9d640 1 -- 172.21.15.129:0/1018275642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8afc101b10 msgr2=0x7f8afc199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8afad9d640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8afc101b10 0x7f8afc199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8afad9d640 1 -- 172.21.15.129:0/1018275642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8afc19e5a0 con 0x7f8afc10c350 2024-05-20T18:10:44.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8afa59c640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8afc101b10 0x7f8afc199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:10:44.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8afad9d640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8afc10c350 0x7f8afc19df60 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f8af4005ab0 tx=0x7f8af4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:44.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8aeb7fe640 1 -- 172.21.15.129:0/1018275642 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af403f030 con 0x7f8afc10c350 2024-05-20T18:10:44.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8aeb7fe640 1 -- 172.21.15.129:0/1018275642 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8af4004440 con 0x7f8afc10c350 2024-05-20T18:10:44.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8b00f65640 1 -- 172.21.15.129:0/1018275642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8afc19e830 con 0x7f8afc10c350 2024-05-20T18:10:44.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8aeb7fe640 1 -- 172.21.15.129:0/1018275642 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4035070 con 0x7f8afc10c350 2024-05-20T18:10:44.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.243+0000 7f8b00f65640 1 -- 172.21.15.129:0/1018275642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8afc19ed20 con 0x7f8afc10c350 2024-05-20T18:10:44.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.244+0000 7f8b00f65640 1 -- 172.21.15.129:0/1018275642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8afc069110 con 0x7f8afc10c350 2024-05-20T18:10:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.245+0000 7f8aeb7fe640 1 -- 172.21.15.129:0/1018275642 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f8af4042030 con 0x7f8afc10c350 2024-05-20T18:10:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.246+0000 7f8aeb7fe640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8ae40a2d00 0x7f8ae40a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.246+0000 7f8aeb7fe640 1 -- 172.21.15.129:0/1018275642 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7f8af40efa20 con 0x7f8afc10c350 2024-05-20T18:10:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.246+0000 7f8afa59c640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8ae40a2d00 0x7f8ae40a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.247+0000 7f8afa59c640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8ae40a2d00 0x7f8ae40a5210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8afc1029f0 tx=0x7f8aec009500 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:44.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.250+0000 7f8aeb7fe640 1 -- 172.21.15.129:0/1018275642 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8af40bc600 con 0x7f8afc10c350 2024-05-20T18:10:44.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:44 smithi129 ceph-mon[32029]: pgmap v382: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.6 KiB/s wr, 37 op/s 2024-05-20T18:10:44.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:44 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/883266633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T18:10:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.430+0000 7f8b00f65640 1 -- 172.21.15.129:0/1018275642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8afc105eb0 con 0x7f8afc10c350 2024-05-20T18:10:44.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.430+0000 7f8aeb7fe640 1 -- 172.21.15.129:0/1018275642 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) ==== 93+0+10 (secure 0 0 0) 0x7f8af40bbf00 con 0x7f8afc10c350 2024-05-20T18:10:44.428 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T18:10:44.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 -- 172.21.15.129:0/1018275642 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8ae40a2d00 msgr2=0x7f8ae40a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8ae40a2d00 0x7f8ae40a5210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8afc1029f0 tx=0x7f8aec009500 comp rx=0 tx=0).stop 2024-05-20T18:10:44.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 -- 172.21.15.129:0/1018275642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8afc10c350 msgr2=0x7f8afc19df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8afc10c350 0x7f8afc19df60 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f8af4005ab0 tx=0x7f8af4004040 comp rx=0 tx=0).stop 2024-05-20T18:10:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 -- 172.21.15.129:0/1018275642 shutdown_connections 2024-05-20T18:10:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8ae40a2d00 0x7f8ae40a5210 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8afc10c350 0x7f8afc19df60 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8afc1079f0 0x7f8afc199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 --2- 172.21.15.129:0/1018275642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8afc101b10 0x7f8afc199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 -- 172.21.15.129:0/1018275642 >> 172.21.15.129:0/1018275642 conn(0x7f8afc0fd210 msgr2=0x7f8afc103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 -- 172.21.15.129:0/1018275642 shutdown_connections 2024-05-20T18:10:44.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.433+0000 7f8b00f65640 1 -- 172.21.15.129:0/1018275642 wait complete. 2024-05-20T18:10:44.442 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-20T18:10:44.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.560+0000 7f522f884640 1 -- 172.21.15.129:0/2621655634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f522810c360 msgr2=0x7f522810e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.560+0000 7f522f884640 1 --2- 172.21.15.129:0/2621655634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f522810c360 0x7f522810e780 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f5220009900 tx=0x7f522002a410 comp rx=0 tx=0).stop 2024-05-20T18:10:44.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.560+0000 7f522f884640 1 -- 172.21.15.129:0/2621655634 shutdown_connections 2024-05-20T18:10:44.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.560+0000 7f522f884640 1 --2- 172.21.15.129:0/2621655634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5228101b20 0x7f5228101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.560+0000 7f522f884640 1 --2- 172.21.15.129:0/2621655634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f522810c360 0x7f522810e780 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.560+0000 7f522f884640 1 --2- 172.21.15.129:0/2621655634 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5228107a00 0x7f52281015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.560+0000 7f522f884640 1 -- 172.21.15.129:0/2621655634 >> 172.21.15.129:0/2621655634 conn(0x7f52280770c0 msgr2=0x7f5228077500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:44.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.560+0000 7f522f884640 1 -- 172.21.15.129:0/2621655634 shutdown_connections 2024-05-20T18:10:44.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.560+0000 7f522f884640 1 -- 172.21.15.129:0/2621655634 wait complete. 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.560+0000 7f522f884640 1 Processor -- start 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.560+0000 7f522f884640 1 -- start start 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522f884640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5228101b20 0x7f52281997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522f884640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5228107a00 0x7f5228199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522f884640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f522810c360 0x7f522819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522f884640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5228111d30 con 0x7f5228107a00 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522f884640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5228111bb0 con 0x7f5228101b20 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522f884640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5228111eb0 con 0x7f522810c360 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5228107a00 0x7f5228199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522ce1f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5228107a00 0x7f5228199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32938/0 (socket says 172.21.15.129:32938) 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522ce1f640 1 -- 172.21.15.129:0/1455197517 learned_addr learned my addr 172.21.15.129:0/1455197517 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522ce1f640 1 -- 172.21.15.129:0/1455197517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f522810c360 msgr2=0x7f522819dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522ce1f640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f522810c360 0x7f522819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522d620640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5228101b20 0x7f52281997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522ce1f640 1 -- 172.21.15.129:0/1455197517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5228101b20 msgr2=0x7f52281997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522ce1f640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5228101b20 0x7f52281997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522ce1f640 1 -- 172.21.15.129:0/1455197517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522819e600 con 0x7f5228107a00 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522d620640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5228101b20 0x7f52281997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522ce1f640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5228107a00 0x7f5228199d00 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f521800d740 tx=0x7f521800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f521e7fc640 1 -- 172.21.15.129:0/1455197517 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218017070 con 0x7f5228107a00 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f521e7fc640 1 -- 172.21.15.129:0/1455197517 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521800de50 con 0x7f5228107a00 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522f884640 1 -- 172.21.15.129:0/1455197517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522819e8f0 con 0x7f5228107a00 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f521e7fc640 1 -- 172.21.15.129:0/1455197517 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218003e00 con 0x7f5228107a00 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.561+0000 7f522f884640 1 -- 172.21.15.129:0/1455197517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52281a6270 con 0x7f5228107a00 2024-05-20T18:10:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.562+0000 7f522f884640 1 -- 172.21.15.129:0/1455197517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5228069110 con 0x7f5228107a00 2024-05-20T18:10:44.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.564+0000 7f521e7fc640 1 -- 172.21.15.129:0/1455197517 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5218003fa0 con 0x7f5228107a00 2024-05-20T18:10:44.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.565+0000 7f521e7fc640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52140a2d00 0x7f52140a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.565+0000 7f522d620640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52140a2d00 0x7f52140a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:44.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.565+0000 7f521e7fc640 1 -- 172.21.15.129:0/1455197517 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7f52180cf330 con 0x7f5228107a00 2024-05-20T18:10:44.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.565+0000 7f522d620640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52140a2d00 0x7f52140a5210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5224005c40 tx=0x7f52240082f0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:44.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.568+0000 7f521e7fc640 1 -- 172.21.15.129:0/1455197517 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f521809bf10 con 0x7f5228107a00 2024-05-20T18:10:44.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:44 smithi154 ceph-mon[36482]: pgmap v382: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.6 KiB/s wr, 37 op/s 2024-05-20T18:10:44.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:44 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/883266633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T18:10:44.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:44 smithi204 ceph-mon[37076]: pgmap v382: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.6 KiB/s wr, 37 op/s 2024-05-20T18:10:44.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:44 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/883266633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T18:10:44.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.744+0000 7f522f884640 1 -- 172.21.15.129:0/1455197517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f52281a6470 con 0x7f5228107a00 2024-05-20T18:10:44.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.744+0000 7f521e7fc640 1 -- 172.21.15.129:0/1455197517 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) ==== 104+0+11 (secure 0 0 0) 0x7f521809b810 con 0x7f5228107a00 2024-05-20T18:10:44.742 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-05-20T18:10:44.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 -- 172.21.15.129:0/1455197517 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52140a2d00 msgr2=0x7f52140a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52140a2d00 0x7f52140a5210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5224005c40 tx=0x7f52240082f0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 -- 172.21.15.129:0/1455197517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5228107a00 msgr2=0x7f5228199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5228107a00 0x7f5228199d00 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f521800d740 tx=0x7f521800dc00 comp rx=0 tx=0).stop 2024-05-20T18:10:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 -- 172.21.15.129:0/1455197517 shutdown_connections 2024-05-20T18:10:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f52140a2d00 0x7f52140a5210 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f522810c360 0x7f522819dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5228107a00 0x7f5228199d00 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 --2- 172.21.15.129:0/1455197517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5228101b20 0x7f52281997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 -- 172.21.15.129:0/1455197517 >> 172.21.15.129:0/1455197517 conn(0x7f52280770c0 msgr2=0x7f5228073d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 -- 172.21.15.129:0/1455197517 shutdown_connections 2024-05-20T18:10:44.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.747+0000 7f522f884640 1 -- 172.21.15.129:0/1455197517 wait complete. 2024-05-20T18:10:44.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-20T18:10:44.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.874+0000 7f8dcc596640 1 -- 172.21.15.129:0/3151741198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dc410d050 msgr2=0x7f8dc410f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.874+0000 7f8dcc596640 1 --2- 172.21.15.129:0/3151741198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dc410d050 0x7f8dc410f540 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc009900 tx=0x7f8dbc02a510 comp rx=0 tx=0).stop 2024-05-20T18:10:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.874+0000 7f8dcc596640 1 -- 172.21.15.129:0/3151741198 shutdown_connections 2024-05-20T18:10:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.874+0000 7f8dcc596640 1 --2- 172.21.15.129:0/3151741198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dc410d050 0x7f8dc410f540 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.874+0000 7f8dcc596640 1 --2- 172.21.15.129:0/3151741198 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc410a6a0 0x7f8dc410cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.874+0000 7f8dcc596640 1 --2- 172.21.15.129:0/3151741198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dc4103c40 0x7f8dc4104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.874+0000 7f8dcc596640 1 -- 172.21.15.129:0/3151741198 >> 172.21.15.129:0/3151741198 conn(0x7f8dc40770c0 msgr2=0x7f8dc4077500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.874+0000 7f8dcc596640 1 -- 172.21.15.129:0/3151741198 shutdown_connections 2024-05-20T18:10:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.874+0000 7f8dcc596640 1 -- 172.21.15.129:0/3151741198 wait complete. 2024-05-20T18:10:44.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dcc596640 1 Processor -- start 2024-05-20T18:10:44.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dcc596640 1 -- start start 2024-05-20T18:10:44.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dcc596640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dc4103c40 0x7f8dc4199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dcc596640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc410a6a0 0x7f8dc4199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dcc596640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dc410d050 0x7f8dc419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dcc596640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc4112e00 con 0x7f8dc4103c40 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dcc596640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc4112c80 con 0x7f8dc410d050 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dcc596640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc4112f80 con 0x7f8dc410a6a0 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dca332640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dc4103c40 0x7f8dc4199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dca332640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dc4103c40 0x7f8dc4199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32964/0 (socket says 172.21.15.129:32964) 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dca332640 1 -- 172.21.15.129:0/2840876375 learned_addr learned my addr 172.21.15.129:0/2840876375 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dc9b31640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc410a6a0 0x7f8dc4199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dca332640 1 -- 172.21.15.129:0/2840876375 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc410a6a0 msgr2=0x7f8dc4199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dca332640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc410a6a0 0x7f8dc4199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dca332640 1 -- 172.21.15.129:0/2840876375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dc410d050 msgr2=0x7f8dc419dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dcab33640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dc410d050 0x7f8dc419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dca332640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dc410d050 0x7f8dc419dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:44.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dca332640 1 -- 172.21.15.129:0/2840876375 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc419e560 con 0x7f8dc4103c40 2024-05-20T18:10:44.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.875+0000 7f8dc9b31640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc410a6a0 0x7f8dc4199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:10:44.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.876+0000 7f8dca332640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dc4103c40 0x7f8dc4199780 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8db400cd10 tx=0x7f8db400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:44.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.876+0000 7f8dc37fe640 1 -- 172.21.15.129:0/2840876375 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db400f650 con 0x7f8dc4103c40 2024-05-20T18:10:44.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.876+0000 7f8dcc596640 1 -- 172.21.15.129:0/2840876375 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc419e850 con 0x7f8dc4103c40 2024-05-20T18:10:44.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.876+0000 7f8dc37fe640 1 -- 172.21.15.129:0/2840876375 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db400fcd0 con 0x7f8dc4103c40 2024-05-20T18:10:44.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.876+0000 7f8dc37fe640 1 -- 172.21.15.129:0/2840876375 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4018cc0 con 0x7f8dc4103c40 2024-05-20T18:10:44.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.876+0000 7f8dcc596640 1 -- 172.21.15.129:0/2840876375 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc41a6250 con 0x7f8dc4103c40 2024-05-20T18:10:44.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.877+0000 7f8dcc596640 1 -- 172.21.15.129:0/2840876375 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dc4068290 con 0x7f8dc4103c40 2024-05-20T18:10:44.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.878+0000 7f8dc37fe640 1 -- 172.21.15.129:0/2840876375 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f8db400f7f0 con 0x7f8dc4103c40 2024-05-20T18:10:44.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.879+0000 7f8dc37fe640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8dac0a2d00 0x7f8dac0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:44.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.880+0000 7f8dc9b31640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8dac0a2d00 0x7f8dac0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:44.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.880+0000 7f8dc37fe640 1 -- 172.21.15.129:0/2840876375 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7f8db40cfcf0 con 0x7f8dc4103c40 2024-05-20T18:10:44.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.880+0000 7f8dc9b31640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8dac0a2d00 0x7f8dac0a5210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8db8005fd0 tx=0x7f8db80094d0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:44.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:44.884+0000 7f8dc37fe640 1 -- 172.21.15.129:0/2840876375 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8db4014030 con 0x7f8dc4103c40 2024-05-20T18:10:45.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.061+0000 7f8dcc596640 1 -- 172.21.15.129:0/2840876375 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8dc41a6420 con 0x7f8dc4103c40 2024-05-20T18:10:45.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.061+0000 7f8dc37fe640 1 -- 172.21.15.129:0/2840876375 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) ==== 100+0+11 (secure 0 0 0) 0x7f8db409c960 con 0x7f8dc4103c40 2024-05-20T18:10:45.059 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-05-20T18:10:45.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.063+0000 7f8dcc596640 1 -- 172.21.15.129:0/2840876375 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8dac0a2d00 msgr2=0x7f8dac0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:45.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.063+0000 7f8dcc596640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8dac0a2d00 0x7f8dac0a5210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8db8005fd0 tx=0x7f8db80094d0 comp rx=0 tx=0).stop 2024-05-20T18:10:45.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.063+0000 7f8dcc596640 1 -- 172.21.15.129:0/2840876375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dc4103c40 msgr2=0x7f8dc4199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:45.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.063+0000 7f8dcc596640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dc4103c40 0x7f8dc4199780 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8db400cd10 tx=0x7f8db400a620 comp rx=0 tx=0).stop 2024-05-20T18:10:45.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.064+0000 7f8dcc596640 1 -- 172.21.15.129:0/2840876375 shutdown_connections 2024-05-20T18:10:45.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.064+0000 7f8dcc596640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f8dac0a2d00 0x7f8dac0a5210 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:45.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.064+0000 7f8dcc596640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dc410d050 0x7f8dc419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:45.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.064+0000 7f8dcc596640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8dc410a6a0 0x7f8dc4199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:45.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.064+0000 7f8dcc596640 1 --2- 172.21.15.129:0/2840876375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dc4103c40 0x7f8dc4199780 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:45.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.064+0000 7f8dcc596640 1 -- 172.21.15.129:0/2840876375 >> 172.21.15.129:0/2840876375 conn(0x7f8dc40770c0 msgr2=0x7f8dc4100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:45.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.064+0000 7f8dcc596640 1 -- 172.21.15.129:0/2840876375 shutdown_connections 2024-05-20T18:10:45.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.064+0000 7f8dcc596640 1 -- 172.21.15.129:0/2840876375 wait complete. 2024-05-20T18:10:45.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-20T18:10:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.187+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/3333585819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4c01079e0 msgr2=0x7fd4c0101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:45.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.187+0000 7fd4c5b8f640 1 --2- 172.21.15.129:0/3333585819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4c01079e0 0x7fd4c0101570 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0009900 tx=0x7fd4b002a410 comp rx=0 tx=0).stop 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.187+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/3333585819 shutdown_connections 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.187+0000 7fd4c5b8f640 1 --2- 172.21.15.129:0/3333585819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4c010c340 0x7fd4c010e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.187+0000 7fd4c5b8f640 1 --2- 172.21.15.129:0/3333585819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c0101ab0 0x7fd4c0101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.187+0000 7fd4c5b8f640 1 --2- 172.21.15.129:0/3333585819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4c01079e0 0x7fd4c0101570 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.187+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/3333585819 >> 172.21.15.129:0/3333585819 conn(0x7fd4c00fd030 msgr2=0x7fd4c00ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.187+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/3333585819 shutdown_connections 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.187+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/3333585819 wait complete. 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.187+0000 7fd4c5b8f640 1 Processor -- start 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.187+0000 7fd4c5b8f640 1 -- start start 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4c5b8f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4c0101ab0 0x7fd4c019f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4c5b8f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4c01079e0 0x7fd4c019fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:45.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4c5b8f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c010c340 0x7fd4c0199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4c5b8f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c0112e10 con 0x7fd4c0101ab0 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4c5b8f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c0112c90 con 0x7fd4c01079e0 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4c5b8f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c0112f90 con 0x7fd4c010c340 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4c0101ab0 0x7fd4c019f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4c0101ab0 0x7fd4c019f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32986/0 (socket says 172.21.15.129:32986) 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bf7fe640 1 -- 172.21.15.129:0/2784549245 learned_addr learned my addr 172.21.15.129:0/2784549245 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bffff640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c010c340 0x7fd4c0199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4b7fff640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4c01079e0 0x7fd4c019fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bf7fe640 1 -- 172.21.15.129:0/2784549245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c010c340 msgr2=0x7fd4c0199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bf7fe640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c010c340 0x7fd4c0199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bf7fe640 1 -- 172.21.15.129:0/2784549245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4c01079e0 msgr2=0x7fd4c019fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bf7fe640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4c01079e0 0x7fd4c019fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bf7fe640 1 -- 172.21.15.129:0/2784549245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c0199d80 con 0x7fd4c0101ab0 2024-05-20T18:10:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4b7fff640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4c01079e0 0x7fd4c019fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-20T18:10:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bffff640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c010c340 0x7fd4c0199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:10:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bf7fe640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4c0101ab0 0x7fd4c019f4d0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0005940 tx=0x7fd4b00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bd7fa640 1 -- 172.21.15.129:0/2784549245 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b003f030 con 0x7fd4c0101ab0 2024-05-20T18:10:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/2784549245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c0199fb0 con 0x7fd4c0101ab0 2024-05-20T18:10:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bd7fa640 1 -- 172.21.15.129:0/2784549245 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4b0003ce0 con 0x7fd4c0101ab0 2024-05-20T18:10:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4bd7fa640 1 -- 172.21.15.129:0/2784549245 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0035070 con 0x7fd4c0101ab0 2024-05-20T18:10:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.188+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/2784549245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c01a6270 con 0x7fd4c0101ab0 2024-05-20T18:10:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.189+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/2784549245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c0102c90 con 0x7fd4c0101ab0 2024-05-20T18:10:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.191+0000 7fd4bd7fa640 1 -- 172.21.15.129:0/2784549245 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fd4b0042030 con 0x7fd4c0101ab0 2024-05-20T18:10:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.191+0000 7fd4bd7fa640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4ac0a2d00 0x7fd4ac0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:10:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.191+0000 7fd4b7fff640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4ac0a2d00 0x7fd4ac0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:10:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.192+0000 7fd4bd7fa640 1 -- 172.21.15.129:0/2784549245 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8587+0+0 (secure 0 0 0) 0x7fd4b00efde0 con 0x7fd4c0101ab0 2024-05-20T18:10:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.192+0000 7fd4b7fff640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4ac0a2d00 0x7fd4ac0a5210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd4c019adf0 tx=0x7fd4a8005e70 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:10:45.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.195+0000 7fd4bd7fa640 1 -- 172.21.15.129:0/2784549245 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd4b00bc960 con 0x7fd4c0101ab0 2024-05-20T18:10:45.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.421+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/2784549245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fd4c01a6440 con 0x7fd4c0101ab0 2024-05-20T18:10:45.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.430+0000 7fd4bd7fa640 1 -- 172.21.15.129:0/2784549245 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+740910 (secure 0 0 0) 0x7fd4b00bc260 con 0x7fd4c0101ab0 2024-05-20T18:10:45.436 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:10:45.436 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:10:45.436 INFO:teuthology.orchestra.run.smithi129.stdout: "epoch": 25, 2024-05-20T18:10:45.436 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: "active_gid": 15228, 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: "active_name": "x", 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: "active_addrs": { 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: "addrvec": [ 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v2", 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "172.21.15.129:6800", 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 3515662734 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v1", 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "172.21.15.129:6801", 2024-05-20T18:10:45.437 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 3515662734 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: "active_addr": "172.21.15.129:6801/3515662734", 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: "active_change": "2024-05-20T17:58:43.562728+0000", 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: "active_mgr_features": 4540701547738038271, 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: "available": true, 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: "standbys": [ 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: "gid": 24967, 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "z", 2024-05-20T18:10:45.438 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr_features": 4540701547738038271, 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "available_modules": [ 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "alerts", 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:45.439 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How frequently to reexamine health status", 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.440 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.441 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.442 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.443 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.444 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_destination": { 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_destination", 2024-05-20T18:10:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Email address to send alerts to", 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_from_name": { 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_from_name", 2024-05-20T18:10:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "Ceph", 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Email From: name", 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_host": { 2024-05-20T18:10:45.447 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_host", 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SMTP server", 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_password": { 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_password", 2024-05-20T18:10:45.448 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Password to authenticate with", 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_port": { 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_port", 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.449 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "465", 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SMTP port", 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_sender": { 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_sender", 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.450 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SMTP envelope sender", 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_ssl": { 2024-05-20T18:10:45.451 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_ssl", 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_user": { 2024-05-20T18:10:45.452 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_user", 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "User to authenticate as", 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.453 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "balancer", 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "active": { 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "active", 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.454 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "automatically balance PGs across cluster", 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "begin_time": { 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "begin_time", 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0000", 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.455 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "beginning time of day to automatically balance", 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "begin_weekday": { 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "begin_weekday", 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "0", 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "6", 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.456 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "crush_compat_max_iterations": { 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crush_compat_max_iterations", 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "25", 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "250", 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-20T18:10:45.457 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "crush_compat_metrics": { 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crush_compat_metrics", 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "pgs,objects,bytes", 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-20T18:10:45.458 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "crush_compat_step": { 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crush_compat_step", 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0.5", 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "0.001", 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "0.999", 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "aggressiveness of optimization", 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.459 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "end_time": { 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "end_time", 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "2359", 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "ending time of day to automatically balance", 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.460 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "end_weekday": { 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "end_weekday", 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "0", 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "6", 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.461 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.462 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.463 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.464 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.465 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "min_score": { 2024-05-20T18:10:45.466 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "min_score", 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": { 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mode", 2024-05-20T18:10:45.467 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "upmap", 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "crush-compat", 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "none", 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "read", 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap", 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap-read" 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.468 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Balancer mode", 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_ids": { 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pool_ids", 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-20T18:10:45.469 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.470 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap_max_deviation": { 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "upmap_max_deviation", 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.471 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap_max_optimizations": { 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "upmap_max_optimizations", 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.472 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephadm", 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: "agent_down_multiplier": { 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "agent_down_multiplier", 2024-05-20T18:10:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "3.0", 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "agent_refresh_rate": { 2024-05-20T18:10:45.474 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "agent_refresh_rate", 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "20", 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "agent_starting_port": { 2024-05-20T18:10:45.475 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "agent_starting_port", 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "4721", 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.476 INFO:teuthology.orchestra.run.smithi129.stdout: "allow_ptrace": { 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "allow_ptrace", 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.477 INFO:teuthology.orchestra.run.smithi129.stdout: "autotune_interval": { 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "autotune_interval", 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "autotune_memory_target_ratio": { 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "autotune_memory_target_ratio", 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0.7", 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "cephadm_log_destination": { 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephadm_log_destination", 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "file", 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "file,syslog", 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "syslog" 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.480 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "cgroups_split": { 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cgroups_split", 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.481 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: "config_checks_enabled": { 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "config_checks_enabled", 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.482 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "config_dashboard": { 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "config_dashboard", 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.483 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_alertmanager": { 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_alertmanager", 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.484 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_base": { 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_base", 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-20T18:10:45.485 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Container image name, without the tag", 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_elasticsearch": { 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_elasticsearch", 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.486 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "elasticsearch container image", 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_grafana": { 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_grafana", 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.487 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_haproxy": { 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_haproxy", 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.488 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-20T18:10:45.489 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.489 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.489 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.489 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "HAproxy container image", 2024-05-20T18:10:45.489 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.489 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.489 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.489 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_jaeger_agent": { 2024-05-20T18:10:45.489 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_jaeger_agent", 2024-05-20T18:10:45.489 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Jaeger agent container image", 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_jaeger_collector": { 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_jaeger_collector", 2024-05-20T18:10:45.490 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Jaeger collector container image", 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_jaeger_query": { 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_jaeger_query", 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.491 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Jaeger query container image", 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_keepalived": { 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_keepalived", 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.492 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Keepalived container image", 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_loki": { 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_loki", 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Loki container image", 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_node_exporter": { 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_node_exporter", 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.494 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_nvmeof": { 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_nvmeof", 2024-05-20T18:10:45.495 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Nvme-of container image", 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_prometheus": { 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_prometheus", 2024-05-20T18:10:45.496 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_promtail": { 2024-05-20T18:10:45.497 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_promtail", 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Promtail container image", 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.498 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_samba": { 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_samba", 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Samba/SMB container image", 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.499 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_snmp_gateway": { 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_snmp_gateway", 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SNMP Gateway container image", 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.500 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "container_init": { 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_init", 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Run podman/docker with `--init`", 2024-05-20T18:10:45.501 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "daemon_cache_timeout": { 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "daemon_cache_timeout", 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "default_cephadm_command_timeout": { 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "default_cephadm_command_timeout", 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "900", 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.503 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "default_registry": { 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "default_registry", 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io", 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.504 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "device_cache_timeout": { 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "device_cache_timeout", 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "1800", 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.505 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "seconds to cache device inventory", 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "device_enhanced_scan": { 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "device_enhanced_scan", 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.506 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "facts_cache_timeout": { 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "facts_cache_timeout", 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.507 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "seconds to cache host facts data", 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "grafana_dashboards_path": { 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "grafana_dashboards_path", 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.508 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "host_check_interval": { 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "host_check_interval", 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.509 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to perform a host check", 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "hw_monitoring": { 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "hw_monitoring", 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.510 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "inventory_list_all": { 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "inventory_list_all", 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.511 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.512 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "log_refresh_metadata": { 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_refresh_metadata", 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.513 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.514 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.515 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.516 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.517 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.518 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "*", 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "max_count_per_host": { 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_count_per_host", 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "max number of daemons per service per host", 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "max_osd_draining_count": { 2024-05-20T18:10:45.520 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_osd_draining_count", 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "migration_current": { 2024-05-20T18:10:45.521 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "migration_current", 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "internal - do not modify", 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": { 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mode", 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "root", 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "cephadm-package", 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "root" 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "mode for remote execution of cephadm", 2024-05-20T18:10:45.523 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: "oob_default_addr": { 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "oob_default_addr", 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "169.254.1.1", 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.524 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "prometheus_alerts_path": { 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "prometheus_alerts_path", 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.525 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_insecure": { 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_insecure", 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.526 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_password": { 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_password", 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.527 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_url": { 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_url", 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.528 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_username": { 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_username", 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.529 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "secure_monitoring_stack": { 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "secure_monitoring_stack", 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.530 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "service_discovery_port": { 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "service_discovery_port", 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8765", 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.531 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "cephadm service discovery port", 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "ssh_config_file": { 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssh_config_file", 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-20T18:10:45.532 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "use_agent": { 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "use_agent", 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-20T18:10:45.533 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "use_repo_digest": { 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "use_repo_digest", 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-20T18:10:45.534 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_on_failed_host_check": { 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_on_failed_host_check", 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise a health warning if the host check fails", 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.535 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_on_stray_daemons": { 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_on_stray_daemons", 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.536 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_on_stray_hosts": { 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_on_stray_hosts", 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.537 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crash", 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.538 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.539 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.540 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.541 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.542 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "retain_interval": { 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "retain_interval", 2024-05-20T18:10:45.543 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "31536000", 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_recent_interval": { 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_recent_interval", 2024-05-20T18:10:45.544 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "1209600", 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "dashboard", 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.546 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "ALERTMANAGER_API_HOST": { 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.547 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "AUDIT_API_ENABLED": { 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "AUDIT_API_ENABLED", 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-20T18:10:45.549 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "ENABLE_BROWSABLE_API": { 2024-05-20T18:10:45.550 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-20T18:10:45.551 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.552 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-20T18:10:45.553 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-20T18:10:45.554 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.555 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.556 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.557 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.558 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_PASSWORD": { 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.559 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "admin", 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.560 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_URL": { 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_URL", 2024-05-20T18:10:45.561 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_USERNAME": { 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-20T18:10:45.562 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "admin", 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-20T18:10:45.563 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-20T18:10:45.564 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.566 INFO:teuthology.orchestra.run.smithi129.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "PROMETHEUS_API_HOST": { 2024-05-20T18:10:45.567 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-20T18:10:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.569 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-20T18:10:45.570 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-20T18:10:45.571 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.572 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.573 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.574 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.575 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_ENABLED": { 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.576 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.577 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.578 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8", 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.579 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "45", 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.580 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_ACCESS_KEY": { 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.581 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "admin", 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.582 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_SECRET_KEY": { 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.583 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_SSL_VERIFY": { 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.584 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:45.585 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.586 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.587 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "cross_origin_url": { 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cross_origin_url", 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.588 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "crt_file": { 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crt_file", 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.589 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "debug": { 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "debug", 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.590 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Enable/disable debug options", 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "jwt_token_ttl": { 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "jwt_token_ttl", 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.591 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "28800", 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "key_file": { 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "key_file", 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.592 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.593 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.594 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.595 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.596 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.597 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "motd": { 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "motd", 2024-05-20T18:10:45.598 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "The message of the day", 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "redirect_resolve_ip_addr": { 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "redirect_resolve_ip_addr", 2024-05-20T18:10:45.599 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "server_addr": { 2024-05-20T18:10:45.600 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_addr", 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "::", 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "server_port": { 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_port", 2024-05-20T18:10:45.601 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8080", 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "ssl": { 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssl", 2024-05-20T18:10:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "ssl_server_port": { 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssl_server_port", 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.603 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8443", 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_behaviour": { 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_behaviour", 2024-05-20T18:10:45.604 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "redirect", 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "redirect" 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_error_status_code": { 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_error_status_code", 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "500", 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "400", 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "599", 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.606 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "url_prefix": { 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "url_prefix", 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.607 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "devicehealth", 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "enable_monitoring": { 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enable_monitoring", 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.608 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "monitor device health metrics", 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.609 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.610 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.611 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.612 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "mark_out_threshold": { 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mark_out_threshold", 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "2419200", 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.614 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_name": { 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pool_name", 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "device_health_metrics", 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.615 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "retention_period": { 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "retention_period", 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "15552000", 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how long to retain device health metrics", 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.616 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "scrape_frequency": { 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "scrape_frequency", 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "86400", 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.617 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "self_heal": { 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "self_heal", 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.618 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to wake up and check device health", 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.619 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_threshold": { 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_threshold", 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "7257600", 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.620 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "diskprediction_local", 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.621 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.622 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.623 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.624 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.625 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "predict_interval": { 2024-05-20T18:10:45.626 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "predict_interval", 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "86400", 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "predictor_model": { 2024-05-20T18:10:45.627 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "predictor_model", 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "prophetstor", 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:45.628 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.629 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "influx", 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": false, 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "influxdb python module not found", 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "batch_size": { 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "batch_size", 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5000", 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.630 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "database": { 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "database", 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "ceph", 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.631 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "hostname": { 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "hostname", 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "InfluxDB server hostname", 2024-05-20T18:10:45.632 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "30", 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "5", 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-20T18:10:45.633 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.634 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.635 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.636 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.637 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.638 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "password": { 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "password", 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "password of InfluxDB server user", 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "port": { 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "port", 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8086", 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.640 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "InfluxDB server port", 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "ssl": { 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssl", 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "false", 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.641 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "threads": { 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "threads", 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:45.642 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "32", 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "username": { 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "username", 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.643 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "username of InfluxDB server user", 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: "verify_ssl": { 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "verify_ssl", 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.644 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "true", 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.645 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "insights", 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.646 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.648 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.649 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.650 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "iostat", 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.651 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.652 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.653 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.654 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.655 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.656 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "k8sevents", 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_event_retention_days": { 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ceph_event_retention_days", 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "7", 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.657 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.658 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.658 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-20T18:10:45.658 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.658 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.658 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.658 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.658 INFO:teuthology.orchestra.run.smithi129.stdout: "config_check_secs": { 2024-05-20T18:10:45.658 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "config_check_secs", 2024-05-20T18:10:45.658 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.662 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.662 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.662 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.662 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "10", 2024-05-20T18:10:45.662 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.662 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.662 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-20T18:10:45.662 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.662 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.662 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.663 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.664 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.665 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.667 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "localpool", 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "failure_domain": { 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "failure_domain", 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "host", 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.668 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "failure domain for any created local pool", 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.669 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.670 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.671 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.672 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.673 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "min_size": { 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "min_size", 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "default min_size for any created local pool", 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.674 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rep": { 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "num_rep", 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "3", 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "default replica count for any created local pool", 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.675 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "pg_num": { 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pg_num", 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "128", 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "default pg_num for any created local pool", 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.676 INFO:teuthology.orchestra.run.smithi129.stdout: "prefix": { 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "prefix", 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "name prefix for any created local pool", 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.677 INFO:teuthology.orchestra.run.smithi129.stdout: "subtree": { 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "subtree", 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "rack", 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.678 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mds_autoscaler", 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.679 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.681 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.682 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.683 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mirroring", 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.684 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.685 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.686 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.687 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.688 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.689 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "nfs", 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.690 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.691 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.692 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.693 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.694 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "orchestrator", 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: "fail_fs": { 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "fail_fs", 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.695 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.696 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.697 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.699 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.700 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "orchestrator": { 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "orchestrator", 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: "cephadm", 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: "rook", 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: "test_orchestrator" 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Orchestrator backend", 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "osd_perf_query", 2024-05-20T18:10:45.702 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.703 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.704 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.705 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.706 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.707 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "osd_support", 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.708 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.709 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.710 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.711 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.712 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pg_autoscaler", 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.713 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.714 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.715 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.716 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.717 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.718 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "threshold": { 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "threshold", 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "3.0", 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1.0", 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.719 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "scaling threshold", 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "progress", 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: "allow_pg_recovery_event": { 2024-05-20T18:10:45.720 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "allow_pg_recovery_event", 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.721 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "enabled": { 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enabled", 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.722 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.723 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.724 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.725 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.726 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.727 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "max_completed_events": { 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_completed_events", 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "50", 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "number of past completed events to remember", 2024-05-20T18:10:45.728 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.729 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how long the module is going to sleep", 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "prometheus", 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: "cache": { 2024-05-20T18:10:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cache", 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.731 INFO:teuthology.orchestra.run.smithi129.stdout: "exclude_perf_counters": { 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "exclude_perf_counters", 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.732 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.733 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.734 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.735 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.736 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.737 INFO:teuthology.orchestra.run.smithi129.stdout: "rbd_stats_pools": { 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rbd_stats_pools", 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.738 INFO:teuthology.orchestra.run.smithi129.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "300", 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.739 INFO:teuthology.orchestra.run.smithi129.stdout: "scrape_interval": { 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "scrape_interval", 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "15.0", 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.740 INFO:teuthology.orchestra.run.smithi129.stdout: "server_addr": { 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_addr", 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "::", 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "server_port": { 2024-05-20T18:10:45.741 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_port", 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "9283", 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "stale_cache_strategy": { 2024-05-20T18:10:45.742 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "stale_cache_strategy", 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "log", 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_behaviour": { 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_behaviour", 2024-05-20T18:10:45.743 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "default", 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "default", 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "error" 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.744 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_error_status_code": { 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_error_status_code", 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "500", 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "400", 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "599", 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rbd_support", 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.746 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.748 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.749 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.750 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "max_concurrent_snap_create": { 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_concurrent_snap_create", 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.751 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "mirror_snapshot_schedule": { 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mirror_snapshot_schedule", 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.752 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "trash_purge_schedule": { 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "trash_purge_schedule", 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.753 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "restful", 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: "enable_auth": { 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enable_auth", 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.754 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "key_file": { 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "key_file", 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.755 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.756 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.756 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.756 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.756 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.756 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.756 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.756 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.756 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.756 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.756 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.756 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.757 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.758 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.759 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.760 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "server_addr": { 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_addr", 2024-05-20T18:10:45.761 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "server_port": { 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_port", 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.762 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.763 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rgw", 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.764 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.765 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.766 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.767 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.768 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rook", 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.769 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.770 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.771 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.772 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.773 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "storage_class": { 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "storage_class", 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "local", 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.774 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "selftest", 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.775 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.776 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.777 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.778 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.779 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.780 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "roption1": { 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "roption1", 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.781 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "roption2": { 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "roption2", 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "xyz", 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.782 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption1": { 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption1", 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption2": { 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption2", 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.784 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption3": { 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption3", 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption4": { 2024-05-20T18:10:45.785 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption4", 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.786 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption5": { 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption5", 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.787 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption6": { 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption6", 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.788 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption7": { 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption7", 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "42", 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "testkey": { 2024-05-20T18:10:45.789 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "testkey", 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "testlkey": { 2024-05-20T18:10:45.790 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "testlkey", 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.791 INFO:teuthology.orchestra.run.smithi129.stdout: "testnewline": { 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "testnewline", 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.792 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "snap_schedule", 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "allow_m_granularity": { 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "allow_m_granularity", 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.793 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "allow minute scheduled snapshots", 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "dump_on_update": { 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "dump_on_update", 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.794 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "dump database to debug log on update", 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.795 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.796 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.797 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.798 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.799 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "stats", 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.800 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.801 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.802 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.803 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.804 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "status", 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.805 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.806 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.807 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.808 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.809 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.810 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "telegraf", 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "address": { 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "address", 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.811 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "15", 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.812 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.813 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.815 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.816 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.817 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "telemetry", 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.818 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_basic": { 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_basic", 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.819 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_crash": { 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_crash", 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.820 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_device": { 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_device", 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-20T18:10:45.821 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_ident": { 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_ident", 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_perf": { 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_perf", 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.823 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "contact": { 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "contact", 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.824 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: "description": { 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "description", 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.825 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "device_url": { 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "device_url", 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.826 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "enabled": { 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enabled", 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.827 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:45.828 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "24", 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "8", 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "last_opt_revision": { 2024-05-20T18:10:45.829 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "last_opt_revision", 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "1", 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.830 INFO:teuthology.orchestra.run.smithi129.stdout: "leaderboard": { 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "leaderboard", 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.831 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "leaderboard_description": { 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "leaderboard_description", 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.832 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.833 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.834 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.835 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.836 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.837 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "organization": { 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "organization", 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.838 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "proxy": { 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "proxy", 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.839 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "url": { 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "url", 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.840 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "test_orchestrator", 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.841 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.842 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.843 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.844 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.845 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.846 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "volumes", 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.847 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.848 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.849 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.850 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.851 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "max_concurrent_clones": { 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_concurrent_clones", 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "4", 2024-05-20T18:10:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "periodic_async_work": { 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "periodic_async_work", 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.853 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.854 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.854 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.854 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Periodically check for async work", 2024-05-20T18:10:45.854 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.854 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.854 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.854 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.854 INFO:teuthology.orchestra.run.smithi129.stdout: "snapshot_clone_delay": { 2024-05-20T18:10:45.854 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "snapshot_clone_delay", 2024-05-20T18:10:45.854 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "snapshot_clone_no_wait": { 2024-05-20T18:10:45.855 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "snapshot_clone_no_wait", 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.856 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix", 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "discovery_interval": { 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "discovery_interval", 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "100", 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.857 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "identifier": { 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "identifier", 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.858 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.859 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.860 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.861 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.862 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.863 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.864 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "zabbix_host": { 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix_host", 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.865 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "zabbix_port": { 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix_port", 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10051", 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.866 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "zabbix_sender": { 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix_sender", 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.867 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: "gid": 24971, 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "y", 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr_features": 4540701547738038271, 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: "available_modules": [ 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.868 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "alerts", 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How frequently to reexamine health status", 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.869 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.870 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.871 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.872 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.873 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.874 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_destination": { 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_destination", 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Email address to send alerts to", 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.875 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_from_name": { 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_from_name", 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "Ceph", 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Email From: name", 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.876 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_host": { 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_host", 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SMTP server", 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.877 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_password": { 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_password", 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Password to authenticate with", 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.878 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_port": { 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_port", 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "465", 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SMTP port", 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_sender": { 2024-05-20T18:10:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_sender", 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SMTP envelope sender", 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_ssl": { 2024-05-20T18:10:45.880 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_ssl", 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_user": { 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_user", 2024-05-20T18:10:45.881 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "User to authenticate as", 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.882 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "balancer", 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "active": { 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "active", 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.883 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "automatically balance PGs across cluster", 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "begin_time": { 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "begin_time", 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0000", 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.884 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "beginning time of day to automatically balance", 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "begin_weekday": { 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "begin_weekday", 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "0", 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "6", 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.885 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: "crush_compat_max_iterations": { 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crush_compat_max_iterations", 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "25", 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:45.886 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "250", 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "crush_compat_metrics": { 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crush_compat_metrics", 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "pgs,objects,bytes", 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.887 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "crush_compat_step": { 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crush_compat_step", 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0.5", 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "0.001", 2024-05-20T18:10:45.888 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "0.999", 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "aggressiveness of optimization", 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "end_time": { 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "end_time", 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "2359", 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.889 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "ending time of day to automatically balance", 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "end_weekday": { 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "end_weekday", 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "0", 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "6", 2024-05-20T18:10:45.890 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.891 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.892 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.893 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.894 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.895 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "min_score": { 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "min_score", 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-20T18:10:45.896 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": { 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mode", 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "upmap", 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "crush-compat", 2024-05-20T18:10:45.897 INFO:teuthology.orchestra.run.smithi129.stdout: "none", 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "read", 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap", 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap-read" 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Balancer mode", 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_ids": { 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pool_ids", 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.898 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.899 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap_max_deviation": { 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "upmap_max_deviation", 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap_max_optimizations": { 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "upmap_max_optimizations", 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.901 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephadm", 2024-05-20T18:10:45.902 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "agent_down_multiplier": { 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "agent_down_multiplier", 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "3.0", 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.903 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "agent_refresh_rate": { 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "agent_refresh_rate", 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "20", 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.904 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "agent_starting_port": { 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "agent_starting_port", 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "4721", 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.905 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "allow_ptrace": { 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "allow_ptrace", 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.906 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "autotune_interval": { 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "autotune_interval", 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.907 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "autotune_memory_target_ratio": { 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "autotune_memory_target_ratio", 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0.7", 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.908 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "cephadm_log_destination": { 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephadm_log_destination", 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "file", 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "file,syslog", 2024-05-20T18:10:45.909 INFO:teuthology.orchestra.run.smithi129.stdout: "syslog" 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "cgroups_split": { 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cgroups_split", 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.910 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "config_checks_enabled": { 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "config_checks_enabled", 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "config_dashboard": { 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "config_dashboard", 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.912 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_alertmanager": { 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_alertmanager", 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.913 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_base": { 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_base", 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-20T18:10:45.914 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Container image name, without the tag", 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_elasticsearch": { 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_elasticsearch", 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.915 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "elasticsearch container image", 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_grafana": { 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_grafana", 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.916 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_haproxy": { 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_haproxy", 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.917 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "HAproxy container image", 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_jaeger_agent": { 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_jaeger_agent", 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-20T18:10:45.918 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Jaeger agent container image", 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_jaeger_collector": { 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_jaeger_collector", 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.919 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Jaeger collector container image", 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_jaeger_query": { 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_jaeger_query", 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.920 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Jaeger query container image", 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_keepalived": { 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_keepalived", 2024-05-20T18:10:45.921 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Keepalived container image", 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_loki": { 2024-05-20T18:10:45.922 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_loki", 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Loki container image", 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_node_exporter": { 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_node_exporter", 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.924 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_nvmeof": { 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_nvmeof", 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Nvme-of container image", 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.925 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_prometheus": { 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_prometheus", 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.926 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_promtail": { 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_promtail", 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Promtail container image", 2024-05-20T18:10:45.927 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_samba": { 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_samba", 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Samba/SMB container image", 2024-05-20T18:10:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_snmp_gateway": { 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_snmp_gateway", 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SNMP Gateway container image", 2024-05-20T18:10:45.929 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "container_init": { 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_init", 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Run podman/docker with `--init`", 2024-05-20T18:10:45.930 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "daemon_cache_timeout": { 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "daemon_cache_timeout", 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-20T18:10:45.931 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "default_cephadm_command_timeout": { 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "default_cephadm_command_timeout", 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "900", 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "default_registry": { 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "default_registry", 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io", 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.933 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "device_cache_timeout": { 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "device_cache_timeout", 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "1800", 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.934 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "seconds to cache device inventory", 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "device_enhanced_scan": { 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "device_enhanced_scan", 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.935 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "facts_cache_timeout": { 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "facts_cache_timeout", 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.936 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "seconds to cache host facts data", 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "grafana_dashboards_path": { 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "grafana_dashboards_path", 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.937 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "host_check_interval": { 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "host_check_interval", 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.938 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to perform a host check", 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "hw_monitoring": { 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "hw_monitoring", 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.939 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "inventory_list_all": { 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "inventory_list_all", 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-20T18:10:45.940 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.941 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "log_refresh_metadata": { 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_refresh_metadata", 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.942 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.943 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.944 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.945 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.946 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.947 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "*", 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.948 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "max_count_per_host": { 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_count_per_host", 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.949 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "max number of daemons per service per host", 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "max_osd_draining_count": { 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_osd_draining_count", 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.950 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "migration_current": { 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "migration_current", 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.951 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "internal - do not modify", 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": { 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mode", 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "root", 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.952 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: "cephadm-package", 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: "root" 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "mode for remote execution of cephadm", 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: "oob_default_addr": { 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "oob_default_addr", 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "169.254.1.1", 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "prometheus_alerts_path": { 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "prometheus_alerts_path", 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.954 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_insecure": { 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_insecure", 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.955 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_password": { 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_password", 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.956 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_url": { 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_url", 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.957 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_username": { 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_username", 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.958 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "secure_monitoring_stack": { 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "secure_monitoring_stack", 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.959 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: "service_discovery_port": { 2024-05-20T18:10:45.960 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "service_discovery_port", 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8765", 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "cephadm service discovery port", 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.961 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "ssh_config_file": { 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssh_config_file", 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.962 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "use_agent": { 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "use_agent", 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.963 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "use_repo_digest": { 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "use_repo_digest", 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.964 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_on_failed_host_check": { 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_on_failed_host_check", 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise a health warning if the host check fails", 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.965 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_on_stray_daemons": { 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_on_stray_daemons", 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.966 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_on_stray_hosts": { 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_on_stray_hosts", 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-20T18:10:45.967 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crash", 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.968 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.969 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.970 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.971 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.972 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.973 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "retain_interval": { 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "retain_interval", 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "31536000", 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.974 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_recent_interval": { 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_recent_interval", 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "1209600", 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.975 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "dashboard", 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:45.976 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "ALERTMANAGER_API_HOST": { 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-20T18:10:45.977 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-20T18:10:45.978 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.979 INFO:teuthology.orchestra.run.smithi129.stdout: "AUDIT_API_ENABLED": { 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "AUDIT_API_ENABLED", 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.980 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.981 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "ENABLE_BROWSABLE_API": { 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.982 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.983 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.984 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.985 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.986 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.988 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.989 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.990 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_PASSWORD": { 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.991 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "admin", 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.992 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_URL": { 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_URL", 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.993 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_USERNAME": { 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.994 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "admin", 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.995 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.996 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:45.997 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:45.998 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "PROMETHEUS_API_HOST": { 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:45.999 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.000 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.001 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.002 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.003 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.004 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.005 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.006 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-20T18:10:46.007 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_ENABLED": { 2024-05-20T18:10:46.008 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.009 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.010 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.011 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8", 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.012 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "45", 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.013 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_ACCESS_KEY": { 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.014 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "admin", 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.015 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_SECRET_KEY": { 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.016 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_SSL_VERIFY": { 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.017 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.018 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.019 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.020 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "cross_origin_url": { 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cross_origin_url", 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.021 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "crt_file": { 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crt_file", 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.022 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: "debug": { 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "debug", 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Enable/disable debug options", 2024-05-20T18:10:46.023 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "jwt_token_ttl": { 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "jwt_token_ttl", 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "28800", 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "key_file": { 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "key_file", 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.026 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.027 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.028 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.029 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.030 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "motd": { 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "motd", 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "The message of the day", 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.031 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "redirect_resolve_ip_addr": { 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "redirect_resolve_ip_addr", 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "server_addr": { 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_addr", 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "::", 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "server_port": { 2024-05-20T18:10:46.033 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_port", 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8080", 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.034 INFO:teuthology.orchestra.run.smithi129.stdout: "ssl": { 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssl", 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "ssl_server_port": { 2024-05-20T18:10:46.035 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssl_server_port", 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8443", 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_behaviour": { 2024-05-20T18:10:46.036 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_behaviour", 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "redirect", 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "redirect" 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.037 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_error_status_code": { 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_error_status_code", 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "500", 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "400", 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "599", 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.038 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "url_prefix": { 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "url_prefix", 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.039 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "devicehealth", 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: "enable_monitoring": { 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enable_monitoring", 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.040 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "monitor device health metrics", 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.041 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.042 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.043 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.044 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.045 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "mark_out_threshold": { 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mark_out_threshold", 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "2419200", 2024-05-20T18:10:46.046 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_name": { 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pool_name", 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "device_health_metrics", 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.047 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: "retention_period": { 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "retention_period", 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "15552000", 2024-05-20T18:10:46.048 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how long to retain device health metrics", 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "scrape_frequency": { 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "scrape_frequency", 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "86400", 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.049 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "self_heal": { 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "self_heal", 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.050 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.051 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to wake up and check device health", 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_threshold": { 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_threshold", 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "7257600", 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.052 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "diskprediction_local", 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.053 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.054 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.055 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.056 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.057 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "predict_interval": { 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "predict_interval", 2024-05-20T18:10:46.058 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "86400", 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "predictor_model": { 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "predictor_model", 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.059 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "prophetstor", 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "influx", 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": false, 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "influxdb python module not found", 2024-05-20T18:10:46.061 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "batch_size": { 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "batch_size", 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5000", 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.062 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "database": { 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "database", 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "ceph", 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.063 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "hostname": { 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "hostname", 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "InfluxDB server hostname", 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.064 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "30", 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "5", 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.065 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.066 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.067 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.068 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.069 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.070 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "password": { 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "password", 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "password of InfluxDB server user", 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.071 INFO:teuthology.orchestra.run.smithi129.stdout: "port": { 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "port", 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8086", 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "InfluxDB server port", 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.072 INFO:teuthology.orchestra.run.smithi129.stdout: "ssl": { 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssl", 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "false", 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.073 INFO:teuthology.orchestra.run.smithi129.stdout: "threads": { 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "threads", 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "32", 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.074 INFO:teuthology.orchestra.run.smithi129.stdout: "username": { 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "username", 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "username of InfluxDB server user", 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "verify_ssl": { 2024-05-20T18:10:46.075 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "verify_ssl", 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "true", 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "insights", 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.077 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.078 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.079 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.080 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.081 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "iostat", 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.082 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.083 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.084 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.085 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.086 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "k8sevents", 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.087 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_event_retention_days": { 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ceph_event_retention_days", 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "7", 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.088 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "config_check_secs": { 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "config_check_secs", 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "10", 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.089 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.090 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.091 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.092 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.093 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.094 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "localpool", 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "failure_domain": { 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "failure_domain", 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "host", 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.095 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "failure domain for any created local pool", 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.096 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.097 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.098 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.099 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.100 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "min_size": { 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "min_size", 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.101 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "default min_size for any created local pool", 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rep": { 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "num_rep", 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "3", 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "default replica count for any created local pool", 2024-05-20T18:10:46.102 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.103 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.103 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.103 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.103 INFO:teuthology.orchestra.run.smithi129.stdout: "pg_num": { 2024-05-20T18:10:46.103 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pg_num", 2024-05-20T18:10:46.103 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.103 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.103 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.103 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "128", 2024-05-20T18:10:46.103 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.103 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "default pg_num for any created local pool", 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "prefix": { 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "prefix", 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.104 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "name prefix for any created local pool", 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "subtree": { 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "subtree", 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "rack", 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.105 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mds_autoscaler", 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.106 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.107 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.108 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.109 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.110 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mirroring", 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.112 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.113 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.114 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.115 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.116 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "nfs", 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.117 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.118 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.119 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.120 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.121 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.122 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "orchestrator", 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: "fail_fs": { 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "fail_fs", 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.123 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.124 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.125 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.126 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.128 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "orchestrator": { 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "orchestrator", 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.129 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: "cephadm", 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: "rook", 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: "test_orchestrator" 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Orchestrator backend", 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.130 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "osd_perf_query", 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.131 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.132 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.133 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.134 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.135 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "osd_support", 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.136 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.137 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.138 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.139 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.140 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.141 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pg_autoscaler", 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.142 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.143 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.144 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.145 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.146 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.147 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "threshold": { 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "threshold", 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "3.0", 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1.0", 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.148 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "scaling threshold", 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "progress", 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: "allow_pg_recovery_event": { 2024-05-20T18:10:46.149 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "allow_pg_recovery_event", 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.150 INFO:teuthology.orchestra.run.smithi129.stdout: "enabled": { 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enabled", 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.151 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.152 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.153 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.154 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.155 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "max_completed_events": { 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_completed_events", 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.156 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "50", 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "number of past completed events to remember", 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.157 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how long the module is going to sleep", 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "prometheus", 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "cache": { 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cache", 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.159 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "exclude_perf_counters": { 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "exclude_perf_counters", 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.160 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.161 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.162 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.163 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.164 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.165 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "rbd_stats_pools": { 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rbd_stats_pools", 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.166 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "300", 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.167 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "scrape_interval": { 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "scrape_interval", 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "15.0", 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.168 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "server_addr": { 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_addr", 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "::", 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.169 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "server_port": { 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_port", 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "9283", 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.170 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "stale_cache_strategy": { 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "stale_cache_strategy", 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "log", 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.171 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_behaviour": { 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_behaviour", 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "default", 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "default", 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "error" 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.172 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_error_status_code": { 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_error_status_code", 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "500", 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "400", 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "599", 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.173 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rbd_support", 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.175 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.176 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.177 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "max_concurrent_snap_create": { 2024-05-20T18:10:46.179 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_concurrent_snap_create", 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "mirror_snapshot_schedule": { 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mirror_snapshot_schedule", 2024-05-20T18:10:46.180 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "trash_purge_schedule": { 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "trash_purge_schedule", 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.181 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.182 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "restful", 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "enable_auth": { 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enable_auth", 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.183 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "key_file": { 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "key_file", 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.184 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.185 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.186 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.187 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.188 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.189 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "server_addr": { 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_addr", 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.190 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "server_port": { 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_port", 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.191 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rgw", 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.192 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.193 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.194 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.195 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.196 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.197 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rook", 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.198 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.199 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.200 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.201 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.202 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: "storage_class": { 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "storage_class", 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.203 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "local", 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "selftest", 2024-05-20T18:10:46.204 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.206 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.207 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.208 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.209 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "roption1": { 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "roption1", 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.210 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "roption2": { 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "roption2", 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "xyz", 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.211 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption1": { 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption1", 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.212 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption2": { 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption2", 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.213 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption3": { 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption3", 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.214 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption4": { 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption4", 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.215 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption5": { 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption5", 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.216 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption6": { 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption6", 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.217 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption7": { 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption7", 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "42", 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.218 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "testkey": { 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "testkey", 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.219 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "testlkey": { 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "testlkey", 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.220 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "testnewline": { 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "testnewline", 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.221 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "snap_schedule", 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: "allow_m_granularity": { 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "allow_m_granularity", 2024-05-20T18:10:46.222 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "allow minute scheduled snapshots", 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.223 INFO:teuthology.orchestra.run.smithi129.stdout: "dump_on_update": { 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "dump_on_update", 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "dump database to debug log on update", 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.224 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.225 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.226 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.227 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.228 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.229 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "stats", 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.230 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.231 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.232 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.233 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.234 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "status", 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.235 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.236 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.237 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.238 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.239 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "telegraf", 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.240 INFO:teuthology.orchestra.run.smithi129.stdout: "address": { 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "address", 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:46.241 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "15", 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.242 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.243 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.245 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.246 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.247 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "telemetry", 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_basic": { 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_basic", 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.248 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_crash": { 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_crash", 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.249 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_device": { 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_device", 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.250 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_ident": { 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_ident", 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.251 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_perf": { 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_perf", 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.252 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "contact": { 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "contact", 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.253 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "description": { 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "description", 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.254 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "device_url": { 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "device_url", 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.255 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "enabled": { 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enabled", 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.256 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "24", 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "8", 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "last_opt_revision": { 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "last_opt_revision", 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "1", 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.258 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "leaderboard": { 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "leaderboard", 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.259 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "leaderboard_description": { 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "leaderboard_description", 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.260 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.261 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.262 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.263 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.265 INFO:teuthology.orchestra.run.smithi129.stdout: "organization": { 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "organization", 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "proxy": { 2024-05-20T18:10:46.266 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "proxy", 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "url": { 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "url", 2024-05-20T18:10:46.267 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "test_orchestrator", 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.269 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.270 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.271 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.272 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.273 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "volumes", 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.274 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.275 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.275 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.275 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.275 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.275 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.275 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.278 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.278 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.278 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.278 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.278 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.278 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.279 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.280 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.281 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.282 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "max_concurrent_clones": { 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_concurrent_clones", 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "4", 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.283 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "periodic_async_work": { 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "periodic_async_work", 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Periodically check for async work", 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.284 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "snapshot_clone_delay": { 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "snapshot_clone_delay", 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.285 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "snapshot_clone_no_wait": { 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "snapshot_clone_no_wait", 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.286 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix", 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: "discovery_interval": { 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "discovery_interval", 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "100", 2024-05-20T18:10:46.287 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "identifier": { 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "identifier", 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.288 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.290 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.291 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.292 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.293 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.294 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "zabbix_host": { 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix_host", 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.295 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "zabbix_port": { 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix_port", 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10051", 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.296 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "zabbix_sender": { 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix_sender", 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.297 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: "modules": [ 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: "cephadm", 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: "dashboard", 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: "iostat", 2024-05-20T18:10:46.298 INFO:teuthology.orchestra.run.smithi129.stdout: "nfs", 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: "restful", 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_schedule" 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: "available_modules": [ 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "alerts", 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.299 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How frequently to reexamine health status", 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.300 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.301 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.302 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.303 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.304 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_destination": { 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_destination", 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.305 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Email address to send alerts to", 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_from_name": { 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_from_name", 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "Ceph", 2024-05-20T18:10:46.306 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Email From: name", 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_host": { 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_host", 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.307 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SMTP server", 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_password": { 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_password", 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.308 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Password to authenticate with", 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_port": { 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_port", 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "465", 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.309 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SMTP port", 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_sender": { 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_sender", 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.310 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SMTP envelope sender", 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_ssl": { 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_ssl", 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.311 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "smtp_user": { 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "smtp_user", 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "User to authenticate as", 2024-05-20T18:10:46.312 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "balancer", 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: "active": { 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "active", 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "automatically balance PGs across cluster", 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "begin_time": { 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "begin_time", 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.314 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0000", 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "beginning time of day to automatically balance", 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "begin_weekday": { 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "begin_weekday", 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.315 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "0", 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "6", 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "crush_compat_max_iterations": { 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crush_compat_max_iterations", 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.316 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "25", 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "250", 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "crush_compat_metrics": { 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crush_compat_metrics", 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.317 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "pgs,objects,bytes", 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "crush_compat_step": { 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crush_compat_step", 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.318 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0.5", 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "0.001", 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "0.999", 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "aggressiveness of optimization", 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "end_time": { 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "end_time", 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.319 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "2359", 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "ending time of day to automatically balance", 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "end_weekday": { 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "end_weekday", 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.320 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "0", 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "6", 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.321 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.322 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.323 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.324 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.325 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "min_score": { 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "min_score", 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.326 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": { 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mode", 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.327 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "upmap", 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "crush-compat", 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "none", 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "read", 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap", 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap-read" 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Balancer mode", 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_ids": { 2024-05-20T18:10:46.328 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pool_ids", 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:46.329 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap_max_deviation": { 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "upmap_max_deviation", 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.330 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: "upmap_max_optimizations": { 2024-05-20T18:10:46.331 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "upmap_max_optimizations", 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.332 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephadm", 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "agent_down_multiplier": { 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "agent_down_multiplier", 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "3.0", 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.333 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "agent_refresh_rate": { 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "agent_refresh_rate", 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "20", 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-20T18:10:46.334 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "agent_starting_port": { 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "agent_starting_port", 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "4721", 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.335 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "allow_ptrace": { 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "allow_ptrace", 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-20T18:10:46.336 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "autotune_interval": { 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "autotune_interval", 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.337 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: "autotune_memory_target_ratio": { 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "autotune_memory_target_ratio", 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0.7", 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.338 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "cephadm_log_destination": { 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephadm_log_destination", 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.339 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "file", 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "file,syslog", 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "syslog" 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "cgroups_split": { 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cgroups_split", 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "config_checks_enabled": { 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "config_checks_enabled", 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.341 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "config_dashboard": { 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "config_dashboard", 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.342 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_alertmanager": { 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_alertmanager", 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.343 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_base": { 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_base", 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.344 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Container image name, without the tag", 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_elasticsearch": { 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_elasticsearch", 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.345 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "elasticsearch container image", 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_grafana": { 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_grafana", 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.346 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_haproxy": { 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_haproxy", 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.347 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "HAproxy container image", 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_jaeger_agent": { 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_jaeger_agent", 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-20T18:10:46.348 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Jaeger agent container image", 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_jaeger_collector": { 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_jaeger_collector", 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.349 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Jaeger collector container image", 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_jaeger_query": { 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_jaeger_query", 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-20T18:10:46.350 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Jaeger query container image", 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_keepalived": { 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_keepalived", 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.351 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Keepalived container image", 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_loki": { 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_loki", 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.352 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Loki container image", 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_node_exporter": { 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_node_exporter", 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.353 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_nvmeof": { 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_nvmeof", 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.354 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Nvme-of container image", 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_prometheus": { 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_prometheus", 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.355 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Prometheus container image", 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_promtail": { 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_promtail", 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.356 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Promtail container image", 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_samba": { 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_samba", 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.357 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Samba/SMB container image", 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: "container_image_snmp_gateway": { 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_image_snmp_gateway", 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-20T18:10:46.358 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "SNMP Gateway container image", 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "container_init": { 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "container_init", 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.359 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Run podman/docker with `--init`", 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "daemon_cache_timeout": { 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "daemon_cache_timeout", 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.360 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "default_cephadm_command_timeout": { 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "default_cephadm_command_timeout", 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.361 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "900", 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "default_registry": { 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "default_registry", 2024-05-20T18:10:46.362 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "docker.io", 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "device_cache_timeout": { 2024-05-20T18:10:46.363 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "device_cache_timeout", 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "1800", 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "seconds to cache device inventory", 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.364 INFO:teuthology.orchestra.run.smithi129.stdout: "device_enhanced_scan": { 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "device_enhanced_scan", 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.365 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "facts_cache_timeout": { 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "facts_cache_timeout", 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "seconds to cache host facts data", 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.366 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "grafana_dashboards_path": { 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "grafana_dashboards_path", 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.367 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "host_check_interval": { 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "host_check_interval", 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to perform a host check", 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "hw_monitoring": { 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "hw_monitoring", 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.369 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "inventory_list_all": { 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "inventory_list_all", 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.370 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.371 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "log_refresh_metadata": { 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_refresh_metadata", 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.372 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.374 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.375 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-20T18:10:46.376 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.377 INFO:teuthology.orchestra.run.smithi129.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "*", 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.378 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "max_count_per_host": { 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_count_per_host", 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "max number of daemons per service per host", 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.379 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "max_osd_draining_count": { 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_osd_draining_count", 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.380 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "migration_current": { 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "migration_current", 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "internal - do not modify", 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.381 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": { 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mode", 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "root", 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "cephadm-package", 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "root" 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "mode for remote execution of cephadm", 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "oob_default_addr": { 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "oob_default_addr", 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "169.254.1.1", 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.383 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "prometheus_alerts_path": { 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "prometheus_alerts_path", 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.384 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_insecure": { 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_insecure", 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-20T18:10:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_password": { 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_password", 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_url": { 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_url", 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.387 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "registry_username": { 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "registry_username", 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.388 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "secure_monitoring_stack": { 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "secure_monitoring_stack", 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.389 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "service_discovery_port": { 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "service_discovery_port", 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8765", 2024-05-20T18:10:46.390 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "cephadm service discovery port", 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "ssh_config_file": { 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssh_config_file", 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.391 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "use_agent": { 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "use_agent", 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.392 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "use_repo_digest": { 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "use_repo_digest", 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.393 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_on_failed_host_check": { 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_on_failed_host_check", 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.394 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise a health warning if the host check fails", 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_on_stray_daemons": { 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_on_stray_daemons", 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.395 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_on_stray_hosts": { 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_on_stray_hosts", 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.396 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.397 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crash", 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.398 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.399 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.400 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.401 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.402 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "retain_interval": { 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "retain_interval", 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "31536000", 2024-05-20T18:10:46.403 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_recent_interval": { 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_recent_interval", 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "1209600", 2024-05-20T18:10:46.404 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "dashboard", 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.405 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.406 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "ALERTMANAGER_API_HOST": { 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.407 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.408 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "AUDIT_API_ENABLED": { 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "AUDIT_API_ENABLED", 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.410 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "ENABLE_BROWSABLE_API": { 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.411 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.412 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.413 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.414 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.415 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.416 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-20T18:10:46.417 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.418 INFO:teuthology.orchestra.run.smithi129.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.419 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.420 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_PASSWORD": { 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "admin", 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.421 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.422 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_URL": { 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_URL", 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.423 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_API_USERNAME": { 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "admin", 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.424 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.425 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.426 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.427 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-20T18:10:46.428 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "PROMETHEUS_API_HOST": { 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-20T18:10:46.429 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-20T18:10:46.430 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-20T18:10:46.431 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.432 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-20T18:10:46.433 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.434 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.435 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.436 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.437 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.438 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_ENABLED": { 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.439 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.440 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.441 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8", 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.442 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "45", 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.443 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_ACCESS_KEY": { 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.444 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "admin", 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.445 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_SECRET_KEY": { 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.446 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "RGW_API_SSL_VERIFY": { 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.447 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.448 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.449 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.450 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "cross_origin_url": { 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cross_origin_url", 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.451 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "crt_file": { 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "crt_file", 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.452 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "debug": { 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "debug", 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Enable/disable debug options", 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.453 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "jwt_token_ttl": { 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "jwt_token_ttl", 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "28800", 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.454 INFO:teuthology.orchestra.run.smithi129.stdout: "key_file": { 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "key_file", 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.455 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.456 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.457 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.458 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "motd": { 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "motd", 2024-05-20T18:10:46.460 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "The message of the day", 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "redirect_resolve_ip_addr": { 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "redirect_resolve_ip_addr", 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.461 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "server_addr": { 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_addr", 2024-05-20T18:10:46.462 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "::", 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "server_port": { 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_port", 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.463 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8080", 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "ssl": { 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssl", 2024-05-20T18:10:46.464 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "ssl_server_port": { 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssl_server_port", 2024-05-20T18:10:46.465 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8443", 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_behaviour": { 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_behaviour", 2024-05-20T18:10:46.466 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "redirect", 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "redirect" 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.467 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_error_status_code": { 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_error_status_code", 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "500", 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "400", 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "599", 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "url_prefix": { 2024-05-20T18:10:46.468 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "url_prefix", 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "devicehealth", 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "enable_monitoring": { 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enable_monitoring", 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.470 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "monitor device health metrics", 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.473 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.474 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.475 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "mark_out_threshold": { 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mark_out_threshold", 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "2419200", 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.476 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_name": { 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pool_name", 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "device_health_metrics", 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.477 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "retention_period": { 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "retention_period", 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "15552000", 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.478 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how long to retain device health metrics", 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "scrape_frequency": { 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "scrape_frequency", 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "86400", 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.479 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "self_heal": { 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "self_heal", 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.480 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.481 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how frequently to wake up and check device health", 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "warn_threshold": { 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "warn_threshold", 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "7257600", 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.482 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "diskprediction_local", 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.484 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.485 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.486 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.487 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "predict_interval": { 2024-05-20T18:10:46.488 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "predict_interval", 2024-05-20T18:10:46.489 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.489 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.489 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.489 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "86400", 2024-05-20T18:10:46.489 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.489 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.489 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.489 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.489 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.489 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: "predictor_model": { 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "predictor_model", 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "prophetstor", 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.490 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "600", 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.491 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "influx", 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": false, 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "influxdb python module not found", 2024-05-20T18:10:46.492 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "batch_size": { 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "batch_size", 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5000", 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.493 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: "database": { 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "database", 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "ceph", 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-20T18:10:46.494 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "hostname": { 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "hostname", 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.495 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "InfluxDB server hostname", 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "30", 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "5", 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.496 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.497 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.499 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.500 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.501 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "password": { 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "password", 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.502 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "password of InfluxDB server user", 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "port": { 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "port", 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "8086", 2024-05-20T18:10:46.503 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "InfluxDB server port", 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "ssl": { 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ssl", 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "false", 2024-05-20T18:10:46.504 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "threads": { 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "threads", 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:46.505 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "32", 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "username": { 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "username", 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.506 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "username of InfluxDB server user", 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "verify_ssl": { 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "verify_ssl", 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "true", 2024-05-20T18:10:46.507 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "insights", 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.508 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.509 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.510 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.511 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.512 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.513 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "iostat", 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.514 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.515 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.516 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.517 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.518 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "k8sevents", 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_event_retention_days": { 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "ceph_event_retention_days", 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.519 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "7", 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "config_check_secs": { 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "config_check_secs", 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.520 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "10", 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.521 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.522 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.524 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.525 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "localpool", 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.526 INFO:teuthology.orchestra.run.smithi129.stdout: "failure_domain": { 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "failure_domain", 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "host", 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "failure domain for any created local pool", 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.528 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.529 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.530 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.531 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "min_size": { 2024-05-20T18:10:46.532 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "min_size", 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "default min_size for any created local pool", 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rep": { 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "num_rep", 2024-05-20T18:10:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "3", 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "default replica count for any created local pool", 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "pg_num": { 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pg_num", 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.534 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "128", 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "default pg_num for any created local pool", 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "prefix": { 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "prefix", 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.535 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "name prefix for any created local pool", 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "subtree": { 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "subtree", 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.536 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "rack", 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.537 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mds_autoscaler", 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.538 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.539 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.541 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.542 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mirroring", 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.543 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.544 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.546 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.547 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "nfs", 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.548 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.549 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.550 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.551 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.552 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "orchestrator", 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "fail_fs": { 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "fail_fs", 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.554 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.555 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.556 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.557 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.558 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.559 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "orchestrator": { 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "orchestrator", 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "cephadm", 2024-05-20T18:10:46.560 INFO:teuthology.orchestra.run.smithi129.stdout: "rook", 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: "test_orchestrator" 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Orchestrator backend", 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "osd_perf_query", 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.561 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.562 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.563 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.564 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.565 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.566 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "osd_support", 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.567 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.568 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.569 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.570 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.571 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "pg_autoscaler", 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.572 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.573 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.574 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.575 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.576 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.577 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "threshold": { 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "threshold", 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:46.578 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "3.0", 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1.0", 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "scaling threshold", 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "progress", 2024-05-20T18:10:46.579 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "allow_pg_recovery_event": { 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "allow_pg_recovery_event", 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.580 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "enabled": { 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enabled", 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.581 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.582 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.583 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.584 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.585 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.586 INFO:teuthology.orchestra.run.smithi129.stdout: "max_completed_events": { 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_completed_events", 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "50", 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "number of past completed events to remember", 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "sleep_interval": { 2024-05-20T18:10:46.587 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "sleep_interval", 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "5", 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "how long the module is going to sleep", 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.588 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "prometheus", 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "cache": { 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cache", 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.589 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "exclude_perf_counters": { 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "exclude_perf_counters", 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-20T18:10:46.590 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.591 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.592 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.593 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.594 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "rbd_stats_pools": { 2024-05-20T18:10:46.595 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rbd_stats_pools", 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-20T18:10:46.596 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "300", 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "scrape_interval": { 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "scrape_interval", 2024-05-20T18:10:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "15.0", 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "server_addr": { 2024-05-20T18:10:46.598 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_addr", 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "::", 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "server_port": { 2024-05-20T18:10:46.599 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_port", 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "9283", 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "stale_cache_strategy": { 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "stale_cache_strategy", 2024-05-20T18:10:46.600 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "log", 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_behaviour": { 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_behaviour", 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.601 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "default", 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "default", 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "error" 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "standby_error_status_code": { 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "standby_error_status_code", 2024-05-20T18:10:46.602 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "500", 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "400", 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "599", 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.603 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rbd_support", 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.604 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.605 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.606 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.607 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.608 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "max_concurrent_snap_create": { 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_concurrent_snap_create", 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10", 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.609 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "mirror_snapshot_schedule": { 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "mirror_snapshot_schedule", 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "trash_purge_schedule": { 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "trash_purge_schedule", 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.611 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "restful", 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "enable_auth": { 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enable_auth", 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.612 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "key_file": { 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "key_file", 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.613 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.614 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.615 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.616 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.617 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.618 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "server_addr": { 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_addr", 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.619 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "server_port": { 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "server_port", 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.620 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rgw", 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.621 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.622 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.623 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.624 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.625 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.626 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rook", 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.627 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.628 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.629 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.630 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "storage_class": { 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "storage_class", 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "local", 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.632 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "selftest", 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.633 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.634 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.635 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "roption1": { 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "roption1", 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.638 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "roption2": { 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "roption2", 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "xyz", 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.639 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption1": { 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption1", 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.640 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption2": { 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption2", 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.641 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption3": { 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption3", 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "float", 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.642 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption4": { 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption4", 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.643 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption5": { 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption5", 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.644 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption6": { 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption6", 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.645 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "rwoption7": { 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rwoption7", 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "1", 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "42", 2024-05-20T18:10:46.646 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "testkey": { 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "testkey", 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.647 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "testlkey": { 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "testlkey", 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.648 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "testnewline": { 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "testnewline", 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.649 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "snap_schedule", 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: "allow_m_granularity": { 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "allow_m_granularity", 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.650 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "allow minute scheduled snapshots", 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "dump_on_update": { 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "dump_on_update", 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.651 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "dump database to debug log on update", 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.652 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.653 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.654 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.655 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.656 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "stats", 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.657 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.658 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.659 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.660 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.661 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.662 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "status", 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.663 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.664 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.665 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.666 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.667 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "telegraf", 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "address": { 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "address", 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.668 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "15", 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.669 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.670 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.671 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.672 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.673 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.674 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "telemetry", 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_basic": { 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_basic", 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.675 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_crash": { 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_crash", 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.676 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_device": { 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_device", 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.677 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_ident": { 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_ident", 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.678 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "channel_perf": { 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "channel_perf", 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.679 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "contact": { 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "contact", 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.680 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "description": { 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "description", 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.681 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "device_url": { 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "device_url", 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.682 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "enabled": { 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "enabled", 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.683 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.684 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "24", 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "8", 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "last_opt_revision": { 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "last_opt_revision", 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "1", 2024-05-20T18:10:46.685 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "leaderboard": { 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "leaderboard", 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.686 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "leaderboard_description": { 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "leaderboard_description", 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.687 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.688 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.689 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.690 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.691 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.692 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "organization": { 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "organization", 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.693 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "proxy": { 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "proxy", 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.694 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "url": { 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "url", 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.695 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "test_orchestrator", 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.696 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.698 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.699 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.700 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.701 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "volumes", 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.702 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.703 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.705 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.706 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: "max_concurrent_clones": { 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "max_concurrent_clones", 2024-05-20T18:10:46.707 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "4", 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "periodic_async_work": { 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "periodic_async_work", 2024-05-20T18:10:46.708 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Periodically check for async work", 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "snapshot_clone_delay": { 2024-05-20T18:10:46.709 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "snapshot_clone_delay", 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "0", 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.710 INFO:teuthology.orchestra.run.smithi129.stdout: "snapshot_clone_no_wait": { 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "snapshot_clone_no_wait", 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "True", 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.711 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix", 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "can_run": true, 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "error_string": "", 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "module_options": { 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "discovery_interval": { 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "discovery_interval", 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "uint", 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "100", 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.712 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "identifier": { 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "identifier", 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.713 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: "interval": { 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "interval", 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "secs", 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "60", 2024-05-20T18:10:46.714 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "log_level": { 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_level", 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.715 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.716 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster": { 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster", 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.717 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_cluster_level": { 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_cluster_level", 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "info", 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [ 2024-05-20T18:10:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: "", 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: "critical", 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: "debug", 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: "error", 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: "info", 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: "warning" 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: "log_to_file": { 2024-05-20T18:10:46.719 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "log_to_file", 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "bool", 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 1, 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "False", 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.720 INFO:teuthology.orchestra.run.smithi129.stdout: "zabbix_host": { 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix_host", 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "", 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.721 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "zabbix_port": { 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix_port", 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "int", 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "10051", 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.722 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "zabbix_sender": { 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "zabbix_sender", 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "str", 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "level": "advanced", 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "flags": 0, 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "min": "", 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "max": "", 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "enum_allowed": [], 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "desc": "", 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "long_desc": "", 2024-05-20T18:10:46.723 INFO:teuthology.orchestra.run.smithi129.stdout: "tags": [], 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: "see_also": [] 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: "services": { 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: "dashboard": "https://172.21.15.129:8443/" 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: "always_on_modules": { 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: "octopus": [ 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: "balancer", 2024-05-20T18:10:46.724 INFO:teuthology.orchestra.run.smithi129.stdout: "crash", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "devicehealth", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "orchestrator", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "pg_autoscaler", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "progress", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "rbd_support", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "status", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "telemetry", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "volumes" 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "pacific": [ 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "balancer", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "crash", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "devicehealth", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "orchestrator", 2024-05-20T18:10:46.725 INFO:teuthology.orchestra.run.smithi129.stdout: "pg_autoscaler", 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "progress", 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "rbd_support", 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "status", 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "telemetry", 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "volumes" 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "quincy": [ 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "balancer", 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "crash", 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "devicehealth", 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "orchestrator", 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "pg_autoscaler", 2024-05-20T18:10:46.726 INFO:teuthology.orchestra.run.smithi129.stdout: "progress", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "rbd_support", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "status", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "telemetry", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "volumes" 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "reef": [ 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "balancer", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "crash", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "devicehealth", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "orchestrator", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "pg_autoscaler", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "progress", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "rbd_support", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "status", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "telemetry", 2024-05-20T18:10:46.727 INFO:teuthology.orchestra.run.smithi129.stdout: "volumes" 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: "squid": [ 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: "balancer", 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: "crash", 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: "devicehealth", 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: "orchestrator", 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: "pg_autoscaler", 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: "progress", 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: "rbd_support", 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: "status", 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: "telemetry", 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: "volumes" 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T18:10:46.728 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: "last_failure_osd_epoch": 89, 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: "active_clients": [ 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "devicehealth", 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: "addrvec": [ 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v2", 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "172.21.15.129:0", 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 950781653 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.729 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "libcephsqlite", 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: "addrvec": [ 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v2", 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "172.21.15.129:0", 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 498384768 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "rbd_support", 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: "addrvec": [ 2024-05-20T18:10:46.730 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v2", 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "172.21.15.129:0", 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 159609743 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "snap_schedule", 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: "addrvec": [ 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v2", 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "172.21.15.129:0", 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 1360196218 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T18:10:46.731 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "snap_schedule", 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: "addrvec": [ 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v2", 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "172.21.15.129:0", 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 4165797502 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "volumes", 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: "addrvec": [ 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-05-20T18:10:46.732 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "v2", 2024-05-20T18:10:46.733 INFO:teuthology.orchestra.run.smithi129.stdout: "addr": "172.21.15.129:0", 2024-05-20T18:10:46.733 INFO:teuthology.orchestra.run.smithi129.stdout: "nonce": 3451245670 2024-05-20T18:10:46.733 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.733 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T18:10:46.733 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:46.733 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-05-20T18:10:46.733 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:10:46.734 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:45 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1018275642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T18:10:46.734 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:45 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1455197517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T18:10:46.735 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:45 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2840876375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T18:10:46.735 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:46 smithi204 ceph-mon[37076]: pgmap v383: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.3 KiB/s wr, 7 op/s 2024-05-20T18:10:46.735 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:46 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2784549245' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-20T18:10:46.735 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:45 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1018275642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T18:10:46.735 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:45 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1455197517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T18:10:46.735 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:45 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2840876375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T18:10:46.735 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:46 smithi154 ceph-mon[36482]: pgmap v383: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.3 KiB/s wr, 7 op/s 2024-05-20T18:10:46.735 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:46 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2784549245' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-20T18:10:46.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/2784549245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4ac0a2d00 msgr2=0x7fd4ac0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:46.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4ac0a2d00 0x7fd4ac0a5210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd4c019adf0 tx=0x7fd4a8005e70 comp rx=0 tx=0).stop 2024-05-20T18:10:46.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/2784549245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4c0101ab0 msgr2=0x7fd4c019f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:10:46.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4c0101ab0 0x7fd4c019f4d0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0005940 tx=0x7fd4b00042f0 comp rx=0 tx=0).stop 2024-05-20T18:10:46.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/2784549245 shutdown_connections 2024-05-20T18:10:46.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fd4ac0a2d00 0x7fd4ac0a5210 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:46.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4c01079e0 0x7fd4c019fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:46.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c010c340 0x7fd4c0199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:46.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 --2- 172.21.15.129:0/2784549245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4c0101ab0 0x7fd4c019f4d0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:10:46.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/2784549245 >> 172.21.15.129:0/2784549245 conn(0x7fd4c00fd030 msgr2=0x7fd4c010d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:10:46.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/2784549245 shutdown_connections 2024-05-20T18:10:46.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:10:45.435+0000 7fd4c5b8f640 1 -- 172.21.15.129:0/2784549245 wait complete. 2024-05-20T18:10:46.742 DEBUG:tasks.check_counter:Getting stats from a 2024-05-20T18:10:46.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-20T18:10:46.743 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:45 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1018275642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T18:10:46.743 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:45 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1455197517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T18:10:46.743 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:45 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2840876375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T18:10:46.743 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:46 smithi129 ceph-mon[32029]: pgmap v383: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.3 KiB/s wr, 7 op/s 2024-05-20T18:10:46.743 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:46 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2784549245' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-20T18:10:47.012 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T18:10:48.534 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:48 smithi129 ceph-mon[32029]: pgmap v384: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-20T18:10:48.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:48 smithi154 ceph-mon[36482]: pgmap v384: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-20T18:10:48.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:48 smithi204 ceph-mon[37076]: pgmap v384: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-20T18:10:48.966 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:10:48.966 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:10:48.966 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 242, 2024-05-20T18:10:48.966 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 230, 2024-05-20T18:10:48.966 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1450448, 2024-05-20T18:10:48.966 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 140672, 2024-05-20T18:10:48.966 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-05-20T18:10:48.967 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 0, 2024-05-20T18:10:48.967 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.060847335, 2024-05-20T18:10:48.967 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.018977298, 2024-05-20T18:10:48.967 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.040260221, 2024-05-20T18:10:48.967 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.007850955, 2024-05-20T18:10:48.967 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:48.967 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 230, 2024-05-20T18:10:48.967 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.048540580, 2024-05-20T18:10:48.967 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000211046 2024-05-20T18:10:48.967 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:48.967 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1450448, 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 140672 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 2, 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 174, 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 320, 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 186096, 2024-05-20T18:10:48.968 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 5, 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.025572341, 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.020052060, 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.002181993, 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 171, 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.026318265, 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000153907 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:48.969 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 186096 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 0, 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 2, 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 0, 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 0, 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 2, 2024-05-20T18:10:48.970 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 1, 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.000289883, 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.000000000, 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.000212550, 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:48.971 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 0 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 0, 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:10:48.972 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:10:48.973 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:10:48.974 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 149017, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 31, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-05-20T18:10:48.975 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 928, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 2, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-05-20T18:10:48.976 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 240, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 1413888, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 240, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-05-20T18:10:48.977 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 240, 2024-05-20T18:10:48.978 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1413888, 2024-05-20T18:10:48.978 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-05-20T18:10:48.978 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:48.978 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:48.978 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:48.978 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:48.978 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:48.978 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:10:49.621 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-05-20T18:10:49.621 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-05-20T18:10:49.621 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-05-20T18:10:49.621 DEBUG:tasks.check_counter:Getting stats from d 2024-05-20T18:10:49.621 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.d perf dump 2024-05-20T18:10:49.876 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T18:10:50.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:50 smithi154 ceph-mon[36482]: pgmap v385: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-20T18:10:50.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:50 smithi129 ceph-mon[32029]: pgmap v385: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-20T18:10:50.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:50 smithi204 ceph-mon[37076]: pgmap v385: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 938 B/s wr, 2 op/s 2024-05-20T18:10:51.630 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:10:51.630 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 244, 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 234, 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1449088, 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 144064, 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 0, 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.062986178, 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.018586021, 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.044015984, 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.007483841, 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 233, 2024-05-20T18:10:51.631 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.044757399, 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000192091 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1449088, 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 144064 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 2, 2024-05-20T18:10:51.632 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 179, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 320, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 186976, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 5, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.018784750, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.012210803, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.002294355, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 176, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.021300562, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000121025 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:51.633 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 186976 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 4, 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 4, 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 2992, 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 448, 2024-05-20T18:10:51.634 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 2, 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 1, 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.000767184, 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.000030537, 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.000700258, 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 3, 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000864294, 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000288098 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:51.635 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 448 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 0, 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-05-20T18:10:51.636 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-05-20T18:10:51.637 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:10:51.638 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 150008, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 31, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-05-20T18:10:51.639 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 928, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 2, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-05-20T18:10:51.640 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 246, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 1414668, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 246, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 246, 2024-05-20T18:10:51.641 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1414668, 2024-05-20T18:10:51.642 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-05-20T18:10:51.642 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:51.642 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:51.642 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:51.642 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:51.642 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:51.642 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:10:52.305 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-05-20T18:10:52.306 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-05-20T18:10:52.306 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-05-20T18:10:52.306 DEBUG:tasks.check_counter:Getting stats from g 2024-05-20T18:10:52.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.g perf dump 2024-05-20T18:10:52.568 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T18:10:52.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:52 smithi154 ceph-mon[36482]: pgmap v386: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 4 op/s 2024-05-20T18:10:52.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:52 smithi129 ceph-mon[32029]: pgmap v386: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 4 op/s 2024-05-20T18:10:52.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:52 smithi204 ceph-mon[37076]: pgmap v386: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 4 op/s 2024-05-20T18:10:54.400 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:10:54.400 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:10:54.400 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 0, 2024-05-20T18:10:54.400 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 2, 2024-05-20T18:10:54.400 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 0, 2024-05-20T18:10:54.400 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 0, 2024-05-20T18:10:54.400 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-05-20T18:10:54.400 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 0, 2024-05-20T18:10:54.400 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.001341762, 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.000000000, 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.000413717, 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:54.401 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:54.402 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-20T18:10:54.402 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 0 2024-05-20T18:10:54.402 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:54.402 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:10:54.402 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 6, 2024-05-20T18:10:54.402 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 178, 2024-05-20T18:10:54.402 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 3312, 2024-05-20T18:10:54.402 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 186896, 2024-05-20T18:10:54.402 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 5, 2024-05-20T18:10:54.402 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-05-20T18:10:54.402 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.019314887, 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.013750270, 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.002792638, 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 176, 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.013481527, 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000076599 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:54.403 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 186896 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 240, 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 230, 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1448000, 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 141504, 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 2, 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 1, 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.062802082, 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.018114830, 2024-05-20T18:10:54.404 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.048428231, 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.008048265, 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 229, 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.036061386, 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000157473 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1448000, 2024-05-20T18:10:54.405 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 141504 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 0, 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:10:54.406 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:10:54.407 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-05-20T18:10:54.408 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 148994, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 31, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-05-20T18:10:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 0, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 0, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 928, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 2, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:54.410 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 242, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 1414148, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 242, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 242, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1414148, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:54.412 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:54.412 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:54.412 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:54.412 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:10:54.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:54 smithi154 ceph-mon[36482]: pgmap v387: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 767 B/s wr, 2 op/s 2024-05-20T18:10:54.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:54 smithi129 ceph-mon[32029]: pgmap v387: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 767 B/s wr, 2 op/s 2024-05-20T18:10:54.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:54 smithi204 ceph-mon[37076]: pgmap v387: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 767 B/s wr, 2 op/s 2024-05-20T18:10:55.093 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-05-20T18:10:55.093 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-05-20T18:10:55.093 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-05-20T18:10:55.093 DEBUG:tasks.check_counter:Getting stats from j 2024-05-20T18:10:55.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.j perf dump 2024-05-20T18:10:55.337 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.a/config 2024-05-20T18:10:56.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:56 smithi154 ceph-mon[36482]: pgmap v388: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-05-20T18:10:56.658 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:56 smithi129 ceph-mon[32029]: pgmap v388: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-05-20T18:10:56.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:56 smithi204 ceph-mon[37076]: pgmap v388: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-05-20T18:10:57.261 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-05-20T18:10:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:10:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 0, 2024-05-20T18:10:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 2, 2024-05-20T18:10:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 0, 2024-05-20T18:10:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 0, 2024-05-20T18:10:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-05-20T18:10:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 0, 2024-05-20T18:10:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.001174483, 2024-05-20T18:10:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.000000000, 2024-05-20T18:10:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.000293344, 2024-05-20T18:10:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:10:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 0 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 2, 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 181, 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 320, 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 187328, 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 5, 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.017781596, 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.011724627, 2024-05-20T18:10:57.263 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.002719420, 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 178, 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.024268506, 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000136339 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:57.264 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 187328 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 250, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 238, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1452624, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 145792, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 2, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 1, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.063062827, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.018601929, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.046728795, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.008776429, 2024-05-20T18:10:57.265 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 238, 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.026273761, 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000110393 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1452624, 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 145792 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-05-20T18:10:57.266 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 0, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:10:57.267 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:10:57.268 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:10:57.269 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 148977, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 30, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:10:57.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 928, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 2, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-05-20T18:10:57.271 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 248, 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 1414928, 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 248, 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 248, 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1414928, 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-05-20T18:10:57.272 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-05-20T18:10:57.965 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-05-20T18:10:57.965 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-05-20T18:10:57.965 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-05-20T18:10:57.965 DEBUG:tasks.check_counter:Getting stats from b 2024-05-20T18:10:57.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.b perf dump 2024-05-20T18:10:58.336 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T18:10:58.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:10:58 smithi154 ceph-mon[36482]: pgmap v389: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-05-20T18:10:58.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:10:58 smithi129 ceph-mon[32029]: pgmap v389: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-05-20T18:10:58.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:10:58 smithi204 ceph-mon[37076]: pgmap v389: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-05-20T18:11:00.151 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-05-20T18:11:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:11:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 5940, 2024-05-20T18:11:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 5256, 2024-05-20T18:11:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 1782955, 2024-05-20T18:11:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 335693797, 2024-05-20T18:11:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 16, 2024-05-20T18:11:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 5, 2024-05-20T18:11:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 2.066048023, 2024-05-20T18:11:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 1.314411434, 2024-05-20T18:11:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.165192440, 2024-05-20T18:11:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.507823786, 2024-05-20T18:11:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 5255, 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.445361164, 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000275044 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1040, 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000256403, 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000246 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 382816, 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 114432 2024-05-20T18:11:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:11:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 4272, 2024-05-20T18:11:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 4129, 2024-05-20T18:11:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 1140958, 2024-05-20T18:11:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 331837783, 2024-05-20T18:11:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 264, 2024-05-20T18:11:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 7, 2024-05-20T18:11:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 2.098574914, 2024-05-20T18:11:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 1.327761592, 2024-05-20T18:11:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.194763849, 2024-05-20T18:11:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.540732471, 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 4126, 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 2.600603829, 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000630296 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 464, 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000078907, 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000170 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 86448, 2024-05-20T18:11:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 134320 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 276581, 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 279168, 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 111800449, 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 257712074, 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 10, 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 6, 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 18.526031290, 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 8.230195114, 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 27.833232012, 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.749518020, 2024-05-20T18:11:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 279167, 2024-05-20T18:11:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 61.551876046, 2024-05-20T18:11:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000220484 2024-05-20T18:11:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 420103, 2024-05-20T18:11:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.136102350, 2024-05-20T18:11:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000323 2024-05-20T18:11:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 111332704, 2024-05-20T18:11:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 111333728 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 3659, 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 241.959708549, 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.066127277 2024-05-20T18:11:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2474, 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.084162095, 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000034018 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 250, 2024-05-20T18:11:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 250, 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 250, 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 14.045943998, 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.056183775 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 0, 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 0, 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 0, 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 149, 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 0, 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 0, 2024-05-20T18:11:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 128, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 45, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 83, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 0, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 63, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 0, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 3, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 4, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 2, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 330, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 322, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 0, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 0, 2024-05-20T18:11:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 0, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 0, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 0, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 0, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 0, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 629, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 4, 2024-05-20T18:11:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 234, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 7, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 629, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 221, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 21, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 200, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 408, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 0, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 48, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 3216, 2024-05-20T18:11:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 273510, 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 273442, 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 1, 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 49 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 0, 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 0, 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 0, 2024-05-20T18:11:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 0, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 0, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 4, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 22, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 18, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-05-20T18:11:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_path": 0, 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_quiesce_inode": 0, 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 21, 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 0, 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 0, 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-05-20T18:11:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "evlrg": 0, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 274519, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 274458, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 274458, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 61, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 10, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 405, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 403, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 403, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 2, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "segmjr": 1, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-05-20T18:11:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 1, 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 947592589, 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 947924482, 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 0, 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 5712, 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1291.594650487, 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.226119511 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 0 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 100, 2024-05-20T18:11:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 139, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 39, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 22, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 22, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 0, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 128, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 146, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 18, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 4, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 41, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 37, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 152284, 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 231828 2024-05-20T18:11:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 288, 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 0, 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 250, 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 34, 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 0, 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 22, 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.026272268, 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001194194 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-05-20T18:11:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 129, 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.078388176, 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000607660 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 20, 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.005863041, 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000293152 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2, 2024-05-20T18:11:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000325607, 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000162803 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-05-20T18:11:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-05-20T18:11:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 12, 2024-05-20T18:11:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.010446647, 2024-05-20T18:11:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000870553 2024-05-20T18:11:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-05-20T18:11:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 10, 2024-05-20T18:11:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.013357027, 2024-05-20T18:11:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001335702 2024-05-20T18:11:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-05-20T18:11:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-05-20T18:11:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 12, 2024-05-20T18:11:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 11.278603561, 2024-05-20T18:11:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.939883630 2024-05-20T18:11:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-05-20T18:11:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1, 2024-05-20T18:11:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000244518, 2024-05-20T18:11:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000244518 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 5, 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.015988478, 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.003197695 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2, 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.003061628, 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001530814 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-05-20T18:11:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-05-20T18:11:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-05-20T18:11:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 4, 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.597018688, 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.149254672 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 5, 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.968237011, 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.393647402 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1, 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000839174, 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000839174 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-05-20T18:11:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 7, 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.021727975, 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.003103996 2024-05-20T18:11:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 18, 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.025570199, 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001420566 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-05-20T18:11:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 2, 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 5, 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 3, 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 2, 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 65, 2024-05-20T18:11:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 32, 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 724, 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:11:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:11:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:11:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 351053, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 156, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 528, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 6, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 8240, 2024-05-20T18:11:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 173, 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 382218, 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 2812, 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 10640, 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 944467475, 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 37, 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 10640, 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 10640, 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 542.533705896, 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.050990009 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-05-20T18:11:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 10640, 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 11025 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 10640, 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 672, 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 9968, 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 127, 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 44, 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 22, 2024-05-20T18:11:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 5713, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 178, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 0, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 3539, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 650, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 88, 2024-05-20T18:11:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-05-20T18:11:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 107, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-05-20T18:11:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 22, 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 12, 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 12, 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 0, 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 647, 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 0, 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 17 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 1, 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-05-20T18:11:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 0 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 516, 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 4, 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed_ops": 2578, 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 18, 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-05-20T18:11:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 286789, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 75124444, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 286789, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 286789, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 75124444, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 10640, 2024-05-20T18:11:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 944470157, 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 10640, 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 944470157, 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-05-20T18:11:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 10640, 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 10640, 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 10640, 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 10640, 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 18, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 2682, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 18, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 1, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 2682, 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-05-20T18:11:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x5606a2dc2b60": { 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 274519, 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 943730178, 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 274519, 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-05-20T18:11:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-05-20T18:11:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 5712, 2024-05-20T18:11:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 943730178, 2024-05-20T18:11:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-05-20T18:11:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:00.202 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-05-20T18:11:00.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:00 smithi154 ceph-mon[36482]: pgmap v390: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-05-20T18:11:00.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:00 smithi129 ceph-mon[32029]: pgmap v390: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-05-20T18:11:00.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:00 smithi204 ceph-mon[37076]: pgmap v390: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-05-20T18:11:00.906 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=7 2024-05-20T18:11:00.906 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=12 2024-05-20T18:11:00.906 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-05-20T18:11:00.906 DEBUG:tasks.check_counter:Getting stats from e 2024-05-20T18:11:00.906 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.e perf dump 2024-05-20T18:11:01.153 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T18:11:02.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:02 smithi154 ceph-mon[36482]: pgmap v391: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-05-20T18:11:02.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:02 smithi129 ceph-mon[32029]: pgmap v391: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-05-20T18:11:02.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:02 smithi204 ceph-mon[37076]: pgmap v391: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-05-20T18:11:02.869 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-05-20T18:11:02.869 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:11:02.869 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 0, 2024-05-20T18:11:02.869 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 2, 2024-05-20T18:11:02.869 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 0, 2024-05-20T18:11:02.869 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 0, 2024-05-20T18:11:02.869 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 4, 2024-05-20T18:11:02.869 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 0, 2024-05-20T18:11:02.869 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.001149531, 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.000000000, 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.000287483, 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:02.870 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 0 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 250, 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 238, 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 1452624, 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 145792, 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 2, 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 1, 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.051503371, 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.015197303, 2024-05-20T18:11:02.871 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.038954365, 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.006895062, 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 238, 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.014315410, 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000060148 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1452624, 2024-05-20T18:11:02.872 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 145792 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 2, 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 181, 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 320, 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 187328, 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 5, 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 3, 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.011482059, 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.007605312, 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.001677400, 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:02.873 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 178, 2024-05-20T18:11:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.021740708, 2024-05-20T18:11:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000122138 2024-05-20T18:11:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-20T18:11:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 187328 2024-05-20T18:11:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-05-20T18:11:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 0, 2024-05-20T18:11:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-05-20T18:11:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-05-20T18:11:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:11:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-05-20T18:11:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:11:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:11:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-05-20T18:11:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-05-20T18:11:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 148977, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 30, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 0, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 0, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-05-20T18:11:02.878 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 0, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 0, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 928, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 2, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-05-20T18:11:02.879 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 248, 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 1414928, 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 248, 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-05-20T18:11:02.880 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 248, 2024-05-20T18:11:02.881 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 1414928, 2024-05-20T18:11:02.881 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-05-20T18:11:02.881 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:02.881 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:02.881 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:02.881 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:02.881 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:02.881 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-05-20T18:11:03.492 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-05-20T18:11:03.492 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-05-20T18:11:03.492 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-05-20T18:11:03.492 DEBUG:tasks.check_counter:Getting stats from h 2024-05-20T18:11:03.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.h perf dump 2024-05-20T18:11:03.732 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T18:11:04.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:04 smithi129 ceph-mon[32029]: pgmap v392: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-05-20T18:11:04.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:04 smithi204 ceph-mon[37076]: pgmap v392: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-05-20T18:11:04.872 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:04 smithi154 ceph-mon[36482]: pgmap v392: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-05-20T18:11:05.486 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 86, 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 83, 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 166496, 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 49280, 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 4, 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 0, 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.015254485, 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.005356209, 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.005934180, 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.001993640, 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:05.487 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 83, 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.005967242, 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000071894 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 166496, 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 49280 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:11:05.488 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 1, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 61, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 160, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 88384, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 3, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 2, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.004580378, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.002789157, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.000996324, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 59, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.004811747, 2024-05-20T18:11:05.489 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000081555 2024-05-20T18:11:05.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:05.490 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:05.490 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:05.490 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:05.490 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:05.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:05.490 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 160, 2024-05-20T18:11:05.490 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 88384 2024-05-20T18:11:05.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:05.490 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:11:05.490 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 0, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 2, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 0, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 0, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 2, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 1, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.000422615, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.000000000, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.000346502, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:05.491 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 0 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 0, 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-05-20T18:11:05.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:11:05.493 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:11:05.494 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:11:05.495 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 148823, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 27, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 0, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 0, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 0, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 0, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:11:05.496 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 928, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 2, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-05-20T18:11:05.497 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 83, 2024-05-20T18:11:05.498 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 151947, 2024-05-20T18:11:05.498 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:05.498 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 83, 2024-05-20T18:11:05.498 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-05-20T18:11:05.498 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-05-20T18:11:05.498 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 83, 2024-05-20T18:11:05.498 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 151947, 2024-05-20T18:11:05.498 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-05-20T18:11:05.498 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:05.499 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:05.499 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:05.499 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:05.499 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:05.499 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-05-20T18:11:06.291 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2024-05-20T18:11:06.291 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2024-05-20T18:11:06.292 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2024-05-20T18:11:06.292 DEBUG:tasks.check_counter:Getting stats from k 2024-05-20T18:11:06.292 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.k perf dump 2024-05-20T18:11:06.536 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.b/config 2024-05-20T18:11:06.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:06 smithi154 ceph-mon[36482]: pgmap v393: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2024-05-20T18:11:06.661 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:06 smithi129 ceph-mon[32029]: pgmap v393: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2024-05-20T18:11:06.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:06 smithi204 ceph-mon[37076]: pgmap v393: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2024-05-20T18:11:08.105 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-05-20T18:11:08.105 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:11:08.105 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 4, 2024-05-20T18:11:08.106 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 4, 2024-05-20T18:11:08.106 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 2992, 2024-05-20T18:11:08.106 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 448, 2024-05-20T18:11:08.106 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 4, 2024-05-20T18:11:08.106 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 0, 2024-05-20T18:11:08.106 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.001587116, 2024-05-20T18:11:08.107 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.000039951, 2024-05-20T18:11:08.107 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.000716658, 2024-05-20T18:11:08.107 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:08.107 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:08.107 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 3, 2024-05-20T18:11:08.107 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000824346, 2024-05-20T18:11:08.108 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000274782 2024-05-20T18:11:08.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:08.108 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:08.108 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:08.108 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:08.108 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:08.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:08.109 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-05-20T18:11:08.109 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 448 2024-05-20T18:11:08.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:08.109 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:11:08.109 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 250, 2024-05-20T18:11:08.109 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 240, 2024-05-20T18:11:08.109 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 1450720, 2024-05-20T18:11:08.109 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 147904, 2024-05-20T18:11:08.110 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 2, 2024-05-20T18:11:08.110 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 1, 2024-05-20T18:11:08.110 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.052967658, 2024-05-20T18:11:08.110 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.014398855, 2024-05-20T18:11:08.110 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.041082755, 2024-05-20T18:11:08.110 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.006909911, 2024-05-20T18:11:08.110 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:08.111 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 239, 2024-05-20T18:11:08.111 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.014919282, 2024-05-20T18:11:08.111 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000062423 2024-05-20T18:11:08.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:08.111 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:08.111 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:08.111 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:08.111 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:08.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:08.112 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1450720, 2024-05-20T18:11:08.112 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 147904 2024-05-20T18:11:08.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:08.112 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:11:08.112 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 2, 2024-05-20T18:11:08.112 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 183, 2024-05-20T18:11:08.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 320, 2024-05-20T18:11:08.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 187680, 2024-05-20T18:11:08.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 5, 2024-05-20T18:11:08.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 3, 2024-05-20T18:11:08.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.011963286, 2024-05-20T18:11:08.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.007924425, 2024-05-20T18:11:08.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.001665471, 2024-05-20T18:11:08.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:08.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:08.114 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 180, 2024-05-20T18:11:08.114 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.018849815, 2024-05-20T18:11:08.114 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000104721 2024-05-20T18:11:08.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:08.114 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:08.114 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:08.114 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:08.114 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:08.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:08.114 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-20T18:11:08.114 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 187680 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 0, 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:11:08.115 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:11:08.116 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:11:08.117 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 149007, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 30, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 0, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 0, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-05-20T18:11:08.118 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 0, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 0, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 928, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 2, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-05-20T18:11:08.119 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 252, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 1415448, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 252, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 252, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 1415448, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-05-20T18:11:08.120 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-05-20T18:11:08.121 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:08.121 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-05-20T18:11:08.121 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-05-20T18:11:08.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:08 smithi204 ceph-mon[37076]: pgmap v394: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:11:08.711 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:08 smithi154 ceph-mon[36482]: pgmap v394: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:11:08.742 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-05-20T18:11:08.742 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-05-20T18:11:08.742 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-05-20T18:11:08.746 DEBUG:tasks.check_counter:Getting stats from c 2024-05-20T18:11:08.746 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.c perf dump 2024-05-20T18:11:08.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:08 smithi129 ceph-mon[32029]: pgmap v394: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:11:08.994 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T18:11:10.432 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:10 smithi204 ceph-mon[37076]: pgmap v395: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:11:10.839 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-05-20T18:11:10.839 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:11:10.839 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 0, 2024-05-20T18:11:10.839 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 2, 2024-05-20T18:11:10.839 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 0, 2024-05-20T18:11:10.839 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 0, 2024-05-20T18:11:10.839 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-05-20T18:11:10.839 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 0, 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.001283290, 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.000000000, 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.000360498, 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:10.840 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 0 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 6, 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 183, 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 3312, 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 187776, 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 5, 2024-05-20T18:11:10.841 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 3, 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.014635421, 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.008232523, 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.002199879, 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 181, 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.012389785, 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000068451 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:10.842 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 187776 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 247, 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 237, 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 1449904, 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 145984, 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 2, 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 1, 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.050694104, 2024-05-20T18:11:10.843 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.014775142, 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.037138607, 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.007319888, 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 236, 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.012521720, 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000053058 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 1449904, 2024-05-20T18:11:10.844 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 145984 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 0, 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:11:10.845 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:11:10.846 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:11:10.847 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 148984, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 30, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 0, 2024-05-20T18:11:10.848 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 0, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 928, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 2, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-05-20T18:11:10.849 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 249, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 1415058, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 249, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 249, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 1415058, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:10.850 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:10.851 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:10.851 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-05-20T18:11:10.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:10 smithi154 ceph-mon[36482]: pgmap v395: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:11:10.910 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:10 smithi129 ceph-mon[32029]: pgmap v395: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-05-20T18:11:11.734 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:11 smithi204 ceph-mon[37076]: osdmap e108: 12 total, 12 up, 12 in 2024-05-20T18:11:11.776 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.c 2024-05-20T18:11:11.777 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.c 2024-05-20T18:11:11.777 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.c 2024-05-20T18:11:11.777 DEBUG:tasks.check_counter:Getting stats from f 2024-05-20T18:11:11.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.f perf dump 2024-05-20T18:11:11.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:11 smithi154 ceph-mon[36482]: osdmap e108: 12 total, 12 up, 12 in 2024-05-20T18:11:11.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:11 smithi129 ceph-mon[32029]: osdmap e108: 12 total, 12 up, 12 in 2024-05-20T18:11:12.188 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T18:11:12.464 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:12 smithi204 ceph-mon[37076]: pgmap v397: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-05-20T18:11:12.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:12 smithi154 ceph-mon[36482]: pgmap v397: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-05-20T18:11:12.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:12 smithi129 ceph-mon[32029]: pgmap v397: 97 pgs: 97 active+clean; 22 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-05-20T18:11:14.606 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:14 smithi204 ceph-mon[37076]: pgmap v398: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 20 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-05-20T18:11:14.755 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-05-20T18:11:14.755 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:11:14.755 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 252, 2024-05-20T18:11:14.755 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 240, 2024-05-20T18:11:14.755 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 1453168, 2024-05-20T18:11:14.755 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 147072, 2024-05-20T18:11:14.755 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-05-20T18:11:14.755 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 0, 2024-05-20T18:11:14.755 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.052126028, 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.015425480, 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.037459361, 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.007104704, 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 240, 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.011848568, 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000049369 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:14.756 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 1453168, 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 147072 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 2, 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 183, 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 320, 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 187680, 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 5, 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 3, 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.012991242, 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.007312922, 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.002290023, 2024-05-20T18:11:14.757 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 180, 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.021095547, 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000117197 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-20T18:11:14.758 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 187680 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 0, 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 2, 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 0, 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 0, 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 2, 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 1, 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.000473895, 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.000000000, 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.000378901, 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:14.759 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 0 2024-05-20T18:11:14.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 0, 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:11:14.761 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:11:14.762 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:11:14.763 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 149007, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 30, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 0, 2024-05-20T18:11:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 928, 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 2, 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-05-20T18:11:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 250, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 1415188, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 250, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 250, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 1415188, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:14.767 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:14.767 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-05-20T18:11:14.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:14 smithi154 ceph-mon[36482]: pgmap v398: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 20 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-05-20T18:11:14.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:14 smithi129 ceph-mon[32029]: pgmap v398: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 20 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-05-20T18:11:15.440 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-05-20T18:11:15.440 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-05-20T18:11:15.440 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-05-20T18:11:15.440 DEBUG:tasks.check_counter:Getting stats from i 2024-05-20T18:11:15.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.i perf dump 2024-05-20T18:11:15.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:15 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:11:15.688 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T18:11:15.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:15 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:11:15.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:15 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-20T18:11:17.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:16 smithi204 ceph-mon[37076]: pgmap v399: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 12 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-05-20T18:11:17.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:16 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:16 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:16 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:16 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:16 smithi154 ceph-mon[36482]: pgmap v399: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 12 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-05-20T18:11:17.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:16 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:16 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:16 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:16 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:16 smithi129 ceph-mon[32029]: pgmap v399: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 12 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-05-20T18:11:17.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:16 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:16 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:16 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:16 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:17.798 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-05-20T18:11:17.798 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:11:17.798 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 236, 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 238, 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 405922, 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 75058, 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 6, 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 2, 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.029247347, 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.009343116, 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.007925465, 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.008710779, 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 236, 2024-05-20T18:11:17.799 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.011766397, 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000049857 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 0 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:11:17.800 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 176, 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 175, 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 218327, 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 63742, 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 3, 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.020738326, 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.006890628, 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.008273770, 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.005261215, 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 174, 2024-05-20T18:11:17.801 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.010429845, 2024-05-20T18:11:17.802 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000059941 2024-05-20T18:11:17.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.802 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:17.802 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.802 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.802 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.802 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 164032, 2024-05-20T18:11:17.802 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 22336 2024-05-20T18:11:17.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.802 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 23, 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 47, 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 8232, 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 136637, 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 5, 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 4, 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.004987463, 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.001586838, 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.002283295, 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000662068, 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 46, 2024-05-20T18:11:17.803 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.003918058, 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000085175 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 130912 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 1, 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-05-20T18:11:17.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "finisher-MDSRank": { 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "queue_len": 0, 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "complete_latency": { 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 622, 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.020104172, 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000032321 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "finisher-PurgeQueue": { 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "queue_len": 0, 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "complete_latency": { 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.805 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: "mds": { 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: "request": 0, 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: "reply": 0, 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: "reply_latency": { 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: "slow_reply": 0, 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: "forward": 0, 2024-05-20T18:11:17.806 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_fetch_complete": 0, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_fetch_keys": 0, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_commit": 0, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_split": 0, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_merge": 0, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes": 71, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_top": 0, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_bottom": 71, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_pin_tail": 0, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_pinned": 71, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_expired": 0, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_with_caps": 0, 2024-05-20T18:11:17.807 INFO:teuthology.orchestra.run.smithi204.stdout: "caps": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "subtrees": 2, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_hit": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_forward": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_discover": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_dir_fetch": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_remote_ino": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_lock": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "load_cent": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "q": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "exported": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "exported_inodes": 0, 2024-05-20T18:11:17.808 INFO:teuthology.orchestra.run.smithi204.stdout: "imported": 0, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "imported_inodes": 0, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_dir_fetch": 0, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_backtrace_fetch": 0, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_peer_discover": 0, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "root_rfiles": 4, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "root_rbytes": 234, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "root_rsnaps": 7, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_backtrace_fetch": 0, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_set_tag": 0, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_backtrace_repaired": 0, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_inotable_repaired": 0, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_dir_inodes": 0, 2024-05-20T18:11:17.809 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_dir_base_inodes": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_dirfrag_rstats": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_file_inodes": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_inode_file_caps": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_revoke": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_grant": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_trunc": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_caps": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_caps_dirty": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_cap_release": 0, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "process_request_cap_release": 0 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.810 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_cache": { 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_update": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_update_receipt": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_try_discover": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_send_discover": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_handle_discover": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays_delayed": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays_enqueuing": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_created": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_enqueued": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_reintegrated": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_migrated": 0, 2024-05-20T18:11:17.811 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_processing": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_enqueued": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_prioritized": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "recovery_started": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "recovery_completed": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_quiesce_path": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_quiesce_inode": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_enqueue_scrub": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_exportdir": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_flush": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_fragmentdir": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_fragstats": 0, 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_inodestats": 0 2024-05-20T18:11:17.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_log": { 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "evlrg": 0, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "evadd": 0, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "evex": 0, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "evtrm": 0, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "ev": 83, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "evexg": 0, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "evexd": 0, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "segadd": 0, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "segex": 0, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "segtrm": 0, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "seg": 2, 2024-05-20T18:11:17.813 INFO:teuthology.orchestra.run.smithi204.stdout: "segmjr": 5, 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: "segexg": 0, 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: "segexd": 0, 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: "expos": 947592589, 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: "wrpos": 947932756, 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: "rdpos": 947932644, 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: "jlat": { 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: "replayed": 83 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.814 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_mem": { 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "ino": 30, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "ino+": 48, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "ino-": 18, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "dir": 11, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "dir+": 11, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "dir-": 0, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "dn": 71, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "dn+": 71, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "dn-": 0, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "cap": 0, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "cap+": 0, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "cap-": 0, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "rss": 114364, 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: "heap": 215444 2024-05-20T18:11:17.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_server": { 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "dispatch_client_request": 0, 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "dispatch_server_request": 0, 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_request": 0, 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_session": 0, 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_peer_request": 0, 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "req_create_latency": { 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "req_getattr_latency": { 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.816 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: "req_getfilelock_latency": { 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: "req_link_latency": { 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookup_latency": { 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.817 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookuphash_latency": { 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupino_latency": { 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.818 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupname_latency": { 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupparent_latency": { 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupsnap_latency": { 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.819 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lssnap_latency": { 2024-05-20T18:11:17.820 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.820 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.820 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.820 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mkdir_latency": { 2024-05-20T18:11:17.820 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.820 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mknod_latency": { 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mksnap_latency": { 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.821 INFO:teuthology.orchestra.run.smithi204.stdout: "req_open_latency": { 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: "req_readdir_latency": { 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rename_latency": { 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.822 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "req_renamesnap_latency": { 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "req_snapdiff_latency": { 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmdir_latency": { 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.823 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmsnap_latency": { 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmxattr_latency": { 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setattr_latency": { 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setdirlayout_latency": { 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setfilelock_latency": { 2024-05-20T18:11:17.825 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setlayout_latency": { 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setxattr_latency": { 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: "req_symlink_latency": { 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: "req_unlink_latency": { 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: "cap_revoke_eviction": 0, 2024-05-20T18:11:17.827 INFO:teuthology.orchestra.run.smithi204.stdout: "cap_acquisition_throttle": 0, 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: "req_getvxattr_latency": { 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_sessions": { 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: "session_count": 2, 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: "session_add": 2, 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: "session_remove": 0, 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: "sessions_open": 0, 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: "sessions_stale": 0, 2024-05-20T18:11:17.828 INFO:teuthology.orchestra.run.smithi204.stdout: "total_load": 0, 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "average_load": 0, 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "avg_session_uptime": 2060, 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "mdthresh_evicted": 0 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:11:17.829 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:11:17.830 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:11:17.831 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 197888, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 86, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 8360, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 176, 2024-05-20T18:11:17.832 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 175774, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 1512, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "objecter": { 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "op_active": 0, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "op_laggy": 0, 2024-05-20T18:11:17.833 INFO:teuthology.orchestra.run.smithi204.stdout: "op_send": 384, 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: "op_send_bytes": 26, 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: "op_resend": 4, 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: "op_reply": 384, 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: "op_latency": { 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 384, 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.230997131, 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000601555 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: "op_inflight": 0, 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: "oplen_avg": { 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 384, 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 385 2024-05-20T18:11:17.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "op": 384, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "op_r": 384, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "op_w": 0, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "op_rmw": 0, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "op_pg": 0, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_stat": 238, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_create": 0, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_read": 143, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_write": 0, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_writefull": 0, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_writesame": 0, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_append": 0, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_zero": 0, 2024-05-20T18:11:17.835 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_truncate": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_delete": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_mapext": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_sparse_read": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_clonerange": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_getxattr": 2, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_setxattr": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_cmpxattr": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_rmxattr": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_resetxattrs": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_call": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_watch": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_notify": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_src_cmpxattr": 0, 2024-05-20T18:11:17.836 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_pgls": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_pgls_filter": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_other": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_active": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_send": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_resend": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_ping": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_active": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_send": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_resend": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_active": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_send": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_resend": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_active": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_send": 0, 2024-05-20T18:11:17.837 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_resend": 0, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "command_active": 0, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "command_send": 0, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "command_resend": 0, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "map_epoch": 108, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "map_full": 0, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "map_inc": 3, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_sessions": 6, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_session_open": 6, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_session_close": 0, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_laggy": 0, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_wr": 0, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_rd": 2, 2024-05-20T18:11:17.838 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_del": 0 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: "oft": { 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_objs": 0, 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_kv_pairs": 0, 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_updates": 0, 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_removes": 0 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: "osd-slow-ops": { 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: "slow_ops_count": 0 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: "purge_queue": { 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing_ops": 0, 2024-05-20T18:11:17.839 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing_ops_high_water": 0, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing": 0, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing_high_water": 0, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executed_ops": 0, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executed": 0, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_item_in_journal": 0 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 431, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 588371, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 431, 2024-05-20T18:11:17.840 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 431, 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 588371, 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-objecter_bytes": { 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-05-20T18:11:17.841 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 384, 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 352019, 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 384, 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 352019, 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.842 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-objecter_ops": { 2024-05-20T18:11:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-05-20T18:11:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 1024, 2024-05-20T18:11:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-05-20T18:11:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-05-20T18:11:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-05-20T18:11:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-05-20T18:11:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 384, 2024-05-20T18:11:17.843 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 384, 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 384, 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 384, 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-write_buf_throttle": { 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 3758096384, 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-05-20T18:11:17.844 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 0, 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 0, 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-write_buf_throttle-0x559814a77040": { 2024-05-20T18:11:17.845 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 3758096384, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 0, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 0, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:17.846 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:17.847 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:17.847 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:17.847 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-05-20T18:11:18.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:17 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:11:18.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:17 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:11:18.104 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:17 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:18.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:17 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:11:18.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:17 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:11:18.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:17 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:18.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:17 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-20T18:11:18.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:17 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-20T18:11:18.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:17 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' 2024-05-20T18:11:19.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:18 smithi204 ceph-mon[37076]: pgmap v400: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-05-20T18:11:19.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:18 smithi204 ceph-mon[37076]: pgmap v401: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 243 B/s wr, 2 op/s 2024-05-20T18:11:19.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:18 smithi204 ceph-mon[37076]: pgmap v402: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 849 B/s rd, 141 B/s wr, 1 op/s 2024-05-20T18:11:19.375 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=7 2024-05-20T18:11:19.375 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-05-20T18:11:19.375 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-05-20T18:11:19.376 DEBUG:tasks.check_counter:Getting stats from l 2024-05-20T18:11:19.376 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d shell --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 -- ceph daemon mds.l perf dump 2024-05-20T18:11:19.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:18 smithi154 ceph-mon[36482]: pgmap v400: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-05-20T18:11:19.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:18 smithi154 ceph-mon[36482]: pgmap v401: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 243 B/s wr, 2 op/s 2024-05-20T18:11:19.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:18 smithi154 ceph-mon[36482]: pgmap v402: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 849 B/s rd, 141 B/s wr, 1 op/s 2024-05-20T18:11:19.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:18 smithi129 ceph-mon[32029]: pgmap v400: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-05-20T18:11:19.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:18 smithi129 ceph-mon[32029]: pgmap v401: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 243 B/s wr, 2 op/s 2024-05-20T18:11:19.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:18 smithi129 ceph-mon[32029]: pgmap v402: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 849 B/s rd, 141 B/s wr, 1 op/s 2024-05-20T18:11:19.622 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/mon.c/config 2024-05-20T18:11:21.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:20 smithi204 ceph-mon[37076]: pgmap v403: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 730 B/s rd, 121 B/s wr, 1 op/s 2024-05-20T18:11:21.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:20 smithi204 ceph-mon[37076]: osdmap e109: 12 total, 12 up, 12 in 2024-05-20T18:11:21.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:20 smithi154 ceph-mon[36482]: pgmap v403: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 730 B/s rd, 121 B/s wr, 1 op/s 2024-05-20T18:11:21.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:20 smithi154 ceph-mon[36482]: osdmap e109: 12 total, 12 up, 12 in 2024-05-20T18:11:21.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:20 smithi129 ceph-mon[32029]: pgmap v403: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 730 B/s rd, 121 B/s wr, 1 op/s 2024-05-20T18:11:21.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:20 smithi129 ceph-mon[32029]: osdmap e109: 12 total, 12 up, 12 in 2024-05-20T18:11:21.526 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-05-20T18:11:21.526 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-05-20T18:11:21.526 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 0, 2024-05-20T18:11:21.526 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 2, 2024-05-20T18:11:21.526 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 0, 2024-05-20T18:11:21.526 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 0, 2024-05-20T18:11:21.526 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-05-20T18:11:21.526 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 0, 2024-05-20T18:11:21.526 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.001112396, 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.000000000, 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.000298062, 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-20T18:11:21.527 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 0 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 6, 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 189, 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 3312, 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 188832, 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 5, 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 3, 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.013874255, 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.008156830, 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.001858752, 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-20T18:11:21.528 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 187, 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.026291687, 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000140597 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 188832 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:21.529 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 255, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 245, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 1452704, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 150800, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 2, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 1, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.052560423, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.015132512, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.040671668, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.006979969, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 244, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.011137929, 2024-05-20T18:11:21.530 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000045647 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 1452704, 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 150800 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 0, 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-05-20T18:11:21.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-05-20T18:11:21.532 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-05-20T18:11:21.533 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 148984, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 30, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-05-20T18:11:21.534 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-05-20T18:11:21.535 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 928, 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 2, 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 257, 2024-05-20T18:11:21.536 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 1416735, 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 257, 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 257, 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 1416735, 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-05-20T18:11:21.537 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-05-20T18:11:22.212 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-05-20T18:11:22.212 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-05-20T18:11:22.212 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-05-20T18:11:22.213 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-20T18:11:22.223 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-20T18:11:22.224 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T18:11:22.224 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-20T18:11:22.255 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-05-20T18:11:22.255 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-20T18:11:22.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T18:11:22.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.484+0000 7f803a5d5640 1 -- 172.21.15.129:0/2125640740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034108ac0 msgr2=0x7f80341026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:22.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.484+0000 7f803a5d5640 1 --2- 172.21.15.129:0/2125640740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034108ac0 0x7f80341026b0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f8024009900 tx=0x7f802402a410 comp rx=0 tx=0).stop 2024-05-20T18:11:22.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.484+0000 7f803a5d5640 1 -- 172.21.15.129:0/2125640740 shutdown_connections 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.484+0000 7f803a5d5640 1 --2- 172.21.15.129:0/2125640740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8034103260 0x7f803410f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.484+0000 7f803a5d5640 1 --2- 172.21.15.129:0/2125640740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034108f60 0x7f8034102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.484+0000 7f803a5d5640 1 --2- 172.21.15.129:0/2125640740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034108ac0 0x7f80341026b0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.484+0000 7f803a5d5640 1 -- 172.21.15.129:0/2125640740 >> 172.21.15.129:0/2125640740 conn(0x7f80340780f0 msgr2=0x7f8034078530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.485+0000 7f803a5d5640 1 -- 172.21.15.129:0/2125640740 shutdown_connections 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.485+0000 7f803a5d5640 1 -- 172.21.15.129:0/2125640740 wait complete. 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.485+0000 7f803a5d5640 1 Processor -- start 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.485+0000 7f803a5d5640 1 -- start start 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f803a5d5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034103260 0x7f80341997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f803a5d5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034108ac0 0x7f8034199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f803a5d5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8034108f60 0x7f803419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f803a5d5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8034112e00 con 0x7f8034108ac0 2024-05-20T18:11:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f803a5d5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8034112c80 con 0x7f8034108f60 2024-05-20T18:11:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f803a5d5640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8034112f80 con 0x7f8034103260 2024-05-20T18:11:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f80337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034108ac0 0x7f8034199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f80337fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034108ac0 0x7f8034199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34866/0 (socket says 172.21.15.129:34866) 2024-05-20T18:11:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f80337fe640 1 -- 172.21.15.129:0/4191400369 learned_addr learned my addr 172.21.15.129:0/4191400369 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f8033fff640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034103260 0x7f80341997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f8038b72640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8034108f60 0x7f803419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f80337fe640 1 -- 172.21.15.129:0/4191400369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034103260 msgr2=0x7f80341997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f80337fe640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034103260 0x7f80341997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f80337fe640 1 -- 172.21.15.129:0/4191400369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8034108f60 msgr2=0x7f803419dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:22.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f80337fe640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8034108f60 0x7f803419dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f80337fe640 1 -- 172.21.15.129:0/4191400369 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803419e620 con 0x7f8034108ac0 2024-05-20T18:11:22.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f80337fe640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034108ac0 0x7f8034199d20 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f802800e9f0 tx=0x7f802800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:22.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.486+0000 7f80317fa640 1 -- 172.21.15.129:0/4191400369 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802800ccc0 con 0x7f8034108ac0 2024-05-20T18:11:22.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.487+0000 7f80317fa640 1 -- 172.21.15.129:0/4191400369 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80280035f0 con 0x7f8034108ac0 2024-05-20T18:11:22.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.487+0000 7f803a5d5640 1 -- 172.21.15.129:0/4191400369 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f803419e910 con 0x7f8034108ac0 2024-05-20T18:11:22.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.487+0000 7f80317fa640 1 -- 172.21.15.129:0/4191400369 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802800ce60 con 0x7f8034108ac0 2024-05-20T18:11:22.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.487+0000 7f803a5d5640 1 -- 172.21.15.129:0/4191400369 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f803419ee10 con 0x7f8034108ac0 2024-05-20T18:11:22.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.488+0000 7f803a5d5640 1 -- 172.21.15.129:0/4191400369 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80340682c0 con 0x7f8034108ac0 2024-05-20T18:11:22.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.489+0000 7f80317fa640 1 -- 172.21.15.129:0/4191400369 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f80280037a0 con 0x7f8034108ac0 2024-05-20T18:11:22.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.490+0000 7f80317fa640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f801c0a2d00 0x7f801c0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:22.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.490+0000 7f8033fff640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f801c0a2d00 0x7f801c0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:22.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.490+0000 7f80317fa640 1 -- 172.21.15.129:0/4191400369 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f80280cf8e0 con 0x7f8034108ac0 2024-05-20T18:11:22.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.490+0000 7f8033fff640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f801c0a2d00 0x7f801c0a5210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8024004770 tx=0x7f80240058d0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:22.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.495+0000 7f80317fa640 1 -- 172.21.15.129:0/4191400369 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f803419ee10 con 0x7f8034108ac0 2024-05-20T18:11:22.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.674+0000 7f803a5d5640 1 -- 172.21.15.129:0/4191400369 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8034106a00 con 0x7f8034108ac0 2024-05-20T18:11:22.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.675+0000 7f80317fa640 1 -- 172.21.15.129:0/4191400369 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+23222 (secure 0 0 0) 0x7f802809c4c0 con 0x7f8034108ac0 2024-05-20T18:11:22.673 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:11:22.673 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":109,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T18:11:20.328361+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":20,"snap_epoch":108,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4},{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2249940021":"2024-05-21T17:58:43.562501+0000","172.21.15.129:6800/2870089245":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/1943655536":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/2870089245":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1369259428":"2024-05-21T17:58:43.562501+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/549964701":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":20,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T18:11:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.677+0000 7f803a5d5640 1 -- 172.21.15.129:0/4191400369 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f801c0a2d00 msgr2=0x7f801c0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.677+0000 7f803a5d5640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f801c0a2d00 0x7f801c0a5210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8024004770 tx=0x7f80240058d0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.677+0000 7f803a5d5640 1 -- 172.21.15.129:0/4191400369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034108ac0 msgr2=0x7f8034199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.677+0000 7f803a5d5640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034108ac0 0x7f8034199d20 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f802800e9f0 tx=0x7f802800eeb0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.677+0000 7f803a5d5640 1 -- 172.21.15.129:0/4191400369 shutdown_connections 2024-05-20T18:11:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.677+0000 7f803a5d5640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f801c0a2d00 0x7f801c0a5210 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.677+0000 7f803a5d5640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8034108f60 0x7f803419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.677+0000 7f803a5d5640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034108ac0 0x7f8034199d20 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.677+0000 7f803a5d5640 1 --2- 172.21.15.129:0/4191400369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034103260 0x7f80341997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.678+0000 7f803a5d5640 1 -- 172.21.15.129:0/4191400369 >> 172.21.15.129:0/4191400369 conn(0x7f80340780f0 msgr2=0x7f8034104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.678+0000 7f803a5d5640 1 -- 172.21.15.129:0/4191400369 shutdown_connections 2024-05-20T18:11:22.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.678+0000 7f803a5d5640 1 -- 172.21.15.129:0/4191400369 wait complete. 2024-05-20T18:11:22.687 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-20T17:52:07.946331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-20T17:56:42.965073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-20T17:56:44.277499+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 20, 'snap_epoch': 108, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-20T18:11:22.687 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-20T18:11:22.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.804+0000 7fee04ec3640 1 -- 172.21.15.129:0/4184774453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee00102230 msgr2=0x7fee00102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:22.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.804+0000 7fee04ec3640 1 --2- 172.21.15.129:0/4184774453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee00102230 0x7fee00102640 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fede8009900 tx=0x7fede802a410 comp rx=0 tx=0).stop 2024-05-20T18:11:22.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.804+0000 7fee04ec3640 1 -- 172.21.15.129:0/4184774453 shutdown_connections 2024-05-20T18:11:22.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.804+0000 7fee04ec3640 1 --2- 172.21.15.129:0/4184774453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0010c350 0x7fee0010e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.804+0000 7fee04ec3640 1 --2- 172.21.15.129:0/4184774453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee00102230 0x7fee00102640 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.804+0000 7fee04ec3640 1 --2- 172.21.15.129:0/4184774453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee001079f0 0x7fee00101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.804+0000 7fee04ec3640 1 -- 172.21.15.129:0/4184774453 >> 172.21.15.129:0/4184774453 conn(0x7fee00076830 msgr2=0x7fee00076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:22.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.804+0000 7fee04ec3640 1 -- 172.21.15.129:0/4184774453 shutdown_connections 2024-05-20T18:11:22.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.804+0000 7fee04ec3640 1 -- 172.21.15.129:0/4184774453 wait complete. 2024-05-20T18:11:22.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.805+0000 7fee04ec3640 1 Processor -- start 2024-05-20T18:11:22.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.805+0000 7fee04ec3640 1 -- start start 2024-05-20T18:11:22.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fee04ec3640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee00102230 0x7fee0019f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:22.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fee04ec3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee001079f0 0x7fee0019fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:22.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fee04ec3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0010c350 0x7fee00199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:22.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fee04ec3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fee00111d00 con 0x7fee001079f0 2024-05-20T18:11:22.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fee04ec3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fee00111b80 con 0x7fee0010c350 2024-05-20T18:11:22.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fee04ec3640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fee00111e80 con 0x7fee00102230 2024-05-20T18:11:22.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee001079f0 0x7fee0019fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:22.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee001079f0 0x7fee0019fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34892/0 (socket says 172.21.15.129:34892) 2024-05-20T18:11:22.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf7fff640 1 -- 172.21.15.129:0/1729465374 learned_addr learned my addr 172.21.15.129:0/1729465374 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:22.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedfe59c640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee00102230 0x7fee0019f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:22.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedfed9d640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0010c350 0x7fee00199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:22.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf7fff640 1 -- 172.21.15.129:0/1729465374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee00102230 msgr2=0x7fee0019f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:22.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf7fff640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee00102230 0x7fee0019f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf7fff640 1 -- 172.21.15.129:0/1729465374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0010c350 msgr2=0x7fee00199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf7fff640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0010c350 0x7fee00199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf7fff640 1 -- 172.21.15.129:0/1729465374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee00199cb0 con 0x7fee001079f0 2024-05-20T18:11:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf7fff640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee001079f0 0x7fee0019fa30 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fede8005940 tx=0x7fede8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf77fe640 1 -- 172.21.15.129:0/1729465374 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede803f030 con 0x7fee001079f0 2024-05-20T18:11:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fee04ec3640 1 -- 172.21.15.129:0/1729465374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee00199f40 con 0x7fee001079f0 2024-05-20T18:11:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf77fe640 1 -- 172.21.15.129:0/1729465374 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fede8003ad0 con 0x7fee001079f0 2024-05-20T18:11:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fedf77fe640 1 -- 172.21.15.129:0/1729465374 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8035070 con 0x7fee001079f0 2024-05-20T18:11:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.806+0000 7fee04ec3640 1 -- 172.21.15.129:0/1729465374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0019a480 con 0x7fee001079f0 2024-05-20T18:11:22.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.807+0000 7fee04ec3640 1 -- 172.21.15.129:0/1729465374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee00102d00 con 0x7fee001079f0 2024-05-20T18:11:22.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.809+0000 7fedf77fe640 1 -- 172.21.15.129:0/1729465374 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fede8042030 con 0x7fee001079f0 2024-05-20T18:11:22.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.810+0000 7fedf77fe640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fedec0a2d00 0x7fedec0a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:22.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.810+0000 7fedfe59c640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fedec0a2d00 0x7fedec0a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:22.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.810+0000 7fedf77fe640 1 -- 172.21.15.129:0/1729465374 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7fede80efe50 con 0x7fee001079f0 2024-05-20T18:11:22.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.810+0000 7fedfe59c640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fedec0a2d00 0x7fedec0a5210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fee0006a210 tx=0x7fedf0005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:22.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.814+0000 7fedf77fe640 1 -- 172.21.15.129:0/1729465374 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fede80bca30 con 0x7fee001079f0 2024-05-20T18:11:22.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.990+0000 7fee04ec3640 1 -- 172.21.15.129:0/1729465374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fee00069110 con 0x7fee001079f0 2024-05-20T18:11:22.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.991+0000 7fedf77fe640 1 -- 172.21.15.129:0/1729465374 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) ==== 93+0+10 (secure 0 0 0) 0x7fede80bc330 con 0x7fee001079f0 2024-05-20T18:11:22.988 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-05-20T18:11:22.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 -- 172.21.15.129:0/1729465374 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fedec0a2d00 msgr2=0x7fedec0a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:22.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fedec0a2d00 0x7fedec0a5210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fee0006a210 tx=0x7fedf0005f90 comp rx=0 tx=0).stop 2024-05-20T18:11:22.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 -- 172.21.15.129:0/1729465374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee001079f0 msgr2=0x7fee0019fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee001079f0 0x7fee0019fa30 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fede8005940 tx=0x7fede8004270 comp rx=0 tx=0).stop 2024-05-20T18:11:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 -- 172.21.15.129:0/1729465374 shutdown_connections 2024-05-20T18:11:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fedec0a2d00 0x7fedec0a5210 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee0010c350 0x7fee00199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee001079f0 0x7fee0019fa30 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 --2- 172.21.15.129:0/1729465374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fee00102230 0x7fee0019f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 -- 172.21.15.129:0/1729465374 >> 172.21.15.129:0/1729465374 conn(0x7fee00076830 msgr2=0x7fee00100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 -- 172.21.15.129:0/1729465374 shutdown_connections 2024-05-20T18:11:22.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:22.993+0000 7fee04ec3640 1 -- 172.21.15.129:0/1729465374 wait complete. 2024-05-20T18:11:23.000 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:22 smithi129 ceph-mon[32029]: pgmap v405: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 159 B/s wr, 1 op/s 2024-05-20T18:11:23.001 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:22 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/4191400369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T18:11:23.002 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-20T18:11:23.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.118+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2553101549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1181027e0 msgr2=0x7fb11810e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.118+0000 7fb11d0f0640 1 --2- 172.21.15.129:0/2553101549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1181027e0 0x7fb11810e4e0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb110009ca0 tx=0x7fb11002a8c0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.118+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2553101549 shutdown_connections 2024-05-20T18:11:23.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.118+0000 7fb11d0f0640 1 --2- 172.21.15.129:0/2553101549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1181027e0 0x7fb11810e4e0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.118+0000 7fb11d0f0640 1 --2- 172.21.15.129:0/2553101549 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb118107e60 0x7fb118102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.118+0000 7fb11d0f0640 1 --2- 172.21.15.129:0/2553101549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1181079c0 0x7fb118101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.118+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2553101549 >> 172.21.15.129:0/2553101549 conn(0x7fb118076830 msgr2=0x7fb118076c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:23.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.118+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2553101549 shutdown_connections 2024-05-20T18:11:23.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.118+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2553101549 wait complete. 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb11d0f0640 1 Processor -- start 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb11d0f0640 1 -- start start 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb11d0f0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1181027e0 0x7fb11819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb11d0f0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb1181079c0 0x7fb11819ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb11d0f0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb118107e60 0x7fb11819b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb11d0f0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb118111d50 con 0x7fb1181027e0 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb11d0f0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb118111bd0 con 0x7fb118107e60 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb11d0f0640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb118111ed0 con 0x7fb1181079c0 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb116d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1181027e0 0x7fb11819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb116d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1181027e0 0x7fb11819a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34912/0 (socket says 172.21.15.129:34912) 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb116d9d640 1 -- 172.21.15.129:0/2565606411 learned_addr learned my addr 172.21.15.129:0/2565606411 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:23.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.119+0000 7fb11659c640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb1181079c0 0x7fb11819ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb11759e640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb118107e60 0x7fb11819b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb116d9d640 1 -- 172.21.15.129:0/2565606411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb1181079c0 msgr2=0x7fb11819ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb116d9d640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb1181079c0 0x7fb11819ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb116d9d640 1 -- 172.21.15.129:0/2565606411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb118107e60 msgr2=0x7fb11819b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb116d9d640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb118107e60 0x7fb11819b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb116d9d640 1 -- 172.21.15.129:0/2565606411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1181a5880 con 0x7fb1181027e0 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb11759e640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb118107e60 0x7fb11819b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb116d9d640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1181027e0 0x7fb11819a5d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb10000d770 tx=0x7fb10000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb10ffff640 1 -- 172.21.15.129:0/2565606411 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb100017070 con 0x7fb1181027e0 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2565606411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1181a5b70 con 0x7fb1181027e0 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb10ffff640 1 -- 172.21.15.129:0/2565606411 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb100009c30 con 0x7fb1181027e0 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb10ffff640 1 -- 172.21.15.129:0/2565606411 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10000f620 con 0x7fb1181027e0 2024-05-20T18:11:23.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.120+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2565606411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1181a6110 con 0x7fb1181027e0 2024-05-20T18:11:23.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.121+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2565606411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb118102e50 con 0x7fb1181027e0 2024-05-20T18:11:23.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.123+0000 7fb10ffff640 1 -- 172.21.15.129:0/2565606411 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fb10001a070 con 0x7fb1181027e0 2024-05-20T18:11:23.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.124+0000 7fb10ffff640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb1040a2d00 0x7fb1040a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.124+0000 7fb11659c640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb1040a2d00 0x7fb1040a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.124+0000 7fb10ffff640 1 -- 172.21.15.129:0/2565606411 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7fb1000cef90 con 0x7fb1181027e0 2024-05-20T18:11:23.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.125+0000 7fb11659c640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb1040a2d00 0x7fb1040a5210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb118068b70 tx=0x7fb108005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:23.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.128+0000 7fb10ffff640 1 -- 172.21.15.129:0/2565606411 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb100009de0 con 0x7fb1181027e0 2024-05-20T18:11:23.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:22 smithi204 ceph-mon[37076]: pgmap v405: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 159 B/s wr, 1 op/s 2024-05-20T18:11:23.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:22 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/4191400369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T18:11:23.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.304+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2565606411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb118109ad0 con 0x7fb1181027e0 2024-05-20T18:11:23.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.304+0000 7fb10ffff640 1 -- 172.21.15.129:0/2565606411 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) ==== 104+0+11 (secure 0 0 0) 0x7fb10009bb70 con 0x7fb1181027e0 2024-05-20T18:11:23.302 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-05-20T18:11:23.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.306+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2565606411 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb1040a2d00 msgr2=0x7fb1040a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.306+0000 7fb11d0f0640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb1040a2d00 0x7fb1040a5210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb118068b70 tx=0x7fb108005f90 comp rx=0 tx=0).stop 2024-05-20T18:11:23.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.306+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2565606411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1181027e0 msgr2=0x7fb11819a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.306+0000 7fb11d0f0640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1181027e0 0x7fb11819a5d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb10000d770 tx=0x7fb10000dc30 comp rx=0 tx=0).stop 2024-05-20T18:11:23.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.306+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2565606411 shutdown_connections 2024-05-20T18:11:23.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.306+0000 7fb11d0f0640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fb1040a2d00 0x7fb1040a5210 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.306+0000 7fb11d0f0640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb118107e60 0x7fb11819b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.306+0000 7fb11d0f0640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb1181079c0 0x7fb11819ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.306+0000 7fb11d0f0640 1 --2- 172.21.15.129:0/2565606411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1181027e0 0x7fb11819a5d0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.307+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2565606411 >> 172.21.15.129:0/2565606411 conn(0x7fb118076830 msgr2=0x7fb1180fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:23.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.307+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2565606411 shutdown_connections 2024-05-20T18:11:23.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.307+0000 7fb11d0f0640 1 -- 172.21.15.129:0/2565606411 wait complete. 2024-05-20T18:11:23.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-20T18:11:23.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:22 smithi154 ceph-mon[36482]: pgmap v405: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 159 B/s wr, 1 op/s 2024-05-20T18:11:23.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:22 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/4191400369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T18:11:23.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.432+0000 7f1021013640 1 -- 172.21.15.129:0/1649307029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f101c107a00 msgr2=0x7f101c1015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.432+0000 7f1021013640 1 --2- 172.21.15.129:0/1649307029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f101c107a00 0x7f101c1015e0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f100c009900 tx=0x7f100c02a410 comp rx=0 tx=0).stop 2024-05-20T18:11:23.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.432+0000 7f1021013640 1 -- 172.21.15.129:0/1649307029 shutdown_connections 2024-05-20T18:11:23.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.432+0000 7f1021013640 1 --2- 172.21.15.129:0/1649307029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f101c101b20 0x7f101c101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.432+0000 7f1021013640 1 --2- 172.21.15.129:0/1649307029 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f101c10c360 0x7f101c10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.432+0000 7f1021013640 1 --2- 172.21.15.129:0/1649307029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f101c107a00 0x7f101c1015e0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.432+0000 7f1021013640 1 -- 172.21.15.129:0/1649307029 >> 172.21.15.129:0/1649307029 conn(0x7f101c0770c0 msgr2=0x7f101c077500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:23.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.432+0000 7f1021013640 1 -- 172.21.15.129:0/1649307029 shutdown_connections 2024-05-20T18:11:23.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.432+0000 7f1021013640 1 -- 172.21.15.129:0/1649307029 wait complete. 2024-05-20T18:11:23.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.433+0000 7f1021013640 1 Processor -- start 2024-05-20T18:11:23.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.433+0000 7f1021013640 1 -- start start 2024-05-20T18:11:23.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.433+0000 7f1021013640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f101c101b20 0x7f101c19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.433+0000 7f1021013640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f101c107a00 0x7f101c19ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.433+0000 7f1021013640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f101c10c360 0x7f101c19b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.433+0000 7f1021013640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f101c111d10 con 0x7f101c10c360 2024-05-20T18:11:23.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.433+0000 7f1021013640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f101c111b90 con 0x7f101c107a00 2024-05-20T18:11:23.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.433+0000 7f1021013640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f101c111e90 con 0x7f101c101b20 2024-05-20T18:11:23.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.433+0000 7f101b59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f101c10c360 0x7f101c19b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.433+0000 7f101b59e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f101c10c360 0x7f101c19b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34924/0 (socket says 172.21.15.129:34924) 2024-05-20T18:11:23.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.433+0000 7f101b59e640 1 -- 172.21.15.129:0/3151654522 learned_addr learned my addr 172.21.15.129:0/3151654522 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:23.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f1013fff640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f101c107a00 0x7f101c19ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f101ad9d640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f101c101b20 0x7f101c19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f101b59e640 1 -- 172.21.15.129:0/3151654522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f101c101b20 msgr2=0x7f101c19a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f101b59e640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f101c101b20 0x7f101c19a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f101b59e640 1 -- 172.21.15.129:0/3151654522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f101c107a00 msgr2=0x7f101c19ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f101b59e640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f101c107a00 0x7f101c19ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f101b59e640 1 -- 172.21.15.129:0/3151654522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101c1a5900 con 0x7f101c10c360 2024-05-20T18:11:23.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f101b59e640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f101c10c360 0x7f101c19b070 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f101400cad0 tx=0x7f101400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:23.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f1018d99640 1 -- 172.21.15.129:0/3151654522 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101400ae20 con 0x7f101c10c360 2024-05-20T18:11:23.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f1018d99640 1 -- 172.21.15.129:0/3151654522 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1014005020 con 0x7f101c10c360 2024-05-20T18:11:23.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f1021013640 1 -- 172.21.15.129:0/3151654522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101c1a5bf0 con 0x7f101c10c360 2024-05-20T18:11:23.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f1018d99640 1 -- 172.21.15.129:0/3151654522 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101400f6f0 con 0x7f101c10c360 2024-05-20T18:11:23.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.434+0000 7f1021013640 1 -- 172.21.15.129:0/3151654522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f101c1a6140 con 0x7f101c10c360 2024-05-20T18:11:23.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.435+0000 7f1021013640 1 -- 172.21.15.129:0/3151654522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f101c069110 con 0x7f101c10c360 2024-05-20T18:11:23.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.436+0000 7f1018d99640 1 -- 172.21.15.129:0/3151654522 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f10140051d0 con 0x7f101c10c360 2024-05-20T18:11:23.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.437+0000 7f1018d99640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10080a2d00 0x7f10080a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.437+0000 7f101ad9d640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10080a2d00 0x7f10080a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.437+0000 7f1018d99640 1 -- 172.21.15.129:0/3151654522 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f10140cf780 con 0x7f101c10c360 2024-05-20T18:11:23.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.438+0000 7f101ad9d640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10080a2d00 0x7f10080a5210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f100c000c00 tx=0x7f100c0095f0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:23.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.441+0000 7f1018d99640 1 -- 172.21.15.129:0/3151654522 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1014014030 con 0x7f101c10c360 2024-05-20T18:11:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.617+0000 7f1021013640 1 -- 172.21.15.129:0/3151654522 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f101c105930 con 0x7f101c10c360 2024-05-20T18:11:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.617+0000 7f1018d99640 1 -- 172.21.15.129:0/3151654522 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) ==== 100+0+11 (secure 0 0 0) 0x7f101409c360 con 0x7f101c10c360 2024-05-20T18:11:23.615 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-05-20T18:11:23.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.619+0000 7f1021013640 1 -- 172.21.15.129:0/3151654522 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10080a2d00 msgr2=0x7f10080a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.619+0000 7f1021013640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10080a2d00 0x7f10080a5210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f100c000c00 tx=0x7f100c0095f0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.619+0000 7f1021013640 1 -- 172.21.15.129:0/3151654522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f101c10c360 msgr2=0x7f101c19b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.619+0000 7f1021013640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f101c10c360 0x7f101c19b070 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f101400cad0 tx=0x7f101400cf90 comp rx=0 tx=0).stop 2024-05-20T18:11:23.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.619+0000 7f1021013640 1 -- 172.21.15.129:0/3151654522 shutdown_connections 2024-05-20T18:11:23.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.619+0000 7f1021013640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f10080a2d00 0x7f10080a5210 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.619+0000 7f1021013640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f101c107a00 0x7f101c19ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.619+0000 7f1021013640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f101c10c360 0x7f101c19b070 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.619+0000 7f1021013640 1 --2- 172.21.15.129:0/3151654522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f101c101b20 0x7f101c19a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.619+0000 7f1021013640 1 -- 172.21.15.129:0/3151654522 >> 172.21.15.129:0/3151654522 conn(0x7f101c0770c0 msgr2=0x7f101c103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.620+0000 7f1021013640 1 -- 172.21.15.129:0/3151654522 shutdown_connections 2024-05-20T18:11:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.620+0000 7f1021013640 1 -- 172.21.15.129:0/3151654522 wait complete. 2024-05-20T18:11:23.628 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-20T18:11:23.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.744+0000 7f0a2b03d640 1 -- 172.21.15.129:0/219675270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a2410c330 msgr2=0x7f0a2410e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.744+0000 7f0a2b03d640 1 --2- 172.21.15.129:0/219675270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a2410c330 0x7f0a2410e750 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009ca0 tx=0x7f0a1c02a8c0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.744+0000 7f0a2b03d640 1 -- 172.21.15.129:0/219675270 shutdown_connections 2024-05-20T18:11:23.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.744+0000 7f0a2b03d640 1 --2- 172.21.15.129:0/219675270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a24101aa0 0x7f0a24101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.744+0000 7f0a2b03d640 1 --2- 172.21.15.129:0/219675270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a2410c330 0x7f0a2410e750 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.744+0000 7f0a2b03d640 1 --2- 172.21.15.129:0/219675270 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a241079d0 0x7f0a24101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.744+0000 7f0a2b03d640 1 -- 172.21.15.129:0/219675270 >> 172.21.15.129:0/219675270 conn(0x7f0a240770b0 msgr2=0x7f0a240774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:23.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.744+0000 7f0a2b03d640 1 -- 172.21.15.129:0/219675270 shutdown_connections 2024-05-20T18:11:23.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.744+0000 7f0a2b03d640 1 -- 172.21.15.129:0/219675270 wait complete. 2024-05-20T18:11:23.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a2b03d640 1 Processor -- start 2024-05-20T18:11:23.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a2b03d640 1 -- start start 2024-05-20T18:11:23.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a2b03d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a24101aa0 0x7f0a2419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a2b03d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a241079d0 0x7f0a2419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a2b03d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a2410c330 0x7f0a2419afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a2b03d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0a24111ce0 con 0x7f0a2410c330 2024-05-20T18:11:23.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a2b03d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a24111b60 con 0x7f0a241079d0 2024-05-20T18:11:23.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a2b03d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0a24111e60 con 0x7f0a24101aa0 2024-05-20T18:11:23.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a295da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a2410c330 0x7f0a2419afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a295da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a2410c330 0x7f0a2419afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34944/0 (socket says 172.21.15.129:34944) 2024-05-20T18:11:23.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a295da640 1 -- 172.21.15.129:0/3506838624 learned_addr learned my addr 172.21.15.129:0/3506838624 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:23.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a23fff640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a241079d0 0x7f0a2419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.745+0000 7f0a28dd9640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a24101aa0 0x7f0a2419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a295da640 1 -- 172.21.15.129:0/3506838624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a24101aa0 msgr2=0x7f0a2419a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a295da640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a24101aa0 0x7f0a2419a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a295da640 1 -- 172.21.15.129:0/3506838624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a241079d0 msgr2=0x7f0a2419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a295da640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a241079d0 0x7f0a2419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a295da640 1 -- 172.21.15.129:0/3506838624 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a241a5830 con 0x7f0a2410c330 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a28dd9640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a24101aa0 0x7f0a2419a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a23fff640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a241079d0 0x7f0a2419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a295da640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a2410c330 0x7f0a2419afa0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c031a20 tx=0x7f0a1c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a21ffb640 1 -- 172.21.15.129:0/3506838624 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c003c30 con 0x7f0a2410c330 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a21ffb640 1 -- 172.21.15.129:0/3506838624 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a1c003dd0 con 0x7f0a2410c330 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a2b03d640 1 -- 172.21.15.129:0/3506838624 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a241a5ac0 con 0x7f0a2410c330 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a21ffb640 1 -- 172.21.15.129:0/3506838624 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c02fcf0 con 0x7f0a2410c330 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.746+0000 7f0a2b03d640 1 -- 172.21.15.129:0/3506838624 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a241a6010 con 0x7f0a2410c330 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.747+0000 7f0a2b03d640 1 -- 172.21.15.129:0/3506838624 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a24069f50 con 0x7f0a2410c330 2024-05-20T18:11:23.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.748+0000 7f0a21ffb640 1 -- 172.21.15.129:0/3506838624 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f0a1c004050 con 0x7f0a2410c330 2024-05-20T18:11:23.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.749+0000 7f0a21ffb640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0a140a2d00 0x7f0a140a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:23.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.749+0000 7f0a28dd9640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0a140a2d00 0x7f0a140a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:23.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.749+0000 7f0a21ffb640 1 -- 172.21.15.129:0/3506838624 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f0a1c0f07e0 con 0x7f0a2410c330 2024-05-20T18:11:23.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.750+0000 7f0a28dd9640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0a140a2d00 0x7f0a140a5210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0a18005d90 tx=0x7f0a18005d20 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:23.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.754+0000 7f0a21ffb640 1 -- 172.21.15.129:0/3506838624 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0a1c0bd560 con 0x7f0a2410c330 2024-05-20T18:11:23.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.946+0000 7f0a2b03d640 1 -- 172.21.15.129:0/3506838624 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0a24069430 con 0x7f0a2410c330 2024-05-20T18:11:23.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.946+0000 7f0a21ffb640 1 -- 172.21.15.129:0/3506838624 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v109) ==== 81+0+958 (secure 0 0 0) 0x7f0a1c02fe90 con 0x7f0a2410c330 2024-05-20T18:11:23.944 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:0/2249940021 2024-05-21T17:58:43.562501+0000 2024-05-20T18:11:23.944 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6800/2870089245 2024-05-21T17:58:43.562501+0000 2024-05-20T18:11:23.944 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:0/2611729606 2024-05-21T17:49:47.618673+0000 2024-05-20T18:11:23.944 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:0/1943655536 2024-05-21T17:58:43.562501+0000 2024-05-20T18:11:23.944 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:0/874085270 2024-05-21T17:49:47.618673+0000 2024-05-20T18:11:23.944 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6801/1916241944 2024-05-21T17:49:47.618673+0000 2024-05-20T18:11:23.944 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6800/1916241944 2024-05-21T17:49:47.618673+0000 2024-05-20T18:11:23.944 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6800/3614898033 2024-05-21T17:49:21.927054+0000 2024-05-20T18:11:23.945 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:0/1985417481 2024-05-21T17:49:21.927054+0000 2024-05-20T18:11:23.945 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6801/2870089245 2024-05-21T17:58:43.562501+0000 2024-05-20T18:11:23.945 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:0/1769091279 2024-05-21T17:49:21.927054+0000 2024-05-20T18:11:23.945 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:0/1369259428 2024-05-21T17:58:43.562501+0000 2024-05-20T18:11:23.945 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6801/3614898033 2024-05-21T17:49:21.927054+0000 2024-05-20T18:11:23.945 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:0/3608430741 2024-05-21T17:49:21.927054+0000 2024-05-20T18:11:23.945 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:0/549964701 2024-05-21T17:58:43.562501+0000 2024-05-20T18:11:23.945 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:0/99394831 2024-05-21T17:49:47.618673+0000 2024-05-20T18:11:23.946 INFO:teuthology.orchestra.run.smithi129.stderr:listed 16 entries 2024-05-20T18:11:23.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 -- 172.21.15.129:0/3506838624 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0a140a2d00 msgr2=0x7f0a140a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0a140a2d00 0x7f0a140a5210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0a18005d90 tx=0x7f0a18005d20 comp rx=0 tx=0).stop 2024-05-20T18:11:23.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 -- 172.21.15.129:0/3506838624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a2410c330 msgr2=0x7f0a2419afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:23.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a2410c330 0x7f0a2419afa0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c031a20 tx=0x7f0a1c031a50 comp rx=0 tx=0).stop 2024-05-20T18:11:23.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 -- 172.21.15.129:0/3506838624 shutdown_connections 2024-05-20T18:11:23.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f0a140a2d00 0x7f0a140a5210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a241079d0 0x7f0a2419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a2410c330 0x7f0a2419afa0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 --2- 172.21.15.129:0/3506838624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0a24101aa0 0x7f0a2419a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:23.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 -- 172.21.15.129:0/3506838624 >> 172.21.15.129:0/3506838624 conn(0x7f0a240770b0 msgr2=0x7f0a24103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:23.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 -- 172.21.15.129:0/3506838624 shutdown_connections 2024-05-20T18:11:23.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:23.948+0000 7f0a2b03d640 1 -- 172.21.15.129:0/3506838624 wait complete. 2024-05-20T18:11:23.957 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-20T18:11:23.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:11:23.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-20T18:11:24.014 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-05-20T18:11:24.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:11:24.014 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-20T18:11:24.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:11:24.070 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-05-20T18:11:24.119 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-20T18:11:24.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:11:24.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-20T18:11:24.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-20T18:11:24.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link delete ceph-brx 2024-05-20T18:11:24.209 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:23 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1729465374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T18:11:24.209 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:23 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2565606411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T18:11:24.209 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:23 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3151654522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T18:11:24.246 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-20T18:11:24.257 INFO:tasks.cephadm:Teardown begin 2024-05-20T18:11:24.257 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-20T18:11:24.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:11:24.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:23 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1729465374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T18:11:24.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:23 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2565606411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T18:11:24.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:23 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3151654522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T18:11:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.402+0000 7f7e400de640 1 -- 172.21.15.129:0/976864011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e3810c330 msgr2=0x7f7e3810e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.402+0000 7f7e400de640 1 --2- 172.21.15.129:0/976864011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e3810c330 0x7f7e3810e750 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009ca0 tx=0x7f7e3002a8c0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.402+0000 7f7e400de640 1 -- 172.21.15.129:0/976864011 shutdown_connections 2024-05-20T18:11:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.402+0000 7f7e400de640 1 --2- 172.21.15.129:0/976864011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e38101aa0 0x7f7e38101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.402+0000 7f7e400de640 1 --2- 172.21.15.129:0/976864011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e3810c330 0x7f7e3810e750 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.402+0000 7f7e400de640 1 --2- 172.21.15.129:0/976864011 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e381079d0 0x7f7e38101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.402+0000 7f7e400de640 1 -- 172.21.15.129:0/976864011 >> 172.21.15.129:0/976864011 conn(0x7f7e380770c0 msgr2=0x7f7e38077500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.402+0000 7f7e400de640 1 -- 172.21.15.129:0/976864011 shutdown_connections 2024-05-20T18:11:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.402+0000 7f7e400de640 1 -- 172.21.15.129:0/976864011 wait complete. 2024-05-20T18:11:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.403+0000 7f7e400de640 1 Processor -- start 2024-05-20T18:11:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.403+0000 7f7e400de640 1 -- start start 2024-05-20T18:11:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e400de640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e38101aa0 0x7f7e3819a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e400de640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e381079d0 0x7f7e3819aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e400de640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e3810c330 0x7f7e3819afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e400de640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e38111d00 con 0x7f7e38101aa0 2024-05-20T18:11:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e400de640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e38111b80 con 0x7f7e381079d0 2024-05-20T18:11:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e400de640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7e38111e80 con 0x7f7e3810c330 2024-05-20T18:11:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e3de7a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e38101aa0 0x7f7e3819a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e3de7a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e38101aa0 0x7f7e3819a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34952/0 (socket says 172.21.15.129:34952) 2024-05-20T18:11:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e3de7a640 1 -- 172.21.15.129:0/2135373726 learned_addr learned my addr 172.21.15.129:0/2135373726 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e3d679640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e381079d0 0x7f7e3819aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e3e67b640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e3810c330 0x7f7e3819afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e3de7a640 1 -- 172.21.15.129:0/2135373726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e3810c330 msgr2=0x7f7e3819afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e3de7a640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e3810c330 0x7f7e3819afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e3de7a640 1 -- 172.21.15.129:0/2135373726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e381079d0 msgr2=0x7f7e3819aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e3de7a640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e381079d0 0x7f7e3819aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e3de7a640 1 -- 172.21.15.129:0/2135373726 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e381a5850 con 0x7f7e38101aa0 2024-05-20T18:11:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e3de7a640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e38101aa0 0x7f7e3819a540 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800e9f0 tx=0x7f7e2800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.404+0000 7f7e2effd640 1 -- 172.21.15.129:0/2135373726 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800ccc0 con 0x7f7e38101aa0 2024-05-20T18:11:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.405+0000 7f7e400de640 1 -- 172.21.15.129:0/2135373726 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e381a5b40 con 0x7f7e38101aa0 2024-05-20T18:11:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.405+0000 7f7e2effd640 1 -- 172.21.15.129:0/2135373726 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e28005500 con 0x7f7e38101aa0 2024-05-20T18:11:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.405+0000 7f7e2effd640 1 -- 172.21.15.129:0/2135373726 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28010690 con 0x7f7e38101aa0 2024-05-20T18:11:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.405+0000 7f7e400de640 1 -- 172.21.15.129:0/2135373726 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e381a6040 con 0x7f7e38101aa0 2024-05-20T18:11:24.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.406+0000 7f7e400de640 1 -- 172.21.15.129:0/2135373726 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e38102c80 con 0x7f7e38101aa0 2024-05-20T18:11:24.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.407+0000 7f7e2effd640 1 -- 172.21.15.129:0/2135373726 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f7e280046c0 con 0x7f7e38101aa0 2024-05-20T18:11:24.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.408+0000 7f7e2effd640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e240a2d00 0x7f7e240a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:24.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.408+0000 7f7e3d679640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e240a2d00 0x7f7e240a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:24.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.408+0000 7f7e2effd640 1 -- 172.21.15.129:0/2135373726 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f7e280cfa30 con 0x7f7e38101aa0 2024-05-20T18:11:24.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.408+0000 7f7e3d679640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e240a2d00 0x7f7e240a5210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7e38100c30 tx=0x7f7e34005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:24.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.412+0000 7f7e2effd640 1 -- 172.21.15.129:0/2135373726 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7e28053850 con 0x7f7e38101aa0 2024-05-20T18:11:24.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:23 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1729465374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-20T18:11:24.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:23 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2565606411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-20T18:11:24.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:23 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3151654522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-20T18:11:24.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.625+0000 7f7e400de640 1 -- 172.21.15.129:0/2135373726 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e38069f50 con 0x7f7e38101aa0 2024-05-20T18:11:24.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.625+0000 7f7e2effd640 1 -- 172.21.15.129:0/2135373726 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11829 (secure 0 0 0) 0x7f7e2809c610 con 0x7f7e38101aa0 2024-05-20T18:11:24.623 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:11:24.623 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":26255,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/2724397229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":2724397229},{"type":"v1","addr":"172.21.15.154:6837","nonce":2724397229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:09:18.904334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26815":{"gid":26815,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/1721642062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":1721642062},{"type":"v1","addr":"172.21.15.204:6839","nonce":1721642062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:11:24.623 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 28 2024-05-20T18:11:24.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.627+0000 7f7e400de640 1 -- 172.21.15.129:0/2135373726 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e240a2d00 msgr2=0x7f7e240a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:24.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.627+0000 7f7e400de640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e240a2d00 0x7f7e240a5210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7e38100c30 tx=0x7f7e34005f90 comp rx=0 tx=0).stop 2024-05-20T18:11:24.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.627+0000 7f7e400de640 1 -- 172.21.15.129:0/2135373726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e38101aa0 msgr2=0x7f7e3819a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:24.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.627+0000 7f7e400de640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e38101aa0 0x7f7e3819a540 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800e9f0 tx=0x7f7e2800eeb0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.627+0000 7f7e400de640 1 -- 172.21.15.129:0/2135373726 shutdown_connections 2024-05-20T18:11:24.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.627+0000 7f7e400de640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f7e240a2d00 0x7f7e240a5210 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.627+0000 7f7e400de640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e381079d0 0x7f7e3819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.627+0000 7f7e400de640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7e3810c330 0x7f7e3819afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.627+0000 7f7e400de640 1 --2- 172.21.15.129:0/2135373726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e38101aa0 0x7f7e3819a540 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.627+0000 7f7e400de640 1 -- 172.21.15.129:0/2135373726 >> 172.21.15.129:0/2135373726 conn(0x7f7e380770c0 msgr2=0x7f7e38103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:24.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.628+0000 7f7e400de640 1 -- 172.21.15.129:0/2135373726 shutdown_connections 2024-05-20T18:11:24.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.628+0000 7f7e400de640 1 -- 172.21.15.129:0/2135373726 wait complete. 2024-05-20T18:11:24.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-20T18:11:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.752+0000 7f6e501ce640 1 -- 172.21.15.129:0/3850353698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e4810c370 msgr2=0x7f6e4810e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.752+0000 7f6e501ce640 1 --2- 172.21.15.129:0/3850353698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e4810c370 0x7f6e4810e790 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6e38009860 tx=0x7f6e3802a340 comp rx=0 tx=0).stop 2024-05-20T18:11:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.752+0000 7f6e501ce640 1 -- 172.21.15.129:0/3850353698 shutdown_connections 2024-05-20T18:11:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.752+0000 7f6e501ce640 1 --2- 172.21.15.129:0/3850353698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e4810c370 0x7f6e4810e790 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.752+0000 7f6e501ce640 1 --2- 172.21.15.129:0/3850353698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e48101b30 0x7f6e48101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.752+0000 7f6e501ce640 1 --2- 172.21.15.129:0/3850353698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e48107a10 0x7f6e481015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.752+0000 7f6e501ce640 1 -- 172.21.15.129:0/3850353698 >> 172.21.15.129:0/3850353698 conn(0x7f6e48077130 msgr2=0x7f6e48077570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.752+0000 7f6e501ce640 1 -- 172.21.15.129:0/3850353698 shutdown_connections 2024-05-20T18:11:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.752+0000 7f6e501ce640 1 -- 172.21.15.129:0/3850353698 wait complete. 2024-05-20T18:11:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e501ce640 1 Processor -- start 2024-05-20T18:11:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e501ce640 1 -- start start 2024-05-20T18:11:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e501ce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e48101b30 0x7f6e48198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e501ce640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e48107a10 0x7f6e48198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e501ce640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e4810c370 0x7f6e4819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e501ce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6e48111d40 con 0x7f6e48101b30 2024-05-20T18:11:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e501ce640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e48111bc0 con 0x7f6e48107a10 2024-05-20T18:11:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e501ce640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6e48111ec0 con 0x7f6e4810c370 2024-05-20T18:11:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e4df6a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e48101b30 0x7f6e48198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e4d769640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e48107a10 0x7f6e48198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e4df6a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e48101b30 0x7f6e48198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34978/0 (socket says 172.21.15.129:34978) 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e4df6a640 1 -- 172.21.15.129:0/1033050900 learned_addr learned my addr 172.21.15.129:0/1033050900 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.753+0000 7f6e4e76b640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e4810c370 0x7f6e4819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e4d769640 1 -- 172.21.15.129:0/1033050900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e4810c370 msgr2=0x7f6e4819ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e4d769640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e4810c370 0x7f6e4819ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e4d769640 1 -- 172.21.15.129:0/1033050900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e48101b30 msgr2=0x7f6e48198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e4d769640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e48101b30 0x7f6e48198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e4d769640 1 -- 172.21.15.129:0/1033050900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4819d4a0 con 0x7f6e48107a10 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e4df6a640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e48101b30 0x7f6e48198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e4d769640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e48107a10 0x7f6e48198ba0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6e480685a0 tx=0x7f6e3c00eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e46ffd640 1 -- 172.21.15.129:0/1033050900 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c00cdb0 con 0x7f6e48107a10 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e501ce640 1 -- 172.21.15.129:0/1033050900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e4819d790 con 0x7f6e48107a10 2024-05-20T18:11:24.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e46ffd640 1 -- 172.21.15.129:0/1033050900 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e3c003e50 con 0x7f6e48107a10 2024-05-20T18:11:24.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e46ffd640 1 -- 172.21.15.129:0/1033050900 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c010690 con 0x7f6e48107a10 2024-05-20T18:11:24.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.754+0000 7f6e501ce640 1 -- 172.21.15.129:0/1033050900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e481a5110 con 0x7f6e48107a10 2024-05-20T18:11:24.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.755+0000 7f6e501ce640 1 -- 172.21.15.129:0/1033050900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e48069140 con 0x7f6e48107a10 2024-05-20T18:11:24.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.756+0000 7f6e46ffd640 1 -- 172.21.15.129:0/1033050900 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f6e3c003970 con 0x7f6e48107a10 2024-05-20T18:11:24.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.757+0000 7f6e46ffd640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6e300a2d00 0x7f6e300a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:24.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.757+0000 7f6e4df6a640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6e300a2d00 0x7f6e300a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:24.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.757+0000 7f6e46ffd640 1 -- 172.21.15.129:0/1033050900 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f6e3c0cfc10 con 0x7f6e48107a10 2024-05-20T18:11:24.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.758+0000 7f6e4df6a640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6e300a2d00 0x7f6e300a5210 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6e400096c0 tx=0x7f6e40009260 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:24.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.761+0000 7f6e46ffd640 1 -- 172.21.15.129:0/1033050900 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6e3c09c7d0 con 0x7f6e48107a10 2024-05-20T18:11:24.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.974+0000 7f6e501ce640 1 -- 172.21.15.129:0/1033050900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e48105f00 con 0x7f6e48107a10 2024-05-20T18:11:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.975+0000 7f6e46ffd640 1 -- 172.21.15.129:0/1033050900 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11829 (secure 0 0 0) 0x7f6e3c09c0d0 con 0x7f6e48107a10 2024-05-20T18:11:24.973 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:11:24.973 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2428051280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2428051280},{"type":"v1","addr":"172.21.15.129:6835","nonce":2428051280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/2359452744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":2359452744},{"type":"v1","addr":"172.21.15.129:6837","nonce":2359452744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/2206181560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":2206181560},{"type":"v1","addr":"172.21.15.129:6839","nonce":2206181560}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6841/1623696468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6840","nonce":1623696468},{"type":"v1","addr":"172.21.15.129:6841","nonce":1623696468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3042064152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3042064152},{"type":"v1","addr":"172.21.15.204:6833","nonce":3042064152}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3269888601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3269888601},{"type":"v1","addr":"172.21.15.154:6833","nonce":3269888601}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24469,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/4037294668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":4037294668},{"type":"v1","addr":"172.21.15.204:6835","nonce":4037294668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/259458619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":259458619},{"type":"v1","addr":"172.21.15.204:6837","nonce":259458619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3512970065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":3512970065},{"type":"v1","addr":"172.21.15.154:6835","nonce":3512970065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":26255,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/2724397229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":2724397229},{"type":"v1","addr":"172.21.15.154:6837","nonce":2724397229}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-20T17:56:45.096192+0000","modified":"2024-05-20T18:09:18.904334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24560},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24560":{"gid":24560,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/793468142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":793468142},{"type":"v1","addr":"172.21.15.154:6839","nonce":793468142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26815":{"gid":26815,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/1721642062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":1721642062},{"type":"v1","addr":"172.21.15.204:6839","nonce":1721642062}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24560,"qdb_cluster":[24560]},"id":1}]} 2024-05-20T18:11:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 28 2024-05-20T18:11:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 -- 172.21.15.129:0/1033050900 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6e300a2d00 msgr2=0x7f6e300a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6e300a2d00 0x7f6e300a5210 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6e400096c0 tx=0x7f6e40009260 comp rx=0 tx=0).stop 2024-05-20T18:11:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 -- 172.21.15.129:0/1033050900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e48107a10 msgr2=0x7f6e48198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e48107a10 0x7f6e48198ba0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6e480685a0 tx=0x7f6e3c00eed0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 -- 172.21.15.129:0/1033050900 shutdown_connections 2024-05-20T18:11:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f6e300a2d00 0x7f6e300a5210 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e48107a10 0x7f6e48198ba0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6e4810c370 0x7f6e4819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 --2- 172.21.15.129:0/1033050900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e48101b30 0x7f6e48198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 -- 172.21.15.129:0/1033050900 >> 172.21.15.129:0/1033050900 conn(0x7f6e48077130 msgr2=0x7f6e481030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 -- 172.21.15.129:0/1033050900 shutdown_connections 2024-05-20T18:11:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:24.978+0000 7f6e501ce640 1 -- 172.21.15.129:0/1033050900 wait complete. 2024-05-20T18:11:24.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-20T18:11:25.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.102+0000 7f570cd4c640 1 -- 172.21.15.129:0/1656179095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f570810d000 msgr2=0x7f570810f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.102+0000 7f570cd4c640 1 --2- 172.21.15.129:0/1656179095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f570810d000 0x7f570810f4f0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009900 tx=0x7f56f802a410 comp rx=0 tx=0).stop 2024-05-20T18:11:25.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.102+0000 7f570cd4c640 1 -- 172.21.15.129:0/1656179095 shutdown_connections 2024-05-20T18:11:25.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.102+0000 7f570cd4c640 1 --2- 172.21.15.129:0/1656179095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f570810d000 0x7f570810f4f0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.102+0000 7f570cd4c640 1 --2- 172.21.15.129:0/1656179095 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f570810a610 0x7f570810ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.102+0000 7f570cd4c640 1 --2- 172.21.15.129:0/1656179095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5708103bb0 0x7f5708103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.102+0000 7f570cd4c640 1 -- 172.21.15.129:0/1656179095 >> 172.21.15.129:0/1656179095 conn(0x7f57080fd030 msgr2=0x7f57080ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:25.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.102+0000 7f570cd4c640 1 -- 172.21.15.129:0/1656179095 shutdown_connections 2024-05-20T18:11:25.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.102+0000 7f570cd4c640 1 -- 172.21.15.129:0/1656179095 wait complete. 2024-05-20T18:11:25.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f570cd4c640 1 Processor -- start 2024-05-20T18:11:25.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f570cd4c640 1 -- start start 2024-05-20T18:11:25.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f570cd4c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5708103bb0 0x7f5708199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f570cd4c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f570810a610 0x7f5708199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f570cd4c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f570810d000 0x7f570819df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f570cd4c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5708112db0 con 0x7f570810d000 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f570cd4c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5708112c30 con 0x7f570810a610 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f570cd4c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5708112f30 con 0x7f5708103bb0 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f5706d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f570810d000 0x7f570819df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f5706d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f570810d000 0x7f570819df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34990/0 (socket says 172.21.15.129:34990) 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f5706d9d640 1 -- 172.21.15.129:0/860961179 learned_addr learned my addr 172.21.15.129:0/860961179 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f5705d9b640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f570810a610 0x7f5708199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.103+0000 7f570659c640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5708103bb0 0x7f5708199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.104+0000 7f5706d9d640 1 -- 172.21.15.129:0/860961179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5708103bb0 msgr2=0x7f5708199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.104+0000 7f5706d9d640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5708103bb0 0x7f5708199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.104+0000 7f5706d9d640 1 -- 172.21.15.129:0/860961179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f570810a610 msgr2=0x7f5708199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.104+0000 7f5706d9d640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f570810a610 0x7f5708199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.104+0000 7f5706d9d640 1 -- 172.21.15.129:0/860961179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570819e570 con 0x7f570810d000 2024-05-20T18:11:25.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.104+0000 7f5706d9d640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f570810d000 0x7f570819df30 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f56f8005ab0 tx=0x7f56f8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:25.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.104+0000 7f56ff7fe640 1 -- 172.21.15.129:0/860961179 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f803f030 con 0x7f570810d000 2024-05-20T18:11:25.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.104+0000 7f56ff7fe640 1 -- 172.21.15.129:0/860961179 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56f8003ad0 con 0x7f570810d000 2024-05-20T18:11:25.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.104+0000 7f570cd4c640 1 -- 172.21.15.129:0/860961179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570819e800 con 0x7f570810d000 2024-05-20T18:11:25.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.104+0000 7f56ff7fe640 1 -- 172.21.15.129:0/860961179 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f8035070 con 0x7f570810d000 2024-05-20T18:11:25.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.104+0000 7f570cd4c640 1 -- 172.21.15.129:0/860961179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f570819ecf0 con 0x7f570810d000 2024-05-20T18:11:25.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.105+0000 7f570cd4c640 1 -- 172.21.15.129:0/860961179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5708069140 con 0x7f570810d000 2024-05-20T18:11:25.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.106+0000 7f56ff7fe640 1 -- 172.21.15.129:0/860961179 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f56f8042030 con 0x7f570810d000 2024-05-20T18:11:25.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.107+0000 7f56ff7fe640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56f40a2d00 0x7f56f40a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.108+0000 7f570659c640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56f40a2d00 0x7f56f40a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.108+0000 7f56ff7fe640 1 -- 172.21.15.129:0/860961179 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f56f80efd90 con 0x7f570810d000 2024-05-20T18:11:25.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.108+0000 7f570659c640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56f40a2d00 0x7f56f40a5210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5700005fd0 tx=0x7f5700009680 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:25.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.112+0000 7f56ff7fe640 1 -- 172.21.15.129:0/860961179 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f56f80bc320 con 0x7f570810d000 2024-05-20T18:11:25.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.296+0000 7f570cd4c640 1 -- 172.21.15.129:0/860961179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57081a6240 con 0x7f570810d000 2024-05-20T18:11:25.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.298+0000 7f56ff7fe640 1 -- 172.21.15.129:0/860961179 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+23222 (secure 0 0 0) 0x7f56f80c0620 con 0x7f570810d000 2024-05-20T18:11:25.298 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-05-20T18:11:25.298 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":109,"fsid":"0f1f0dd2-16d1-11ef-bc9a-c7b262605968","created":"2024-05-20T17:48:49.505110+0000","modified":"2024-05-20T18:11:20.328361+0000","last_up_change":"2024-05-20T17:55:56.773476+0000","last_in_change":"2024-05-20T17:55:37.148467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-20T17:52:07.946331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-20T17:56:42.965073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-20T17:56:44.277499+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":20,"snap_epoch":108,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e5e9e31-cbcd-4bc8-89aa-219d7b9f94bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6803","nonce":4065863746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6805","nonce":4065863746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6809","nonce":4065863746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4065863746},{"type":"v1","addr":"172.21.15.129:6807","nonce":4065863746}]},"public_addr":"172.21.15.129:6803/4065863746","cluster_addr":"172.21.15.129:6805/4065863746","heartbeat_back_addr":"172.21.15.129:6809/4065863746","heartbeat_front_addr":"172.21.15.129:6807/4065863746","state":["exists","up"]},{"osd":1,"uuid":"9b41acf7-4a61-4152-9ec6-a0c980cb29c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6801","nonce":792053596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6803","nonce":792053596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6807","nonce":792053596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":792053596},{"type":"v1","addr":"172.21.15.154:6805","nonce":792053596}]},"public_addr":"172.21.15.154:6801/792053596","cluster_addr":"172.21.15.154:6803/792053596","heartbeat_back_addr":"172.21.15.154:6807/792053596","heartbeat_front_addr":"172.21.15.154:6805/792053596","state":["exists","up"]},{"osd":2,"uuid":"c6f8c70f-8cd2-46ee-93da-e1ac0b73de83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6801","nonce":278897720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6803","nonce":278897720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6807","nonce":278897720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":278897720},{"type":"v1","addr":"172.21.15.204:6805","nonce":278897720}]},"public_addr":"172.21.15.204:6801/278897720","cluster_addr":"172.21.15.204:6803/278897720","heartbeat_back_addr":"172.21.15.204:6807/278897720","heartbeat_front_addr":"172.21.15.204:6805/278897720","state":["exists","up"]},{"osd":3,"uuid":"3fbfcecf-5f49-429e-a60f-2aae1d18eb16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6811","nonce":310906415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6813","nonce":310906415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6817","nonce":310906415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":310906415},{"type":"v1","addr":"172.21.15.129:6815","nonce":310906415}]},"public_addr":"172.21.15.129:6811/310906415","cluster_addr":"172.21.15.129:6813/310906415","heartbeat_back_addr":"172.21.15.129:6817/310906415","heartbeat_front_addr":"172.21.15.129:6815/310906415","state":["exists","up"]},{"osd":4,"uuid":"3393945c-4ccc-4f2c-ad95-5e884497d5d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6809","nonce":3850885593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6811","nonce":3850885593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6815","nonce":3850885593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3850885593},{"type":"v1","addr":"172.21.15.154:6813","nonce":3850885593}]},"public_addr":"172.21.15.154:6809/3850885593","cluster_addr":"172.21.15.154:6811/3850885593","heartbeat_back_addr":"172.21.15.154:6815/3850885593","heartbeat_front_addr":"172.21.15.154:6813/3850885593","state":["exists","up"]},{"osd":5,"uuid":"9dbdb424-101b-488e-826c-fac4b7f35dfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6809","nonce":12527578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6811","nonce":12527578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6815","nonce":12527578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":12527578},{"type":"v1","addr":"172.21.15.204:6813","nonce":12527578}]},"public_addr":"172.21.15.204:6809/12527578","cluster_addr":"172.21.15.204:6811/12527578","heartbeat_back_addr":"172.21.15.204:6815/12527578","heartbeat_front_addr":"172.21.15.204:6813/12527578","state":["exists","up"]},{"osd":6,"uuid":"b87d90ab-d39b-464a-b835-0a1cc50d0eb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6819","nonce":2994413067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6821","nonce":2994413067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6825","nonce":2994413067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":2994413067},{"type":"v1","addr":"172.21.15.129:6823","nonce":2994413067}]},"public_addr":"172.21.15.129:6819/2994413067","cluster_addr":"172.21.15.129:6821/2994413067","heartbeat_back_addr":"172.21.15.129:6825/2994413067","heartbeat_front_addr":"172.21.15.129:6823/2994413067","state":["exists","up"]},{"osd":7,"uuid":"e850ec9e-e2a9-4d09-86c3-9f8115e516ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6817","nonce":2792653271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6819","nonce":2792653271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6823","nonce":2792653271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2792653271},{"type":"v1","addr":"172.21.15.154:6821","nonce":2792653271}]},"public_addr":"172.21.15.154:6817/2792653271","cluster_addr":"172.21.15.154:6819/2792653271","heartbeat_back_addr":"172.21.15.154:6823/2792653271","heartbeat_front_addr":"172.21.15.154:6821/2792653271","state":["exists","up"]},{"osd":8,"uuid":"a2371db0-2b8e-4a7c-86de-7c2036efca54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6817","nonce":4118068284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6819","nonce":4118068284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6823","nonce":4118068284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":4118068284},{"type":"v1","addr":"172.21.15.204:6821","nonce":4118068284}]},"public_addr":"172.21.15.204:6817/4118068284","cluster_addr":"172.21.15.204:6819/4118068284","heartbeat_back_addr":"172.21.15.204:6823/4118068284","heartbeat_front_addr":"172.21.15.204:6821/4118068284","state":["exists","up"]},{"osd":9,"uuid":"e40eb83d-8e9f-4f69-af66-b505b01d2836","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6827","nonce":2231439755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6829","nonce":2231439755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6833","nonce":2231439755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2231439755},{"type":"v1","addr":"172.21.15.129:6831","nonce":2231439755}]},"public_addr":"172.21.15.129:6827/2231439755","cluster_addr":"172.21.15.129:6829/2231439755","heartbeat_back_addr":"172.21.15.129:6833/2231439755","heartbeat_front_addr":"172.21.15.129:6831/2231439755","state":["exists","up"]},{"osd":10,"uuid":"fe45b366-5c36-454e-ae75-d6170e11aff1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6825","nonce":1707430880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6827","nonce":1707430880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6831","nonce":1707430880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1707430880},{"type":"v1","addr":"172.21.15.154:6829","nonce":1707430880}]},"public_addr":"172.21.15.154:6825/1707430880","cluster_addr":"172.21.15.154:6827/1707430880","heartbeat_back_addr":"172.21.15.154:6831/1707430880","heartbeat_front_addr":"172.21.15.154:6829/1707430880","state":["exists","up"]},{"osd":11,"uuid":"cc59d526-8616-4b3f-913a-c706b4d575c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6825","nonce":329726241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6827","nonce":329726241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6831","nonce":329726241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":329726241},{"type":"v1","addr":"172.21.15.204:6829","nonce":329726241}]},"public_addr":"172.21.15.204:6825/329726241","cluster_addr":"172.21.15.204:6827/329726241","heartbeat_back_addr":"172.21.15.204:6831/329726241","heartbeat_front_addr":"172.21.15.204:6829/329726241","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:18.884686+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:51:42.347852+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:06.046738+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:30.686231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:52:55.227516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:20.375334+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:53:46.034528+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:11.610095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:54:36.514792+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:03.460167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:29.520471+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-20T17:55:55.257822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4},{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":4,"to":6},{"from":5,"to":9}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2249940021":"2024-05-21T17:58:43.562501+0000","172.21.15.129:6800/2870089245":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/2611729606":"2024-05-21T17:49:47.618673+0000","172.21.15.129:0/1943655536":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/874085270":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6801/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/1916241944":"2024-05-21T17:49:47.618673+0000","172.21.15.129:6800/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1985417481":"2024-05-21T17:49:21.927054+0000","172.21.15.129:6801/2870089245":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/1769091279":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/1369259428":"2024-05-21T17:58:43.562501+0000","172.21.15.129:6801/3614898033":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/3608430741":"2024-05-21T17:49:21.927054+0000","172.21.15.129:0/549964701":"2024-05-21T17:58:43.562501+0000","172.21.15.129:0/99394831":"2024-05-21T17:49:47.618673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":20,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-20T18:11:25.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 -- 172.21.15.129:0/860961179 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56f40a2d00 msgr2=0x7f56f40a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56f40a2d00 0x7f56f40a5210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5700005fd0 tx=0x7f5700009680 comp rx=0 tx=0).stop 2024-05-20T18:11:25.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 -- 172.21.15.129:0/860961179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f570810d000 msgr2=0x7f570819df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f570810d000 0x7f570819df30 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f56f8005ab0 tx=0x7f56f8004270 comp rx=0 tx=0).stop 2024-05-20T18:11:25.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 -- 172.21.15.129:0/860961179 shutdown_connections 2024-05-20T18:11:25.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f56f40a2d00 0x7f56f40a5210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f570810a610 0x7f5708199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f570810d000 0x7f570819df30 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 --2- 172.21.15.129:0/860961179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5708103bb0 0x7f5708199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 -- 172.21.15.129:0/860961179 >> 172.21.15.129:0/860961179 conn(0x7f57080fd030 msgr2=0x7f570810beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:25.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 -- 172.21.15.129:0/860961179 shutdown_connections 2024-05-20T18:11:25.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.300+0000 7f570cd4c640 1 -- 172.21.15.129:0/860961179 wait complete. 2024-05-20T18:11:25.307 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:24 smithi129 ceph-mon[32029]: pgmap v406: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 159 B/s wr, 1 op/s 2024-05-20T18:11:25.307 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:24 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/3506838624' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-20T18:11:25.307 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:24 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2135373726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:11:25.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-20T18:11:25.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:24 smithi154 ceph-mon[36482]: pgmap v406: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 159 B/s wr, 1 op/s 2024-05-20T18:11:25.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:24 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/3506838624' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-20T18:11:25.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:24 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2135373726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:11:25.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.424+0000 7ff3affc3640 1 -- 172.21.15.129:0/1674622896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a8107a00 msgr2=0x7ff3a81015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.424+0000 7ff3affc3640 1 --2- 172.21.15.129:0/1674622896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a8107a00 0x7ff3a81015e0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4009ca0 tx=0x7ff3a402a8c0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.424+0000 7ff3affc3640 1 -- 172.21.15.129:0/1674622896 shutdown_connections 2024-05-20T18:11:25.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.424+0000 7ff3affc3640 1 --2- 172.21.15.129:0/1674622896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a8101b20 0x7ff3a8101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.424+0000 7ff3affc3640 1 --2- 172.21.15.129:0/1674622896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3a810c360 0x7ff3a810e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.424+0000 7ff3affc3640 1 --2- 172.21.15.129:0/1674622896 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a8107a00 0x7ff3a81015e0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.424+0000 7ff3affc3640 1 -- 172.21.15.129:0/1674622896 >> 172.21.15.129:0/1674622896 conn(0x7ff3a80770c0 msgr2=0x7ff3a8077500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:25.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.424+0000 7ff3affc3640 1 -- 172.21.15.129:0/1674622896 shutdown_connections 2024-05-20T18:11:25.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.424+0000 7ff3affc3640 1 -- 172.21.15.129:0/1674622896 wait complete. 2024-05-20T18:11:25.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3affc3640 1 Processor -- start 2024-05-20T18:11:25.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3affc3640 1 -- start start 2024-05-20T18:11:25.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3affc3640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3a8101b20 0x7ff3a8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3affc3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a8107a00 0x7ff3a8198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3affc3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a810c360 0x7ff3a819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3affc3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a8111d10 con 0x7ff3a8107a00 2024-05-20T18:11:25.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3affc3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a8111b90 con 0x7ff3a810c360 2024-05-20T18:11:25.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3affc3640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a8111e90 con 0x7ff3a8101b20 2024-05-20T18:11:25.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3ad55e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a8107a00 0x7ff3a8198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3ad55e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a8107a00 0x7ff3a8198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34996/0 (socket says 172.21.15.129:34996) 2024-05-20T18:11:25.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3ad55e640 1 -- 172.21.15.129:0/663110978 learned_addr learned my addr 172.21.15.129:0/663110978 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:25.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.425+0000 7ff3ae560640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a810c360 0x7ff3a819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff3add5f640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3a8101b20 0x7ff3a8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff3ad55e640 1 -- 172.21.15.129:0/663110978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3a8101b20 msgr2=0x7ff3a8198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff3ad55e640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3a8101b20 0x7ff3a8198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff3ad55e640 1 -- 172.21.15.129:0/663110978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a810c360 msgr2=0x7ff3a819ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff3ad55e640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a810c360 0x7ff3a819ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff3ad55e640 1 -- 172.21.15.129:0/663110978 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a819d450 con 0x7ff3a8107a00 2024-05-20T18:11:25.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff3ae560640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a810c360 0x7ff3a819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-20T18:11:25.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff3ad55e640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a8107a00 0x7ff3a8198bd0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7ff39800d770 tx=0x7ff39800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:25.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff39effd640 1 -- 172.21.15.129:0/663110978 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff398017070 con 0x7ff3a8107a00 2024-05-20T18:11:25.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff3affc3640 1 -- 172.21.15.129:0/663110978 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a819d740 con 0x7ff3a8107a00 2024-05-20T18:11:25.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff39effd640 1 -- 172.21.15.129:0/663110978 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff39800de80 con 0x7ff3a8107a00 2024-05-20T18:11:25.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff39effd640 1 -- 172.21.15.129:0/663110978 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff398003e00 con 0x7ff3a8107a00 2024-05-20T18:11:25.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.426+0000 7ff3affc3640 1 -- 172.21.15.129:0/663110978 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a819dc90 con 0x7ff3a8107a00 2024-05-20T18:11:25.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.427+0000 7ff3affc3640 1 -- 172.21.15.129:0/663110978 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a8069110 con 0x7ff3a8107a00 2024-05-20T18:11:25.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.428+0000 7ff39effd640 1 -- 172.21.15.129:0/663110978 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ff398003fa0 con 0x7ff3a8107a00 2024-05-20T18:11:25.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.429+0000 7ff39effd640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff3940a2d00 0x7ff3940a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.429+0000 7ff3add5f640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff3940a2d00 0x7ff3940a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.430+0000 7ff39effd640 1 -- 172.21.15.129:0/663110978 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7ff3980cef50 con 0x7ff3a8107a00 2024-05-20T18:11:25.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.430+0000 7ff3add5f640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff3940a2d00 0x7ff3940a5210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4004af0 tx=0x7ff3a4005fd0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:25.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:24 smithi204 ceph-mon[37076]: pgmap v406: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 159 B/s wr, 1 op/s 2024-05-20T18:11:25.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:24 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/3506838624' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-20T18:11:25.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:24 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2135373726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:11:25.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.434+0000 7ff39effd640 1 -- 172.21.15.129:0/663110978 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff39809bb30 con 0x7ff3a8107a00 2024-05-20T18:11:25.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.651+0000 7ff3affc3640 1 -- 172.21.15.129:0/663110978 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff3a8105ef0 con 0x7ff3a8107a00 2024-05-20T18:11:25.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.653+0000 7ff39effd640 1 -- 172.21.15.129:0/663110978 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) ==== 161+0+0 (secure 0 0 0) 0x7ff39809b430 con 0x7ff3a8107a00 2024-05-20T18:11:25.651 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-20T18:11:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 -- 172.21.15.129:0/663110978 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff3940a2d00 msgr2=0x7ff3940a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff3940a2d00 0x7ff3940a5210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4004af0 tx=0x7ff3a4005fd0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 -- 172.21.15.129:0/663110978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a8107a00 msgr2=0x7ff3a8198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a8107a00 0x7ff3a8198bd0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7ff39800d770 tx=0x7ff39800dc30 comp rx=0 tx=0).stop 2024-05-20T18:11:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 -- 172.21.15.129:0/663110978 shutdown_connections 2024-05-20T18:11:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ff3940a2d00 0x7ff3940a5210 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3a810c360 0x7ff3a819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a8107a00 0x7ff3a8198bd0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 --2- 172.21.15.129:0/663110978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3a8101b20 0x7ff3a8198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 -- 172.21.15.129:0/663110978 >> 172.21.15.129:0/663110978 conn(0x7ff3a80770c0 msgr2=0x7ff3a8103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:25.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 -- 172.21.15.129:0/663110978 shutdown_connections 2024-05-20T18:11:25.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.656+0000 7ff3affc3640 1 -- 172.21.15.129:0/663110978 wait complete. 2024-05-20T18:11:25.665 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-20T18:11:25.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.782+0000 7f5d5e321640 1 -- 172.21.15.129:0/46889816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d5810c360 msgr2=0x7f5d5810e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.782+0000 7f5d5e321640 1 --2- 172.21.15.129:0/46889816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d5810c360 0x7f5d5810e780 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f5d50009be0 tx=0x7f5d5002a6c0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.782+0000 7f5d5e321640 1 -- 172.21.15.129:0/46889816 shutdown_connections 2024-05-20T18:11:25.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.782+0000 7f5d5e321640 1 --2- 172.21.15.129:0/46889816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d58101ad0 0x7f5d58101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.782+0000 7f5d5e321640 1 --2- 172.21.15.129:0/46889816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d5810c360 0x7f5d5810e780 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.782+0000 7f5d5e321640 1 --2- 172.21.15.129:0/46889816 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d58107a00 0x7f5d58101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.782+0000 7f5d5e321640 1 -- 172.21.15.129:0/46889816 >> 172.21.15.129:0/46889816 conn(0x7f5d58077100 msgr2=0x7f5d58077540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:25.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.782+0000 7f5d5e321640 1 -- 172.21.15.129:0/46889816 shutdown_connections 2024-05-20T18:11:25.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.782+0000 7f5d5e321640 1 -- 172.21.15.129:0/46889816 wait complete. 2024-05-20T18:11:25.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d5e321640 1 Processor -- start 2024-05-20T18:11:25.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d5e321640 1 -- start start 2024-05-20T18:11:25.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d5e321640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d58101ad0 0x7f5d58199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d5e321640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d58107a00 0x7f5d58199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d5e321640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5810c360 0x7f5d5819df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d5e321640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d58111cc0 con 0x7f5d58101ad0 2024-05-20T18:11:25.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d5e321640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d58111b40 con 0x7f5d5810c360 2024-05-20T18:11:25.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d5e321640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d58111e40 con 0x7f5d58107a00 2024-05-20T18:11:25.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d57fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d58101ad0 0x7f5d58199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d57fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d58101ad0 0x7f5d58199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35014/0 (socket says 172.21.15.129:35014) 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d57fff640 1 -- 172.21.15.129:0/2035538152 learned_addr learned my addr 172.21.15.129:0/2035538152 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.783+0000 7f5d5c8be640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5810c360 0x7f5d5819df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d4ffff640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d58107a00 0x7f5d58199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d57fff640 1 -- 172.21.15.129:0/2035538152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d58107a00 msgr2=0x7f5d58199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d57fff640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d58107a00 0x7f5d58199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d57fff640 1 -- 172.21.15.129:0/2035538152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5810c360 msgr2=0x7f5d5819df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d57fff640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5810c360 0x7f5d5819df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d57fff640 1 -- 172.21.15.129:0/2035538152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5819e570 con 0x7f5d58101ad0 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d57fff640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d58101ad0 0x7f5d58199710 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f5d48005fd0 tx=0x7f5d4800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d55ffb640 1 -- 172.21.15.129:0/2035538152 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d48017070 con 0x7f5d58101ad0 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d55ffb640 1 -- 172.21.15.129:0/2035538152 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d48009c30 con 0x7f5d58101ad0 2024-05-20T18:11:25.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d5e321640 1 -- 172.21.15.129:0/2035538152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d5819e860 con 0x7f5d58101ad0 2024-05-20T18:11:25.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d55ffb640 1 -- 172.21.15.129:0/2035538152 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d48010460 con 0x7f5d58101ad0 2024-05-20T18:11:25.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.784+0000 7f5d5e321640 1 -- 172.21.15.129:0/2035538152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d581a61e0 con 0x7f5d58101ad0 2024-05-20T18:11:25.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.785+0000 7f5d5e321640 1 -- 172.21.15.129:0/2035538152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d58069110 con 0x7f5d58101ad0 2024-05-20T18:11:25.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.787+0000 7f5d55ffb640 1 -- 172.21.15.129:0/2035538152 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7f5d48003e70 con 0x7f5d58101ad0 2024-05-20T18:11:25.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.787+0000 7f5d55ffb640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d440a2d00 0x7f5d440a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:25.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.787+0000 7f5d4ffff640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d440a2d00 0x7f5d440a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:25.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.787+0000 7f5d55ffb640 1 -- 172.21.15.129:0/2035538152 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8629+0+0 (secure 0 0 0) 0x7f5d480cf8c0 con 0x7f5d58101ad0 2024-05-20T18:11:25.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.788+0000 7f5d4ffff640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d440a2d00 0x7f5d440a5210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5d400096c0 tx=0x7f5d40009260 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:25.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:25.791+0000 7f5d55ffb640 1 -- 172.21.15.129:0/2035538152 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5d48020030 con 0x7f5d58101ad0 2024-05-20T18:11:25.910 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:25 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a[32025]: 2024-05-20T18:11:25.651+0000 7f2c95bf5700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-20T18:11:26.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.008+0000 7f5d5e321640 1 -- 172.21.15.129:0/2035538152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f5d581a63b0 con 0x7f5d58101ad0 2024-05-20T18:11:26.141 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1033050900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:11:26.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/860961179' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T18:11:26.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/663110978' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-20T18:11:26.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-mon[36482]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-20T18:11:26.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-mon[36482]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-20T18:11:26.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/663110978' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-20T18:11:26.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-mon[36482]: osdmap e110: 12 total, 12 up, 12 in 2024-05-20T18:11:26.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-mon[36482]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-05-20T18:11:26.142 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b[66692]: ignoring --setuser ceph since I am not root 2024-05-20T18:11:26.142 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b[66692]: ignoring --setgroup ceph since I am not root 2024-05-20T18:11:26.143 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-mds[66696]: -- 172.21.15.154:0/304550153 <== mon.0 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ab7ca2da40 con 0x55ab7caa5600 2024-05-20T18:11:26.143 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 18:11:25 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b[66692]: starting mds.b at 2024-05-20T18:11:26.178 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:11:25 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[72933]: ignoring --setuser ceph since I am not root 2024-05-20T18:11:26.178 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:11:25 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[72933]: ignoring --setgroup ceph since I am not root 2024-05-20T18:11:26.178 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:11:25 smithi204 ceph-mds[72937]: -- 172.21.15.204:0/1715238453 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f3e709a40 con 0x556f3e781180 2024-05-20T18:11:26.178 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:11:25 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[72933]: starting mds.i at 2024-05-20T18:11:26.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:25 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1033050900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:11:26.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:25 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/860961179' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T18:11:26.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:25 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/663110978' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-20T18:11:26.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:25 smithi204 ceph-mon[37076]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-20T18:11:26.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:25 smithi204 ceph-mon[37076]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-20T18:11:26.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:25 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/663110978' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-20T18:11:26.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:25 smithi204 ceph-mon[37076]: osdmap e110: 12 total, 12 up, 12 in 2024-05-20T18:11:26.179 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:25 smithi204 ceph-mon[37076]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-05-20T18:11:26.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:25 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1033050900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-20T18:11:26.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:25 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/860961179' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-20T18:11:26.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:25 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/663110978' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-20T18:11:26.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:25 smithi129 ceph-mon[32029]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-20T18:11:26.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:25 smithi129 ceph-mon[32029]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-20T18:11:26.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:25 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/663110978' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-20T18:11:26.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:25 smithi129 ceph-mon[32029]: osdmap e110: 12 total, 12 up, 12 in 2024-05-20T18:11:26.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:25 smithi129 ceph-mon[32029]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-05-20T18:11:26.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.655+0000 7f5d55ffb640 1 -- 172.21.15.129:0/2035538152 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) ==== 104+0+0 (secure 0 0 0) 0x7f5d4809c470 con 0x7f5d58101ad0 2024-05-20T18:11:26.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 -- 172.21.15.129:0/2035538152 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d440a2d00 msgr2=0x7f5d440a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d440a2d00 0x7f5d440a5210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5d400096c0 tx=0x7f5d40009260 comp rx=0 tx=0).stop 2024-05-20T18:11:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 -- 172.21.15.129:0/2035538152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d58101ad0 msgr2=0x7f5d58199710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d58101ad0 0x7f5d58199710 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f5d48005fd0 tx=0x7f5d4800dde0 comp rx=0 tx=0).stop 2024-05-20T18:11:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 -- 172.21.15.129:0/2035538152 shutdown_connections 2024-05-20T18:11:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7f5d440a2d00 0x7f5d440a5210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5810c360 0x7f5d5819df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d58107a00 0x7f5d58199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 --2- 172.21.15.129:0/2035538152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d58101ad0 0x7f5d58199710 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 -- 172.21.15.129:0/2035538152 >> 172.21.15.129:0/2035538152 conn(0x7f5d58077100 msgr2=0x7f5d58073d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 -- 172.21.15.129:0/2035538152 shutdown_connections 2024-05-20T18:11:26.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.657+0000 7f5d5e321640 1 -- 172.21.15.129:0/2035538152 wait complete. 2024-05-20T18:11:26.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-20T18:11:26.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.790+0000 7fc0f08f7640 1 -- 172.21.15.129:0/3038300599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0ec101aa0 msgr2=0x7fc0ec101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:26.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.790+0000 7fc0f08f7640 1 --2- 172.21.15.129:0/3038300599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0ec101aa0 0x7fc0ec101eb0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009900 tx=0x7fc0e002a410 comp rx=0 tx=0).stop 2024-05-20T18:11:26.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.790+0000 7fc0f08f7640 1 -- 172.21.15.129:0/3038300599 shutdown_connections 2024-05-20T18:11:26.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.790+0000 7fc0f08f7640 1 --2- 172.21.15.129:0/3038300599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0ec10c330 0x7fc0ec10e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:26.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.790+0000 7fc0f08f7640 1 --2- 172.21.15.129:0/3038300599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0ec101aa0 0x7fc0ec101eb0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:26.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.790+0000 7fc0f08f7640 1 --2- 172.21.15.129:0/3038300599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0ec1079d0 0x7fc0ec101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:26.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.790+0000 7fc0f08f7640 1 -- 172.21.15.129:0/3038300599 >> 172.21.15.129:0/3038300599 conn(0x7fc0ec0770b0 msgr2=0x7fc0ec0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:26.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.790+0000 7fc0f08f7640 1 -- 172.21.15.129:0/3038300599 shutdown_connections 2024-05-20T18:11:26.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.790+0000 7fc0f08f7640 1 -- 172.21.15.129:0/3038300599 wait complete. 2024-05-20T18:11:26.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0f08f7640 1 Processor -- start 2024-05-20T18:11:26.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0f08f7640 1 -- start start 2024-05-20T18:11:26.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0f08f7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0ec101aa0 0x7fc0ec1a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:26.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0f08f7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0ec1079d0 0x7fc0ec1a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:26.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0f08f7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0ec10c330 0x7fc0ec19b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:26.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0f08f7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec111d30 con 0x7fc0ec101aa0 2024-05-20T18:11:26.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0f08f7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec111bb0 con 0x7fc0ec1079d0 2024-05-20T18:11:26.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0f08f7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ec111eb0 con 0x7fc0ec10c330 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0ea59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0ec101aa0 0x7fc0ec1a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0ea59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0ec101aa0 0x7fc0ec1a1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35022/0 (socket says 172.21.15.129:35022) 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0ea59c640 1 -- 172.21.15.129:0/1548548117 learned_addr learned my addr 172.21.15.129:0/1548548117 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0e9d9b640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0ec1079d0 0x7fc0ec1a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0ead9d640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0ec10c330 0x7fc0ec19b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0ea59c640 1 -- 172.21.15.129:0/1548548117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0ec10c330 msgr2=0x7fc0ec19b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0ea59c640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0ec10c330 0x7fc0ec19b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0ea59c640 1 -- 172.21.15.129:0/1548548117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0ec1079d0 msgr2=0x7fc0ec1a1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0ea59c640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0ec1079d0 0x7fc0ec1a1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.791+0000 7fc0ea59c640 1 -- 172.21.15.129:0/1548548117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ec19bec0 con 0x7fc0ec101aa0 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.792+0000 7fc0ea59c640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0ec101aa0 0x7fc0ec1a1610 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00e9c0 tx=0x7fc0dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.792+0000 7fc0db7fe640 1 -- 172.21.15.129:0/1548548117 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc00cd80 con 0x7fc0ec101aa0 2024-05-20T18:11:26.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.792+0000 7fc0db7fe640 1 -- 172.21.15.129:0/1548548117 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0dc005500 con 0x7fc0ec101aa0 2024-05-20T18:11:26.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.792+0000 7fc0f08f7640 1 -- 172.21.15.129:0/1548548117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0ec19c150 con 0x7fc0ec101aa0 2024-05-20T18:11:26.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.792+0000 7fc0db7fe640 1 -- 172.21.15.129:0/1548548117 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc010690 con 0x7fc0ec101aa0 2024-05-20T18:11:26.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.792+0000 7fc0f08f7640 1 -- 172.21.15.129:0/1548548117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0ec1a83b0 con 0x7fc0ec101aa0 2024-05-20T18:11:26.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.793+0000 7fc0f08f7640 1 -- 172.21.15.129:0/1548548117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0ec102c80 con 0x7fc0ec101aa0 2024-05-20T18:11:26.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.794+0000 7fc0db7fe640 1 -- 172.21.15.129:0/1548548117 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7fc0dc010830 con 0x7fc0ec101aa0 2024-05-20T18:11:26.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.795+0000 7fc0db7fe640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0d40a2d00 0x7fc0d40a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:26.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.795+0000 7fc0e9d9b640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0d40a2d00 0x7fc0d40a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:26.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.795+0000 7fc0db7fe640 1 -- 172.21.15.129:0/1548548117 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8629+0+0 (secure 0 0 0) 0x7fc0dc0cf9e0 con 0x7fc0ec101aa0 2024-05-20T18:11:26.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.795+0000 7fc0e9d9b640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0d40a2d00 0x7fc0d40a5210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc0e002aa00 tx=0x7fc0e0000bc0 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:26.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.799+0000 7fc0db7fe640 1 -- 172.21.15.129:0/1548548117 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc0dc09c590 con 0x7fc0ec101aa0 2024-05-20T18:11:26.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:26.985+0000 7fc0f08f7640 1 -- 172.21.15.129:0/1548548117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc0ec069f50 con 0x7fc0ec101aa0 2024-05-20T18:11:27.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:26 smithi154 ceph-mon[36482]: pgmap v407: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-20T18:11:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:26 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2035538152' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-20T18:11:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:26 smithi154 ceph-mon[36482]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-20T18:11:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:26 smithi154 ceph-mon[36482]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-20T18:11:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:26 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/2035538152' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-20T18:11:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:26 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.204:6838/1094424557,v1:172.21.15.204:6839/1094424557] up:boot 2024-05-20T18:11:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:26 smithi154 ceph-mon[36482]: mds.? [v2:172.21.15.154:6838/3439686810,v1:172.21.15.154:6839/3439686810] up:boot 2024-05-20T18:11:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:26 smithi154 ceph-mon[36482]: fsmap 12 up:standby 2024-05-20T18:11:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:26 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T18:11:27.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:26 smithi154 ceph-mon[36482]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-20T18:11:27.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:26 smithi129 ceph-mon[32029]: pgmap v407: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-20T18:11:27.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:26 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2035538152' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-20T18:11:27.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:26 smithi129 ceph-mon[32029]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-20T18:11:27.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:26 smithi129 ceph-mon[32029]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-20T18:11:27.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:26 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/2035538152' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-20T18:11:27.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:26 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.204:6838/1094424557,v1:172.21.15.204:6839/1094424557] up:boot 2024-05-20T18:11:27.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:26 smithi129 ceph-mon[32029]: mds.? [v2:172.21.15.154:6838/3439686810,v1:172.21.15.154:6839/3439686810] up:boot 2024-05-20T18:11:27.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:26 smithi129 ceph-mon[32029]: fsmap 12 up:standby 2024-05-20T18:11:27.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:26 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T18:11:27.412 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:26 smithi129 ceph-mon[32029]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-20T18:11:27.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:26 smithi204 ceph-mon[37076]: pgmap v407: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-20T18:11:27.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:26 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2035538152' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-20T18:11:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:26 smithi204 ceph-mon[37076]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-20T18:11:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:26 smithi204 ceph-mon[37076]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-20T18:11:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:26 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/2035538152' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-20T18:11:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:26 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.204:6838/1094424557,v1:172.21.15.204:6839/1094424557] up:boot 2024-05-20T18:11:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:26 smithi204 ceph-mon[37076]: mds.? [v2:172.21.15.154:6838/3439686810,v1:172.21.15.154:6839/3439686810] up:boot 2024-05-20T18:11:27.429 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:26 smithi204 ceph-mon[37076]: fsmap 12 up:standby 2024-05-20T18:11:27.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:26 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-20T18:11:27.430 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:26 smithi204 ceph-mon[37076]: from='mgr.15228 172.21.15.129:0/2095075189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-20T18:11:27.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.953+0000 7fc0db7fe640 1 -- 172.21.15.129:0/1548548117 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v111) ==== 181+0+0 (secure 0 0 0) 0x7fc0dc09be90 con 0x7fc0ec101aa0 2024-05-20T18:11:27.951 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-05-20T18:11:27.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.956+0000 7fc0f08f7640 1 -- 172.21.15.129:0/1548548117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0d40a2d00 msgr2=0x7fc0d40a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:27.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.956+0000 7fc0f08f7640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0d40a2d00 0x7fc0d40a5210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc0e002aa00 tx=0x7fc0e0000bc0 comp rx=0 tx=0).stop 2024-05-20T18:11:27.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.956+0000 7fc0f08f7640 1 -- 172.21.15.129:0/1548548117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0ec101aa0 msgr2=0x7fc0ec1a1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:27.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.956+0000 7fc0f08f7640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0ec101aa0 0x7fc0ec1a1610 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00e9c0 tx=0x7fc0dc00ee80 comp rx=0 tx=0).stop 2024-05-20T18:11:27.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.956+0000 7fc0f08f7640 1 -- 172.21.15.129:0/1548548117 shutdown_connections 2024-05-20T18:11:27.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.956+0000 7fc0f08f7640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7fc0d40a2d00 0x7fc0d40a5210 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:27.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.956+0000 7fc0f08f7640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0ec1079d0 0x7fc0ec1a1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:27.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.956+0000 7fc0f08f7640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc0ec10c330 0x7fc0ec19b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:27.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.956+0000 7fc0f08f7640 1 --2- 172.21.15.129:0/1548548117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0ec101aa0 0x7fc0ec1a1610 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:27.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.956+0000 7fc0f08f7640 1 -- 172.21.15.129:0/1548548117 >> 172.21.15.129:0/1548548117 conn(0x7fc0ec0770b0 msgr2=0x7fc0ec100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:27.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.957+0000 7fc0f08f7640 1 -- 172.21.15.129:0/1548548117 shutdown_connections 2024-05-20T18:11:27.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:27.957+0000 7fc0f08f7640 1 -- 172.21.15.129:0/1548548117 wait complete. 2024-05-20T18:11:27.967 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-20T18:11:28.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.086+0000 7ffbde916640 1 -- 172.21.15.129:0/4198548260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbd810c360 msgr2=0x7ffbd810e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:28.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.086+0000 7ffbde916640 1 --2- 172.21.15.129:0/4198548260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbd810c360 0x7ffbd810e780 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0009ca0 tx=0x7ffbd002a8c0 comp rx=0 tx=0).stop 2024-05-20T18:11:28.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.087+0000 7ffbde916640 1 -- 172.21.15.129:0/4198548260 shutdown_connections 2024-05-20T18:11:28.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.087+0000 7ffbde916640 1 --2- 172.21.15.129:0/4198548260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbd810c360 0x7ffbd810e780 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:28.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.087+0000 7ffbde916640 1 --2- 172.21.15.129:0/4198548260 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbd8101ad0 0x7ffbd8101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:28.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.087+0000 7ffbde916640 1 --2- 172.21.15.129:0/4198548260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd8107a00 0x7ffbd8101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.087+0000 7ffbde916640 1 -- 172.21.15.129:0/4198548260 >> 172.21.15.129:0/4198548260 conn(0x7ffbd8077100 msgr2=0x7ffbd8077540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.087+0000 7ffbde916640 1 -- 172.21.15.129:0/4198548260 shutdown_connections 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.087+0000 7ffbde916640 1 -- 172.21.15.129:0/4198548260 wait complete. 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.087+0000 7ffbde916640 1 Processor -- start 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.087+0000 7ffbde916640 1 -- start start 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbde916640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbd8101ad0 0x7ffbd819a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbde916640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbd8107a00 0x7ffbd819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbde916640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd810c360 0x7ffbd81a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbde916640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd8111d30 con 0x7ffbd8107a00 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbde916640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd8111bb0 con 0x7ffbd810c360 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbde916640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd8111eb0 con 0x7ffbd8101ad0 2024-05-20T18:11:28.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbd8107a00 0x7ffbd819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd77fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbd8107a00 0x7ffbd819aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35032/0 (socket says 172.21.15.129:35032) 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd77fe640 1 -- 172.21.15.129:0/1034829787 learned_addr learned my addr 172.21.15.129:0/1034829787 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd7fff640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbd8101ad0 0x7ffbd819a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd77fe640 1 -- 172.21.15.129:0/1034829787 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbd8101ad0 msgr2=0x7ffbd819a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbdceb3640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd810c360 0x7ffbd81a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd77fe640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbd8101ad0 0x7ffbd819a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd77fe640 1 -- 172.21.15.129:0/1034829787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd810c360 msgr2=0x7ffbd81a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd77fe640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd810c360 0x7ffbd81a55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd77fe640 1 -- 172.21.15.129:0/1034829787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd819b310 con 0x7ffbd8107a00 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd77fe640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbd8107a00 0x7ffbd819aa90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc00c970 tx=0x7ffbcc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd57fa640 1 -- 172.21.15.129:0/1034829787 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc004220 con 0x7ffbd8107a00 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd57fa640 1 -- 172.21.15.129:0/1034829787 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbcc0043c0 con 0x7ffbd8107a00 2024-05-20T18:11:28.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbde916640 1 -- 172.21.15.129:0/1034829787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd81a5bd0 con 0x7ffbd8107a00 2024-05-20T18:11:28.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.088+0000 7ffbd57fa640 1 -- 172.21.15.129:0/1034829787 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc004a00 con 0x7ffbd8107a00 2024-05-20T18:11:28.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.089+0000 7ffbde916640 1 -- 172.21.15.129:0/1034829787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd81a60d0 con 0x7ffbd8107a00 2024-05-20T18:11:28.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.089+0000 7ffbde916640 1 -- 172.21.15.129:0/1034829787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbd8069110 con 0x7ffbd8107a00 2024-05-20T18:11:28.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.091+0000 7ffbd57fa640 1 -- 172.21.15.129:0/1034829787 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 137965+0+0 (secure 0 0 0) 0x7ffbcc0079b0 con 0x7ffbd8107a00 2024-05-20T18:11:28.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.091+0000 7ffbd57fa640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffbc00a2d00 0x7ffbc00a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-20T18:11:28.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.091+0000 7ffbd7fff640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffbc00a2d00 0x7ffbc00a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-20T18:11:28.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.092+0000 7ffbd57fa640 1 -- 172.21.15.129:0/1034829787 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8067+0+0 (secure 0 0 0) 0x7ffbcc0cf0e0 con 0x7ffbd8107a00 2024-05-20T18:11:28.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.092+0000 7ffbd7fff640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffbc00a2d00 0x7ffbc00a5210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8102a00 tx=0x7ffbc8005f90 comp rx=0 tx=0).ready entity=mgr.15228 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-20T18:11:28.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.096+0000 7ffbd57fa640 1 -- 172.21.15.129:0/1034829787 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffbcc014030 con 0x7ffbd8107a00 2024-05-20T18:11:28.161 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:27 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1548548117' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-20T18:11:28.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.273+0000 7ffbde916640 1 -- 172.21.15.129:0/1034829787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ffbd802d2a0 con 0x7ffbd8107a00 2024-05-20T18:11:28.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:27 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1548548117' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-20T18:11:28.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:27 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1548548117' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-20T18:11:28.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.960+0000 7ffbd57fa640 1 -- 172.21.15.129:0/1034829787 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v112) ==== 169+0+0 (secure 0 0 0) 0x7ffbcc09bf10 con 0x7ffbd8107a00 2024-05-20T18:11:28.958 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-05-20T18:11:28.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.966+0000 7ffbde916640 1 -- 172.21.15.129:0/1034829787 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffbc00a2d00 msgr2=0x7ffbc00a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:28.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.966+0000 7ffbde916640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffbc00a2d00 0x7ffbc00a5210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8102a00 tx=0x7ffbc8005f90 comp rx=0 tx=0).stop 2024-05-20T18:11:28.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.966+0000 7ffbde916640 1 -- 172.21.15.129:0/1034829787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbd8107a00 msgr2=0x7ffbd819aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-20T18:11:28.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.966+0000 7ffbde916640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbd8107a00 0x7ffbd819aa90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc00c970 tx=0x7ffbcc00ce30 comp rx=0 tx=0).stop 2024-05-20T18:11:28.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.966+0000 7ffbde916640 1 -- 172.21.15.129:0/1034829787 shutdown_connections 2024-05-20T18:11:28.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.966+0000 7ffbde916640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.129:6800/3515662734,v1:172.21.15.129:6801/3515662734] conn(0x7ffbc00a2d00 0x7ffbc00a5210 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:28.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.966+0000 7ffbde916640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbd810c360 0x7ffbd81a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:28.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.966+0000 7ffbde916640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbd8107a00 0x7ffbd819aa90 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:28.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.966+0000 7ffbde916640 1 --2- 172.21.15.129:0/1034829787 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ffbd8101ad0 0x7ffbd819a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-20T18:11:28.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.966+0000 7ffbde916640 1 -- 172.21.15.129:0/1034829787 >> 172.21.15.129:0/1034829787 conn(0x7ffbd8077100 msgr2=0x7ffbd8103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-20T18:11:28.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.966+0000 7ffbde916640 1 -- 172.21.15.129:0/1034829787 shutdown_connections 2024-05-20T18:11:28.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-05-20T18:11:28.967+0000 7ffbde916640 1 -- 172.21.15.129:0/1034829787 wait complete. 2024-05-20T18:11:28.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-20T18:11:29.024 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-20T18:11:29.056 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-20T18:11:29.087 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-20T18:11:29.087 DEBUG:teuthology.orchestra.run.smithi129:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-20T18:11:29.104 DEBUG:teuthology.orchestra.run.smithi154:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-20T18:11:29.120 DEBUG:teuthology.orchestra.run.smithi204:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-20T18:11:29.143 INFO:tasks.cephadm:Stopping all daemons... 2024-05-20T18:11:29.143 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-20T18:11:29.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.a 2024-05-20T18:11:29.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:28 smithi154 ceph-mon[36482]: pgmap v409: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-20T18:11:29.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:28 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1548548117' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-20T18:11:29.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:28 smithi154 ceph-mon[36482]: osdmap e111: 12 total, 12 up, 12 in 2024-05-20T18:11:29.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:28 smithi154 ceph-mon[36482]: from='client.? 172.21.15.129:0/1034829787' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-20T18:11:29.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:28 smithi129 ceph-mon[32029]: pgmap v409: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-20T18:11:29.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:28 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1548548117' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-20T18:11:29.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:28 smithi129 ceph-mon[32029]: osdmap e111: 12 total, 12 up, 12 in 2024-05-20T18:11:29.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:28 smithi129 ceph-mon[32029]: from='client.? 172.21.15.129:0/1034829787' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-20T18:11:29.411 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:29 smithi129 systemd[1]: Stopping Ceph mon.a for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:11:29.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:28 smithi204 ceph-mon[37076]: pgmap v409: 97 pgs: 97 active+clean; 12 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-05-20T18:11:29.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:28 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1548548117' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-20T18:11:29.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:28 smithi204 ceph-mon[37076]: osdmap e111: 12 total, 12 up, 12 in 2024-05-20T18:11:29.428 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:28 smithi204 ceph-mon[37076]: from='client.? 172.21.15.129:0/1034829787' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-20T18:11:29.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:29 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a[32025]: 2024-05-20T18:11:29.628+0000 7f2c9b400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:11:29.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:29 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a[32025]: 2024-05-20T18:11:29.628+0000 7f2c9b400700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-20T18:11:29.911 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:29 smithi129 podman[96735]: 2024-05-20 18:11:29.691904013 +0000 UTC m=+0.212196261 container died 532f464357bd499cee617b2abc843bf027ef1bdd71505be37b3d46216e1aa8d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-20T18:11:30.376 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:30 smithi129 podman[96735]: 2024-05-20 18:11:30.102201086 +0000 UTC m=+0.622493328 container cleanup 532f464357bd499cee617b2abc843bf027ef1bdd71505be37b3d46216e1aa8d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240514) 2024-05-20T18:11:30.376 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:30 smithi129 bash[96735]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a 2024-05-20T18:11:30.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.a.service' 2024-05-20T18:11:30.639 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:30 smithi129 podman[96748]: 2024-05-20 18:11:30.379266383 +0000 UTC m=+0.683262442 container remove 532f464357bd499cee617b2abc843bf027ef1bdd71505be37b3d46216e1aa8d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-a, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-20T18:11:30.639 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:30 smithi129 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-20T18:11:30.639 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:30 smithi129 systemd[1]: Stopped Ceph mon.a for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T18:11:30.640 INFO:journalctl@ceph.mon.a.smithi129.stdout:May 20 18:11:30 smithi129 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.a.service: Consumed 27.126s CPU time. 2024-05-20T18:11:30.658 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:11:30.658 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-20T18:11:30.658 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-20T18:11:30.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.b 2024-05-20T18:11:30.997 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:30 smithi154 systemd[1]: Stopping Ceph mon.b for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:11:31.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:31 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-b[36478]: 2024-05-20T18:11:31.000+0000 7fefe203b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:11:31.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:31 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-b[36478]: 2024-05-20T18:11:31.000+0000 7fefe203b700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-20T18:11:31.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:31 smithi154 podman[72975]: 2024-05-20 18:11:31.052503197 +0000 UTC m=+0.185936529 container died d52372016c01d90b428e254277ef200845bb0fc7bd9b5b61523704eaa2c3b22a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-b, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514) 2024-05-20T18:11:31.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:31 smithi154 podman[72975]: 2024-05-20 18:11:31.417840115 +0000 UTC m=+0.551273434 container cleanup d52372016c01d90b428e254277ef200845bb0fc7bd9b5b61523704eaa2c3b22a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-20T18:11:31.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:May 20 18:11:31 smithi154 bash[72975]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-b 2024-05-20T18:11:31.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.b.service' 2024-05-20T18:11:31.906 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:11:31.907 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-20T18:11:31.907 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-20T18:11:31.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.c 2024-05-20T18:11:32.222 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:31 smithi204 systemd[1]: Stopping Ceph mon.c for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:11:32.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:32 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-c[37059]: 2024-05-20T18:11:32.225+0000 7fb530517700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:11:32.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:32 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-c[37059]: 2024-05-20T18:11:32.225+0000 7fb530517700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-20T18:11:32.678 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:32 smithi204 podman[74015]: 2024-05-20 18:11:32.440567354 +0000 UTC m=+0.344354493 container died 9a5d5ce86e415ed75bde17d5e1a4226712af369d99856cd846d0e19805eccaa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-05-20T18:11:33.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:32 smithi204 podman[74015]: 2024-05-20 18:11:32.934857556 +0000 UTC m=+0.838644693 container cleanup 9a5d5ce86e415ed75bde17d5e1a4226712af369d99856cd846d0e19805eccaa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-20T18:11:33.178 INFO:journalctl@ceph.mon.c.smithi204.stdout:May 20 18:11:32 smithi204 bash[74015]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mon-c 2024-05-20T18:11:33.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mon.c.service' 2024-05-20T18:11:33.402 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:11:33.402 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-20T18:11:33.402 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-20T18:11:33.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.x 2024-05-20T18:11:33.755 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 18:11:33 smithi129 systemd[1]: Stopping Ceph mgr.x for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:11:34.106 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 18:11:33 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T18:11:33.757+0000 7fdde9c47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:11:34.106 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 18:11:33 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x[32254]: 2024-05-20T18:11:33.757+0000 7fdde9c47700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-20T18:11:34.106 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 18:11:33 smithi129 podman[96853]: 2024-05-20 18:11:33.799870601 +0000 UTC m=+0.202594413 container died 6908fde9dc21654a293de409c9376bca8ba2c350f4f114c0f06a584285cfb6c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-20T18:11:34.411 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 18:11:34 smithi129 podman[96853]: 2024-05-20 18:11:34.200562166 +0000 UTC m=+0.603285977 container cleanup 6908fde9dc21654a293de409c9376bca8ba2c350f4f114c0f06a584285cfb6c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2b507c724ef6c8295ee0a35d185cdab167fd61d, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-20T18:11:34.411 INFO:journalctl@ceph.mgr.x.smithi129.stdout:May 20 18:11:34 smithi129 bash[96853]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-x 2024-05-20T18:11:34.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.x.service' 2024-05-20T18:11:34.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:11:34.596 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-20T18:11:34.596 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-20T18:11:34.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.y 2024-05-20T18:11:34.891 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 18:11:34 smithi154 systemd[1]: Stopping Ceph mgr.y for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:11:35.232 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 18:11:34 smithi154 podman[73107]: 2024-05-20 18:11:34.945400916 +0000 UTC m=+0.161227028 container died f3b4f8606f46b99edb408c628588263e44f0446995edc78e3899700acd42ba62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-20T18:11:35.604 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 18:11:35 smithi154 podman[73107]: 2024-05-20 18:11:35.331768066 +0000 UTC m=+0.547594164 container cleanup f3b4f8606f46b99edb408c628588263e44f0446995edc78e3899700acd42ba62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514) 2024-05-20T18:11:35.605 INFO:journalctl@ceph.mgr.y.smithi154.stdout:May 20 18:11:35 smithi154 bash[73107]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-y 2024-05-20T18:11:35.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.y.service' 2024-05-20T18:11:35.762 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:11:35.762 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-20T18:11:35.762 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-20T18:11:35.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.z 2024-05-20T18:11:36.109 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 18:11:35 smithi204 systemd[1]: Stopping Ceph mgr.z for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:11:36.380 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 18:11:36 smithi204 podman[74137]: 2024-05-20 18:11:36.113382811 +0000 UTC m=+0.150366476 container died bd030ed21de2a015e08e5965ab244e7b16996c1c7dd0eb5400edb526c3ac92d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux ) 2024-05-20T18:11:36.678 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 18:11:36 smithi204 podman[74137]: 2024-05-20 18:11:36.475271319 +0000 UTC m=+0.512254982 container cleanup bd030ed21de2a015e08e5965ab244e7b16996c1c7dd0eb5400edb526c3ac92d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=) 2024-05-20T18:11:36.678 INFO:journalctl@ceph.mgr.z.smithi204.stdout:May 20 18:11:36 smithi204 bash[74137]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mgr-z 2024-05-20T18:11:36.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mgr.z.service' 2024-05-20T18:11:36.940 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:11:36.940 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-20T18:11:36.940 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-20T18:11:36.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.0 2024-05-20T18:11:37.247 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:36 smithi129 systemd[1]: Stopping Ceph osd.0 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:11:37.661 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:37 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0[44123]: 2024-05-20T18:11:37.250+0000 7f4259e06700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:11:37.661 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:37 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0[44123]: 2024-05-20T18:11:37.250+0000 7f4259e06700 -1 osd.0 112 *** Got signal Terminated *** 2024-05-20T18:11:37.661 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:37 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0[44123]: 2024-05-20T18:11:37.250+0000 7f4259e06700 -1 osd.0 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:11:43.004 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:42 smithi129 podman[96975]: 2024-05-20 18:11:42.715675774 +0000 UTC m=+5.583340842 container died d05547a54800a2eacaa9f417423648419885f60816b46bafb330588ae17b266d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-20T18:11:43.411 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:43 smithi129 podman[96975]: 2024-05-20 18:11:43.106326735 +0000 UTC m=+5.973991795 container cleanup d05547a54800a2eacaa9f417423648419885f60816b46bafb330588ae17b266d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux ) 2024-05-20T18:11:43.411 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:43 smithi129 bash[96975]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0 2024-05-20T18:11:43.897 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:43 smithi129 podman[96992]: 2024-05-20 18:11:43.598014639 +0000 UTC m=+0.865527057 container remove d05547a54800a2eacaa9f417423648419885f60816b46bafb330588ae17b266d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:11:44.159 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:43 smithi129 podman[97081]: 2024-05-20 18:11:43.900854161 +0000 UTC m=+0.090758188 container create 984fde6f63c55be6dd5115e95aa8e4c775e1070f0b973d098cd1156f4220f36a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-20T18:11:44.159 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:43 smithi129 podman[97081]: 2024-05-20 18:11:43.826859957 +0000 UTC m=+0.016763962 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:11:44.159 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:44 smithi129 podman[97081]: 2024-05-20 18:11:44.162435535 +0000 UTC m=+0.352339540 container init 984fde6f63c55be6dd5115e95aa8e4c775e1070f0b973d098cd1156f4220f36a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-20T18:11:44.411 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:44 smithi129 podman[97081]: 2024-05-20 18:11:44.168996361 +0000 UTC m=+0.358900366 container start 984fde6f63c55be6dd5115e95aa8e4c775e1070f0b973d098cd1156f4220f36a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:11:44.411 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:44 smithi129 podman[97081]: 2024-05-20 18:11:44.279353025 +0000 UTC m=+0.469257035 container attach 984fde6f63c55be6dd5115e95aa8e4c775e1070f0b973d098cd1156f4220f36a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-20T18:11:44.988 INFO:journalctl@ceph.osd.0.smithi129.stdout:May 20 18:11:44 smithi129 podman[97081]: 2024-05-20 18:11:44.672893129 +0000 UTC m=+0.862797135 container died 984fde6f63c55be6dd5115e95aa8e4c775e1070f0b973d098cd1156f4220f36a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-20T18:11:45.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.0.service' 2024-05-20T18:11:45.306 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:11:45.306 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-20T18:11:45.307 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-20T18:11:45.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.1 2024-05-20T18:11:45.632 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:45 smithi154 systemd[1]: Stopping Ceph osd.1 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:11:45.891 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:45 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1[41040]: 2024-05-20T18:11:45.635+0000 7f0e62c6e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:11:45.891 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:45 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1[41040]: 2024-05-20T18:11:45.635+0000 7f0e62c6e700 -1 osd.1 112 *** Got signal Terminated *** 2024-05-20T18:11:45.892 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:45 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1[41040]: 2024-05-20T18:11:45.635+0000 7f0e62c6e700 -1 osd.1 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:11:51.392 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:51 smithi154 podman[73234]: 2024-05-20 18:11:51.121643188 +0000 UTC m=+5.612405764 container died 3370cb90079b79e5b171b61c3ad46e573c0253e5399936a1c03a16f525b7179a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T18:11:51.767 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:51 smithi154 podman[73234]: 2024-05-20 18:11:51.560857015 +0000 UTC m=+6.051619580 container cleanup 3370cb90079b79e5b171b61c3ad46e573c0253e5399936a1c03a16f525b7179a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:11:51.767 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:51 smithi154 bash[73234]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1 2024-05-20T18:11:52.141 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:51 smithi154 podman[73252]: 2024-05-20 18:11:51.770745994 +0000 UTC m=+0.641912005 container remove 3370cb90079b79e5b171b61c3ad46e573c0253e5399936a1c03a16f525b7179a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, GIT_CLEAN=True, ceph=True) 2024-05-20T18:11:52.453 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:52 smithi154 podman[73339]: 2024-05-20 18:11:52.092558862 +0000 UTC m=+0.017538764 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:11:52.453 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:52 smithi154 podman[73339]: 2024-05-20 18:11:52.205017215 +0000 UTC m=+0.129997103 container create 21a1ac8f205e2dfb5509d58cae27342540e7448556b5ecdda0598d2287f3eca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-20T18:11:52.785 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:52 smithi154 podman[73339]: 2024-05-20 18:11:52.457479952 +0000 UTC m=+0.382459831 container init 21a1ac8f205e2dfb5509d58cae27342540e7448556b5ecdda0598d2287f3eca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-20T18:11:52.785 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:52 smithi154 podman[73339]: 2024-05-20 18:11:52.460834482 +0000 UTC m=+0.385814372 container start 21a1ac8f205e2dfb5509d58cae27342540e7448556b5ecdda0598d2287f3eca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1-deactivate, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-05-20T18:11:52.786 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:52 smithi154 podman[73339]: 2024-05-20 18:11:52.565874854 +0000 UTC m=+0.490854754 container attach 21a1ac8f205e2dfb5509d58cae27342540e7448556b5ecdda0598d2287f3eca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-20T18:11:53.074 INFO:journalctl@ceph.osd.1.smithi154.stdout:May 20 18:11:52 smithi154 podman[73339]: 2024-05-20 18:11:52.78923173 +0000 UTC m=+0.714211622 container died 21a1ac8f205e2dfb5509d58cae27342540e7448556b5ecdda0598d2287f3eca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T18:11:53.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.1.service' 2024-05-20T18:11:53.440 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:11:53.440 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-20T18:11:53.441 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-20T18:11:53.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.2 2024-05-20T18:11:53.786 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:11:53 smithi204 systemd[1]: Stopping Ceph osd.2 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:11:54.178 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:11:53 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2[42150]: 2024-05-20T18:11:53.788+0000 7fa3ecef0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:11:54.178 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:11:53 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2[42150]: 2024-05-20T18:11:53.788+0000 7fa3ecef0700 -1 osd.2 112 *** Got signal Terminated *** 2024-05-20T18:11:54.178 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:11:53 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2[42150]: 2024-05-20T18:11:53.788+0000 7fa3ecef0700 -1 osd.2 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:11:59.537 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:11:59 smithi204 podman[74275]: 2024-05-20 18:11:59.21948192 +0000 UTC m=+5.573647317 container died 5edf29a0299f167e9c81849e2eb772f15f4e3ffd97ca92793db5ed8c1ddbb944 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-20T18:11:59.884 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:11:59 smithi204 podman[74275]: 2024-05-20 18:11:59.622869582 +0000 UTC m=+5.977034976 container cleanup 5edf29a0299f167e9c81849e2eb772f15f4e3ffd97ca92793db5ed8c1ddbb944 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-20T18:11:59.884 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:11:59 smithi204 bash[74275]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2 2024-05-20T18:12:00.178 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:11:59 smithi204 podman[74291]: 2024-05-20 18:11:59.887878938 +0000 UTC m=+0.657894702 container remove 5edf29a0299f167e9c81849e2eb772f15f4e3ffd97ca92793db5ed8c1ddbb944 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-05-20T18:12:00.554 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:12:00 smithi204 podman[74382]: 2024-05-20 18:12:00.14082273 +0000 UTC m=+0.017076721 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:12:00.555 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:12:00 smithi204 podman[74382]: 2024-05-20 18:12:00.26378684 +0000 UTC m=+0.140040819 container create 0383bf68ba62272a4cf07098a1c7754790e428d1fc519dba4e67a8df5877d40c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T18:12:00.845 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:12:00 smithi204 podman[74382]: 2024-05-20 18:12:00.558170034 +0000 UTC m=+0.434424018 container init 0383bf68ba62272a4cf07098a1c7754790e428d1fc519dba4e67a8df5877d40c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:12:00.845 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:12:00 smithi204 podman[74382]: 2024-05-20 18:12:00.561489682 +0000 UTC m=+0.437743668 container start 0383bf68ba62272a4cf07098a1c7754790e428d1fc519dba4e67a8df5877d40c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-20T18:12:00.845 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:12:00 smithi204 podman[74382]: 2024-05-20 18:12:00.650093969 +0000 UTC m=+0.526347967 container attach 0383bf68ba62272a4cf07098a1c7754790e428d1fc519dba4e67a8df5877d40c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-20T18:12:01.149 INFO:journalctl@ceph.osd.2.smithi204.stdout:May 20 18:12:00 smithi204 podman[74382]: 2024-05-20 18:12:00.849209597 +0000 UTC m=+0.725463583 container died 0383bf68ba62272a4cf07098a1c7754790e428d1fc519dba4e67a8df5877d40c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-20T18:12:01.487 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.2.service' 2024-05-20T18:12:01.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:12:01.526 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-20T18:12:01.526 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-20T18:12:01.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.3 2024-05-20T18:12:01.858 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:01 smithi129 systemd[1]: Stopping Ceph osd.3 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:12:02.125 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:01 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3[51041]: 2024-05-20T18:12:01.860+0000 7f5e99928700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:12:02.125 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:01 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3[51041]: 2024-05-20T18:12:01.860+0000 7f5e99928700 -1 osd.3 112 *** Got signal Terminated *** 2024-05-20T18:12:02.125 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:01 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3[51041]: 2024-05-20T18:12:01.860+0000 7f5e99928700 -1 osd.3 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:12:03.641 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:03.345+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:03.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:03 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:03.581+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:03.929 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:03 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:03.595+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:04.641 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:04 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:04.354+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:04.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:04 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:04.614+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:04.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:04 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:04.583+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:05.411 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:04 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:04.970+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:05.641 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:05 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:05.314+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:05.911 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:05 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3[51041]: 2024-05-20T18:12:05.437+0000 7f5e95ede700 -1 osd.3 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.375597+0000 front 2024-05-20T18:11:40.375403+0000 (oldest deadline 2024-05-20T18:12:05.075362+0000) 2024-05-20T18:12:05.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:05 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:05.551+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:05.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:05 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:05.596+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:06.291 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:05 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:05.977+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:06.641 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:06 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:06.342+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:06.661 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:06 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3[51041]: 2024-05-20T18:12:06.399+0000 7f5e95ede700 -1 osd.3 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.375597+0000 front 2024-05-20T18:11:40.375403+0000 (oldest deadline 2024-05-20T18:12:05.075362+0000) 2024-05-20T18:12:06.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:06 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:06.294+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:06.829 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:06.579+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:06.829 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:06.599+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:07.280 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:06 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:06.947+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:07.538 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:07 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:07.283+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:07.641 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:07 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:07.302+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:07.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:07 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:07.373+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:07.842 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:07 smithi129 podman[97188]: 2024-05-20 18:12:07.54197629 +0000 UTC m=+5.805771229 container died 718af1f5aca88c6707349a25a5095427a166be173bfda08fa07f0ef7a660ed3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3, org.label-schema.build-date=20240514, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-20T18:12:07.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:07.610+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:07.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:07.549+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:08.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:07 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:07.932+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:08.161 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:07 smithi129 podman[97188]: 2024-05-20 18:12:07.936158105 +0000 UTC m=+6.199953046 container cleanup 718af1f5aca88c6707349a25a5095427a166be173bfda08fa07f0ef7a660ed3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c) 2024-05-20T18:12:08.161 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:07 smithi129 bash[97188]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3 2024-05-20T18:12:08.568 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:08 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:08.307+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:08.568 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:08 smithi129 podman[97215]: 2024-05-20 18:12:08.212789058 +0000 UTC m=+0.664785062 container remove 718af1f5aca88c6707349a25a5095427a166be173bfda08fa07f0ef7a660ed3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-20T18:12:08.641 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:08 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:08.272+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:08.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:08 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:08.329+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:08.829 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:08 smithi129 podman[97304]: 2024-05-20 18:12:08.475330758 +0000 UTC m=+0.018098601 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:12:08.829 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:08 smithi129 podman[97304]: 2024-05-20 18:12:08.605488143 +0000 UTC m=+0.148256013 container create 01c3bed5d0333f3784ddceabd504b2d606e6133bccd4744fe19dc95004158610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-20T18:12:08.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:08 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:08.632+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:08.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:08 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:08.548+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:09.122 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:08 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:08.961+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:09.122 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:08 smithi129 podman[97304]: 2024-05-20 18:12:08.833173431 +0000 UTC m=+0.375941283 container init 01c3bed5d0333f3784ddceabd504b2d606e6133bccd4744fe19dc95004158610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-20T18:12:09.122 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:08 smithi129 podman[97304]: 2024-05-20 18:12:08.836744607 +0000 UTC m=+0.379512452 container start 01c3bed5d0333f3784ddceabd504b2d606e6133bccd4744fe19dc95004158610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514) 2024-05-20T18:12:09.122 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:08 smithi129 podman[97304]: 2024-05-20 18:12:08.933502858 +0000 UTC m=+0.476270710 container attach 01c3bed5d0333f3784ddceabd504b2d606e6133bccd4744fe19dc95004158610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=) 2024-05-20T18:12:09.411 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:09 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:09.321+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:09.411 INFO:journalctl@ceph.osd.3.smithi129.stdout:May 20 18:12:09 smithi129 podman[97304]: 2024-05-20 18:12:09.125873936 +0000 UTC m=+0.668641779 container died 01c3bed5d0333f3784ddceabd504b2d606e6133bccd4744fe19dc95004158610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-3-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-20T18:12:09.641 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:09 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:09.232+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:09.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:09 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:09.357+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:09.772 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.3.service' 2024-05-20T18:12:09.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:12:09.813 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-20T18:12:09.813 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-20T18:12:09.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.4 2024-05-20T18:12:09.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:09 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:09.672+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:09.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:09 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:09.581+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:10.135 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:09 smithi154 systemd[1]: Stopping Ceph osd.4 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:12:10.160 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:09 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:09.973+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:10.391 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:10 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:10.362+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:10.392 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:10 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:10.138+0000 7f5324424700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:12:10.392 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:10 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:10.138+0000 7f5324424700 -1 osd.4 112 *** Got signal Terminated *** 2024-05-20T18:12:10.392 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:10 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:10.138+0000 7f5324424700 -1 osd.4 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:12:10.392 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:10 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:10.204+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:10.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:10 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:10.355+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:10.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:10 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:10.644+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:10.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:10 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:10.568+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:11.362 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:11 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:11.001+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:11.641 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:11 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:11.184+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:11.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:11 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:11.348+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:11.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:11 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:11.348+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:11.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:11 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:11.364+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:11.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:11 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:11.605+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:11.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:11 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:11.548+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:12.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:11 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:11.802+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:12.342 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:11 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:11.981+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:12.342 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:11 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:11.981+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:12.641 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:12 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:12.195+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:12.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:12 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:12.350+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:12.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:12 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:12.350+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:12.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:12 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:12.345+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:12.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:12 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:12.345+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:12.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:12 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:12.596+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:12.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:12 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:12.623+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:13.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:12 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:12.846+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:13.367 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:12 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:12.952+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:13.368 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:12 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:12.952+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:13.641 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:13 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:13.162+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:13.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:13 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:13.345+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:13.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:13 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:13.345+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:13.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:13 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:13.370+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:13.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:13 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:13.370+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:13.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:13 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:13.613+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:13.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:13 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:13.613+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.936752+0000 front 2024-05-20T18:11:46.936734+0000 (oldest deadline 2024-05-20T18:12:12.836770+0000) 2024-05-20T18:12:13.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:13 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:13.613+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:13.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:13 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:13.613+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:14.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:13 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:13.856+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:14.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:13 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:13.908+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:14.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:13 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:13.908+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:14.641 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:14 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4[47664]: 2024-05-20T18:12:14.191+0000 7f531c9d2700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:40.349923+0000 front 2024-05-20T18:11:40.349990+0000 (oldest deadline 2024-05-20T18:12:03.249955+0000) 2024-05-20T18:12:14.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:14 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:14.302+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:14.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:14 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:14.302+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:14.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:14 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:14.404+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:14.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:14 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:14.404+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:14.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:14 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:14.594+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:14.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:14 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:14.594+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.936752+0000 front 2024-05-20T18:11:46.936734+0000 (oldest deadline 2024-05-20T18:12:12.836770+0000) 2024-05-20T18:12:14.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:14 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:14.630+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:14.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:14 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:14.630+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:15.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:14 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:14.852+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:15.160 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:14 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:14.897+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:15.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:14 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:14.897+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:15.601 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:15 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:15.326+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:15.601 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:15 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:15.326+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:15.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:15 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:15.393+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:15.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:15 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:15.393+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:15.891 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:15 smithi154 podman[73455]: 2024-05-20 18:12:15.605322798 +0000 UTC m=+5.593719991 container died 3eed87ec9193f1fe0c337fb796bac6ca9f8fa89ce86393536c7c747709b6fcbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-20T18:12:15.892 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:15 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:15.839+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:15.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:15 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:15.584+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:15.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:15 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:15.584+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.936752+0000 front 2024-05-20T18:11:46.936734+0000 (oldest deadline 2024-05-20T18:12:12.836770+0000) 2024-05-20T18:12:15.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:15 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:15.667+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:15.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:15 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:15.667+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:16.252 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:15 smithi154 podman[73455]: 2024-05-20 18:12:15.996588749 +0000 UTC m=+5.984985936 container cleanup 3eed87ec9193f1fe0c337fb796bac6ca9f8fa89ce86393536c7c747709b6fcbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.vendor=CentOS) 2024-05-20T18:12:16.252 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:15 smithi154 bash[73455]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4 2024-05-20T18:12:16.411 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:15 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:15.918+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:16.411 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:15 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:15.918+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:16.428 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:15 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:15.970+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:16.568 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:16 smithi154 podman[73473]: 2024-05-20 18:12:16.256089434 +0000 UTC m=+0.644742429 container remove 3eed87ec9193f1fe0c337fb796bac6ca9f8fa89ce86393536c7c747709b6fcbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-20T18:12:16.568 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:16 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:16.370+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:16.568 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:16 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:16.370+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:16.831 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:16 smithi154 podman[73561]: 2024-05-20 18:12:16.474833262 +0000 UTC m=+0.016743954 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:12:16.831 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:16 smithi154 podman[73561]: 2024-05-20 18:12:16.582089449 +0000 UTC m=+0.124000150 container create 2dee5051c8e860e93b0db45a896ce5fa8ba3ccf3c4a27985d319c86f2b944ccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-20T18:12:16.910 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:16 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:16.438+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:16.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:16 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:16.438+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:16.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:16 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:16.562+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:16.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:16 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:16.562+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.936752+0000 front 2024-05-20T18:11:46.936734+0000 (oldest deadline 2024-05-20T18:12:12.836770+0000) 2024-05-20T18:12:16.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:16 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:16.699+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:16.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:16 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:16.699+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:17.136 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:16 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:16.854+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:17.136 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:16 smithi154 podman[73561]: 2024-05-20 18:12:16.834729819 +0000 UTC m=+0.376640498 container init 2dee5051c8e860e93b0db45a896ce5fa8ba3ccf3c4a27985d319c86f2b944ccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-20T18:12:17.137 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:16 smithi154 podman[73561]: 2024-05-20 18:12:16.837803684 +0000 UTC m=+0.379714367 container start 2dee5051c8e860e93b0db45a896ce5fa8ba3ccf3c4a27985d319c86f2b944ccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-20T18:12:17.137 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:16 smithi154 podman[73561]: 2024-05-20 18:12:16.934696831 +0000 UTC m=+0.476607525 container attach 2dee5051c8e860e93b0db45a896ce5fa8ba3ccf3c4a27985d319c86f2b944ccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.license=GPLv2) 2024-05-20T18:12:17.386 INFO:journalctl@ceph.osd.4.smithi154.stdout:May 20 18:12:17 smithi154 podman[73561]: 2024-05-20 18:12:17.140370648 +0000 UTC m=+0.682281330 container died 2dee5051c8e860e93b0db45a896ce5fa8ba3ccf3c4a27985d319c86f2b944ccc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True) 2024-05-20T18:12:17.400 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:16 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:16.928+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:17.400 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:16 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:16.928+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:17.428 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:16 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:16.933+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:17.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:17 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:17.389+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:17.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:17 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:17.389+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:17.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:17 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:17.403+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:17.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:17 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:17.403+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:17.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.4.service' 2024-05-20T18:12:17.853 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:12:17.853 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-20T18:12:17.853 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-20T18:12:17.853 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.5 2024-05-20T18:12:17.880 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:17 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:17.607+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:17.880 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:17 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:17.607+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.936752+0000 front 2024-05-20T18:11:46.936734+0000 (oldest deadline 2024-05-20T18:12:12.836770+0000) 2024-05-20T18:12:17.880 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:17 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:17.661+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:17.880 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:17 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:17.661+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:18.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:17 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:17.835+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:18.171 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:17 smithi204 systemd[1]: Stopping Ceph osd.5 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:12:18.171 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:17 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:17.892+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:18.351 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:17 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:17.960+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:18.351 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:17 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:17.960+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:18.428 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:18 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:18.174+0000 7ff859128700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:12:18.428 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:18 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:18.174+0000 7ff859128700 -1 osd.5 112 *** Got signal Terminated *** 2024-05-20T18:12:18.428 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:18 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:18.174+0000 7ff859128700 -1 osd.5 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:12:18.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:18 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:18.354+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:18.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:18 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:18.354+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:18.868 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:18 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:18.420+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:18.869 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:18 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:18.420+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:18.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:18 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:18.673+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:18.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:18 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:18.673+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:18.929 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:18 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:18.644+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:18.929 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:18 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:18.644+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.936752+0000 front 2024-05-20T18:11:46.936734+0000 (oldest deadline 2024-05-20T18:12:12.836770+0000) 2024-05-20T18:12:18.929 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:18 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:18.878+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:19.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:18 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:18.872+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:19.332 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:18 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:18.943+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:19.332 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:18 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:18.943+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:19.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:19 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:19.373+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:19.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:19 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:19.373+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:19.662 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:19 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:19.335+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:19.662 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:19 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:19.335+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:19.928 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:19 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:19.862+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:19.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:19 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:19.670+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:19.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:19 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:19.670+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.936752+0000 front 2024-05-20T18:11:46.936734+0000 (oldest deadline 2024-05-20T18:12:12.836770+0000) 2024-05-20T18:12:19.929 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:19 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:19.715+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:19.929 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:19 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:19.715+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:20.361 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:19 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:19.953+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:20.362 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:19 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:19.953+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:20.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:19 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:19.910+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:20.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:19 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:19.910+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:20.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:20.365+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:20.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:20.365+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:20.863 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:20 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:20.405+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:20.863 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:20 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:20.405+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:20.928 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:20 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:20.871+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:20.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:20 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:20.633+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:20.928 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:20 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:20.633+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.936752+0000 front 2024-05-20T18:11:46.936734+0000 (oldest deadline 2024-05-20T18:12:12.836770+0000) 2024-05-20T18:12:20.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:20 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:20.702+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:20.929 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:20 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:20.702+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:21.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:20 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:20.866+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:21.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:20 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:20.866+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:21.386 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:20.970+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:21.386 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:20 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:20.970+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:21.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:21 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:21.362+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:21.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:21 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:21.362+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:21.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:21 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:21.389+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:21.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:21 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:21.389+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:21.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:21 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:21.725+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:21.882 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:21 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:21.725+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:21.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:21 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:21.587+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:21.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:21 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:21.587+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.936752+0000 front 2024-05-20T18:11:46.936734+0000 (oldest deadline 2024-05-20T18:12:12.836770+0000) 2024-05-20T18:12:21.882 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:21 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:21.587+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:55.038247+0000 front 2024-05-20T18:11:55.038319+0000 (oldest deadline 2024-05-20T18:12:20.938359+0000) 2024-05-20T18:12:22.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:21 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:21.833+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:22.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:21 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:21.833+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:22.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:21 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:21.884+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:22.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:21 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:21.884+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:22.364 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:21 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:21.936+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:22.365 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:21 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:21.936+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:22.365 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:21 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:21.936+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.152393+0000 front 2024-05-20T18:11:58.152199+0000 (oldest deadline 2024-05-20T18:12:21.652053+0000) 2024-05-20T18:12:22.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:22 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:22.354+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:22.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:22 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:22.354+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:22.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:22 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:22.368+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:22.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:22 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:22.368+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:22.856 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:22 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:22.691+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:22.860 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:22 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:22.691+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:22.860 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:22 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:22.544+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.435338+0000 front 2024-05-20T18:11:39.435497+0000 (oldest deadline 2024-05-20T18:12:02.935282+0000) 2024-05-20T18:12:22.860 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:22 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:22.544+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.936752+0000 front 2024-05-20T18:11:46.936734+0000 (oldest deadline 2024-05-20T18:12:12.836770+0000) 2024-05-20T18:12:22.860 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:22 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5[49089]: 2024-05-20T18:12:22.544+0000 7ff8516d6700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:55.038247+0000 front 2024-05-20T18:11:55.038319+0000 (oldest deadline 2024-05-20T18:12:20.938359+0000) 2024-05-20T18:12:23.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:22 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:22.823+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:23.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:22 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:22.823+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:23.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:22 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:22.889+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:23.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:22 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:22.889+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:23.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:22 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:22.889+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.152393+0000 front 2024-05-20T18:11:58.152199+0000 (oldest deadline 2024-05-20T18:12:21.652053+0000) 2024-05-20T18:12:23.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:22 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:22.858+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:23.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:22 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:22.858+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:23.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:23 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:23.377+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:23.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:23 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:23.377+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:23.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:23 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:23.330+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:23.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:23 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:23.330+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:23.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:23 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:23.330+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:23.905 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:23 smithi204 podman[74499]: 2024-05-20 18:12:23.645327812 +0000 UTC m=+5.586063629 container died 77ed5e1c6f5a0bdc4e87f3259d48b7b462587b7e57852251309b7d17da52c204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:12:23.905 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:23 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:23.810+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:23.905 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:23 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:23.810+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:23.906 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:23 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:23.642+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:23.906 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:23 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:23.642+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:24.142 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:23 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:23.866+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:24.142 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:23 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:23.866+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:24.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:23 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:23.862+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:24.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:23 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:23.862+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:24.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:23 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:23.862+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.152393+0000 front 2024-05-20T18:11:58.152199+0000 (oldest deadline 2024-05-20T18:12:21.652053+0000) 2024-05-20T18:12:24.178 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:23 smithi204 podman[74499]: 2024-05-20 18:12:23.991604316 +0000 UTC m=+5.932340122 container cleanup 77ed5e1c6f5a0bdc4e87f3259d48b7b462587b7e57852251309b7d17da52c204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c) 2024-05-20T18:12:24.178 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:23 smithi204 bash[74499]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5 2024-05-20T18:12:24.611 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:24 smithi204 podman[74515]: 2024-05-20 18:12:24.201318754 +0000 UTC m=+0.552538141 container remove 77ed5e1c6f5a0bdc4e87f3259d48b7b462587b7e57852251309b7d17da52c204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-05-20T18:12:24.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:24 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:24.363+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:24.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:24 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:24.363+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:24.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:24 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:24.363+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:24.830 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:24 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:24.427+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:24.831 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:24 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:24.427+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:24.831 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:24 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:24.427+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:24.892 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:24 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:24.791+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:24.893 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:24 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:24.791+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:24.893 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:24 smithi204 podman[74607]: 2024-05-20 18:12:24.517619483 +0000 UTC m=+0.017291792 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:12:24.893 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:24 smithi204 podman[74607]: 2024-05-20 18:12:24.63587785 +0000 UTC m=+0.135550186 container create 6b6ba85d60fd0d2072bd78a0d42537d3016b59ef2d4425f3b2efac89888f09b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-20T18:12:24.893 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:24 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:24.684+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:24.893 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:24 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:24.684+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:24.893 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:24 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:24.684+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:25.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:24 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:24.834+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:25.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:24 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:24.834+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:25.160 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:24 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:24.878+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:25.160 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:24 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:24.878+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:25.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:24 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:24.878+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.152393+0000 front 2024-05-20T18:11:58.152199+0000 (oldest deadline 2024-05-20T18:12:21.652053+0000) 2024-05-20T18:12:25.178 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:24 smithi204 podman[74607]: 2024-05-20 18:12:24.896704607 +0000 UTC m=+0.396376898 container init 6b6ba85d60fd0d2072bd78a0d42537d3016b59ef2d4425f3b2efac89888f09b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T18:12:25.178 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:24 smithi204 podman[74607]: 2024-05-20 18:12:24.900198527 +0000 UTC m=+0.399870824 container start 6b6ba85d60fd0d2072bd78a0d42537d3016b59ef2d4425f3b2efac89888f09b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-05-20T18:12:25.178 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:24 smithi204 podman[74607]: 2024-05-20 18:12:24.996823125 +0000 UTC m=+0.496495429 container attach 6b6ba85d60fd0d2072bd78a0d42537d3016b59ef2d4425f3b2efac89888f09b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-20T18:12:25.471 INFO:journalctl@ceph.osd.5.smithi204.stdout:May 20 18:12:25 smithi204 podman[74607]: 2024-05-20 18:12:25.195571223 +0000 UTC m=+0.695243522 container died 6b6ba85d60fd0d2072bd78a0d42537d3016b59ef2d4425f3b2efac89888f09b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-20T18:12:25.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:25 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:25.388+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:25.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:25 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:25.388+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:25.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:25 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:25.388+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:25.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:25 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:25.335+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:25.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:25 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:25.335+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:25.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:25 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:25.335+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:25.747 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:25 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:25.704+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:25.747 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:25 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:25.704+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:25.747 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:25 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:25.704+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:25.770 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.5.service' 2024-05-20T18:12:25.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:12:25.852 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-20T18:12:25.852 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-20T18:12:25.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.6 2024-05-20T18:12:26.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:25 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:25.828+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:26.142 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:25 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:25.828+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:26.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:25 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:25.895+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:26.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:25 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:25.896+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:26.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:25 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:25.896+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.152393+0000 front 2024-05-20T18:11:58.152199+0000 (oldest deadline 2024-05-20T18:12:21.652053+0000) 2024-05-20T18:12:26.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:25 smithi129 systemd[1]: Stopping Ceph osd.6 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:12:26.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:25 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:25.795+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:26.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:25 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:25.795+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:26.661 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:26 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:26.190+0000 7f77ed2b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:12:26.661 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:26 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:26.190+0000 7f77ed2b1700 -1 osd.6 112 *** Got signal Terminated *** 2024-05-20T18:12:26.661 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:26 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:26.190+0000 7f77ed2b1700 -1 osd.6 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:12:26.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:26 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:26.310+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:26.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:26 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:26.310+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:26.662 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:26 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:26.310+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:26.853 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:26 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:26.424+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:26.853 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:26 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:26.424+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:26.853 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:26 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:26.424+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:27.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:26 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:26.856+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:27.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:26 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:26.856+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:27.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:26 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:26.854+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:27.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:26 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:26.854+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:27.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:26 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:26.854+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.152393+0000 front 2024-05-20T18:11:58.152199+0000 (oldest deadline 2024-05-20T18:12:21.652053+0000) 2024-05-20T18:12:27.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:26 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:26.840+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:27.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:26 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:26.840+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:27.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:26 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:26.687+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:27.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:26 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:26.687+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:27.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:26 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:26.687+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:27.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:27.319+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:27.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:27.319+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:27.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:27.319+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:27.823 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:27 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:27.434+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:27.823 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:27 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:27.434+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:27.823 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:27 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:27.434+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:28.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:27 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:27.825+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:28.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:27 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:27.825+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:28.160 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:27.820+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:28.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:27.820+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:28.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:27.820+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.152393+0000 front 2024-05-20T18:11:58.152199+0000 (oldest deadline 2024-05-20T18:12:21.652053+0000) 2024-05-20T18:12:28.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:27 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:27.802+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:28.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:27 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:27.802+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:28.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:27 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:27.685+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:28.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:27 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:27.685+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:28.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:27 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:27.685+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:28.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:28 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:28.365+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:28.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:28 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:28.365+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:28.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:28 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:28.365+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:28.802 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:28 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:28.478+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:28.802 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:28 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:28.478+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:28.802 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:28 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:28.478+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:29.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:28 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:28.805+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:29.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:28 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:28.805+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:29.160 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:28 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:28.816+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:29.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:28 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:28.816+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:29.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:28 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:28.816+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.152393+0000 front 2024-05-20T18:11:58.152199+0000 (oldest deadline 2024-05-20T18:12:21.652053+0000) 2024-05-20T18:12:29.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:28 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:28.816+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.353085+0000 front 2024-05-20T18:12:06.353120+0000 (oldest deadline 2024-05-20T18:12:28.653203+0000) 2024-05-20T18:12:29.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:28 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:28.814+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:29.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:28 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:28.814+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:29.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:28 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:28.728+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:29.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:28 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:28.728+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:29.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:28 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:28.728+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:29.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:29 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:29.334+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:29.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:29 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:29.334+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:29.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:29 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:29.334+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:29.819 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:29 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:29.480+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:29.819 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:29 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:29.480+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:29.819 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:29 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:29.480+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:30.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:29 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:29.822+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:30.142 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:29 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:29.822+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:30.142 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:29 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:29.822+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:04.058645+0000 front 2024-05-20T18:12:04.058839+0000 (oldest deadline 2024-05-20T18:12:29.358620+0000) 2024-05-20T18:12:30.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:29 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:29.861+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:30.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:29 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:29.861+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:30.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:29 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:29.861+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.152393+0000 front 2024-05-20T18:11:58.152199+0000 (oldest deadline 2024-05-20T18:12:21.652053+0000) 2024-05-20T18:12:30.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:29 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:29.861+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.353085+0000 front 2024-05-20T18:12:06.353120+0000 (oldest deadline 2024-05-20T18:12:28.653203+0000) 2024-05-20T18:12:30.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:29 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:29.855+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:30.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:29 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:29.855+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:30.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:29 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:29.777+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:30.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:29 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:29.777+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:30.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:29 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:29.777+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:30.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:30 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:30.338+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:30.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:30 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:30.338+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:30.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:30 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:30.338+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:30.838 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:30 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:30.527+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:30.838 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:30 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:30.527+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:30.838 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:30 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:30.527+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:31.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:30 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:30.841+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:31.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:30 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:30.841+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:31.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:30 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:30.841+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:04.058645+0000 front 2024-05-20T18:12:04.058839+0000 (oldest deadline 2024-05-20T18:12:29.358620+0000) 2024-05-20T18:12:31.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:30 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:30.889+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:38.948517+0000 front 2024-05-20T18:11:38.948547+0000 (oldest deadline 2024-05-20T18:12:04.848756+0000) 2024-05-20T18:12:31.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:30 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:30.889+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.850316+0000 front 2024-05-20T18:11:48.850456+0000 (oldest deadline 2024-05-20T18:12:11.750320+0000) 2024-05-20T18:12:31.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:30 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:30.889+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.152393+0000 front 2024-05-20T18:11:58.152199+0000 (oldest deadline 2024-05-20T18:12:21.652053+0000) 2024-05-20T18:12:31.161 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:30 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6[57840]: 2024-05-20T18:12:30.889+0000 7f77e9867700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.353085+0000 front 2024-05-20T18:12:06.353120+0000 (oldest deadline 2024-05-20T18:12:28.653203+0000) 2024-05-20T18:12:31.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:30 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:30.809+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:31.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:30 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:30.809+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:31.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:30 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:30.751+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:31.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:30 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:30.751+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:31.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:30 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:30.751+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:31.488 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:31 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:31.370+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:31.488 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:31 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:31.370+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:31.489 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:31 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:31.370+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:31.858 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:31 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:31.517+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:31.858 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:31 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:31.517+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:31.858 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:31 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:31.517+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:32.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:31 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:31.862+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:32.141 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:31 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:31.862+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:32.142 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:31 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:31.862+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:04.058645+0000 front 2024-05-20T18:12:04.058839+0000 (oldest deadline 2024-05-20T18:12:29.358620+0000) 2024-05-20T18:12:32.160 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:31 smithi129 podman[97411]: 2024-05-20 18:12:31.887952894 +0000 UTC m=+5.832875487 container died dbe7a1fde5a179d972885de713cd3b30b2ece1ec88dd9007475e8f6bb584e88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS) 2024-05-20T18:12:32.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:31 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:31.823+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:32.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:31 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:31.823+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:32.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:31 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:31.823+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:32.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:31 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:31.782+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:32.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:31 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:31.782+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:32.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:31 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:31.782+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:32.446 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:32 smithi129 podman[97411]: 2024-05-20 18:12:32.282299536 +0000 UTC m=+6.227222119 container cleanup dbe7a1fde5a179d972885de713cd3b30b2ece1ec88dd9007475e8f6bb584e88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6, org.label-schema.build-date=20240514, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True) 2024-05-20T18:12:32.446 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:32 smithi129 bash[97411]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6 2024-05-20T18:12:32.447 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:32 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:32.341+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:32.447 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:32 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:32.341+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:32.447 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:32 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:32.341+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:32.447 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:32 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:32.341+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:32.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:32 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:32.522+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:32.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:32 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:32.522+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:32.892 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:32 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:32.522+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:32.892 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:32 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:32.522+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:32.910 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:32 smithi129 podman[97435]: 2024-05-20 18:12:32.575756872 +0000 UTC m=+0.691483560 container remove dbe7a1fde5a179d972885de713cd3b30b2ece1ec88dd9007475e8f6bb584e88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-20T18:12:33.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:32 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:32.847+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:33.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:32 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:32.847+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:33.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:32 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:32.847+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:33.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:32 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:32.822+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:33.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:32 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:32.822+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:33.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:32 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:32.822+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:33.308 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:33 smithi129 podman[97530]: 2024-05-20 18:12:33.059942938 +0000 UTC m=+0.112189103 container create 8c2588c10757542cef50de10b798354f56ce7d94597da1908abab18e449735c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:12:33.309 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:33 smithi129 podman[97530]: 2024-05-20 18:12:32.965384893 +0000 UTC m=+0.017631020 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:12:33.309 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:33 smithi129 podman[97530]: 2024-05-20 18:12:33.312503098 +0000 UTC m=+0.364749233 container init 8c2588c10757542cef50de10b798354f56ce7d94597da1908abab18e449735c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6-deactivate, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-20T18:12:33.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:32 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:32.906+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:33.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:32 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:32.906+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:33.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:32 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:32.906+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:04.058645+0000 front 2024-05-20T18:12:04.058839+0000 (oldest deadline 2024-05-20T18:12:29.358620+0000) 2024-05-20T18:12:33.597 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:33 smithi129 podman[97530]: 2024-05-20 18:12:33.316203452 +0000 UTC m=+0.368449574 container start 8c2588c10757542cef50de10b798354f56ce7d94597da1908abab18e449735c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:12:33.597 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:33 smithi129 podman[97530]: 2024-05-20 18:12:33.395719202 +0000 UTC m=+0.447965335 container attach 8c2588c10757542cef50de10b798354f56ce7d94597da1908abab18e449735c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-20T18:12:33.597 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:33 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:33.313+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:33.598 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:33 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:33.313+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:33.598 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:33 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:33.313+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:33.598 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:33 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:33.314+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:33.671 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:33 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:33.497+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:33.671 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:33 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:33.497+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:33.672 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:33 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:33.497+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:33.672 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:33 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:33.497+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:33.887 INFO:journalctl@ceph.osd.6.smithi129.stdout:May 20 18:12:33 smithi129 podman[97530]: 2024-05-20 18:12:33.600942235 +0000 UTC m=+0.653188366 container died 8c2588c10757542cef50de10b798354f56ce7d94597da1908abab18e449735c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-6-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-20T18:12:34.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:33 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:33.810+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:34.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:33 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:33.810+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:34.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:33 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:33.810+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:34.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:33 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:33.867+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:34.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:33 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:33.867+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:34.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:33 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:33.867+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:34.253 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.6.service' 2024-05-20T18:12:34.295 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:12:34.295 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-20T18:12:34.296 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-20T18:12:34.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.7 2024-05-20T18:12:34.319 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:33 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:33.930+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:34.319 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:33 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:33.930+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:34.319 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:33 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:33.930+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:04.058645+0000 front 2024-05-20T18:12:04.058839+0000 (oldest deadline 2024-05-20T18:12:29.358620+0000) 2024-05-20T18:12:34.586 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:34 smithi154 systemd[1]: Stopping Ceph osd.7 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:12:34.587 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:34.467+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:34.587 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:34.467+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:34.587 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:34.467+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:34.587 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:34.467+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:34.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:34 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:34.364+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:34.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:34 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:34.364+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:34.662 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:34 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:34.364+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:34.662 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:34 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:34.364+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:34.891 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:34.588+0000 7f2f28a19700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:12:34.891 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:34.588+0000 7f2f28a19700 -1 osd.7 112 *** Got signal Terminated *** 2024-05-20T18:12:34.892 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:34.588+0000 7f2f28a19700 -1 osd.7 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:12:35.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:34 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:34.788+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:35.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:34 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:34.788+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:35.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:34 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:34.788+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:35.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:34 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:34.831+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:35.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:34 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:34.831+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:35.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:34 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:34.831+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:35.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:34.916+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:35.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:34.916+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:35.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:34.916+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:04.058645+0000 front 2024-05-20T18:12:04.058839+0000 (oldest deadline 2024-05-20T18:12:29.358620+0000) 2024-05-20T18:12:35.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:35 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:35.392+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:35.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:35 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:35.392+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:35.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:35 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:35.392+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:35.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:35 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:35.392+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:35.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:35 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:35.427+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:35.892 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:35 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:35.427+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:35.892 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:35 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:35.427+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:35.892 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:35 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:35.427+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:36.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:35 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:35.802+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:36.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:35 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:35.802+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:36.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:35 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:35.802+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:36.080 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:35 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:35.762+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:36.080 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:35 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:35.762+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:36.080 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:35 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:35.763+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:36.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:35 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:35.954+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:36.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:35 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:35.954+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:36.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:35 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:35.954+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:04.058645+0000 front 2024-05-20T18:12:04.058839+0000 (oldest deadline 2024-05-20T18:12:29.358620+0000) 2024-05-20T18:12:36.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:36 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:36.373+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:36.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:36 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:36.373+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:36.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:36 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:36.373+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:36.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:36 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:36.373+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:36.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:36.469+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:36.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:36.469+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:36.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:36.469+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:36.892 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:36.469+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:37.091 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:36.782+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:37.091 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:36.782+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:37.091 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:36.782+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:37.091 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:36.782+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.200526+0000 front 2024-05-20T18:12:11.200703+0000 (oldest deadline 2024-05-20T18:12:35.801104+0000) 2024-05-20T18:12:37.092 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:36.804+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:37.092 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:36.804+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:37.092 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:36.804+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:37.092 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:36 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:36.804+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:37.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:36.948+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:37.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:36.948+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:37.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:36 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:36.948+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:04.058645+0000 front 2024-05-20T18:12:04.058839+0000 (oldest deadline 2024-05-20T18:12:29.358620+0000) 2024-05-20T18:12:37.441 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:37 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:37.442+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:37.441 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:37 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:37.442+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:37.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:37 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:37.442+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:37.892 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:37 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:37.443+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:37.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:37 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:37.422+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:37.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:37 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:37.422+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:37.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:37 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:37.422+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:37.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:37 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:37.422+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:38.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:37.756+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:38.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:37.756+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:38.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:37.756+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:38.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:37.756+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.200526+0000 front 2024-05-20T18:12:11.200703+0000 (oldest deadline 2024-05-20T18:12:35.801104+0000) 2024-05-20T18:12:38.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:37.771+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:38.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:37.771+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:38.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:37.771+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:38.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:37 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:37.771+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:38.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:37 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:37.985+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:38.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:37 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:37.985+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:38.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:37 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:37.985+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:04.058645+0000 front 2024-05-20T18:12:04.058839+0000 (oldest deadline 2024-05-20T18:12:29.358620+0000) 2024-05-20T18:12:38.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:38 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:38.398+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:38.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:38 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:38.398+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:38.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:38 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:38.398+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:38.892 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:38 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:38.398+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:38.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:38 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:38.425+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:38.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:38 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:38.425+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:38.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:38 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:38.425+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:38.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:38 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:38.425+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:39.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:38.796+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:39.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:38.796+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:39.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:38.796+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:39.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:38.796+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:39.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:38.797+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:39.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:38.797+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:39.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:38.797+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:39.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:38 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:38.797+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.200526+0000 front 2024-05-20T18:12:11.200703+0000 (oldest deadline 2024-05-20T18:12:35.801104+0000) 2024-05-20T18:12:39.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:39 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:39.015+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:46.456108+0000 front 2024-05-20T18:11:46.456331+0000 (oldest deadline 2024-05-20T18:12:11.756274+0000) 2024-05-20T18:12:39.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:39 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:39.015+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:54.057460+0000 front 2024-05-20T18:11:54.057374+0000 (oldest deadline 2024-05-20T18:12:19.357472+0000) 2024-05-20T18:12:39.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:39 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7[54476]: 2024-05-20T18:12:39.015+0000 7f2f24fcf700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:04.058645+0000 front 2024-05-20T18:12:04.058839+0000 (oldest deadline 2024-05-20T18:12:29.358620+0000) 2024-05-20T18:12:39.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:39 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:39.407+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:39.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:39 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:39.407+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:39.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:39 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:39.407+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:39.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:39 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:39.407+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:39.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:39 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:39.414+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:39.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:39 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:39.414+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:39.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:39 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:39.414+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:39.891 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:39 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:39.414+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:39.892 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:39 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:39.414+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:40.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:39 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:39.811+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:40.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:39 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:39.811+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:40.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:39 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:39.811+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:40.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:39 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:39.811+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.200526+0000 front 2024-05-20T18:12:11.200703+0000 (oldest deadline 2024-05-20T18:12:35.801104+0000) 2024-05-20T18:12:40.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:39 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:39.783+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:40.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:39 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:39.783+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:40.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:39 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:39.783+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:40.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:39 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:39.783+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:40.470 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:40 smithi154 podman[73679]: 2024-05-20 18:12:40.195175348 +0000 UTC m=+5.685627981 container died 7c61d5d202235e3b7ffa57aaa9fd61b1dbdc6d97329023c9a33c7bc150bdba9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-20T18:12:40.470 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:40 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:40.438+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:40.471 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:40 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:40.438+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:40.471 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:40 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:40.438+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:40.471 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:40 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:40.438+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:40.471 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:40 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:40.438+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:40.820 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:40 smithi154 podman[73679]: 2024-05-20 18:12:40.581504852 +0000 UTC m=+6.071957479 container cleanup 7c61d5d202235e3b7ffa57aaa9fd61b1dbdc6d97329023c9a33c7bc150bdba9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-20T18:12:40.820 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:40 smithi154 bash[73679]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7 2024-05-20T18:12:40.910 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:40 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:40.430+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:40.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:40 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:40.430+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:40.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:40 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:40.430+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:40.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:40 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:40.430+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:41.140 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:40 smithi154 podman[73697]: 2024-05-20 18:12:40.824495262 +0000 UTC m=+0.623592907 container remove 7c61d5d202235e3b7ffa57aaa9fd61b1dbdc6d97329023c9a33c7bc150bdba9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.license=GPLv2) 2024-05-20T18:12:41.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:40.842+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:41.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:40.843+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:41.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:40.843+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:41.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:40.843+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.200526+0000 front 2024-05-20T18:12:11.200703+0000 (oldest deadline 2024-05-20T18:12:35.801104+0000) 2024-05-20T18:12:41.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:40.826+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:41.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:40.826+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:41.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:40.826+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:41.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:40 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:40.826+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:41.391 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:41 smithi154 podman[73785]: 2024-05-20 18:12:41.047112692 +0000 UTC m=+0.017005368 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:12:41.392 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:41 smithi154 podman[73785]: 2024-05-20 18:12:41.150066162 +0000 UTC m=+0.119958841 container create 45006a56495714dd546dfdd5cab99872dec3e2dc65d405b11ac0ff036161f5a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c) 2024-05-20T18:12:41.392 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:41 smithi154 podman[73785]: 2024-05-20 18:12:41.360914472 +0000 UTC m=+0.330807142 container init 45006a56495714dd546dfdd5cab99872dec3e2dc65d405b11ac0ff036161f5a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-05-20T18:12:41.392 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:41 smithi154 podman[73785]: 2024-05-20 18:12:41.364933545 +0000 UTC m=+0.334826220 container start 45006a56495714dd546dfdd5cab99872dec3e2dc65d405b11ac0ff036161f5a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:12:41.659 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:41 smithi154 podman[73785]: 2024-05-20 18:12:41.460996616 +0000 UTC m=+0.430889303 container attach 45006a56495714dd546dfdd5cab99872dec3e2dc65d405b11ac0ff036161f5a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True) 2024-05-20T18:12:41.659 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:41 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:41.401+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:41.659 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:41 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:41.401+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:41.659 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:41 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:41.401+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:41.659 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:41 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:41.401+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:41.659 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:41 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:41.401+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:41.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:41 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:41.398+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:41.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:41 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:41.398+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:41.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:41 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:41.398+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:41.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:41 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:41.398+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:41.968 INFO:journalctl@ceph.osd.7.smithi154.stdout:May 20 18:12:41 smithi154 podman[73785]: 2024-05-20 18:12:41.663050275 +0000 UTC m=+0.632942947 container died 45006a56495714dd546dfdd5cab99872dec3e2dc65d405b11ac0ff036161f5a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-20T18:12:42.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:41.800+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:42.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:41.800+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:42.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:41.800+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:42.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:41.800+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.200526+0000 front 2024-05-20T18:12:11.200703+0000 (oldest deadline 2024-05-20T18:12:35.801104+0000) 2024-05-20T18:12:42.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:41.871+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:42.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:41.871+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:42.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:41.871+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:42.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:41 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:41.871+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:42.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.7.service' 2024-05-20T18:12:42.377 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:12:42.377 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-20T18:12:42.377 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-20T18:12:42.377 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.8 2024-05-20T18:12:42.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:42 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:42.393+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:42.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:42 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:42.393+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:42.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:42 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:42.393+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:42.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:42 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:42.393+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:42.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:42 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:42.393+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:42.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:42 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:42.382+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:42.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:42 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:42.382+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:42.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:42 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:42.382+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:42.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:42 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:42.382+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:42.678 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:42 smithi204 systemd[1]: Stopping Ceph osd.8 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:12:43.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:42.894+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:43.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:42.894+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:43.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:42.894+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:43.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:42.894+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:43.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:42.701+0000 7f84adbca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:12:43.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:42.701+0000 7f84adbca700 -1 osd.8 112 *** Got signal Terminated *** 2024-05-20T18:12:43.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:42.701+0000 7f84adbca700 -1 osd.8 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:12:43.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:42.820+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:43.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:42.820+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:43.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:42.820+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:43.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:42 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:42.820+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.200526+0000 front 2024-05-20T18:12:11.200703+0000 (oldest deadline 2024-05-20T18:12:35.801104+0000) 2024-05-20T18:12:43.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:43 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:43.345+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:43.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:43 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:43.345+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:43.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:43 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:43.345+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:43.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:43 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:43.345+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:43.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:43 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:43.345+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:43.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:43.412+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:43.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:43.412+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:43.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:43.412+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:43.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:43 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:43.412+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:44.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:43 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:43.796+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:44.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:43 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:43.796+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:44.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:43 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:43.796+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:44.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:43 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:43.796+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.200526+0000 front 2024-05-20T18:12:11.200703+0000 (oldest deadline 2024-05-20T18:12:35.801104+0000) 2024-05-20T18:12:44.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:43 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:43.861+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:44.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:43 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:43.861+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:44.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:43 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:43.861+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:44.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:43 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:43.861+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:44.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:44 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:44.297+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:44.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:44 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:44.297+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:44.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:44 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:44.297+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:44.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:44 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:44.297+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:44.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:44 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:44.297+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:44.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:44 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:44.391+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:44.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:44 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:44.391+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:44.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:44 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:44.391+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:44.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:44 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:44.391+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:45.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:44 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:44.835+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:45.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:44 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:44.835+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:45.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:44 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:44.835+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:45.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:44 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:44.835+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.200526+0000 front 2024-05-20T18:12:11.200703+0000 (oldest deadline 2024-05-20T18:12:35.801104+0000) 2024-05-20T18:12:45.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:44 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:44.873+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:45.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:44 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:44.873+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:45.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:44 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:44.873+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:45.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:44 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:44.873+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:45.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:45 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:45.258+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:45.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:45 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:45.258+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:45.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:45 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:45.258+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:45.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:45 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:45.258+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:45.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:45 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:45.258+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:45.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:45 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:45.431+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:45.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:45 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:45.431+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:45.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:45 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:45.431+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:45.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:45 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:45.431+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:46.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:45 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:45.901+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:46.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:45 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:45.901+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:46.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:45 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:45.901+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:46.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:45 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:45.901+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:46.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:45 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:45.857+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:46.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:45 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:45.857+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:46.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:45 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:45.857+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:46.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:45 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:45.857+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.200526+0000 front 2024-05-20T18:12:11.200703+0000 (oldest deadline 2024-05-20T18:12:35.801104+0000) 2024-05-20T18:12:46.179 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:45 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:45.857+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:22.802685+0000 front 2024-05-20T18:12:22.802662+0000 (oldest deadline 2024-05-20T18:12:45.702725+0000) 2024-05-20T18:12:46.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:46 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:46.252+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:46.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:46 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:46.252+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:46.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:46 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:46.252+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:46.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:46 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:46.252+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:46.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:46 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:46.252+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:46.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:46 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:46.252+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:46.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:46.383+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:46.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:46.383+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:46.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:46.383+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:46.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:46.383+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:46.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:46 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:46.383+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:20.161912+0000 front 2024-05-20T18:12:20.161878+0000 (oldest deadline 2024-05-20T18:12:46.061594+0000) 2024-05-20T18:12:47.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:46 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:46.860+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:49.497755+0000 front 2024-05-20T18:11:49.497835+0000 (oldest deadline 2024-05-20T18:12:15.397746+0000) 2024-05-20T18:12:47.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:46 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:46.860+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.098723+0000 front 2024-05-20T18:11:57.098686+0000 (oldest deadline 2024-05-20T18:12:21.198860+0000) 2024-05-20T18:12:47.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:46 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:46.860+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.500192+0000 front 2024-05-20T18:12:06.500064+0000 (oldest deadline 2024-05-20T18:12:31.199941+0000) 2024-05-20T18:12:47.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:46 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:46.860+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.200526+0000 front 2024-05-20T18:12:11.200703+0000 (oldest deadline 2024-05-20T18:12:35.801104+0000) 2024-05-20T18:12:47.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:46 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8[55652]: 2024-05-20T18:12:46.860+0000 7f84a6178700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:22.802685+0000 front 2024-05-20T18:12:22.802662+0000 (oldest deadline 2024-05-20T18:12:45.702725+0000) 2024-05-20T18:12:47.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:46 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:46.860+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:47.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:46 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:46.860+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:47.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:46 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:46.860+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:47.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:46 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:46.860+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:47.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:47 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:47.240+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:47.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:47 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:47.240+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:47.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:47 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:47.240+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:47.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:47 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:47.240+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:47.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:47 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:47.240+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:47.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:47 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:47.240+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:47.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:47 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:47.410+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:47.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:47 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:47.410+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:47.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:47 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:47.410+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:47.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:47 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:47.410+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:47.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:47 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:47.410+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:20.161912+0000 front 2024-05-20T18:12:20.161878+0000 (oldest deadline 2024-05-20T18:12:46.061594+0000) 2024-05-20T18:12:48.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:47 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:47.879+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:48.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:47 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:47.879+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:48.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:47 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:47.879+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:48.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:47 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:47.879+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:48.543 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:48 smithi204 podman[74724]: 2024-05-20 18:12:48.275633674 +0000 UTC m=+5.698348724 container died ebef3b669d5eb0b891b4a3519631f72878ff565795f5888f15d663a39cccf72e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0) 2024-05-20T18:12:48.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:48 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:48.216+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:48.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:48 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:48.216+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:48.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:48 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:48.216+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:48.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:48 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:48.216+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:48.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:48 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:48.216+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:48.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:48 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:48.216+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:48.845 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:48 smithi204 podman[74724]: 2024-05-20 18:12:48.637775658 +0000 UTC m=+6.060490703 container cleanup ebef3b669d5eb0b891b4a3519631f72878ff565795f5888f15d663a39cccf72e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240514) 2024-05-20T18:12:48.845 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:48 smithi204 bash[74724]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8 2024-05-20T18:12:48.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:48 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:48.454+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:48.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:48 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:48.454+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:48.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:48 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:48.454+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:48.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:48 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:48.454+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:48.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:48 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:48.454+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:20.161912+0000 front 2024-05-20T18:12:20.161878+0000 (oldest deadline 2024-05-20T18:12:46.061594+0000) 2024-05-20T18:12:49.178 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:48 smithi204 podman[74741]: 2024-05-20 18:12:48.922712482 +0000 UTC m=+0.646075543 container remove ebef3b669d5eb0b891b4a3519631f72878ff565795f5888f15d663a39cccf72e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-20T18:12:49.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:48 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:48.848+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:49.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:48 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:48.848+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:49.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:48 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:48.848+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:49.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:48 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:48.848+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:49.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:48 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:48.848+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:12:49.514 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:49 smithi204 podman[74829]: 2024-05-20 18:12:49.151839553 +0000 UTC m=+0.017461931 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:12:49.514 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:49 smithi204 podman[74829]: 2024-05-20 18:12:49.273935421 +0000 UTC m=+0.139557825 container create 52350f48593766840ee7db47bf4d7a17e37944e51d6e31e09fae3a819654c549 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8-deactivate, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-20T18:12:49.514 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:49 smithi204 podman[74829]: 2024-05-20 18:12:49.517929843 +0000 UTC m=+0.383552218 container init 52350f48593766840ee7db47bf4d7a17e37944e51d6e31e09fae3a819654c549 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, ceph=True) 2024-05-20T18:12:49.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:49 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:49.246+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:49.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:49 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:49.246+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:49.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:49 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:49.246+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:49.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:49 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:49.246+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:49.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:49 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:49.246+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:49.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:49 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:49.246+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:49.810 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:49 smithi204 podman[74829]: 2024-05-20 18:12:49.521405655 +0000 UTC m=+0.387028027 container start 52350f48593766840ee7db47bf4d7a17e37944e51d6e31e09fae3a819654c549 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:12:49.810 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:49 smithi204 podman[74829]: 2024-05-20 18:12:49.61801119 +0000 UTC m=+0.483633562 container attach 52350f48593766840ee7db47bf4d7a17e37944e51d6e31e09fae3a819654c549 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:12:49.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:49 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:49.494+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:49.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:49 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:49.494+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:49.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:49 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:49.494+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:49.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:49 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:49.494+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:49.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:49 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:49.494+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:20.161912+0000 front 2024-05-20T18:12:20.161878+0000 (oldest deadline 2024-05-20T18:12:46.061594+0000) 2024-05-20T18:12:50.159 INFO:journalctl@ceph.osd.8.smithi204.stdout:May 20 18:12:49 smithi204 podman[74829]: 2024-05-20 18:12:49.814290209 +0000 UTC m=+0.679912584 container died 52350f48593766840ee7db47bf4d7a17e37944e51d6e31e09fae3a819654c549 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-20T18:12:50.159 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:49 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:49.886+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:50.159 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:49 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:49.886+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:50.159 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:49 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:49.886+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:50.159 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:49 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:49.886+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:50.160 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:49 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:49.886+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:12:50.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.8.service' 2024-05-20T18:12:50.530 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:12:50.531 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-20T18:12:50.531 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-20T18:12:50.531 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.9 2024-05-20T18:12:50.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:50 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:50.270+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:50.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:50 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:50.270+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:50.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:50 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:50.270+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:50.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:50 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:50.270+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:50.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:50 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:50.270+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:50.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:50 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:50.270+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:50.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:50.446+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:50.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:50.446+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:50.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:50.446+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:50.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:50.446+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:50.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:50.446+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:20.161912+0000 front 2024-05-20T18:12:20.161878+0000 (oldest deadline 2024-05-20T18:12:46.061594+0000) 2024-05-20T18:12:50.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:50 smithi129 systemd[1]: Stopping Ceph osd.9 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:12:51.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:50 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:50.900+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:51.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:50 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:50.900+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:51.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:50 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:50.900+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:51.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:50 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:50.900+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:51.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:50 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:50.900+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:12:51.411 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:50.935+0000 7faef94c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:12:51.411 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:50.935+0000 7faef94c7700 -1 osd.9 112 *** Got signal Terminated *** 2024-05-20T18:12:51.411 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:50 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:50.935+0000 7faef94c7700 -1 osd.9 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:12:51.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:51 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:51.237+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:51.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:51 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:51.237+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:51.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:51 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:51.237+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:51.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:51 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:51.237+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:51.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:51 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:51.237+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:51.643 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:51 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:51.237+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:51.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:51 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:51.423+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:51.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:51 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:51.423+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:51.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:51 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:51.423+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:51.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:51 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:51.423+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:51.912 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:51 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:51.423+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:20.161912+0000 front 2024-05-20T18:12:20.161878+0000 (oldest deadline 2024-05-20T18:12:46.061594+0000) 2024-05-20T18:12:52.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:51 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:51.929+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:52.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:51 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:51.929+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:52.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:51 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:51.929+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:52.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:51 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:51.929+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:52.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:51 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:51.929+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:12:52.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:52 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:52.265+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:52.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:52 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:52.265+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:52.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:52 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:52.265+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:52.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:52 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:52.265+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:52.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:52 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:52.265+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:52.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:52 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:52.265+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:52.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:52 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:52.408+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:52.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:52 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:52.408+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:52.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:52 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:52.408+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:52.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:52 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:52.408+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:52.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:52 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:52.408+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:20.161912+0000 front 2024-05-20T18:12:20.161878+0000 (oldest deadline 2024-05-20T18:12:46.061594+0000) 2024-05-20T18:12:53.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:52 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:52.947+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:53.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:52 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:52.947+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:53.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:52 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:52.947+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:53.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:52 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:52.947+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:53.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:52 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:52.947+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:12:53.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:53 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:53.241+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:53.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:53 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:53.241+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:53.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:53 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:53.241+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:53.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:53 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:53.241+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:53.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:53 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:53.241+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:53.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:53 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:53.241+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:53.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:53 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:53.241+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:26.424073+0000 front 2024-05-20T18:12:26.424182+0000 (oldest deadline 2024-05-20T18:12:52.324024+0000) 2024-05-20T18:12:53.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:53 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:53.373+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:53.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:53 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:53.373+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:53.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:53 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:53.373+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:53.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:53 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:53.373+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:53.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:53 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:53.373+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:20.161912+0000 front 2024-05-20T18:12:20.161878+0000 (oldest deadline 2024-05-20T18:12:46.061594+0000) 2024-05-20T18:12:54.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:53 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:53.926+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:54.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:53 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:53.926+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:54.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:53 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:53.926+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:54.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:53 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:53.926+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:54.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:53 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:53.926+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:12:54.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:53 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:53.926+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:12:54.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:54 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:54.244+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:54.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:54 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:54.244+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:54.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:54 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:54.244+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:54.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:54 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:54.244+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:54.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:54 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:54.244+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:54.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:54 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:54.244+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:54.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:54 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:54.244+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:26.424073+0000 front 2024-05-20T18:12:26.424182+0000 (oldest deadline 2024-05-20T18:12:52.324024+0000) 2024-05-20T18:12:54.660 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:54 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:54.407+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:54.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:54 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:54.407+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:54.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:54 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:54.407+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:54.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:54 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:54.407+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:54.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:54 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:54.407+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:20.161912+0000 front 2024-05-20T18:12:20.161878+0000 (oldest deadline 2024-05-20T18:12:46.061594+0000) 2024-05-20T18:12:54.661 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:54 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:54.407+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.362437+0000 front 2024-05-20T18:12:28.362503+0000 (oldest deadline 2024-05-20T18:12:54.262462+0000) 2024-05-20T18:12:55.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:54 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:54.940+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:55.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:54 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:54.940+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:55.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:54 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:54.940+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:55.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:54 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:54.940+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:55.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:54 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:54.940+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:12:55.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:54 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:54.940+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:12:55.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:55 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:55.249+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:55.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:55 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:55.249+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:55.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:55 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:55.249+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:55.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:55 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:55.249+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:55.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:55 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:55.249+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:55.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:55 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:55.249+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:55.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:55 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:55.249+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:26.424073+0000 front 2024-05-20T18:12:26.424182+0000 (oldest deadline 2024-05-20T18:12:52.324024+0000) 2024-05-20T18:12:55.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:55 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:55.433+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.855690+0000 front 2024-05-20T18:11:39.855744+0000 (oldest deadline 2024-05-20T18:12:05.755860+0000) 2024-05-20T18:12:55.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:55 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:55.433+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:48.057141+0000 front 2024-05-20T18:11:48.057113+0000 (oldest deadline 2024-05-20T18:12:12.157130+0000) 2024-05-20T18:12:55.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:55 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:55.433+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:58.558881+0000 front 2024-05-20T18:11:58.558931+0000 (oldest deadline 2024-05-20T18:12:23.258948+0000) 2024-05-20T18:12:55.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:55 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:55.433+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:03.259291+0000 front 2024-05-20T18:12:03.259364+0000 (oldest deadline 2024-05-20T18:12:31.460038+0000) 2024-05-20T18:12:55.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:55 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:55.433+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:20.161912+0000 front 2024-05-20T18:12:20.161878+0000 (oldest deadline 2024-05-20T18:12:46.061594+0000) 2024-05-20T18:12:55.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:55 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9[64848]: 2024-05-20T18:12:55.433+0000 7faef1a75700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.362437+0000 front 2024-05-20T18:12:28.362503+0000 (oldest deadline 2024-05-20T18:12:54.262462+0000) 2024-05-20T18:12:56.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:55 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:55.899+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:56.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:55 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:55.899+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:56.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:55 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:55.899+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:56.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:55 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:55.899+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:56.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:55 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:55.899+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:12:56.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:55 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:55.899+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:12:56.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:56 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:56.296+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:56.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:56 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:56.296+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:56.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:56 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:56.296+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:56.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:56 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:56.296+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:56.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:56 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:56.296+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:56.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:56 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:56.296+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:56.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:56 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:56.296+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:26.424073+0000 front 2024-05-20T18:12:26.424182+0000 (oldest deadline 2024-05-20T18:12:52.324024+0000) 2024-05-20T18:12:56.831 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:56 smithi129 podman[97635]: 2024-05-20 18:12:56.526303105 +0000 UTC m=+5.798036855 container died 40d2e565f682f79446b05bc0810c67c991843e6341ab5888a508c868d9c8dd5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-20T18:12:56.831 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:56 smithi129 podman[97635]: 2024-05-20 18:12:56.835189657 +0000 UTC m=+6.106923394 container cleanup 40d2e565f682f79446b05bc0810c67c991843e6341ab5888a508c868d9c8dd5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:12:57.160 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:56 smithi129 bash[97635]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9 2024-05-20T18:12:57.161 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:57 smithi129 podman[97650]: 2024-05-20 18:12:57.06171239 +0000 UTC m=+0.535815918 container remove 40d2e565f682f79446b05bc0810c67c991843e6341ab5888a508c868d9c8dd5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:12:57.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:56 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:56.913+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:57.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:56 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:56.913+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:57.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:56 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:56.913+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:57.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:56 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:56.913+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:57.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:56 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:56.913+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:12:57.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:56 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:56.913+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:12:57.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:57.307+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:57.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:57.307+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:57.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:57.307+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:57.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:57.308+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:57.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:57.308+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:57.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:57.308+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:57.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:57 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:57.308+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:26.424073+0000 front 2024-05-20T18:12:26.424182+0000 (oldest deadline 2024-05-20T18:12:52.324024+0000) 2024-05-20T18:12:57.653 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:57 smithi129 podman[97738]: 2024-05-20 18:12:57.362600439 +0000 UTC m=+0.100485325 container create dc73fb6c30f0356ad604ac5ad79e9943eac5ba45f8c8616feca5b536025f15a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9-deactivate, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T18:12:57.653 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:57 smithi129 podman[97738]: 2024-05-20 18:12:57.279322135 +0000 UTC m=+0.017207034 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:12:57.653 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:57 smithi129 podman[97738]: 2024-05-20 18:12:57.565004015 +0000 UTC m=+0.302888901 container init dc73fb6c30f0356ad604ac5ad79e9943eac5ba45f8c8616feca5b536025f15a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-20T18:12:57.653 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:57 smithi129 podman[97738]: 2024-05-20 18:12:57.568355961 +0000 UTC m=+0.306240857 container start dc73fb6c30f0356ad604ac5ad79e9943eac5ba45f8c8616feca5b536025f15a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:12:57.653 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:57 smithi129 podman[97738]: 2024-05-20 18:12:57.656763835 +0000 UTC m=+0.394648734 container attach dc73fb6c30f0356ad604ac5ad79e9943eac5ba45f8c8616feca5b536025f15a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c) 2024-05-20T18:12:57.911 INFO:journalctl@ceph.osd.9.smithi129.stdout:May 20 18:12:57 smithi129 podman[97738]: 2024-05-20 18:12:57.861465517 +0000 UTC m=+0.599350411 container died dc73fb6c30f0356ad604ac5ad79e9943eac5ba45f8c8616feca5b536025f15a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-20T18:12:58.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:57 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:57.904+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:58.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:57 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:57.904+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:58.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:57 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:57.904+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:58.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:57 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:57.904+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:58.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:57 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:57.904+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:12:58.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:57 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:57.904+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:12:58.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.9.service' 2024-05-20T18:12:58.565 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:12:58.566 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-20T18:12:58.566 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-20T18:12:58.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.10 2024-05-20T18:12:58.592 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:58.313+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:58.592 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:58.313+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:58.592 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:58.313+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:58.593 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:58.313+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:58.593 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:58.313+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:58.593 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:58.313+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:58.593 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:58.313+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:26.424073+0000 front 2024-05-20T18:12:26.424182+0000 (oldest deadline 2024-05-20T18:12:52.324024+0000) 2024-05-20T18:12:58.888 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:58 smithi154 systemd[1]: Stopping Ceph osd.10 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:12:59.141 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:58.891+0000 7fcf38bf1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:12:59.141 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:58.891+0000 7fcf38bf1700 -1 osd.10 112 *** Got signal Terminated *** 2024-05-20T18:12:59.141 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:58 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:58.891+0000 7fcf38bf1700 -1 osd.10 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:12:59.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:58 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:58.895+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:12:59.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:58 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:58.895+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:12:59.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:58 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:58.895+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:12:59.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:58 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:58.895+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:12:59.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:58 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:58.895+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:12:59.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:58 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:58.895+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:12:59.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:59 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:59.338+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:12:59.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:59 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:59.338+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:12:59.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:59 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:59.338+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:12:59.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:59 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:59.338+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:12:59.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:59 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:59.338+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:12:59.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:59 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:59.338+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:12:59.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:12:59 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:12:59.338+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:26.424073+0000 front 2024-05-20T18:12:26.424182+0000 (oldest deadline 2024-05-20T18:12:52.324024+0000) 2024-05-20T18:13:00.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:59 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:59.922+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:00.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:59 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:59.922+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:00.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:59 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:59.922+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:00.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:59 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:59.922+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:00.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:59 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:59.922+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:00.179 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:12:59 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:12:59.922+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:00.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:00 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:00.339+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:13:00.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:00 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:00.339+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:13:00.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:00 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:00.339+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:13:00.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:00 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:00.339+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:13:00.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:00 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:00.339+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:13:00.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:00 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:00.339+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:13:00.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:00 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:00.339+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:26.424073+0000 front 2024-05-20T18:12:26.424182+0000 (oldest deadline 2024-05-20T18:12:52.324024+0000) 2024-05-20T18:13:01.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:00 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:00.917+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:01.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:00 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:00.917+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:01.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:00 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:00.917+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:01.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:00 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:00.917+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:01.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:00 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:00.917+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:01.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:00 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:00.917+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:01.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:01 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:01.359+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:13:01.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:01 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:01.359+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:13:01.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:01 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:01.359+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:13:01.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:01 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:01.359+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:13:01.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:01 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:01.359+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:13:01.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:01 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:01.359+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:13:01.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:01 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:01.359+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:26.424073+0000 front 2024-05-20T18:12:26.424182+0000 (oldest deadline 2024-05-20T18:12:52.324024+0000) 2024-05-20T18:13:02.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:01.942+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:02.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:01.942+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:02.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:01.942+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:02.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:01.942+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:02.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:01.942+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:02.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:01 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:01.942+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:02.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:02 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:02.371+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:13:02.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:02 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:02.371+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:13:02.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:02 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:02.371+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:13:02.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:02 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:02.371+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:13:02.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:02 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:02.371+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:13:02.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:02 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:02.371+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:13:02.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:02 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:02.371+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:26.424073+0000 front 2024-05-20T18:12:26.424182+0000 (oldest deadline 2024-05-20T18:12:52.324024+0000) 2024-05-20T18:13:03.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:02 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:02.962+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:03.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:02 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:02.962+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:03.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:02 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:02.962+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:03.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:02 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:02.962+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:03.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:02 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:02.962+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:03.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:02 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:02.962+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:03.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:02 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:02.962+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-05-20T18:12:37.108336+0000 front 2024-05-20T18:12:37.108179+0000 (oldest deadline 2024-05-20T18:13:02.407859+0000) 2024-05-20T18:13:03.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:03.342+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:41.716033+0000 front 2024-05-20T18:11:41.715894+0000 (oldest deadline 2024-05-20T18:12:07.015839+0000) 2024-05-20T18:13:03.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:03.342+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:47.016313+0000 front 2024-05-20T18:11:47.016350+0000 (oldest deadline 2024-05-20T18:12:11.116422+0000) 2024-05-20T18:13:03.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:03.342+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.518367+0000 front 2024-05-20T18:11:57.518278+0000 (oldest deadline 2024-05-20T18:12:23.418300+0000) 2024-05-20T18:13:03.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:03.342+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6814 osd.3 since back 2024-05-20T18:12:06.820274+0000 front 2024-05-20T18:12:06.820049+0000 (oldest deadline 2024-05-20T18:12:32.120022+0000) 2024-05-20T18:13:03.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:03.342+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:14.921614+0000 front 2024-05-20T18:12:14.921634+0000 (oldest deadline 2024-05-20T18:12:38.922109+0000) 2024-05-20T18:13:03.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:03.342+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:21.223342+0000 front 2024-05-20T18:12:21.223506+0000 (oldest deadline 2024-05-20T18:12:45.923147+0000) 2024-05-20T18:13:03.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:03 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10[61486]: 2024-05-20T18:13:03.342+0000 7fcf3119f700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:26.424073+0000 front 2024-05-20T18:12:26.424182+0000 (oldest deadline 2024-05-20T18:12:52.324024+0000) 2024-05-20T18:13:04.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:03 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:03.934+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:04.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:03 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:03.934+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:04.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:03 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:03.934+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:04.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:03 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:03.934+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:04.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:03 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:03.934+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:04.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:03 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:03.934+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:04.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:03 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:03.934+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-05-20T18:12:37.108336+0000 front 2024-05-20T18:12:37.108179+0000 (oldest deadline 2024-05-20T18:13:02.407859+0000) 2024-05-20T18:13:04.737 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:04 smithi154 podman[73890]: 2024-05-20 18:13:04.481014421 +0000 UTC m=+5.713872350 container died 19ad0d44f55ddf99cda38c1df5e6c25295185d02ecb0a1784e4943b09a6b2e3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-20T18:13:05.071 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:04 smithi154 podman[73890]: 2024-05-20 18:13:04.849168817 +0000 UTC m=+6.082026739 container cleanup 19ad0d44f55ddf99cda38c1df5e6c25295185d02ecb0a1784e4943b09a6b2e3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-20T18:13:05.071 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:04 smithi154 bash[73890]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10 2024-05-20T18:13:05.380 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:05 smithi154 podman[73915]: 2024-05-20 18:13:05.075551068 +0000 UTC m=+0.595596478 container remove 19ad0d44f55ddf99cda38c1df5e6c25295185d02ecb0a1784e4943b09a6b2e3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-20T18:13:05.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:04 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:04.976+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:05.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:04 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:04.976+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:05.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:04 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:04.976+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:05.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:04 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:04.976+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:05.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:04 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:04.976+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:05.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:04 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:04.976+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:05.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:04 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:04.976+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-05-20T18:12:37.108336+0000 front 2024-05-20T18:12:37.108179+0000 (oldest deadline 2024-05-20T18:13:02.407859+0000) 2024-05-20T18:13:05.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:05 smithi154 podman[74009]: 2024-05-20 18:13:05.384736553 +0000 UTC m=+0.110011529 container create 1ac41f9463de3636d5ddf4c09ddabbc5680c56909084c2d12edb70453fa96e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-20T18:13:05.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:05 smithi154 podman[74009]: 2024-05-20 18:13:05.291953688 +0000 UTC m=+0.017228672 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:13:05.641 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:05 smithi154 podman[74009]: 2024-05-20 18:13:05.596137714 +0000 UTC m=+0.321412705 container init 1ac41f9463de3636d5ddf4c09ddabbc5680c56909084c2d12edb70453fa96e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, ceph=True) 2024-05-20T18:13:05.642 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:05 smithi154 podman[74009]: 2024-05-20 18:13:05.599432912 +0000 UTC m=+0.324707897 container start 1ac41f9463de3636d5ddf4c09ddabbc5680c56909084c2d12edb70453fa96e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-20T18:13:06.141 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:05 smithi154 podman[74009]: 2024-05-20 18:13:05.678690426 +0000 UTC m=+0.403965426 container attach 1ac41f9463de3636d5ddf4c09ddabbc5680c56909084c2d12edb70453fa96e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-20T18:13:06.142 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:05 smithi154 podman[74009]: 2024-05-20 18:13:05.889078942 +0000 UTC m=+0.614353926 container died 1ac41f9463de3636d5ddf4c09ddabbc5680c56909084c2d12edb70453fa96e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10-deactivate, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-20T18:13:06.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:06.013+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:06.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:06.013+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:06.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:06.013+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:06.330 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:06.013+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:06.330 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:06.013+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:06.330 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:06.013+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:06.330 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:06.013+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-05-20T18:12:37.108336+0000 front 2024-05-20T18:12:37.108179+0000 (oldest deadline 2024-05-20T18:13:02.407859+0000) 2024-05-20T18:13:06.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.10.service' 2024-05-20T18:13:06.469 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:06 smithi154 podman[74035]: 2024-05-20 18:13:06.389876865 +0000 UTC m=+0.494871705 container remove 1ac41f9463de3636d5ddf4c09ddabbc5680c56909084c2d12edb70453fa96e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:13:06.469 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:06 smithi154 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.10.service: Deactivated successfully. 2024-05-20T18:13:06.469 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:06 smithi154 systemd[1]: Stopped Ceph osd.10 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T18:13:06.469 INFO:journalctl@ceph.osd.10.smithi154.stdout:May 20 18:13:06 smithi154 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.10.service: Consumed 5min 16.134s CPU time. 2024-05-20T18:13:06.484 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:06.484 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-20T18:13:06.484 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-20T18:13:06.484 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.11 2024-05-20T18:13:06.613 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:06 smithi204 systemd[1]: Stopping Ceph osd.11 for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:06.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:06.824+0000 7fee6f17a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:06.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:06.824+0000 7fee6f17a700 -1 osd.11 112 *** Got signal Terminated *** 2024-05-20T18:13:06.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:06 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:06.824+0000 7fee6f17a700 -1 osd.11 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-20T18:13:07.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.014+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:07.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.014+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:07.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.014+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:07.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.014+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:07.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.014+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:07.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.014+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:07.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.014+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-05-20T18:12:37.108336+0000 front 2024-05-20T18:12:37.108179+0000 (oldest deadline 2024-05-20T18:13:02.407859+0000) 2024-05-20T18:13:08.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.994+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:08.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.994+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:08.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.994+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:08.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.994+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:08.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.994+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:08.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.994+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:08.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:07 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:07.994+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-05-20T18:12:37.108336+0000 front 2024-05-20T18:12:37.108179+0000 (oldest deadline 2024-05-20T18:13:02.407859+0000) 2024-05-20T18:13:09.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:09 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:09.040+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:09.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:09 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:09.040+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:09.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:09 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:09.040+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:09.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:09 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:09.040+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:09.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:09 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:09.040+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:09.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:09 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:09.040+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:09.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:09 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:09.040+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-05-20T18:12:37.108336+0000 front 2024-05-20T18:12:37.108179+0000 (oldest deadline 2024-05-20T18:13:02.407859+0000) 2024-05-20T18:13:10.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:10 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:10.075+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:10.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:10 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:10.075+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:10.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:10 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:10.075+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:10.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:10 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:10.075+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:10.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:10 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:10.075+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:10.430 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:10 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:10.075+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:10.430 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:10 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:10.075+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-05-20T18:12:37.108336+0000 front 2024-05-20T18:12:37.108179+0000 (oldest deadline 2024-05-20T18:13:02.407859+0000) 2024-05-20T18:13:11.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:11 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:11.114+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-05-20T18:11:39.800229+0000 front 2024-05-20T18:11:39.799998+0000 (oldest deadline 2024-05-20T18:12:03.300019+0000) 2024-05-20T18:13:11.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:11 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:11.114+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-05-20T18:11:50.301869+0000 front 2024-05-20T18:11:50.301648+0000 (oldest deadline 2024-05-20T18:12:13.201460+0000) 2024-05-20T18:13:11.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:11 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:11.114+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-05-20T18:11:57.902563+0000 front 2024-05-20T18:11:57.902600+0000 (oldest deadline 2024-05-20T18:12:23.802790+0000) 2024-05-20T18:13:11.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:11 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:11.114+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-05-20T18:12:11.305138+0000 front 2024-05-20T18:12:11.305087+0000 (oldest deadline 2024-05-20T18:12:36.605029+0000) 2024-05-20T18:13:11.428 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:11 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:11.114+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-05-20T18:12:23.006455+0000 front 2024-05-20T18:12:23.006487+0000 (oldest deadline 2024-05-20T18:12:48.306476+0000) 2024-05-20T18:13:11.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:11 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:11.114+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.129:6822 osd.6 since back 2024-05-20T18:12:28.307159+0000 front 2024-05-20T18:12:28.307202+0000 (oldest deadline 2024-05-20T18:12:53.606916+0000) 2024-05-20T18:13:11.429 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:11 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11[62563]: 2024-05-20T18:13:11.114+0000 7fee67728700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-05-20T18:12:37.108336+0000 front 2024-05-20T18:12:37.108179+0000 (oldest deadline 2024-05-20T18:13:02.407859+0000) 2024-05-20T18:13:12.648 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:12 smithi204 podman[74947]: 2024-05-20 18:13:12.396430028 +0000 UTC m=+5.706024645 container died 9ca218df3a6b8964b47f66d948b828c3877493bb5deac28d03b8a8aea0b8c369 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514) 2024-05-20T18:13:12.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:12 smithi204 podman[74947]: 2024-05-20 18:13:12.741717022 +0000 UTC m=+6.051311638 container cleanup 9ca218df3a6b8964b47f66d948b828c3877493bb5deac28d03b8a8aea0b8c369 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:13:12.928 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:12 smithi204 bash[74947]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11 2024-05-20T18:13:13.348 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:12 smithi204 podman[74973]: 2024-05-20 18:13:12.934537025 +0000 UTC m=+0.536643725 container remove 9ca218df3a6b8964b47f66d948b828c3877493bb5deac28d03b8a8aea0b8c369 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T18:13:13.609 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:13 smithi204 podman[75061]: 2024-05-20 18:13:13.35221841 +0000 UTC m=+0.106968296 container create a1ed366c3e84b359d7e4f9d760e451ecc3f67da54512a6eee5046887550fcd9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11-deactivate, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-20T18:13:13.609 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:13 smithi204 podman[75061]: 2024-05-20 18:13:13.262631796 +0000 UTC m=+0.017381674 image pull cfcc61b2d476f611479310eb042aa05f11dd5f60dbb65e3b0d7e0e11d024cdb3 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708 2024-05-20T18:13:13.609 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:13 smithi204 podman[75061]: 2024-05-20 18:13:13.613158453 +0000 UTC m=+0.367908320 container init a1ed366c3e84b359d7e4f9d760e451ecc3f67da54512a6eee5046887550fcd9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-20T18:13:13.906 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:13 smithi204 podman[75061]: 2024-05-20 18:13:13.616336175 +0000 UTC m=+0.371086046 container start a1ed366c3e84b359d7e4f9d760e451ecc3f67da54512a6eee5046887550fcd9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514) 2024-05-20T18:13:13.906 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:13 smithi204 podman[75061]: 2024-05-20 18:13:13.679764396 +0000 UTC m=+0.434514261 container attach a1ed366c3e84b359d7e4f9d760e451ecc3f67da54512a6eee5046887550fcd9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-20T18:13:14.178 INFO:journalctl@ceph.osd.11.smithi204.stdout:May 20 18:13:13 smithi204 podman[75061]: 2024-05-20 18:13:13.910124981 +0000 UTC m=+0.664874853 container died a1ed366c3e84b359d7e4f9d760e451ecc3f67da54512a6eee5046887550fcd9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:13:14.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@osd.11.service' 2024-05-20T18:13:14.535 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:14.535 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-20T18:13:14.535 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-20T18:13:14.535 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.a 2024-05-20T18:13:14.843 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 18:13:14 smithi129 systemd[1]: Stopping Ceph mds.a for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:15.161 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 18:13:14 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-a[71631]: 2024-05-20T18:13:14.845+0000 7fa7f7214700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:15.161 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 18:13:14 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-a[71631]: 2024-05-20T18:13:14.845+0000 7fa7f7214700 -1 mds.a *** got signal Terminated *** 2024-05-20T18:13:17.788 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 18:13:17 smithi129 podman[97858]: 2024-05-20 18:13:17.506800807 +0000 UTC m=+2.768532201 container died 9e62bdb20c4361caec038a550b211c705c20265a496cbafef53d4bd7ccea1ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-05-20T18:13:18.110 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 18:13:17 smithi129 podman[97858]: 2024-05-20 18:13:17.870758436 +0000 UTC m=+3.132489826 container cleanup 9e62bdb20c4361caec038a550b211c705c20265a496cbafef53d4bd7ccea1ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=) 2024-05-20T18:13:18.110 INFO:journalctl@ceph.mds.a.smithi129.stdout:May 20 18:13:17 smithi129 bash[97858]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-a 2024-05-20T18:13:18.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.a.service' 2024-05-20T18:13:18.279 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:18.279 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-20T18:13:18.280 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-20T18:13:18.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.d 2024-05-20T18:13:18.411 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 18:13:18 smithi129 systemd[1]: Stopping Ceph mds.d for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:18.910 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 18:13:18 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-d[70672]: 2024-05-20T18:13:18.616+0000 7fb440fce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:18.911 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 18:13:18 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-d[70672]: 2024-05-20T18:13:18.616+0000 7fb440fce700 -1 mds.d *** got signal Terminated *** 2024-05-20T18:13:22.120 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 18:13:21 smithi129 podman[97975]: 2024-05-20 18:13:21.855426407 +0000 UTC m=+3.345839133 container died 9acc43aa3ab0895e27120ec5c40793d1288c981345e21884be0ac08b7b3ac2ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:13:22.410 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 18:13:22 smithi129 podman[97975]: 2024-05-20 18:13:22.225104884 +0000 UTC m=+3.715517600 container cleanup 9acc43aa3ab0895e27120ec5c40793d1288c981345e21884be0ac08b7b3ac2ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514) 2024-05-20T18:13:22.411 INFO:journalctl@ceph.mds.d.smithi129.stdout:May 20 18:13:22 smithi129 bash[97975]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-d 2024-05-20T18:13:22.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.d.service' 2024-05-20T18:13:22.640 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:22.640 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-20T18:13:22.640 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-20T18:13:22.641 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.g 2024-05-20T18:13:22.910 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 18:13:22 smithi129 systemd[1]: Stopping Ceph mds.g for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:23.410 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 18:13:22 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-g[71278]: 2024-05-20T18:13:22.995+0000 7f2cf0271700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:23.411 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 18:13:22 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-g[71278]: 2024-05-20T18:13:22.995+0000 7f2cf0271700 -1 mds.g *** got signal Terminated *** 2024-05-20T18:13:27.161 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 18:13:26 smithi129 podman[98095]: 2024-05-20 18:13:26.889103469 +0000 UTC m=+4.027115431 container died ebdb95f64391801f02a02f3058e98fc409644346dd130852f8f208f28fa9656e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-g, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:13:27.566 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 18:13:27 smithi129 podman[98095]: 2024-05-20 18:13:27.203866683 +0000 UTC m=+4.341878642 container cleanup ebdb95f64391801f02a02f3058e98fc409644346dd130852f8f208f28fa9656e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-g, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:13:27.566 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 18:13:27 smithi129 bash[98095]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-g 2024-05-20T18:13:27.566 INFO:journalctl@ceph.mds.g.smithi129.stdout:May 20 18:13:27 smithi129 podman[98106]: 2024-05-20 18:13:27.363402208 +0000 UTC m=+0.476441656 container remove ebdb95f64391801f02a02f3058e98fc409644346dd130852f8f208f28fa9656e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:13:27.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.g.service' 2024-05-20T18:13:27.614 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:27.615 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-20T18:13:27.615 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-20T18:13:27.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.j 2024-05-20T18:13:27.911 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 18:13:27 smithi129 systemd[1]: Stopping Ceph mds.j for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:28.410 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 18:13:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-j[70291]: 2024-05-20T18:13:27.952+0000 7f18ce242700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:28.411 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 18:13:27 smithi129 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-j[70291]: 2024-05-20T18:13:27.952+0000 7f18ce242700 -1 mds.j *** got signal Terminated *** 2024-05-20T18:13:31.146 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 18:13:30 smithi129 podman[98216]: 2024-05-20 18:13:30.870569349 +0000 UTC m=+3.028294127 container died 9f47465ebe1d0e8040c0c6b5509a5fa83d0b851ed5eb70199d35e8480f31f67e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-20T18:13:31.410 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 18:13:31 smithi129 podman[98216]: 2024-05-20 18:13:31.232697945 +0000 UTC m=+3.390422716 container cleanup 9f47465ebe1d0e8040c0c6b5509a5fa83d0b851ed5eb70199d35e8480f31f67e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-j, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-20T18:13:31.411 INFO:journalctl@ceph.mds.j.smithi129.stdout:May 20 18:13:31 smithi129 bash[98216]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-j 2024-05-20T18:13:31.622 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.j.service' 2024-05-20T18:13:31.653 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:31.653 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-20T18:13:31.653 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-20T18:13:31.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.b 2024-05-20T18:13:31.978 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 18:13:31 smithi154 systemd[1]: Stopping Ceph mds.b for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:32.391 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 18:13:31 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b[66692]: 2024-05-20T18:13:31.981+0000 7fd21e584700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:32.391 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 18:13:31 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b[66692]: 2024-05-20T18:13:31.981+0000 7fd21e584700 -1 mds.b *** got signal Terminated *** 2024-05-20T18:13:34.020 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 18:13:33 smithi154 podman[74113]: 2024-05-20 18:13:33.753990859 +0000 UTC m=+1.903994452 container died 7b06ee3220d54cf332ab1907d16b6c8ce4329782bef29749f5a49c3f2e654797 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T18:13:34.391 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 18:13:34 smithi154 podman[74113]: 2024-05-20 18:13:34.122867244 +0000 UTC m=+2.272870871 container cleanup 7b06ee3220d54cf332ab1907d16b6c8ce4329782bef29749f5a49c3f2e654797 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:13:34.391 INFO:journalctl@ceph.mds.b.smithi154.stdout:May 20 18:13:34 smithi154 bash[74113]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-b 2024-05-20T18:13:34.599 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.b.service' 2024-05-20T18:13:34.632 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:34.633 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-20T18:13:34.633 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-20T18:13:34.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.e 2024-05-20T18:13:35.141 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 18:13:34 smithi154 systemd[1]: Stopping Ceph mds.e for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:35.141 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 18:13:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-e[66081]: 2024-05-20T18:13:34.934+0000 7fe87e0a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:35.141 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 18:13:34 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-e[66081]: 2024-05-20T18:13:34.934+0000 7fe87e0a5700 -1 mds.e *** got signal Terminated *** 2024-05-20T18:13:36.891 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 18:13:36 smithi154 podman[74233]: 2024-05-20 18:13:36.638014036 +0000 UTC m=+1.784991527 container died 9cd4a95b1ef887af37c8bc5afc3e9d61c62718e6f543151a1bdc8fd244e36e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-20T18:13:37.141 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 18:13:36 smithi154 podman[74233]: 2024-05-20 18:13:36.968125357 +0000 UTC m=+2.115102829 container cleanup 9cd4a95b1ef887af37c8bc5afc3e9d61c62718e6f543151a1bdc8fd244e36e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_BRANCH=HEAD) 2024-05-20T18:13:37.142 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 18:13:36 smithi154 bash[74233]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-e 2024-05-20T18:13:37.511 INFO:journalctl@ceph.mds.e.smithi154.stdout:May 20 18:13:37 smithi154 podman[74257]: 2024-05-20 18:13:37.244548298 +0000 UTC m=+0.602943197 container remove 9cd4a95b1ef887af37c8bc5afc3e9d61c62718e6f543151a1bdc8fd244e36e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-20T18:13:37.524 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.e.service' 2024-05-20T18:13:37.557 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:37.558 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-20T18:13:37.558 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-20T18:13:37.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.h 2024-05-20T18:13:37.860 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:13:37 smithi154 systemd[1]: Stopping Ceph mds.h for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:38.141 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:13:37 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[71186]: 2024-05-20T18:13:37.863+0000 7f885db4b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:38.141 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:13:37 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h[71186]: 2024-05-20T18:13:37.863+0000 7f885db4b700 -1 mds.h *** got signal Terminated *** 2024-05-20T18:13:41.310 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:13:41 smithi154 podman[74366]: 2024-05-20 18:13:41.04993211 +0000 UTC m=+3.275519825 container died 69763b3db3d969b5e3ed082bb228c583f7cba641d860e1d5399349caa0cb14c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-20T18:13:41.641 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:13:41 smithi154 podman[74366]: 2024-05-20 18:13:41.396579525 +0000 UTC m=+3.622167239 container cleanup 69763b3db3d969b5e3ed082bb228c583f7cba641d860e1d5399349caa0cb14c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.schema-version=1.0) 2024-05-20T18:13:41.641 INFO:journalctl@ceph.mds.h.smithi154.stdout:May 20 18:13:41 smithi154 bash[74366]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-h 2024-05-20T18:13:41.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.h.service' 2024-05-20T18:13:41.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:41.813 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-20T18:13:41.813 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-20T18:13:41.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.k 2024-05-20T18:13:42.113 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 18:13:41 smithi154 systemd[1]: Stopping Ceph mds.k for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:42.391 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 18:13:42 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-k[65770]: 2024-05-20T18:13:42.116+0000 7f11c823a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:42.391 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 18:13:42 smithi154 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-k[65770]: 2024-05-20T18:13:42.116+0000 7f11c823a700 -1 mds.k *** got signal Terminated *** 2024-05-20T18:13:44.171 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 18:13:43 smithi154 podman[74486]: 2024-05-20 18:13:43.905369975 +0000 UTC m=+1.864032856 container died 32548813b8d3335971d78b45035be0cc0c5b1136b24805f173988cf94603b609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-k, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:13:44.538 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 18:13:44 smithi154 podman[74486]: 2024-05-20 18:13:44.274812202 +0000 UTC m=+2.233475076 container cleanup 32548813b8d3335971d78b45035be0cc0c5b1136b24805f173988cf94603b609 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-20T18:13:44.539 INFO:journalctl@ceph.mds.k.smithi154.stdout:May 20 18:13:44 smithi154 bash[74486]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-k 2024-05-20T18:13:44.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.k.service' 2024-05-20T18:13:44.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:44.726 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-20T18:13:44.726 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-20T18:13:44.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.c 2024-05-20T18:13:45.039 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 18:13:44 smithi204 systemd[1]: Stopping Ceph mds.c for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:45.384 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 18:13:45 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-c[67749]: 2024-05-20T18:13:45.041+0000 7f2f9ba20700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:45.384 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 18:13:45 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-c[67749]: 2024-05-20T18:13:45.041+0000 7f2f9ba20700 -1 mds.c *** got signal Terminated *** 2024-05-20T18:13:46.388 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 18:13:46 smithi204 podman[75167]: 2024-05-20 18:13:46.140348155 +0000 UTC m=+1.215390748 container died 3938857b9ea6d929322ba977e094c8a10325b3942297a0c01902d40043797f29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-c, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T18:13:46.678 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 18:13:46 smithi204 podman[75167]: 2024-05-20 18:13:46.492545719 +0000 UTC m=+1.567588312 container cleanup 3938857b9ea6d929322ba977e094c8a10325b3942297a0c01902d40043797f29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-c, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-20T18:13:46.678 INFO:journalctl@ceph.mds.c.smithi204.stdout:May 20 18:13:46 smithi204 bash[75167]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-c 2024-05-20T18:13:46.922 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.c.service' 2024-05-20T18:13:46.957 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:46.957 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-20T18:13:46.957 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-20T18:13:46.957 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.f 2024-05-20T18:13:47.334 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 18:13:47 smithi204 systemd[1]: Stopping Ceph mds.f for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:47.678 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 18:13:47 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-f[67444]: 2024-05-20T18:13:47.337+0000 7fe9f8b4a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:47.678 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 18:13:47 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-f[67444]: 2024-05-20T18:13:47.337+0000 7fe9f8b4a700 -1 mds.f *** got signal Terminated *** 2024-05-20T18:13:51.593 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 18:13:51 smithi204 podman[75304]: 2024-05-20 18:13:51.342466733 +0000 UTC m=+4.165644925 container died 756cb0b9c6660f7338ccb14ea57796b55c7e2d977cd5e2fcbc462abc9a79d9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-f, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240514, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c) 2024-05-20T18:13:51.860 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 18:13:51 smithi204 podman[75304]: 2024-05-20 18:13:51.687889707 +0000 UTC m=+4.511067889 container cleanup 756cb0b9c6660f7338ccb14ea57796b55c7e2d977cd5e2fcbc462abc9a79d9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-20T18:13:51.860 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 18:13:51 smithi204 bash[75304]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-f 2024-05-20T18:13:52.111 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 18:13:51 smithi204 podman[75319]: 2024-05-20 18:13:51.864114299 +0000 UTC m=+0.514863344 container remove 756cb0b9c6660f7338ccb14ea57796b55c7e2d977cd5e2fcbc462abc9a79d9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-f, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2) 2024-05-20T18:13:52.111 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 18:13:52 smithi204 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.f.service: Deactivated successfully. 2024-05-20T18:13:52.111 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 18:13:52 smithi204 systemd[1]: Stopped Ceph mds.f for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T18:13:52.111 INFO:journalctl@ceph.mds.f.smithi204.stdout:May 20 18:13:52 smithi204 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.f.service: Consumed 1.136s CPU time. 2024-05-20T18:13:52.113 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.f.service' 2024-05-20T18:13:52.146 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:52.146 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-20T18:13:52.147 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-20T18:13:52.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.i 2024-05-20T18:13:52.428 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:13:52 smithi204 systemd[1]: Stopping Ceph mds.i for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:52.927 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:13:52 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[72933]: 2024-05-20T18:13:52.483+0000 7f2dae76d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:52.928 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:13:52 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i[72933]: 2024-05-20T18:13:52.483+0000 7f2dae76d700 -1 mds.i *** got signal Terminated *** 2024-05-20T18:13:54.005 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:13:53 smithi204 podman[75424]: 2024-05-20 18:13:53.741655861 +0000 UTC m=+1.373556257 container died 7bc558f440fca562e41b7968183dca304bb12c3a097604a3d7a68b3d6ee82f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-20T18:13:54.256 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:13:54 smithi204 podman[75424]: 2024-05-20 18:13:54.083644966 +0000 UTC m=+1.715545368 container cleanup 7bc558f440fca562e41b7968183dca304bb12c3a097604a3d7a68b3d6ee82f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, org.label-schema.build-date=20240514) 2024-05-20T18:13:54.256 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:13:54 smithi204 bash[75424]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i 2024-05-20T18:13:54.506 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:13:54 smithi204 podman[75437]: 2024-05-20 18:13:54.259845819 +0000 UTC m=+0.515189174 container remove 7bc558f440fca562e41b7968183dca304bb12c3a097604a3d7a68b3d6ee82f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-20T18:13:54.506 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:13:54 smithi204 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.i.service: Deactivated successfully. 2024-05-20T18:13:54.507 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:13:54 smithi204 systemd[1]: Stopped Ceph mds.i for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968. 2024-05-20T18:13:54.507 INFO:journalctl@ceph.mds.i.smithi204.stdout:May 20 18:13:54 smithi204 systemd[1]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.i.service: Consumed 1.202s CPU time. 2024-05-20T18:13:54.508 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.i.service' 2024-05-20T18:13:54.538 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:54.539 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-20T18:13:54.539 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-20T18:13:54.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.l 2024-05-20T18:13:54.875 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 18:13:54 smithi204 systemd[1]: Stopping Ceph mds.l for 0f1f0dd2-16d1-11ef-bc9a-c7b262605968... 2024-05-20T18:13:55.177 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 18:13:54 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-l[67140]: 2024-05-20T18:13:54.878+0000 7fa3f6aa6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-20T18:13:55.178 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 18:13:54 smithi204 ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-l[67140]: 2024-05-20T18:13:54.878+0000 7fa3f6aa6700 -1 mds.l *** got signal Terminated *** 2024-05-20T18:13:57.326 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 18:13:57 smithi204 podman[75541]: 2024-05-20 18:13:57.008321221 +0000 UTC m=+2.236483949 container died eab0016ca770245aea1614ff5aa25a79ba6b1a253daf5cebb3c6afd5ccdccfce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-20T18:13:57.326 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 18:13:57 smithi204 podman[75541]: 2024-05-20 18:13:57.329399608 +0000 UTC m=+2.557562327 container cleanup eab0016ca770245aea1614ff5aa25a79ba6b1a253daf5cebb3c6afd5ccdccfce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d5395c4020ac9059aec12c85bcc41cf8b1b75bc8c18339e31fbf77c838aee708, name=ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240517.203521-squid-debug-d2b507c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-05-20T18:13:57.643 INFO:journalctl@ceph.mds.l.smithi204.stdout:May 20 18:13:57 smithi204 bash[75541]: ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968-mds-l 2024-05-20T18:13:57.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0f1f0dd2-16d1-11ef-bc9a-c7b262605968@mds.l.service' 2024-05-20T18:13:57.803 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-20T18:13:57.803 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-20T18:13:57.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 --force --keep-logs 2024-05-20T18:13:57.994 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T18:14:00.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 --force --keep-logs 2024-05-20T18:14:00.725 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T18:14:03.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 --force --keep-logs 2024-05-20T18:14:03.330 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T18:14:05.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-20T18:14:05.703 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-20T18:14:05.734 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-20T18:14:05.760 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-20T18:14:05.762 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661/remote/smithi129/crash 2024-05-20T18:14:05.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/crash -- . 2024-05-20T18:14:05.800 INFO:teuthology.orchestra.run.smithi129.stderr:tar: /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/crash: Cannot open: No such file or directory 2024-05-20T18:14:05.800 INFO:teuthology.orchestra.run.smithi129.stderr:tar: Error is not recoverable: exiting now 2024-05-20T18:14:05.802 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661/remote/smithi154/crash 2024-05-20T18:14:05.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/crash -- . 2024-05-20T18:14:05.848 INFO:teuthology.orchestra.run.smithi154.stderr:tar: /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/crash: Cannot open: No such file or directory 2024-05-20T18:14:05.848 INFO:teuthology.orchestra.run.smithi154.stderr:tar: Error is not recoverable: exiting now 2024-05-20T18:14:05.849 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661/remote/smithi204/crash 2024-05-20T18:14:05.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/crash -- . 2024-05-20T18:14:05.915 INFO:teuthology.orchestra.run.smithi204.stderr:tar: /var/lib/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/crash: Cannot open: No such file or directory 2024-05-20T18:14:05.915 INFO:teuthology.orchestra.run.smithi204.stderr:tar: Error is not recoverable: exiting now 2024-05-20T18:14:05.916 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-20T18:14:05.916 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-20T18:14:06.011 INFO:teuthology.orchestra.run.smithi129.stdout:2024-05-20T18:10:00.000204+0000 mon.a (mon.0) 1361 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-20T18:14:06.011 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-20T18:14:06.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[SEC\]' /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-20T18:14:06.046 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]' /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-20T18:14:06.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[WRN\]' /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-20T18:14:06.192 INFO:teuthology.orchestra.run.smithi129.stdout:2024-05-20T18:10:00.000204+0000 mon.a (mon.0) 1361 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-05-20T18:14:06.193 INFO:tasks.cephadm:Compressing logs... 2024-05-20T18:14:06.193 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-20T18:14:06.235 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-20T18:14:06.237 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-20T18:14:06.261 INFO:teuthology.orchestra.run.smithi204.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log 2024-05-20T18:14:06.261 INFO:teuthology.orchestra.run.smithi204.stderr:: No such file or directory 2024-05-20T18:14:06.261 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.2.log 2024-05-20T18:14:06.261 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.f.log 2024-05-20T18:14:06.261 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.audit.log 2024-05-20T18:14:06.270 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.5.log 2024-05-20T18:14:06.270 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.c.log 2024-05-20T18:14:06.270 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.audit.log: /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mon.c.log 2024-05-20T18:14:06.270 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mgr.z.log 2024-05-20T18:14:06.270 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.i.log 2024-05-20T18:14:06.270 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mgr.z.log: 86.6% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log.gz 2024-05-20T18:14:06.270 INFO:teuthology.orchestra.run.smithi204.stderr: 90.3% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.audit.log.gz 2024-05-20T18:14:06.270 INFO:teuthology.orchestra.run.smithi204.stderr: 94.4% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.f.log.gz 2024-05-20T18:14:06.270 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.cephadm.log 2024-05-20T18:14:06.271 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.11.log 2024-05-20T18:14:06.271 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.cephadm.log: 94.5%gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.8.log 2024-05-20T18:14:06.271 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.c.log.gz 2024-05-20T18:14:06.271 INFO:teuthology.orchestra.run.smithi204.stderr: 83.7% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.cephadm.log.gz 2024-05-20T18:14:06.271 INFO:teuthology.orchestra.run.smithi129.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.audit.log 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.g.log 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log: /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.j.log 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.3.log 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.0.log 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mgr.x.log 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mon.a.log 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.6.log 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.cephadm.log 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.a.log 2024-05-20T18:14:06.272 INFO:teuthology.orchestra.run.smithi129.stderr: 86.6% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log.gz 2024-05-20T18:14:06.273 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-volume.log 2024-05-20T18:14:06.273 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.d.log 2024-05-20T18:14:06.273 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.9.log 2024-05-20T18:14:06.273 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-20T18:14:06.273 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 94.5% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.a.log.gz 2024-05-20T18:14:06.276 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.11.log: 92.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mgr.z.log.gz 2024-05-20T18:14:06.276 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-volume.log 2024-05-20T18:14:06.277 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.l.log 2024-05-20T18:14:06.277 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-20T18:14:06.278 INFO:teuthology.orchestra.run.smithi154.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-20T18:14:06.278 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log 2024-05-20T18:14:06.278 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.audit.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.7.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log: /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.b.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.4.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.1.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mgr.y.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.h.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.10.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mgr.y.log: /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.cephadm.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.e.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-volume.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mon.b.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.k.log 2024-05-20T18:14:06.279 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-20T18:14:06.280 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.l.log: 94.5% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.l.log.gz 2024-05-20T18:14:06.282 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-20T18:14:06.287 INFO:teuthology.orchestra.run.smithi204.stderr: 91.0% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-volume.log.gz 2024-05-20T18:14:06.377 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 86.6% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.log.gz 2024-05-20T18:14:06.400 INFO:teuthology.orchestra.run.smithi129.stderr: 84.0% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.cephadm.log.gz 2024-05-20T18:14:06.420 INFO:teuthology.orchestra.run.smithi129.stderr: 90.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.audit.log.gz 2024-05-20T18:14:06.453 INFO:teuthology.orchestra.run.smithi129.stderr: 91.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-volume.log.gz 2024-05-20T18:14:06.473 INFO:teuthology.orchestra.run.smithi129.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-20T18:14:06.499 INFO:teuthology.orchestra.run.smithi154.stderr: 83.7% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.cephadm.log.gz 2024-05-20T18:14:06.520 INFO:teuthology.orchestra.run.smithi154.stderr: 90.3% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph.audit.log.gz 2024-05-20T18:14:06.597 INFO:teuthology.orchestra.run.smithi204.stderr: 91.8% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mon.c.log.gz 2024-05-20T18:14:06.623 INFO:teuthology.orchestra.run.smithi129.stderr: 94.5% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.g.log.gz 2024-05-20T18:14:06.624 INFO:teuthology.orchestra.run.smithi154.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-20T18:14:06.713 INFO:teuthology.orchestra.run.smithi204.stderr: 96.8% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.i.log.gz 2024-05-20T18:14:06.832 INFO:teuthology.orchestra.run.smithi154.stderr: 91.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-volume.log.gz 2024-05-20T18:14:06.848 INFO:teuthology.orchestra.run.smithi154.stderr: 94.4% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.e.log.gz 2024-05-20T18:14:06.852 INFO:teuthology.orchestra.run.smithi154.stderr: 94.4% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.k.log.gz 2024-05-20T18:14:06.884 INFO:teuthology.orchestra.run.smithi154.stderr: 92.0% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mgr.y.log.gz 2024-05-20T18:14:07.083 INFO:teuthology.orchestra.run.smithi129.stderr: 94.5% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.d.log.gz 2024-05-20T18:14:07.158 INFO:teuthology.orchestra.run.smithi129.stderr: 94.5% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.j.log.gz 2024-05-20T18:14:07.327 INFO:teuthology.orchestra.run.smithi129.stderr: 90.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mgr.x.log.gz 2024-05-20T18:14:08.874 INFO:teuthology.orchestra.run.smithi154.stderr: 91.8% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mon.b.log.gz 2024-05-20T18:14:10.732 INFO:teuthology.orchestra.run.smithi129.stderr: 90.3% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mon.a.log.gz 2024-05-20T18:14:51.478 INFO:teuthology.orchestra.run.smithi204.stderr: 93.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.8.log.gz 2024-05-20T18:14:51.951 INFO:teuthology.orchestra.run.smithi129.stderr: 93.0% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.6.log.gz 2024-05-20T18:14:54.525 INFO:teuthology.orchestra.run.smithi204.stderr: 93.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.2.log.gz 2024-05-20T18:14:57.797 INFO:teuthology.orchestra.run.smithi204.stderr: 93.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.5.log.gz 2024-05-20T18:15:02.157 INFO:teuthology.orchestra.run.smithi204.stderr: 93.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.11.log.gz 2024-05-20T18:15:02.159 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-05-20T18:15:02.159 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m55.907s 2024-05-20T18:15:02.159 INFO:teuthology.orchestra.run.smithi204.stderr:user 3m14.524s 2024-05-20T18:15:02.159 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m6.725s 2024-05-20T18:15:37.760 INFO:teuthology.orchestra.run.smithi129.stderr: 93.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.9.log.gz 2024-05-20T18:15:48.837 INFO:teuthology.orchestra.run.smithi129.stderr: 93.0% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.3.log.gz 2024-05-20T18:15:59.341 INFO:teuthology.orchestra.run.smithi129.stderr: 92.9% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.0.log.gz 2024-05-20T18:15:59.343 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-05-20T18:15:59.343 INFO:teuthology.orchestra.run.smithi129.stderr:real 1m53.095s 2024-05-20T18:15:59.343 INFO:teuthology.orchestra.run.smithi129.stderr:user 3m11.557s 2024-05-20T18:15:59.343 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m7.558s 2024-05-20T18:17:55.666 INFO:teuthology.orchestra.run.smithi154.stderr: 93.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.1.log.gz 2024-05-20T18:18:54.977 INFO:teuthology.orchestra.run.smithi154.stderr: 97.4% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.h.log.gz 2024-05-20T18:18:57.304 INFO:teuthology.orchestra.run.smithi154.stderr: 93.2% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.4.log.gz 2024-05-20T18:20:05.139 INFO:teuthology.orchestra.run.smithi154.stderr: 93.0% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.7.log.gz 2024-05-20T18:22:15.619 INFO:teuthology.orchestra.run.smithi154.stderr: 93.1% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-osd.10.log.gz 2024-05-20T18:23:09.895 INFO:teuthology.orchestra.run.smithi154.stderr: 94.0% -- replaced with /var/log/ceph/0f1f0dd2-16d1-11ef-bc9a-c7b262605968/ceph-mds.b.log.gz 2024-05-20T18:23:09.897 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-05-20T18:23:09.897 INFO:teuthology.orchestra.run.smithi154.stderr:real 9m3.646s 2024-05-20T18:23:09.897 INFO:teuthology.orchestra.run.smithi154.stderr:user 5m10.487s 2024-05-20T18:23:09.897 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m14.580s 2024-05-20T18:23:09.897 INFO:tasks.cephadm:Archiving logs... 2024-05-20T18:23:09.898 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661/remote/smithi129/log 2024-05-20T18:23:09.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-20T18:23:40.772 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661/remote/smithi154/log 2024-05-20T18:23:40.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-20T18:24:28.690 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661/remote/smithi204/log 2024-05-20T18:24:28.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-20T18:25:00.831 INFO:tasks.cephadm:Removing cluster... 2024-05-20T18:25:00.832 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 --force 2024-05-20T18:25:01.024 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T18:25:01.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 --force 2024-05-20T18:25:02.003 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T18:25:02.627 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 --force 2024-05-20T18:25:02.819 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: 0f1f0dd2-16d1-11ef-bc9a-c7b262605968 2024-05-20T18:25:03.395 INFO:tasks.cephadm:Removing cephadm ... 2024-05-20T18:25:03.396 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-20T18:25:03.414 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-20T18:25:03.430 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-20T18:25:03.445 INFO:tasks.cephadm:Teardown complete 2024-05-20T18:25:03.446 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-20T18:25:03.460 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-20T18:25:03.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-20T18:25:03.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-20T18:25:03.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-20T18:25:03.607 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-20T18:25:03.619 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-20T18:25:03.620 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-20T18:25:03.622 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-20T18:25:03.625 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-20T18:25:03.635 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-05-20T18:25:03.636 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-05-20T18:25:03.639 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-05-20T18:25:03.640 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-20T18:25:03.640 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-05-20T18:25:03.640 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv01.front.sepia.ceph.com 3 6 377 24 +723us[ +744us] +/- 56ms 2024-05-20T18:25:03.640 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 -1983us[-1962us] +/- 51ms 2024-05-20T18:25:03.640 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +1656us[+1676us] +/- 68ms 2024-05-20T18:25:03.640 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T18:25:03.642 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-20T18:25:03.642 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-05-20T18:25:03.643 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv01.front.sepia.ceph.com 3 6 377 28 +719us[ +711us] +/- 56ms 2024-05-20T18:25:03.643 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 -1979us[-1987us] +/- 50ms 2024-05-20T18:25:03.643 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +1652us[+1652us] +/- 68ms 2024-05-20T18:25:03.643 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T18:25:03.661 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-20T18:25:03.661 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-05-20T18:25:03.661 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv01.front.sepia.ceph.com 3 6 377 29 +716us[ +589us] +/- 56ms 2024-05-20T18:25:03.661 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 27 -2001us[-2001us] +/- 51ms 2024-05-20T18:25:03.661 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +1640us[+1513us] +/- 68ms 2024-05-20T18:25:03.662 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-20T18:25:03.662 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-20T18:25:03.673 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-20T18:25:03.674 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-20T18:25:03.690 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-20T18:25:03.812 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-20T18:25:03.912 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-20T18:25:03.979 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-20T18:25:04.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T18:25:04.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-20T18:25:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T18:25:04.036 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-20T18:25:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T18:25:04.063 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-20T18:25:04.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-20T18:25:04.089 INFO:teuthology.task.internal:Duration was 2650.108678 seconds 2024-05-20T18:25:04.090 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-20T18:25:04.123 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-20T18:25:04.124 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-20T18:25:04.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-20T18:25:04.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-20T18:25:04.165 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-20T18:25:04.182 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-20T18:25:04.183 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-20T18:25:04.686 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-20T18:25:04.686 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-05-20T18:25:04.686 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-20T18:25:04.708 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-05-20T18:25:04.708 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-20T18:25:04.733 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-05-20T18:25:04.733 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-20T18:25:04.760 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-20T18:25:04.760 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-20T18:25:04.763 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-20T18:25:04.775 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-20T18:25:04.826 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-20T18:25:04.826 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-20T18:25:04.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-20T18:25:04.841 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-20T18:25:05.200 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-20T18:25:05.212 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-20T18:25:05.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-20T18:25:05.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-20T18:25:05.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-20T18:25:05.331 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-20T18:25:05.342 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-20T18:25:05.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-20T18:25:05.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-20T18:25:05.366 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-05-20T18:25:05.368 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-05-20T18:25:05.393 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-05-20T18:25:05.413 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-20T18:25:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T18:25:05.435 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-20T18:25:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T18:25:05.449 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-20T18:25:05.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-20T18:25:05.467 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-20T18:25:05.481 INFO:teuthology.task.internal:Transferring archived files... 2024-05-20T18:25:05.482 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661/remote/smithi129 2024-05-20T18:25:05.482 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-20T18:25:05.526 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661/remote/smithi154 2024-05-20T18:25:05.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-20T18:25:05.561 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-05-20_15:44:59-fs-wip-pdonnell-testing-20240517.203521-squid-debug-distro-default-smithi/7715661/remote/smithi204 2024-05-20T18:25:05.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-20T18:25:05.597 INFO:teuthology.task.internal:Removing archive directory... 2024-05-20T18:25:05.597 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-20T18:25:05.600 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-20T18:25:05.602 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-20T18:25:05.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-20T18:25:05.657 INFO:teuthology.task.internal:Not uploading archives. 2024-05-20T18:25:05.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-20T18:25:05.681 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-20T18:25:05.682 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-20T18:25:05.684 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-20T18:25:05.686 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-20T18:25:05.697 INFO:teuthology.orchestra.run.smithi129.stdout: 266019 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 20 18:25 /home/ubuntu/cephtest 2024-05-20T18:25:05.698 INFO:teuthology.orchestra.run.smithi154.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 20 18:25 /home/ubuntu/cephtest 2024-05-20T18:25:05.700 INFO:teuthology.orchestra.run.smithi204.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 20 18:25 /home/ubuntu/cephtest 2024-05-20T18:25:05.701 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-20T18:25:05.715 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-20T18:25:05.786 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} duration: 2650.1086778640747 failure_reason: '"2024-05-20T18:10:00.000204+0000 mon.a (mon.0) 1361 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-05-20T18:25:05.786 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-20T18:25:05.868 INFO:teuthology.run:FAIL